IRC channel logs

2013-11-17.log

back to list of logs

<unknown_lamer>mark_weaver: the second NAN_BOXING_PRELIM_25_handles_negative_addrs.patch ... formatting problems in comments mostly, some stuff moved around a bit in numbers.h, and then a weird change in hash.c
<unknown_lamer>I may have broken hashing slightly, I'm not sure
<unknown_lamer>only took about 15 minutes of looking at the few rejected hunks to fix it all up, and It Works (tm)
<mark_weaver>unknown_lamer: cool! but still stutter. hmm.
<mark_weaver>so even 15ms is too much, I guess?
<unknown_lamer>you have 16.6ms to go from zero to frame is on the screen
<unknown_lamer>at 60fps. At 120fps, ... good thing I don't own any displays that go that fast any more ;)
<unknown_lamer>hrm, I kind of want to grab my 75Hz @ 1920x1200 SGI monitor out of the closet
<mark_weaver>what if you forced gc to happen more often? might it have less to do? dunno.
<unknown_lamer>too bad something brokei nternally causing the image to be projected on the inside of a drum
<unknown_lamer>I tried that by binding GC_collect_a_little ... nothing
<unknown_lamer>there are two possible problems... or maybe three
<mark_weaver>then I guess maybe the thing is to reduce the total size of the heap.
<mark_weaver>it's possible that modern master is better in this regard, with the new ELF linker.
<unknown_lamer>1. the fps accumulator isn't accurate enough (possible, I think it is counting the time it takes to generate and print the fps count as frame time)
<unknown_lamer>2. something in guile is being called from an async or something
<unknown_lamer>3. frames are rendering *too* fast (sometimes as little as 3ms) and then the time spent blocked after the second swap-buffers is being counted
<unknown_lamer>mark_weaver: gc is no longer an issue!
<unknown_lamer>it's spending like 30ms per minute in gc now :)
<mark_weaver>ah, okay.
<unknown_lamer>and I think most of that is because I am pushing frame times into a list and generating a chart from them
<unknown_lamer>I also think I can get this patch onto master without tooooo much trouble
<mark_weaver>yeah, it shouldn't be too bad.
<unknown_lamer>obviously rewriting the parts that touched the vm, but when you said assembly I thought you meant machine assembly and not vm opcodes!
<mark_weaver>but master has regressed its math performance somewhat, especially for multiplication.
<unknown_lamer>I guess my next step is to grab the patch where you merged scm_i_from_double, make a new patch, try to apply the new patch on master, sort through rejects and see if it'll compile...
<mark_weaver>there were nice optimized assembly bits for the stack VM, for add/sub/mul on x86/arm/mips.
<unknown_lamer>also understanding the rtl vm enough
<unknown_lamer>ah
<mark_weaver>and that's not yet in the
<mark_weaver>TRL VM.
<mark_weaver>*RTL
<unknown_lamer>"I wanted to write a video game and I ended up patching libgc and guile's numeric support instead"
<unknown_lamer>this is where the #lisp folks would tell me to just go back to common lisp ;)
<mark_weaver>fairly soon, I plan to add a bunch of new RTL VM instructions to support the R6RS fixnum and flonum operations efficiently.
<mark_weaver>(actually, the optimized MUL is not even in 2.0.9. it's only on stable-2.0)
<unknown_lamer>I saw that numbers.c has become ... messier than in 1.8
<mark_weaver>heh
<unknown_lamer>I suspect it might be better to move generic operators into pure scheme...
<unknown_lamer>and then take advantage of goops's dispatch mechanism
<unknown_lamer>which in theory also caches its decisions
<mark_weaver>that would be great, when we have native compilation and goops dispatch is fast enough. at this point, I think it would cause a serious slowness.
<mark_weaver>slowdown.
<unknown_lamer>maybe
<mark_weaver>but yes, in the long term, that's where I'd like to go.
<unknown_lamer>goops dispatch should be as fast as normal dispatch
<unknown_lamer>if it isn't, there are published methods for making it so!
*unknown_lamer is digging into how sbcl does its generic operations
<unknown_lamer>mostly by avoiding them at compile time afaict
<mark_weaver>you'll see that in my monster guile todo list, which I posted to guile-devel at some point, there's an item about making goops dispatch fast enough that we can use it for generic numeric operators.
<unknown_lamer> http://docs.racket-lang.org/ts-guide/types.html I'm sort of wondering...
<mark_weaver>(unless I added it to my list since my last post)
<mark_weaver>but first, we need polymorphic inline caches.
<mark_weaver>and native compilation too.
<mark_weaver>yeah, having something like typed racket in guile sounds good to me.
<mark_weaver>as long as it's optional, of course.
***sneek_ is now known as sneek
<mark_weaver>anyway, thanks for working on this!
<mark_weaver>unknown_lamer: btw, when I said assembly, I _did_ mean x86/asm/mips assembly.
<mark_weaver>the patches I sent you patch those bits for x86.
<mark_weaver>unknown_lamer: btw, I don't think you should spend too much time adapting that patch to latest master. I have a newer patch that I didn't give you, because I don't think it was quite working yet, and I have newer better ideas on how to do the tagging.
<mark_weaver>I sent it to you just so you could get an idea of what would happen if floats didn't generate garbage.
<unknown_lamer>ah
<unknown_lamer>dangit, there goes my chance to be helpful!
<unknown_lamer>oh nice, sbcl moved to git, finally
<unknown_lamer>(from cvs of all things)
<gzg>Was SCSH ever a "big thing" in the Scheme community? Looking at guile-scsh, it looks pretty nifty!
<mark_weaver>unknown_lamer: well, if you could figure out how to do nice stutter-free animation using guile-figl and the nan-boxing patch, I think that would be *very* useful.
<unknown_lamer>indeed
<unknown_lamer>I added the freeglut extension that prevents glut from killing the program when you close a window (the only annoying bit is that figl keeps its own already-initialized flag to prevent you from calling initialize-glut more than once per program0
<unknown_lamer>)
<unknown_lamer>but, good enough to call statprof for now
<mark_weaver>I'm totally ignorant of opengl stuff (learning about it some day is on my TODO :), so I'm not quite sure what the issues are, but it would be nice to get figl fixed.
<unknown_lamer>I think figl is fine, but guile is not optimized for real time operation
<unknown_lamer>I am pretty sure there's just something or other running from an async that runs for a few ms occasionally
<mark_weaver>how about putting a print statement in the code that runs asyncs? (to verify that hypothesis)
<mark_weaver>maybe measure the time spent in the async as well.
<mark_weaver>asyncs are kind of a mess. really, they should be deprecated.
<mark_weaver>anyway, gotta go afk for a bit ...
<unknown_lamer>exciting
<unknown_lamer>I think statprof and glut both use setitimer, and then boom
***haroldwu_irssi is now known as haroldwu
<stis>evening guilers!
<mark_weaver>tupi: hi tupi! I literally just (a few seconds ago) sent you an updated version of the patch set. the one I sent before didn't reap pipes properly (actually, I think the reaper has been broken for a long time).
<mark_weaver>I *think* this is a completely proper fix now. It runs the stress test indefinitely without problems while reaping pipes, etc.
<mark_weaver>s/reaping pipes/reaping unclosed pipes (or pipes closed with 'close-port' instead of 'close-pipe')/
<tupi>mark_weaver: hello! thank you for your work! i just started to look at it right now
<tupi>i can see a branch origin/thread-safe-popen but not the wip-... i guess it thjat one i should use ?
<tupi>then i wanted to make sure that branch was based on stable ?
<mark_weaver>yeah, I deleted the 'wip' branch, to prevent confusion. 'thread-safe-popen' is the one to use now.
<mark_weaver>yes, it's based on stable-2.0.
<tupi>perfect, let me try all this and get back asap. i will compile install and try here and on the machine of our lab, they 12 core based, but the 'real' test :) will have to wait untill or monday or tuesday (i m fly there on tuesaday but will try to make them try before...)
<mark_weaver>okay, sounds good!
<mark_weaver>now that I have a test that quickly reproduces the problem you saw on even modest (2-4 core) machines, that gives me some confidence that the problem is really fixed.
<mark_weaver>plus, I thought about it really carefully :)
<tupi>yes i saw it and ws pelased to see you found a way to reproduce, it obviously help a lt
<mark_weaver>tupi: also, I want to stress again that you should make sure they're using libgc-7.2d
<mark_weaver>all of my testing has been with that version of libgc, and the older version in debian stable (7.1) is not nearly as robust, and lacks some features such that finalization happens differently.
<tupi>yes, all machines use that version
<tupi>they're all on testing
<tupi>using testing i mean
<tupi>this is the propor git command right ?
<tupi>git checkout --track -b origin/thread-safe-popen origin/stable-2.0
<mark_weaver>wow, I don't even know what that does. I just use "git checkout thread-safe-popen"
<tupi>that is because you created locally, but i need to track 'your' changes ...
<mark_weaver>no, I used that command on other machines where I hadn't created it locally.
<mark_weaver>(I pushed it from one machine, and then checked it out from another debian stable machine that had never seen it before)
<tupi>ok, i am not a git specialist, that's why i did ask...
<mark_weaver>well, neither am I. I learned just what I need to know :)
<mark_weaver>but it worked for me. however, I did have to "git pull" first, from any branch (I did it from stable-2.0), before "git checkout thread-safe-popen".
<tupi>i did a git pull of stable-2.0 before as well, just to make sure
<tupi>i will create an /opt2 dir to make sure and compare with and wothout th patch...
<mark_weaver>okay
<mark_weaver>the main thing I think is probably wrong with the command you suggested is the "origin/stable-2.0" at the end. I don't see why that should be mentioned at all.
<mark_weaver>in any case, I'd go with what I know works. and I know that "git checkout thread-safe-popen" works.
<tupi>ok
<tupi>i 'copy' that from other examples and because there is origin/master ... but
<tupi>ok
<mark_weaver>hi civodul!
<civodul>Hello Guilers! :-)
<mark_weaver>civodul: I believe I successfully made (ice-9 popen) thread safe, and part of that work involved making guardians thread-safe, and laying the groundwork for blocking asyncs when certain mutexes are held. If you'd like to take a look, see the 'thread-safe-popen' branch.
<civodul>nice!
<civodul>i'll look into it later today
<tupi>mark_weaver: almost there ... how can i locallu delete a branch ?
<mark_weaver>tupi: git branch -D <branch-name>
<tupi>tx
<tupi>it occurs to me that, maybe, guile compilation could use multicore in some of his steps [not all of course...]
<civodul>the warning passes could easily run in parallel
<civodul>but i don't think it represents much of the compile time
<tupi>mark_weaver: so far so good on the 12 cores machine of the lab
<tupi>and it crashed immediately if using previous guile version
<tupi>congratulation! i was so concerned about that problem, i can't wait to test it on the customer 32 cores box ...
<tupi>i think you could/should immediately merge that good work in stable-2.0
<mark_weaver>tupi: I should wait for feedback first.
<mark_weaver>glad to hear it's working on 12 cores :)
<mark_weaver>tupi: but not to worry, we'll have it fixed in 2.0.10. speaking of which, we should probably release 2.0.10 in a few weeks. I'm willing to take the lead on that. I have a few other things to get in before release though.
<mark_weaver>civodul: ^^
<tupi>mark_weaver: i use git but glad to here for others that 2.0.10 is coming
<tupi>i just have to remember to switch to stable-2.0 when you'll have merged...
<mark_weaver>tupi: I'll close the bug when it's merged, and CC you on that email.
<tupi>great, tx
<mark_weaver>sneek: seen davexunit
<sneek>davexunit was here Nov 15 at 01:57 pm UTC, saying: I think haskell is great, and I imagine many other schemers agree..
<stis>mark_weaver: what is holding a nan boxed version of guile back?
<mark_weaver>stis: well, I'll soon post patches to make it an compile-time option, but it's not clear that it's the right thing to do by default. there are downsides.
<stis>ok!
<dsmith>mark_weaver, Might want to mention 7.2d in the README. Currently says 7.0
<mark_weaver>downsides include: (1) it drastically reduces the range of fixnums (from 62 bits to about 50 bits), (2) it slows down fixnum math and all tag checks somewhat, (3) it won't work at all on systems with more than 49 bits of virtual address space, which modern systems will soon have.
<mark_weaver>stis: with type inference, it should be possible to make the compile unbox floats in many cases, especially when using the R6RS flonum ops.
<mark_weaver>(and perhaps using explicit type annotations as well)
<stis>Hmm yeah, this shows some serious problems!
<stis>I really don't like getting fixnum with only 50 bits.
<stis>(You know that I use bitvectors for represetnting ordered sets)
<stis>e.g. the fixnums if I have say 59 iobjects
<stis>it's really a powerful trick!
<stis>else I would argue to use nan boxing, because it will be here tomorrow, type inference will take some time
<stis>and perhaps adress spaces will just reach trhat limit at the time we have good type inference and can take advantage of it!
<stis>and then we can perhaps ingibit nan boxing for those platforms!
<mark_weaver>the problem is, we can't know at compile time whether the compiled binary might be used on a system that supports more than 49 bits of virtual address space.
<mark_weaver>bitvectors wouldn't be affected by this. however, if you use normal exact integers as bits, then that would be affected.
<mark_weaver>(e.g. using SRFI-60)
<mark_weaver>it's likely that some systems will support 56 bit address spaces fairly soon.
<stis>ok, then it is clearly a hairy concept!
*stis wishes that hardware where designed with type bits in mind
<tupi>i guess i can safely tar /opt2 and untar in /opt, right?
<mark_weaver>yeah, that would be nice :)
<mark_weaver>tupi: no, the $PREFIX is hardcoded into the compiled C code.
<mark_weaver>tupi: you can keep the .go files though.
<mark_weaver>tupi: just rerun configure and then make again, and that should be fine I think. (if you want to be extra careful, cd into libguile and make clean there).
<tupi>ok
<mark_weaver>the .go files, on the other hand, are quite portable. as long as the word size and endianness is the same, they can be even be moved to a different architecture.
<mark_weaver>(in master, they might not be quite as portable, dunno)
<tupi>but as far as using guile is concerned, tar /opt2 and untar in /opt should be ok, so i could send the tar file to somebody there ... then when i have access to their machine would do the house keeping of their git clone ... ?
<mark_weaver>tupi: no, the installed guile cannot be moved to a different directory.
<tupi>ah, ok
<mark_weaver>tupi: it could perhaps be moved to a different machine with the exact same OS, as long as it ends up in the same PREFIX.
<mark_weaver>(if both machines have the same architecture and same version of debian, for example)
<tupi>i'll prepare an exact same prefix version then
<mark_weaver>(and assuming that no libraries outside of debian got linked in)
<mark_weaver>of course, when the "same version of debian" that we're talking about is debian testing, which is somewhat of a moving target, things get a bit less certain. but as long as both systems are reasonably up-to-date, it will probably be okay.
<mark_weaver>just make sure that nothing in /usr/local gets linked in.
<tupi>mark_weaver: nothing in /usr/local is linked, same OS, same debian versions .. it's gone a be fine if i send a /opt tar [just with guile] i think
<mark_weaver>tupi: also, remember that guile will need to find the right libguile library, which means either setting LD_LIBRARY_PATH before running guile, or configuring /etc/ld.so.conf (or /etc/ld.so.conf.d/*) and running 'ldconfig'.
<mark_weaver>you have to be careful with that. you could easily end up linking to the libguile in /usr/lib, which is still buggy.
<mark_weaver>however, these issues exist even when you build on your own system, so I guess you've taken care of it somehow.
<mark_weaver>well, the other option is to use -rpath to tell the compiled guile executable where to find its matching libguile.
<mark_weaver>but I don't have much experience with that, so I don't remember how to do that off-hand. civodul would probably know, since Guix uses that technique.
<tupi>i don't use LD_LIBRARY_PATH, did not change /etc/ld.so.conf.d the only variable is this one:
<tupi>export PKG_CONFIG_PATH="/opt/lib/pkgconfig:/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
<mark_weaver>tupi: well, I think that's not enough. I'm not sure how it's working for you.
<mark_weaver>I'm not an expert on build systems; maybe there's something I don't know.
<mark_weaver>civodul is the one who would know.
<tupi>mark_weaver: not either, but i can tell you it works, even for guile-gnome and guile-clutter
<tupi>guile probably sets things in its intial steps i guess
<mark_weaver>tupi: also, in this case, I would remove the "/use/local/lib/pkgconfig" from there, because we're trying to make sure nothing from /usr/local gets used.
<mark_weaver>tupi: yes, it would work for guile-gnome and guile-clutter, because guile does set some things when it uses lt_dlopen.
<tupi>yes i should, it is an old setting, i don't use /usr/local anymore
<mark_weaver>but libguile gets loaded before guile has a chance to do any of that.
<mark_weaver>tupi: are you sure you don't have anything about /opt in your /etc/ld.so.conf*
<mark_weaver>?
<tupi>sure, no settings there
<mark_weaver>tupi: when you run "make install" doesn't it show you a warning like this one? http://paste.lisp.org/+300I
<mark_weaver>(it's buried in the middle of the install log, so it's easily missed if you're not paying attention)
<tupi>i did miss it
<tupi>damned
<mark_weaver>based on that message, it sounds like you might be able to just "export LD_RUN_PATH=/opt/lib" before building guile.
<tupi>ok
<mark_weaver>or, as I said, you can "export LD_LIBRARY_PATH=/opt/lib" before running guile.
<tupi>i guess i can add a guile.conf in /etc/ld.conf.d
<mark_weaver>if you do that, you have to make sure the one in /opt/lib is searched before any other directories containing libguile.
***anderson is now known as carer
<mark_weaver>(you can use "ldconfig -p | grep libguile-2.0" to check that)
***carer is now known as anderson
<tupi>wow, pretty messy [my installation]
<tupi>ldconfig -p | grep libguile-2.0
<tupi>
<mark_weaver>rebuilding with LD_RUN_PATH set is probably the best option. you'd only have to "make clean" in libguile.
<tupi>i'll do that
<tupi>i think it is because some ebian packages need guile-2.0-libs and i thought i didn't have any manually installed guile at this point ...
<mark_weaver>civodul: is "export LD_RUN_PATH=/opt/lib" while building guile a good way to ensure that the installed executable will find the right libguile? (assuming --prefix=/opt) ?
<mark_weaver>(this on a debian system where all other libraries are from debian, and where nothing else is in /opt)
<mark_weaver>s/installed executable/installed 'guile' executable'
<mark_weaver>tupi: if you want to be on a safe side, "rm -rf /opt" before rebuilding guile like that.
<tupi>mark_weaver: ok
<mark_weaver>(I'm not 100% sure, but it might end up linking with the libguile you already installed in /opt, instead of the new one in the build dir)
<tupi>make clean ?
<tupi>make uninstall i mean
<tupi>becasue i have other things there
<mark_weaver>what other things?
<mark_weaver>I thought the idea was that /opt would have nothing else, so you could tar it up for them?
<tupi>yes, i am doing 2 thngs: 1 is on my machine and 2-, preparing an empty /opt for them
<tupi>on my machine [which i still have to avoid these probs we're talking about as well, i have guile-gnome, clutter, guiledhall...
*mark_weaver reads the ld manual...
<mark_weaver>I guess it should be fine. just 'make uninstall' then.
<tupi>ok
<mark_weaver>but when you build the guile that you'll tar up for your customer, make sure /opt is empty before building guile, and export LD_RUN_PATH=/opt/lib before building guile.
<mark_weaver>(assuming --prefix=/opt)
<tupi>yes, that is what i am doing now
<mark_weaver>probably some of these things are needed, but I'm not 100% clear on all the details of what LD_RUN_PATH does, so I'm trying to be careful.
<tupi>i temp moved the /opt to /opt.prev and mkdir /opt
<mark_weaver>s/are needed/are not needed/
<mark_weaver>okay
<mark_weaver>I wish civodul was here. I hope I'm understanding LD_RUN_PATH correctly.
<mark_weaver>well, in the worst case, if it doesn't work, you can always set LD_LIBRARY_PATH before running guile. I know that works; I've used it quite a bit.
<tupi>i still had the warning message
<tupi>i am confused, i admit :)
<mark_weaver>well, that's okay. I think it would probably warn no matter what.
<tupi>but i still get this:
<tupi>ldconfig -p | grep libguile-2.0
<tupi>
<mark_weaver>that's expected.
<tupi>anyway, if popen does not crash anymore, i am running the proper libguile i guess
<mark_weaver>'ldconfig -p' shows the system default paths, which does not include rpaths included in the guile executable.
<mark_weaver>tupi: well, not quite. there are two components of the fixes: one in popen.scm, and some things in libguile.
<mark_weaver>if you link against the wrong libguile, then problems will be considerably less likely, but there are still thread safety issues.
<tupi>can i ask guile what libguile it is using ? [like a internal ?}
<mark_weaver>in order to be completely thread safe, you need both.
<mark_weaver>I think "ldd /opt/bin/guile" will tell you.
<tupi>ah, that's ok then, i can always use that to double check after a new build
<mark_weaver>tupi: if LD_RUN_PATH doesn't work, then probably the right thing is to add "LDFLAGS=-Wl,-rpath -Wl,/opt/lib" as an argument to Guile's ./configure
<tupi>mark_weaver: it did work, otherwise ldd woudlnot have answered libguile-2.0.so.22 => /opt/lib/libguile-2.0.so.22 (0x00007f882012d000) i guess
<mark_weaver>btw, make sure that LD_RUN_PATH is *not* set when you run "ldd"
<mark_weaver>(just in case that taints the experiment)
<mark_weaver>if "ldd" prints that when LD_RUN_PATH is _not_ set, then I'm sure we're good.
<tupi>ok
<mark_weaver>does it?
<tupi>no, i did make [and set LD_RUN..] in a different terminal, so pretty sure it is not tainted
<mark_weaver>excellent!
<tupi>pfuut! many tx, i feel a little embarassed not to know these things
<mark_weaver>heh, me too. there's always more to learn :)
<tupi>here is the final proof :)
<tupi>david@capac:/opt/lib 50 $ echo $LD_RUN_PATH
<tupi>
<tupi>david@capac:/opt/lib 51 $ ldd /opt/bin/guile | grep libguile
<tupi>
<mark_weaver>indeed, that's definitive :)
<mark_weaver>out of paranoia, also please check "echo $LD_LIBRARY_PATH" please.
<tupi>david@capac:/opt/lib 52 $ echo $LD_LIBRARY_PATH
<tupi>
<tupi>
<mark_weaver>good!
<tupi>we're fine!
<mark_weaver>woohoo!
<tupi>all clean, houston can send to texas :)
<mark_weaver>(sorry to others, in retrospect we probably should have taken this to a private channel :)
<tupi>true, sorry! we should have made a thread safe [to others] talk :)
<civodul>mark_weaver: LD_RUN_PATH means that is only used when no -rpath option is passed
<civodul>libtool links with -rpath
<mark_weaver>civodul: ah, okay. so how does the installed 'guile' in /opt find its libguile, given that tupi didn't have /opt in LD_LIBRARY_PATH or /etc/ld.so.conf* ?
<mark_weaver>is there some magic going on that I don't know about?
<mark_weaver>I guess libtool is taking care of it?
<mark_weaver>it's adding an -rpath for $PREFIX/lib ?
<civodul>mark_weaver: normally the installed guile has libguile in its RUNPATH
<civodul>"objdump -x /opt/bin/guile | grep PATH" should show it
<civodul>but that can be overridden by $LD_LIBRARY_PATH and (i think) /etc/ld.so.conf
<civodul>i never use the latter, though
<mark_weaver>oh, well, if /etc/ld.so.conf overrides it, that would be bad in this case.
<mark_weaver>but I think that /etc/ld.so.conf is searched after all rpaths.
***sneek_ is now known as sneek
<tupi>i am experiencing a terrible reduction od the use of the cores after a couple of minutes, from 12 cores up to 100% to all cores below 12%, actuallyt most of them down to 0.1%
<civodul>tupi: using futures or par-map?
*civodul recalls a bug report...
<tupi>par-map
<civodul>blech
<civodul>i remember you reported that some time ago, no?
<tupi>could it be the OS ?
<civodul>apologies...
<civodul>unlikelty
<civodul>more likely is a future or par-map bug
<tupi>is there a way for you to reproduce this ? i hardly can send you anything, my processes triggers octave, clojure, java, c++ ...
<mark_weaver>tupi: it probably doesn't matter what external processes are being run.
<tupi>maybe based on marks thread-safe test, just making the threads a little more cpu consumming ?
<mark_weaver>tupi: in the worst case, you could always avoid futures altogether and just use plain threads.
<tupi>mark_weaver: i agree, i am concverned how to reproduce in a easy way so that civodul can debug
<mark_weaver>I could write you a version of 'par-map' that doesn't use futures.
<mark_weaver>(not an ideal solution, but we need to get things working for you asap)
<tupi>my problem is indeed that without proper multicore, the solution i proposed simply does not work [it would take weeks to get the results]
<mark_weaver>what primitives do you need? just par-map/par-for-each, or others as well?
<mark_weaver>because those should be fairly straightforward
<tupi>and i can see that once it starts not using all cores, it also is not using any core at 100%
<tupi>mark_weaver: i only use par-map
<mark_weaver>okay, I'll code something up.
<tupi>great!
*civodul had forgotten about %port-alist
<mark_weaver>civodul: it didn't exist
<mark_weaver>(well, it existed from C, but not from scheme)
<civodul>yeah, i mean i forgot about port alists
<civodul>mark_weaver: so the thread-safe-popen looks good to me, thanks for working on it!
<civodul>i have minor stylistic comments
<mark_weaver>cool!
<civodul>but otherwise that looks great
<civodul>do you plan to git-send-email it?
<mark_weaver>yes. so, the mutex_lock_with_async things (not sure about the name) are important, I think. I'd like to do a comprehensive search around the tree to find other places where they should be used.
<civodul>yeah i wasn't all that clear on that
<mark_weaver>tupi: okay, I have implementations of par-for-each and par-map.
<tupi>mark_weaver: fantastic
<tupi>can you put it in the thread-popen-safe branch ?
<mark_weaver>(par-map is implemented in terms of par-for-each)
<tupi>thread-safe-popen i mean
<tupi>or another branch ?
<mark_weaver>I think another branch. give me a few minutes please.
<tupi>sure
<mark_weaver>(I had just made an external module, but now I'm putting the new procedures into (ice-9 threads)
<mark_weaver>)
<tupi>ok
<mark_weaver>tupi: okay, it's the 'alt-par-map-and-thread-safe-popen' branch.
<tupi>ok trying it
<mark_weaver>git fetch && git checkout alt-par-map-and-thread-safe-popen
<tupi>compiling ...
<tupi>ok, testing
<tupi>they all drop again