<Rashack>DeeEff: that i can't explain, seems strange to me <DeeEff>In fact, AFAIK, I think you should only be able to have num-processors * 2 number of threads <DeeEff>so isn't n-par-map doing something it isn't supposed to? *paroneayea is being dumb <Rashack>i'd say n-par-map does exactly what you tell it <DeeEff>well depends, I thought n-par-map and par-map were utilizing native threads <Rashack>the maximum numbers of threads depends on the kind of work they are doing <DeeEff>but, if they're using green threads then it makes more sense <Rashack>if they're just sleepin, it's moslty memory contsraints <ijp>guile only has native threads <Rashack>isn't it the same thing these days on a recent linux system? <DeeEff>to be fair I don't fully understand parallelism primitives or how the OS manages them, so I couldn't say <DeeEff>I just think it's strange that you can open 400 threads on a machine with 4 processors <Rashack>you have to differentiate between running processes (threads) and physical processors <Rashack>a thread is just a promise of execution (or something like that :) <Rashack>the os (linux, or the guile runtime depending on what you are doing) is responsible for giving the threads time on the cpu <Rashack>think of a thread as a program that you can suspend <Rashack>which is basically what you are doing when you call (sleep ...) <Rashack>and the system notices (well, sleep tells it) and knows that it can let another thread run for a while <DeeEff>still though, if a thread is supposed to sleep for a bit, why does the rest of the body of the lambda execute and return before the sleep is over? <Rashack>moving the focus back to par-map and n-par-map: when you use par-map, it is restricted to (current-processor-count) number of threads <Rashack>and n-par map starts as many new it needs to, up to the number you specify <Rashack>well, it doesnt really, but par-map is using the same lambda over and over again <DeeEff>okay then, so when using par-map, isn't it just a better idea to call n-par-map with the number of threads equal to the length of whatever you're mapping over? What's to stop someone from starting 900000 threads? Sure they won't run all at once, but do they take up a lot of additional memory? <Rashack>DeeEff: try this: ,t (par-map (lambda (x) (sleep 1) (display "foo") (* x x)) (iota 9)) <Rashack>and you'll see that the foos start coming in chunks, after each sleep is finished <Rashack>DeeEff: 900000 will probably eat most of your memory, unless you use Erlang :) <Rashack>it all depends on the work your functions are doing <Rashack>if they are waiting a lot, you can have a lot of the running <DeeEff>so a future depends on the size of the content passed into the body? <Rashack>if all of them are using an "old" spinning harddrive at different ends of the drive, you'd better jus have one of them at a time or it'll take a long time <Rashack>not that simple, all of them could potentially generate loads of data and allocate all of the available heap <DeeEff>so in practice, when is it a good idea to use n-par-map and what are typically the rules for the number of threads you want? <DeeEff>even in high level terms, if possible? <DeeEff>sorry for all the questions, I'm somewhat new to this parallelism thing <Rashack>it's easier to say this or that for specific use cases of course <Rashack>but if yout threads are doing calculation intesive tasks, it's probably wiser to go with par-map <DeeEff>any specific reason? just so I don't hit memory limits with a vast number of threads? <DeeEff>because I always thought 2 * num-processors was the usual number of threads you'd use for very computationally intensive tasks <Rashack>i wouldn't using n-par-map on, hmm, "dynamic structures", such as user dependent input or lists with wildly varying length, since you could easily exhaust the memory on your system <Rashack>i'm by no means an expert down to the metal regarding these things, as processors evolve things get more and more complicated <DeeEff>hrm... Well, as far as I could tell, par-map provides a drop-in replacement for clojure's pmap, so I guess I'll just have to experiment with it and see what happens. <Rashack>imagine you have 2*n-procs threads running cpu-intesive tasks, half of them have to wait all the time, right? <DeeEff>I suppose that's true. The mock example I chose to test it out was probably bad, considering sleep will tell the thread to do just that <Rashack>the semantics of pmap are the same, but the specifics...? look at the code ;) <Rashack>remember that clojure iz lazy, so i guess the results can be even more interesting there <DeeEff>this is true. From the looks of it, they use number of processors + 2 as the number of threads. <DeeEff>Likewise, I'm unsure if the JVM has native threads, so I think it's different in that respect too <DeeEff>rather, if _every_ JVM has native threads <Rashack>the same as guile's (according to ijp) <Rashack>Erlang has it's own implementation, but erlang philosophy is designed around processes and message passing <Rashack>DeeEff: if you want to scrape a few hundred web-sites, you'd probably want to use n-par-map in favor of par-map <DeeEff>okay, so computation heavy tasks -> par-map, short lived tasks -> n-par-map <DeeEff>though, wouldn't you use n-for-each in that case? <Rashack>par-map will only attempt to suck down four (on my system) at a time, whereas n-par-map will try as many as you ask for, and since fetching a web-page is i/o-bound and a lot of waiting, it will be more efficient <Rashack>it's about intensity and wait states <Rashack>n-for-each is perhaps better here, but there are probably even better solutions ;) <Rashack>just trying to make the difference obvious <DeeEff>alright, thanks for the help then <DeeEff>I'll try to keep that in mind in the future, but I expect I'll be back eventually <Rashack>and the fashionable solution here is of course to use a message queue :P <Rashack>one process produces the links and another consumes them and passes them on to a thread pool for downloading :) <please_help>Do I have to redo the scm_make_smob_type (etc.) calls if I have a second C module that will operate on inputs of a smob type described in the first C module? ***__zxq9__ is now known as zxq9
<please_help>I got one module to work under guile correctly, but when I try to (load-extension ...) my second module in a clean guile environment it just says "file not found" despite the file being here. <davexunit>paroneayea: it is, but there's something about it, something I can't put my finger on, that makes me shy away from it and use regular record types instead. <paroneayea>davexunit: could be realllly helpful in the enemy example tho <paroneayea>davexunit: so if I wanted to draw layers of things, am I right that the best way to do it would be to make each layer a signal, then make a meta-signal that group-move'd each one to different z values? <please_help>Is the best way to go from bytevector to f32 vector (given the encoding is known to match) mapping through the bytevector and using bytevector-ieee-float-ref? <boxofrox>ok, very new to guile and lisp in general. using guile 2.0.11. trying to write my first macro and running into a glitch with (quote). <boxofrox>trying to match arguments that are lists of the form (a b), but catching symbols in addition because they expand to (quote symbol). <nalaginrut>boxofrox: '(('fred . 0) you don't need one more quote here, '((fred . 0) is enough <boxofrox>the first ' was for debugging, is the second messing up my debugging output? <nalaginrut>boxofrox: I don't know your purpose, but if you use another quote here, you can't fetch it with assoc operations correctly <boxofrox>the assoc list was a quick alternative to make a simple use case for my question here. I'm actually using records in my main code. <boxofrox>ultimate objective is to organize my counters I use for profiling how often parts of my program run. <boxofrox>despite my code using (counter/add counters ('fred 2) 'ted), can I expect the syntax-rule to analyze (counter/add counters ((quote fred) 2) (quote ted))? <nalaginrut>boxofrox: can you remove the second quote than try what you said? <boxofrox>nalaginrut: sorry, i did apply your change. same error as described at the bottom of the pastebin. the error crashes guile during the macro evaluation before the assoc list is accessed. <boxofrox>trying to wrap my head around how I distinguish ('fred 2) from (quote ted). i'm using symbols for testing, but anything usable as an assoc list key is intended to work. may have to scrub that dream. <ijp>boxofrox: in syntax-rules/case, yes, you can use the names quote/unquote/quasiquote <ijp>for one, the pattern matching macro does this <boxofrox>ijp: i'm not sure I follow. I suspect you mean to add another (pattern template) specifically for the (quote ted) scenario. I'll try that and read up on unquote and quasiquote. <ijp>boxofrox: I thought you were asking if a pattern (quote foo) matched 'foo <boxofrox>ijp: not quite. my pattern (_ (a b)) is intended to match something like ('foo 1), but is catching 'ted because it expands to (quote ted). <boxofrox>but adding quote to the syntax-rules literals, and adding a pattern specifically for (quote whatever) seems to be working past my error. <ijp>oh right, you wanted a symbol match <ijp>right, that is the way to do it <please_help>I have a smob definition in one module, and another module wants to operate on the same kind of smob. Is it safe to redo the calls to scm_make_smob_type et al., or is there a better way to communicate the smob type between modules? <nalaginrut>boxofrox: in your case, I think you want to call (counter/add counters ('fred 2)), no? <nalaginrut>I don't understand the meaning of (counter/add counters ('fred 2) 'ted) <zacts>my cat caught a mouse last night <wleslie>today I literally did nothing but write code. I think I probably should have had some kind of exercise. <wingo>tuesdays are for eating food <wingo>wednesdays are for exercising <wingo>you can really focus on one task without interruptions <wingo>civodul: i got slot-ref and object creation performance to a level that they are faster than 2.0 when accessed from Scheme but slower than pre-refactor 2.2 <wingo>i think those are the only speed-sensitive operations; nothing much changed regarding generic function dispatch until now <wingo>for better and for worse, i introduced "slot definition" objects that replace the format of the old "slots", "direct-slots", and "getters-n-setters" properties <wingo>it's better because it's more structured, more expressive (if we add a protocol there), and closer to CLOS <wingo>and all the old accessors and protocols still work <wingo>but the actual format of the slots is different -- (slot-ref class 'slots) is different, though it was undocumented before, and there is no more `getters-n-setters' slot <wingo>dunno though, it seems like the right thing... <wingo>anyway, still working on commenting and documenting after that speed golfing <boxofrox>nalaginrut: sorry, went to bed. here's the follow up. I started with (counters/add lst name), which incremented the counter with `name` in list `list` by one. while writing unit tests, i found myself calling (counter/add) many times to bump multiple counters twice or thrice. So I decided to condense all that into one call. (counters/add lst ('fred 2) 'ted)) is the same as (counters/add lst ('fred 2) ('ted <boxofrox>1)), and results in 'fred having +2 to its count, and 'ted having +1 to its count. <boxofrox>not sure if i'll use that extended functionality in my other code, but (counter/add lst 'fred) still works as before, my unit tests are less verbose, and I wrote my first macro. <paroneayea>something I haven't found documented anywhere, but sometimes see <paroneayea>but it looks like if you add the procedure-or-something, the procedure-or-something becomes a function? <davexunit>paroneayea: check out "6.13.4 Iteration mechanisms" in the manual for docs. <davexunit>if viewing the manual in Emacs, which I know you are ;), press 'i' and then type 'named let' <paroneayea>maybe a "see also 'named let' for an extension of the let syntax" should be added to the texinfo manual, or something... <mark_weaver>I don't much like the solution Hans suggested though. If there's a long chain of weak-key hash table entries, such that each one's value points to the next one's key, then I guess only one link of that chain would be freed per GC cycle. <mark_weaver>IMO, a proper solution requires marking the value of a given weak-key hash table entry only after the key itself has been found to be reachable. <civodul>i don't feel like implementing it right now ;-) <civodul>but the problem with make-fresh-user-module is different no? <civodul>it's just that modules are registered in a global name space, isn't it? <mark_weaver>well, my idea is that (make-fresh-user-module) would create a module with name (%ephemeral <GENSYM>), where (%ephemeral) is a special module whose 'submodules' table is weak-key. <mark_weaver>and then, if there are no more references to <GENSYM>, the symbol itself is collected and then the module. <mark_weaver>but, it doesn't work because of the problem you raised on the bdwgc list. <mark_weaver>actually, I think Hans's proposed solution might not work at all if there are two weak-key hash table entries that form a cycle, i.e. one entry A->B and another B->A. <civodul>oh, the %ephemeral name space is a nice trick <civodul>but yeah, what Hans suggests is hairy at best