<janneke>mescc compiles strcmp to handle --help :-) <davexunit>paroneayea: resuming the actor-alive? discussion <davexunit>I think the recursive messaging thing is nice, but it's kind of a bummer that you couldn't have a local loop without calling actor-alive? on every iteration <davexunit>(while #t (foo) (something-that-yields) (bar)) <davexunit>the reason why I have so much of an opinion here is because I ran into this same problem <paroneayea>though it would require another check on every time we resume any coroutine <paroneayea>well, there's maybe another way to do it I can think of <paroneayea>I've been thinking of wrapping a registered actor in a record anyway, and I could set a dead/alive flag on it <davexunit>in my library I didn't have actors, it was more like fibers, but I wanted an enemy to be able to die and have all their threads killed. <davexunit>wrapping each continuation in a lambda that tested for aliveness did the trick <davexunit>then I was only doing the check in one place <paroneayea>davexunit: I'll make a TODO item to play with it. <paroneayea>davexunit: I recently added a wrapper around continuations for another reason <davexunit>it's analogous to escaping strings in templating engines to me <davexunit>enforce the rule in one place and no one can screw up <paroneayea>so I could make use of that to add a resume? thunk <davexunit>you said it was simple to test if an actor was alive <davexunit>I wouldn't worry about the number of hash-ref calls unless you see a clear performance hit from it. <davexunit>ACTION says as he ruthlessly tries to optimize some of his own code <paroneayea>davexunit: some of the artifacts of 8sync were that initially I thought the agenda itself was going to be a lot more full-featured <paroneayea>but over time the actor model has become where everything interesting has happened <paroneayea>as something that kicks off another thing scheduled on the agenda <paroneayea>but, I'm convinced that i shouldn't feel bad about this historical aspect <paroneayea>I saw somewhere an advocacy of lisp programming / repl driven development / live coding as "finding the right design through successive approximation" <paroneayea>and 8sync has definitely been a product of that. <linas>so how does one examine the heap? <paroneayea>luckily I think we're starting to get to the right core ideas in 8sync now. <paroneayea>linas: guile's heap? maybe I'm wrong but I don't think you do? <davexunit>doo dee doo, don't mind me just replacing core bindings min/max with macros... <paroneayea>linas: I could be wrong but I think the heap is not much exposed, especially since the garbage collector does a lot of management there and we don't have a lot of control over it <linas>well, my heap seems to be grwoing without bound, and I want to get a clue about why <linas>probably some crazy bug in my code... but I don't know where to start <davexunit>you can run the program with GC_PRINT_STATS=1 to get some info <paroneayea>linas: there's also a gcprof procedure in the statprof library <davexunit>and maybe gcprof will have some info but I don't think it will help in this case <linas>that displays something more than just (gc-stats) ? <davexunit>I don't think you can get the info you are after <paroneayea>davexunit: so there are some things I should adjust for sure... but you think the general direction of 8sync looks good from what you read in the tutorial? <linas>+1 for 8sync, btw .. I just got off a long jag of nasty C++ netowrking code that was ... tediously buggy <paroneayea>one thing I'm excited about is that the design does lend itself well to mixing in a lot of different types of actors <paroneayea>so it would not be hard to be running a MUD game like in mudsync, and toss in a web server that shows a leaderboard <paroneayea>or even add a web interface with websockets for the GUI alongside the telnet option <paroneayea>I hope we can make 8sync reasonably batteries-included for things like web stuff, basic worker queues, websocket support, etc <davexunit>I had visions in my head of using it for games <davexunit>one thing I thought of was a network multiplayer bomberman clone <paroneayea>oh man I accidentally got myself sucked into a distraction vortex <paroneayea>now I'm watching gameplay footage from Creatures for the c64 <paroneayea>I guess there was some weird stuff int hat game too <davexunit>artyom-poptsov1: neat. ICS is a calendar format, I take it. <artyom-poptsov1>Oh, well, I could parse Google Calendar stream with it, for example. <artyom-poptsov1>And I could use it with an IRC bot living in our hackerspace's chat room to handle requests about events or something. <artyom-poptsov1>There's some work to be done before the Guile-ICS release; at least I should write Texinfo documentation for it. :-) ***baijum is now known as Guest68954
<add^_>Hey Ludo, long time no see! (Although I wouldn't be surprised if you've forgotten about me) <paroneayea>davexunit: I've thought a lot about your suggestion to just make all coroutines simply not resume if an actor dies <paroneayea>right enough where I should delay the 8sync 0.4 release to change this <linas>Sooo .. I'm pondering this pseudo mem-leak, which I totally don't understand. <linas>I have some guile c++ code that mallocs a 100-byte object. <linas>I have some scheme code that calls the above as fast as possible in inf loop. <linas>For 5-10 minutes, the guile heap grows fast, but then settles down. <linas>after 10 minutes, the malloc has been called about 100 million times, and the guile heap is at about 100 million bytes. <linas>after 20 minutes, its 200M calls and 120MB heap <linas>after 40 minutes its about 400M calls and 140MB heap. <linas>and the guile heap usage seems to slowly slowly crawl upward. <linas>Also: `top` shows that VIRT is about 10x larger than the guile heap size, and RESIDENT is about 2x larger than the guile heap size <linas>I'm actually using c++ operator new <wingo>linas: malloc malloc or scm_gc_malloc ? <wingo>so, i have the answer, let me find it <linas>operator new which is malloc and then scm_gc_register_collectable_memory <linas>its vaguely like memory-fragmentation <wingo>you are calling register_allocation or no? <linas>except all the objects are the same size, so it shouldn't really fragment <linas>guile 2.1 with scm_gc_register_collectable_memory <linas>although I beleive I see the same thing with guile 2.0 also <wingo>wait you are leaking the memory? <linas>I call scm_gc_register_collectable_memory and then guile does whatever it does. <linas>Its currently "leaking" at the rate of maybe 0.1 bytes per malloc. The "leak" gets slower and slower <wingo>i didn't see where in your description btw where you were allocating managed memory <wingo>if you just call a function in a loop that shouldn't allocate anything, right? <wingo>when do you free the memory? <linas>I never free the memory, I let guile do it. I poke the memory into a smob <linas>Its almost asymptotic. It almost stops growing after an hour (and 600M mallocs) <wingo>i think everything might be ok then. what do you do with the smob? <wingo>do you use finalizers to free the memory then? <linas>I registered the free with scm_set_smob_free <linas>Its a "psuedo-leak" -- after an hour, maybe 60GB has been malloced and freed, the guile heap is about 140MB so in that sense "its all OK" <linas>but after 90 minutes, the guile heap is now 200MB .. still quite mangeable. <linas>but seemingly way too large for a loop that is only mallocing 100 bytes at a time <wingo>i encourage you to look into it if you are interested in these things :) <linas>The production system zooms up to 10GB of guile heap before it "stabilizes" -- that's what is driving this <linas>yeah .. and it will stay there for a few hours and then go up to 12 GB... <random-nick>linas: did you test with using C's malloc instead of C++'s new with the same code? <linas>the production system has maybe 50 or 100 guile threads doing all kinds of crap, allocating smobs almost as fast as possible, pushing things around. <linas>I mean, it seems to all work great, except that the mem usage is a bit outlandish. <linas>wingo I am trying to decide if I should write a tiny stand-alone demo for this. <linas>the problem is.. I'll write it, run it, reproduce the problem, and then.. what, dive into the guts of bdw-gc? That's maybe more than I want to do. <random-nick>linas: well, when you reproduce the problem you can try to experiment with different things <random-nick>linas: like running it with an older guile version (2.0?) <linas>the "productin" system is on 2.0, I beleive the problem exists there too <linas>I can double-check to see if its better or worse, but I suspect there's no change <paroneayea>davexunit: I just realized i'll need to move 8yield and 8sleep to be being managed by the actor system to make that work. Maybe I'll make this a 0.4.1 thing :) <paroneayea>I'll just add a footnote that in 0.4.1, checking actor-alive? will likely not be necessary. <wingo>see the impl of scm_gc_register_allocation, and note that if you have free functions on your smobs, those smobs are asynchronously finalized, so to an extent you have the main prog racing with the finalization thread, and free() takes more time than malloc()... <paroneayea>but 0.4.0 is so close, I should get it out the door I think! <paroneayea>thanks for prompting me on it though... I think it's the right direction :) <linas>wingo: ahh! racing! thank you! that is actually something I can fiddle with! I will track that and see! ***dje is now known as dje42
<wingo>ACTION tries to get parallel fibers + guile web server working <linas>wingo your hypothesis about racing seems competeley correct <linas>(p.s. love what I read about fibers) <linas>so adding atomic counters to the maloc/free code completely eliminates the growth of the guile heap <linas>the atomic counters apparently have a side-effect of stalling any malloc threads, until free catches up. <wingo>linas: i was referring to a malloc/free race between mutator and finalizer; afaiu there was no race in bytes_until_gc, was there? <wingo>or was there a race somewhere else? <wingo>the race i was referring to would just mean that after a GC, you don't actually decrease heap size by much because there's finalizers, it takes two gc's to free the heap and by then maybe you have grown the heap again or something <wingo>looking forward to your writeup tho :) <wingo>ACTION has parallel fibers working with the web server, whee <linas>I put one atomic counter in the malloc right before SCM_NEWSMOB <linas>I put the other atomic counter right before the free, which runs in guiles/bdw-gcs finalizer <linas>i.e. in the free routine that is called by scm_set_smob_free <wingo>linas: oh you mean a race in your code, between the code that creates the smob and the code that frees it? <linas>"race in my code" is not how I would put it. I have multiple threads, many of them calling SCM_NEWSMOB rapidly <wingo>davexunit: yes -- though there's a knob. by default it uses the main thread only for the handler that threads the server state, and uses other cores for reading the request and writing the response <wingo>linas: so what counter did you change to be atomic? <linas>and then, asynchronously, guile calls the free routine I registered with scm_set_smob_free() <linas>I just added the c++ std::atomic<unsigned long> so that I could count how many mallocs and frees there are <linas>I beleive c++ std::atomic<unsigned long> uses a mutex under the covers <wingo>hm i don't think so, it uses atomic operations depending on your compilation target <wingo>but yeah, it guarantees atomicity <linas>Simply adding those counters has the side effect of preventing the growth of the guile heap. No other changes <wingo>i believe you but that is pretty weird :) <linas>well, I am guessing that the atomic in the async-finalizer-free must somehow making the malloc thread(s) stall <linas>but yeah, theat's weird, I'd have to do hand-waving about cache-line flushes to make it vaguely beleivable <linas>(since atomics interat with the cpu cache lines, I believe, and maybe the malloc thread just keeps loosing over and over which the free thread spins madly, freeing everything...!?) <linas>now to see if the production system can be cured the same way... <wingo>well inserting an atomic forces your whole program into a sequential-consistency mode around those atomic operations <sneek>Welcome back civodul, you have 1 message. <sneek>civodul, davexunit says: I tried removing ld-wrapper from a build using package-with-explicit-inputs but the resulting shared library still has hardcoded reference to glibc libgcc. what's up with that? <paroneayea>davexunit: btw, I'm wondering where I'm going to "store" the captured stacks from errors <paroneayea>and I might want to still access the captured stacks there <paroneayea>I could also stick the errors on the hive, which might be a fun, different kind of hackiness. <paroneayea>I kind of don't want to keep tacking slots forever onto the default hive though. <wingo>lloda`: was your lock a livelock or a deadlock? <wingo>i found a livelock recently and will push a fix tomorrow