<robin>wingo, immix looks pretty promising; i like your idea of extending it for gengc (it was slighly concerning that they only tested it on 1/2-cpu systems in the original paper but i didn't notice any obvious barriers to scaling up) <robin>is there a 'canonical' implementation? there seem to be ~5 at least (the original for mmtk, scala-native, haxe, crystal?, some language called inko...) <robin>i wonder whether it might make sense to adopt something like ravenbrook's memory pool system (TIL originally implemented by harlequin!) or mmtk <robin>(and i assume java has something similar with its multiple GC options) <robin>as it would be quite nice to have a flexible gc framework for experimentation (perhaps some of them can even just act as a minimal wrapper for bdwgc as a baseline for testing) <robin>scala-native's immix implementation (https://github.com/scala-native/immix) is under 2 kLOC so it's presumably not terribly complicated to implement on its own (though it might differ from the original version described in the 2008 pldi paper, ofc) <robin>i haven't really looked closely at any of mps, mmtk, etc. but mps is certainly lisp-friendly for obvious reasons and was proposed as a replacement for sbcl's gc (as a gsoc project idea) <drakonis>robin: mps is currently being worked on by sbcl developers <drakonis>there is parallel gc getting worked on too <robin>drakonis, i was curious about that, from the docs it sounds like there's currently a single mps thread used for any nontrivial operations (well, "per arena" but guile would only use one arena) <drakonis>by the way, what exactly are you replying to? <drakonis>not seeing anything on the logs about immix besides what you said <robin>drakonis, oh, wingo sent me a link to the immix paper here on irc (https://sci-hub.se/10.1145/1379022.1375586), noting that it might be better than bdwgc (maybe combined with a generational collector for scheme objects + pinning C-allocated objects in old space iirc) <robin>drakonis, is the sbcl-related work public? not obviously in the sbcl or official mps repos <robin>not sure if parallel gc is critical for the average guile program, but certainly useful for large systems <robin>ty drakonis, no public recordings but i should be able to find something (or just ping him about it) <robin>wingo, mps lacks support for parallel gc :/ (sbcl developers are hacking on it but no progress on the parallelism front, according to someone who watched katzman's talk at https://boston-lisp.common-lisp.dev/ last year :/ <wingo>robin: so first thing for guile to do is to take control of its gc. not generational, just parallel stop-the-world immix. mutator threads mark their own stacks. cooperative stop via safepoints, and sometimes wakeups for threads that are polling or so. (see asyncs.c; needs some tweaks to let GC run even when asyncs are disabled, and to let GC run before asyncs) <wingo>stacks for threads not in guile mode get marked "remotely" <wingo>that should already yield a speedup i think, because of decreasing L1 misses because of preserving allocation order. also we get to specialize the gc a bit more to guile <wingo>then once that's working we see about doing generational collection. need write barriers though, for that, so it's a gnarlier change <robin>wingo, plus parallel+concurrent "global" defragmentation for immix iiuc <wingo>like those threads don't mark their own stacks because they are not in guile mode <wingo>we can't cooperatively ask them to stop <wingo>so other threads mark their stacks <wingo>up to the point they left guile mode <wingo>robin: anything concurrent is going to be really tricky i think <wingo>in the sense of marker / etc running concurrently with the mutator <wingo>i was interested to see the result in the ocaml paper that their parallel stop-the-world gc worked better than the concurrent one <robin>yeah, concurrency seems harder than parallelism <robin>fwiw there are sbcl (and future sicl?) users who want parallel mps, that may be harder than just dropping in mmtk though (i'm not enthusiastic about adding a rust dependency, but if it works out-of-the-box...) <wingo>honestly i would reimplement in guile <robin>old-school lispm style with non-consing code? <wingo>that libpas always_inline trick gives you template monomorphisation in c :) <wingo>could be an interesting tool <robin>ah, mps (ansi c) may be even more promising there, because you get everything 'for free' except parallelism and immix <wingo>what is mps? mark-sweep, mark-compact, something else? <robin>lots of options, there are at least nine 'pool' classes (for automatic management: mostly-copying, mostly-copying zero rank, mark and sweep, weak linked -- no, i don't know what all of those actually mean, but you can write your own too) <stis>Hmm how is erlang doing gc? <stis>erlang is probably a sphagetti stack VM <stis>wingo: what's the goal about compacting gc, is it even doable? <robin>each process has an independent heap (shared-nothing) except for a global pool of binaries (~=bytevectors), dunno the details beyond that, but the requirements are certainly different than most languages (i think) <stis>Hmm that's interesting, that can make things speedy? <stis>e.g. have a big slow managing thread and let the small speedy thingy live for themselfs <robin>yes, there are lots of libraries for process management (a supervisor process that can restart failing subprocesses being the most common, maybe) <robin>fast enough for telecom applications and whatsapp (i'm not sure if it's 'fast' so much as 'fast enough and extremely reliable', though it has both a bytecode VM and a native compiler) <robin>wingo, that always_inline trick is quite clever! <robin>the blocker with immix and mps, btw, is that mps has effectively one per-process thread for nontrivial operations -- so, minimally, independent collectors for every thread plus global parallelism for defrag would take some work (hard to say how much, esp. since it's a rather large framework and that kind of change would probably affect nonobvious components, see e.g. https://www.ravenbrook.com/project/mps/master/manual/html/) <robin>so it may indeed be easier to just diy :) <robin>i think i really just need to skim the mps source to guesstimate how tricky it'd be though, as you really do get a lot of nice things for zero effort (e.g. fine-grained gc 'telemetry', trivial-ish implementations of things like ephemerons and blobs as new pool classes iiuc) <robin>i can imagine some obvious ways to 'modernize' mps but they're probably subtly wrong (i still have that ~500-page gc handbook to slog through -- nearly as long as graeber's (rip) final book aiming to upend half the field of cultural anthropology and i've barely started that one! though i did get through liu cixin's _a remembrance of earth's past_ trilogy in a week or so, maybe it won't be so bad, especially if the gc handbook happens to have a compelling <robin>narrative structure and a large cast of interesting characters ;)) <robin>no hardcover of the second edition afaict :( but it sounds like the first edition + errata is roughly equivalent <stis>Hmm, combining sphagettis stacks and exceptions/continuatoins are not easy <wingo>stis: is doable! immix compacts. references from conservative roots (c stack, c globals) are pinned but in a granular way (128 or 256 byte chunks). <stis>wingo: I think that sphagetti stacks would enable better performance for guile fibers!! <wingo>probably, yes! would require some reworks tho <stis>yep, I can try to benchmark if you help me. <stis>wingo: got 60M iterations out of simple generator code, that's 20X better then with prompts <stis>got 3X form making jit working with it <stis>wingo: great work on the compiler and jit engine. I fond the architechture really nice <wingo>there is another 2x-3x or so on the table one day, if i can manage to find the time... <RSBach>Hi all. I am encountering an issue buiilding guile 3.0.8 for arch linux from the AUR package guile3. Here is the PKGBUILD and the errors I am getting https://bpa.st/VZHQ <RSBach>It seems some header providing 'struct pollfd' in 'fports.c' is missing. <RSBach>I tried to solve this same problem last year, but now I am trying again. Please let me know if there is any more information I can provide. Sorry to provide so much OS specific context to this; I would like for my package manager to be aware of this bleeding edgeness ;) <dsmith-work>Onmy deb box, that's provided by the libc6-dev package *daviid worked for harlequin - long time ago :), not on mps though, but as part of the lispworks group <RSBach>I have poll.h. It just includes <sys/poll.h> <RSBach>I chased the problem much deeper into the bits headers related to sys/polll.h last year at some point. I am forgetting a lot about it though. <RSBach>This makes me even more confused that `POLL_IN` is suggested to replace `POLLIN` in my error dump, as `POLLIN` is defined in <bits/poll.h> for me <drakonis>robin: to be honest, what guile presently needs in the need is to reach a level of metacircularity as sbcl does <drakonis>C still comprises the majority of the project's language distribution <drakonis>sbcl is almost entirely just common lisp <drakonis>but then, i don't think i'm being fair here, it has had a significant lead for decades now <RSBach>drakonis: how would you see such a change affectign guile's embeddability? <drakonis>i suppose that would sacrifice much of it <drakonis>but on the other hand, it would open guile to much grander things <drakonis>perhaps it could still be retained with some work? <drakonis>its really not trivial work to bring it up to a level equivalent to sbcl's <RSBach>drakonis: yeah, I think it should still be possible. It has been a while since I embedded guile into any projects (they were onky toy projects to prove how to do it), but we need to "initialize" the lisp engine anyway, right? Haskell can make shared objects that need to be initialized, so maybe guile could expose some of the libguile api for C without actually writing it in C. <drakonis>i was reading guile's documentation and source, yet i did not find any indications that guile in its present form can provide a convenient way to do something similar <drakonis>not without pulling apart its guts to make room for it <stis> FYIdrakonis, I have an assembler ported from sbcl <stis>actually used it to jit guile in an earlier attempt <dthompson>guile's C code is slowly being replaced by Scheme code, as advances in the compiler make it feasible. <stis>yeah my pet peve is that dynamic wind code is executed from C and is slow <drakonis>there arent infinite warm bodies to throw at problems <drakonis>there are some C bits that are kept around because doing them with scheme right now is too slow <stis>serializing nonserialising of scheme datastructures such as lambdas ckontinuations and other not spoeaasilly. " approaches actuually <stis>my python on guile can serialize python generators!! <stis>can't do that in CPython <stis>also contains a superfast factorial function and other goodies form gmp <stis>one of the the serialsizer are streamable, so you do not burn much gc and memory serializing large datastructures <stis>and can be handled at C speed <stis>(moving chunks of memory around in stead of sinfgle values) <civodul>wingo: read hash extensions are still supposed to use source properties, right? <RSBach>I was acting brain dead. I had '/usr/include/guile/3.0' in my C_INCLUDE_PATH from when I linked C to it last year. I guess fports.h was getting the poll.h file from libguile instead of poll.h from my system. <civodul>wingo: commit 54bbe0b2846c5b1aa366c91d679ba724869c8cda removed the non syntax? case in source-annotation, and i think that case was useful typically when you have an sexp returned by a read hash extension <lilyp>can we make read hash extensions produce syntax objects? <robin>dsmith-work, ooo that's awesome (lisp hacking at harlequin) :D <robin>drakonis, i concur, in the long run; imho it *could* be done without sacrificing much in the way of embeddability <robin>[...that more of guile ought to be written in guile itself] <drakonis>there's endless improvements that can be achieved by doing that <robin>guile's already taken some large steps in that direction in the last decade+, ofc <robin>sbcl has 10x more lisp than c, for guile it's about even <daviid>robin: yes, quite an experience :) <drakonis>and the places where C exists are largely interaction with the host operating system <robin>maybe some of the *current* scheme<->c interop would become a bit less efficient or more difficult, but that might not matter if guile itself is significantly faster <drakonis>being able to provide much greater optimization flexibility is worth it <dsmith-work>robin: Ahh, you prob tab-completed me instesad of daviid <robin>daviid, i'll have to ask you about harlequin war stories someday, but did you ever happen to meet kmp? <daviid>no, i left just before he joined <robin>ah right, he probably went down with the ship at symbolics or something <robin>i learned so much from his essays and usenet posts back in the day (even some interesting ideas for cl<->scheme interop, if i can ever dig up the notes on his website again :p) <singpolyma>Is there anything like apply but for syntax-transformer? <daviid>robin: this was 93-94 (my 'trip' at harlequin) <daviid>drakonis: aiscm has a tensorflow partial binding, if everything goes well, i plan to add one to guile-cv <drakonis>in my opinion, it is impressive that it is competitive with other options <singpolyma>Ok, so I found (macro-binding) which gives me a procedure I can call, but it seems like while it does call the macro ish it does not expect a list quite the same as the direct macro would ***civodul` is now known as civodul
<sneek>I've been running for one month and 15 days <sneek>This system has been up 29 weeks, 6 days, 7 hours, 36 minutes