<ryanwatk`>Anybody know how to fix a 'missed test plan', test ERROR? <paroneayea>janneke: are you talking about adding match to your minimal scheme? <janneke>paroneayea: no, it's about goops and typed-methods vs plain function with matcher <ecraven>ArneBab_: thanks for the ping, I'll try to run them tonight <ecraven>just rebuilding guile now, that'll take a bit ;P <ArneBab_>ecraven: when using the release, building guile takes just 15 minutes for me (it comes with prepared *.go files) <ArneBab_>ecraven: nalaginrut reports actually beating chez on simple benchmarks. <lloda>have any of you built nalaginrut's tjit branch? <wingo>i checked out the source but didn't build it <wingo>it could lead to increased memory usage but when i installed that, i was struggling with finding a leak <wingo>and thinking it was related to our weak hash tables <wingo>civodul: i think i found an interesting thing <wingo>regarding the gc-too-many-times issue <wingo>which in this example where we're rapidly expanding the live object count, we want to expand <wingo>it is choosing instead to collect, too often. <wingo>the test as to whether to try to collect looks like this: <wingo> if (!GC_incremental && !GC_dont_gc && <wingo> ((GC_dont_expand && GC_bytes_allocd > 0) <wingo> || (GC_fo_entries > (last_fo_entries + 500) <wingo> && (last_bytes_finalized | GC_bytes_finalized) != 0) <wingo>GC_incremental is false. GC_dont_gc is false. GC_dont_expand is false also. <wingo>GC_fo_entries is the list of objects with finalizers i think; or is it objects on the finalization queue? i don't know. <wingo>but the fo is for finalizable object. <wingo>my current suspicion is that it's that middle case that's true <wingo>indeed it's the count of finalizable objects <wingo>so... currently we have a mechanism to run a callback on an object after gc, as long as that object is alive <wingo>the way it works is it makes a fresh weak reference to that object, attaches a finalizer, then in the finalizer if the weak reference still holds, it will run the callback and reattach a finalizer <wingo>or rather, s/reattach a finalizer/begin the process again with a fresh weak reference and associated finalizer/ <wingo>so if there are more than 500 objects needing this kind of treatment -- and currently in 2.2, every weak table and weak set has this treatment, and there are a few ways to cause many weak tables to come into existence, notably loading many modules -- then it's quite possible to peg libgc into this collect-rather-than-expand case <civodul>i think we should arrange to reduce as much as possible the number of finalizers in use <civodul>and there are several kinds of finalizers, notably disappearing links and actual finalizers <wingo>disappearing links aren't finalizers <wingo>they are cleared by gc before releasing the alloc lock <wingo>they aren't enqueued to the finalizer thread <wingo>i am a bit hesitant to remove the "vacuum" phase -- if we removed it, we'd have to incrementally vacuum when adding entries to a weak table and that might have bad hysteresis effects (you go to expand -> you vacuum first -> you get back 10% of entries, no need to expand -> next time 9% -> next time 8% -> etc) <wingo>so i think what we can do is just make one finalizer to vacuum all weak tables and sets <wingo>those are the only uses of scm_i_register_weak_gc_callback or whatever <wingo>and those are the ones that add new finalizers in response to finalization, defeating this logic in bdw-gc -- but adding just 1 finalizer won't be a problem <civodul>why do we need to scm_i_set_finalizer (ptr, weak_gc_finalizer, data); after CALLBACK has been called? <wingo>the goal is to get a callback after each garbage collection. the way to get a callback after garbage collection is to use a finalizer. <wingo>we could use asyncs but then it could stall a main thread <civodul>i understand, but the fact that we re-add the callback after we've called makes it look like the callback might be called more than once <wingo>finalizers are de-registered once they are called once <wingo>so at the time that we are re-adding the callback, there is no finalizer registered <wingo>s/once they are called once/before they are called/ <wingo>it makes it so that the weak gc callback is called once and never again <civodul>ok, i hadn't understood that this was about being called after each gc <civodul>so yes, having just one finalizer for all the weak things would be a solution <civodul>esp. since Hans says: "In well-written programs there will typically be very few uses of finalization." :-) <wingo>yeah tricky code, twisty ideas, bad communication on my part :) <civodul>i'm sure Guix will perform better as a result <civodul>i recall reducing the number of finalizers with good results back in... 2009 <civodul>like 8cf0dd6104c58f68e0f0ecdf0ab28a033754acf1 <davexunit>sometimes I use weak hash tables to deal with doubly-linked structures. <civodul>davexunit: using weak hash tables currently triggers GC much more often, but wingo has just proposed a fix <civodul>davexunit: BTW, wouldn't it be great to have (ice-9 json) in 2.2.0? :-) <wingo>i think we could release a 2.2.0 this week fwiw <civodul>so yeah, re (ice-9 json), it all depends on its status <civodul>i vaguely recall that it was in a good shape already but i don't know <davexunit>civodul: there were problems that I never got around to fixing. <davexunit>the latest version of guile-json more or less does what my patch does <davexunit>had it been that way a couple years ago I probably wouldn't have wrote (ice-9 json) <civodul>i think there's value in having a json module in Guile proper <civodul>but i agree that guile-json works great too <davexunit>it's just been hard to pick that patch back up and finish it, for some reason. ***dsmith is now known as dsmith-work
<wingo>actually i tried an unrelated case, i should try the iota case again... <wingo>before, (length (iota #e80e6)) was 75s. now it is 3s :) <wingo>i should try fibers again then <wingo>maybe i get the nice linear scaling that i was looking for <wingo>a simple heuristic causing gc to go quadratic <davexunit>man we gonna go quadratic in this place tonight <janneke>paroneayea: i said to amz3 that i sometimes use[d] goops but without methods <janneke>paroneayea: and instead plain functions with (match o (($ <foo) ..) (($ <bar>) ...)) <janneke>i'm not really happy with either..both bring something useful <janneke>and i was wondering what your idea of that is, paroneayea <paroneayea>janneke: using a matcher is often nice in a single procedure, but if you have code that might be extended by other types added fluidly later, or esp if by 3rd party libs, generic methods are just great. Generic methods doesn't necessarily mean GOOPS style... Structure and Interpretation of Classical Mechanics uses a generic method system heavily <wingo>aaaah, this probably fixes that bad lilypond-on-big-files regression too <davexunit>you can use generics without using classes, even, though in guile 2.2 this is clunky :/ <wingo>lilypond itself creates a couple hundred weak hash tables, and then guile probably has another hundred or so <paroneayea>it should be possible, I think even without the <<foo>> unhygenic stuff <wingo>so in an expanding heap GC_collect_or_expand will almost always choose "collect" instead of / before "expand" <paroneayea>you can check that a value is a record, and extract what record type it is, so... <davexunit>I don't know what needs to be done to make that happen, though <davexunit>but guile obviously knows how to make a class from a record type <paroneayea>davexunit: I think class-of could probably even just return the record type itself <davexunit>my current hack is to do something like (define <<foo>> (class-of (make-foo ...))) <wingo>it would be nice if the record type were a class for the purposes of goops. i think that's where we'd like to go. not there yet tho <davexunit>sounds reasonable to me but there might be something I'm overlooking <paroneayea>janneke: so, Sussman has a couple of lectures he gave recently which you can find on youtube which go into why generic methods are useful <davexunit>I use that class-of trick in my new game library <paroneayea>and he's also teaching a class on symbolic computation which goes into both combinators and generic methods <davexunit>because I have a guardian that can garbage collect GPU-allocated objects (textures, vertex arrays, etc.) <davexunit>and a generic function that calls the finalizer for that type of resource <davexunit>so I need goops classes from record types in a handful of places <paroneayea>the result that comes out of it basically seems to be that combinators are really powerful and beautiful abstractions, though they can become fairly rigid <paroneayea>and generic methods are kind of messy, but are super extensible and easy to keep building on <paroneayea>"APL is like a diamond. It has a beautiful crystal structure; all of its parts are related in a uniform and elegant way. But if you try to extend this structure in any way - even by adding another diamond - you get an ugly kludge. LISP, on the other hand, is like a ball of mud. You can add any amount of mud to it and it still looks like a ball of mud." <paroneayea>so SICM (and those problem sets linked) use a super inefficient but easy to implement generic method system <paroneayea>pretty flexible, but I think it will always be O(n) per number of implementors there :) <paroneayea>whereas GOOPS is just currently O(n) per number of implementers, but it can be fixed IIUC :) <davexunit>like how alists are faster than hash tables up to a certain size <paroneayea>though I've written some things which have a large number of generic implementors <paroneayea>I know that some CLOS implementations try to switch strategies depending on how many implementaters there are <davexunit>I wonder if we could speed up guile's hash tables <paroneayea>guile's hash tables seem fast enough, if we used them <davexunit>remember: you never need to shower if you don't leave your house! <davexunit>I've become 10% more productive with this lifehack! <paroneayea>ACTION isn't really productive until finishing their morning routine, regardless of whether leaving <paroneayea>but certainly, I don't judge others who can utilize that lifehack in their own homes ;) <janneke>wow, thanks -- gives me something to think about <paroneayea>janneke: the upside of SICM predicate check style generics is you can implement them in like 10-15 lines of code... maybe useful for mes minimalism! ;) <dmiles>probly a good usecase for minbikanren <janneke>paroneayea: great, gonna look at that! <davexunit>"we do absolutely bananas crazy things with the guile interpreter, thus guile 2.0 sucks!" ***dje is now known as xdje
<wingo>i think with that gc thing fixed, i just need to add more prebuilt/ links, arrange to issue a warning and instructions if prebuilt/ isn't present, and figure out goops slot design issue <wingo>about slots overriding initialization values, or overriding slots with the same name in subclasses, etc <civodul>wingo: AFAICS it's fixed, indeed! thank you! <dsmith-work>ArneBab_: Thank you so much for your cheerfullness! (in responding to lilypond people) <stis>ArneBab_: did you see what wingo and civodule came up with. Maybe really good news for lilypond with large files <wingo>daviid: i think i just mentioned it in the channel? <daviid>wingo: this gc bug you fixed is awsome! <amz3>paroneayea: interesting course you have here <amz3>paroneayea: how do you do to know that the course changed with wget --recursive? <amz3>you check manually the difference between the page? <paroneayea>amz3: yeah I knew he was teaching it again so I viewed the page again <paroneayea>it looks like he's exposing the problem sets one by one as the class goes through them, and there are new or changed ones <paroneayea>too much mutation though! old valuable stuff isn't there right now ;) <amz3>I just discovered that feature structures are a thing and that NLTK (the python nlp library) already has them <amz3>I think one can define 'match' over alists using unify operator of feature structures <amz3>paroneayea: those are nested alists <daviid>wingo: i did read the log till the end now :) so, to clarify, my email (link above) is not a 'goops slot design' email, it is related to a bug in the expansion code for setters for slots that are _not_ being redefined, but merely inherited <daviid>wingo: it really is an annoying bug :) if you could re read this email carefully, it would be really nice! <paroneayea>wingo: daviid asked me to look at the bug again, and I think I've verified their bug... it can be simplified from the example in there <paroneayea>it doesn't have to do with module export boundaries, can be done in same module <paroneayea>but yes it's surprising that setters seem to not be inherited with accessors, even without slots being redefined <daviid>paroneayea: now, comment the definition of the accessor in <a>, then call the setter, and then call the accessor (which should have been defined automatically) on a <b> instance <daviid>paroneayea: wingo wrt to this, I beleive that (1) the expansion of (setter generic-with-setter) should create an accessor if the user did not create one, then (b) the user should be able to create the accessor, in which case the setter should (also) be inherited <paroneayea>I suspect that a slot mechanics refactor might involve changes to this anyway though <daviid>paroneayea: I don't uderstand that sentence <wingo>ArneBab_: you found a nice CSE bug :) <civodul>wingo: on a quick unscientific benchmark, we have a 33% speed improvement on: ,time (guix-build "libreoffice" "ghc-shelly" "-n" "--no-substitutes") <wingo>civodul: relative to the bad gc behavior in 2.1.8 or relative to 2.0? :) <wingo>i guess you picked up the gc fix <civodul>i think synthetic benchmarks like those ecraven runs don't capture this sort of issue <wingo>i mean, that has never been what i was working against <wingo>but yeah definitely looking forward to more adoption and bugs <wingo>i enjoy seeing ecraven's results tho :) <cky>I'm really excited to see 2.2 come out. :-) Curious, does 2.2 (or even 2.0 for that matter) provide a secure way to create sandboxes? Or do we really have to create syscall interceptors (a la systrace) for that kind of thing? <spk121>Cool, a fix in finalizers.c. When I get back home, I'll have to see if it helps with the strange bugs I have in Cygwin with finalizers. <cky>I'm thinking of two scenarios: 1. some kind of IRC execution bot type thing (e.g., rudybot), or something like pasterack, and 2. this question came up on SO and I was really hoping there'd be a useful answer to it (feel free to answer yourself if you want): http://stackoverflow.com/q/42769212/13 <spk121>wingo: should I worry if different architectures (both x86_64) generate different .go files from the same input? <wingo>spk121: i don't know. i think that shouldn't happen. if you can minimize a test case i'd be happy to look at it <cky>spk121: When you say both x86_64, do you mean, say, that you get different outputs on GNU/Linux vs FreeBSD, or something different? Or different distros of the same OS? <wingo>cky: it's complicated :) sandboxing a set of primitives is doable. sandboxing cpu time is doable if you exclude dynamic-wind. <wingo>sandboxing memory usage is hard <spk121>wingo: the former. GNU/Linux vs Cygwin <wingo>i think that was for cky also <cky>wingo: Thanks! Sounds like doing it at the OS level is probably the way to go for the moment. <cky>spk121: Interesting. I don't have any free time at the moment but if this also differs for other Unixes, then we know it's not a Windows-specific or Cygwin-specific issue. <spk121>wingo: Well I know that building 2.1.x on OpenBSD is a long way from working, but, maybe I still have a FreeBSD VM someplace. <cky>spk121: Now that's piqued my interest. For many years OpenBSD was my primary OS, so I have a (sentimental if nothing else) interesting in getting modern Guile to work on OpenBSD. <cky>s/interesting/interest/ <wingo>spk121: again i think that is for cky :) <spk121>cky: three problems. Pretty simple. CPP expects that there are no flags after the filename as in guile-snarf. CPP expects no empty includes, breaking guile snarf. <spk121>cky: and some make vs gmake stuff <dsmith-work>spk121: Is obsd using a different compiler these days? <cky>spk121: Thanks! Things I can hone in on in spare time. :-) <spk121>dsmith-work: on the contrary, OpenBSD 5.9 out of the box comes with a very old gcc 4.x. I haven't moved up to OpenBSD 6.0, so this may all be moot. <paroneayea>wingo: I think we need to make a big deal out of the 2.2 release <paroneayea>I'm not totally sure what that means; maybe we can commission a piece of artwork from sirgazil or etc <dsmith-work>So what's the major changes in 2.2? Register VM instead of stack VM. Go files are now elf. What else? <paroneayea>dsmith-work: some types were unboxed (floats, ???)