IRC channel logs


back to list of logs

<ryanwatk`>Anybody know how to fix a 'missed test plan', test ERROR?
<ryanwatkinz> - I got this when trying to run guix -f guix.scm --with-input=guile=guile-next for the guile chickadee library :(
<paroneayea>huh what
<paroneayea>janneke: are you talking about adding match to your minimal scheme?
<janneke>paroneayea: no, it's about goops and typed-methods vs plain function with matcher
<ecraven>ArneBab_: thanks for the ping, I'll try to run them tonight
<ecraven>just rebuilding guile now, that'll take a bit ;P
<ArneBab_>ecraven: thank you!
<ArneBab_>ecraven: when using the release, building guile takes just 15 minutes for me (it comes with prepared *.go files)
<ArneBab_>ecraven: also it would be interesting to see how the jit-version fares in the tests, but that might be more work:
<ArneBab_>ecraven: nalaginrut reports actually beating chez on simple benchmarks.
<lloda>have any of you built nalaginrut's tjit branch?
<wingo>ACTION has not
<wingo>i checked out the source but didn't build it
<ArneBab_>I did not build it yet, either
<civodul>wingo: fixes our "too frequent stop-world GCs" problem
<civodul>and the test suite still passes
<civodul>but you'd better double check ;-)
<civodul>ACTION goes off-line for a while
<wingo>civodul: tx :)
<wingo>it could lead to increased memory usage but when i installed that, i was struggling with finding a leak
<wingo>and thinking it was related to our weak hash tables
<wingo>anyway i will give it a go
<wingo>civodul: i think i found an interesting thing
<wingo>regarding the gc-too-many-times issue
<wingo>so in GC_collect_or_expand
<wingo>which in this example where we're rapidly expanding the live object count, we want to expand
<wingo>it is choosing instead to collect, too often.
<wingo>the test as to whether to try to collect looks like this:
<wingo> if (!GC_incremental && !GC_dont_gc &&
<wingo> ((GC_dont_expand && GC_bytes_allocd > 0)
<wingo> || (GC_fo_entries > (last_fo_entries + 500)
<wingo> && (last_bytes_finalized | GC_bytes_finalized) != 0)
<wingo> || GC_should_collect())) {
<wingo>GC_incremental is false. GC_dont_gc is false. GC_dont_expand is false also.
<wingo>GC_fo_entries is the list of objects with finalizers i think; or is it objects on the finalization queue? i don't know.
<wingo>but the fo is for finalizable object.
<wingo>my current suspicion is that it's that middle case that's true
<wingo>indeed it's the count of finalizable objects
<wingo>so... currently we have a mechanism to run a callback on an object after gc, as long as that object is alive
<wingo>the way it works is it makes a fresh weak reference to that object, attaches a finalizer, then in the finalizer if the weak reference still holds, it will run the callback and reattach a finalizer
<wingo>or rather, s/reattach a finalizer/begin the process again with a fresh weak reference and associated finalizer/
<wingo>so if there are more than 500 objects needing this kind of treatment -- and currently in 2.2, every weak table and weak set has this treatment, and there are a few ways to cause many weak tables to come into existence, notably loading many modules -- then it's quite possible to peg libgc into this collect-rather-than-expand case
<civodul>wingo: got it
<civodul>i think we should arrange to reduce as much as possible the number of finalizers in use
<civodul>and there are several kinds of finalizers, notably disappearing links and actual finalizers
<wingo>disappearing links aren't finalizers
<civodul>they're similar in practice, IIRC
<wingo>not in implementation
<civodul>hmm, ok
<wingo>they are cleared by gc before releasing the alloc lock
<wingo>they aren't enqueued to the finalizer thread
<civodul>oh, isee
<wingo>i am a bit hesitant to remove the "vacuum" phase -- if we removed it, we'd have to incrementally vacuum when adding entries to a weak table and that might have bad hysteresis effects (you go to expand -> you vacuum first -> you get back 10% of entries, no need to expand -> next time 9% -> next time 8% -> etc)
<wingo>so i think what we can do is just make one finalizer to vacuum all weak tables and sets
<wingo>those are the only uses of scm_i_register_weak_gc_callback or whatever
<wingo>and those are the ones that add new finalizers in response to finalization, defeating this logic in bdw-gc -- but adding just 1 finalizer won't be a problem
<civodul>why do we need to scm_i_set_finalizer (ptr, weak_gc_finalizer, data); after CALLBACK has been called?
<wingo>the goal is to get a callback after each garbage collection. the way to get a callback after garbage collection is to use a finalizer.
<wingo>we could use asyncs but then it could stall a main thread
<civodul>i understand, but the fact that we re-add the callback after we've called makes it look like the callback might be called more than once
<civodul>or am i missing something?
<wingo>finalizers are de-registered once they are called once
<wingo>so at the time that we are re-adding the callback, there is no finalizer registered
<wingo>s/once they are called once/before they are called/
<civodul>to put it differently, what's the effect of ?
<wingo>it makes it so that the weak gc callback is called once and never again
<wingo>instead of after each gc
<civodul>ok, i hadn't understood that this was about being called after each gc
<civodul>good! :-)
<civodul>so yes, having just one finalizer for all the weak things would be a solution
<civodul>esp. since Hans says: "In well-written programs there will typically be very few uses of finalization." :-)
<wingo>yeah tricky code, twisty ideas, bad communication on my part :)
<wingo>i'll make that change :)
<civodul>i'm sure Guix will perform better as a result
<civodul>it uses lots of weak hash tables
<civodul>thanks for analyzing & explaining!
<civodul>i recall reducing the number of finalizers with good results back in... 2009
<civodul>like 8cf0dd6104c58f68e0f0ecdf0ab28a033754acf1
<davexunit>so weak hash tables will see improvements?
<davexunit>sometimes I use weak hash tables to deal with doubly-linked structures.
<civodul>davexunit: using weak hash tables currently triggers GC much more often, but wingo has just proposed a fix
<civodul>davexunit: BTW, wouldn't it be great to have (ice-9 json) in 2.2.0? :-)
<civodul>i don't recall, what's the status?
<wingo>2.2.1 surely :)
<wingo>i think we could release a 2.2.0 this week fwiw
<civodul>that would be awesome
<civodul>so yeah, re (ice-9 json), it all depends on its status
<civodul>i vaguely recall that it was in a good shape already but i don't know
<davexunit>civodul: there were problems that I never got around to fixing.
<davexunit>some day...
<davexunit>the latest version of guile-json more or less does what my patch does
<davexunit>had it been that way a couple years ago I probably wouldn't have wrote (ice-9 json)
<civodul>i think there's value in having a json module in Guile proper
<civodul>but i agree that guile-json works great too
<davexunit>civodul: I agree
<davexunit>it's just been hard to pick that patch back up and finish it, for some reason.
<civodul>i understand, np
***dsmith is now known as dsmith-work
<dsmith-work>{appropriate time} Greetings, Guilers
<paroneayea>janneke: ah... what about it?
<wingo>civodul: fixed, i think!
<wingo>actually i tried an unrelated case, i should try the iota case again...
<wingo>yep, fixed
<wingo>that's great
<wingo>before, (length (iota #e80e6)) was 75s. now it is 3s :)
<wingo>guile 2.0 is 7s
<wingo>i should try fibers again then
<wingo>maybe i get the nice linear scaling that i was looking for
<davexunit>this was a gc issue/
<wingo>a simple heuristic causing gc to go quadratic
<davexunit>man we gonna go quadratic in this place tonight
<janneke>paroneayea: i said to amz3 that i sometimes use[d] goops but without methods
<davexunit>75s to 3s is a mighty fine improvement.
<janneke>paroneayea: and instead plain functions with (match o (($ <foo) ..) (($ <bar>) ...))
<janneke>i'm not really happy with either..both bring something useful
<janneke>and i was wondering what your idea of that is, paroneayea
<paroneayea>janneke: using a matcher is often nice in a single procedure, but if you have code that might be extended by other types added fluidly later, or esp if by 3rd party libs, generic methods are just great. Generic methods doesn't necessarily mean GOOPS style... Structure and Interpretation of Classical Mechanics uses a generic method system heavily
<wingo>aaaah, this probably fixes that bad lilypond-on-big-files regression too
<davexunit>you can use generics without using classes, even, though in guile 2.2 this is clunky :/
<wingo>lilypond itself creates a couple hundred weak hash tables, and then guile probably has another hundred or so
<paroneayea>davexunit: yeah we need to add that back :)
<paroneayea>it should be possible, I think even without the <<foo>> unhygenic stuff
<wingo>so in an expanding heap GC_collect_or_expand will almost always choose "collect" instead of / before "expand"
<paroneayea>you can check that a value is a record, and extract what record type it is, so...
<davexunit>paroneayea: yeah it should be automagic
<davexunit>I don't know what needs to be done to make that happen, though
<davexunit>but guile obviously knows how to make a class from a record type
<paroneayea>davexunit: I think class-of could probably even just return the record type itself
<davexunit>my current hack is to do something like (define <<foo>> (class-of (make-foo ...)))
<davexunit>paroneayea: yeah
<davexunit>would be nice if that were possible
<wingo>it would be nice if the record type were a class for the purposes of goops. i think that's where we'd like to go. not there yet tho
<davexunit>sounds reasonable to me but there might be something I'm overlooking
<davexunit>oh cool wingo agrees
<davexunit>wonder how much extra work it would be
<paroneayea>janneke: so, Sussman has a couple of lectures he gave recently which you can find on youtube which go into why generic methods are useful
<davexunit>I use that class-of trick in my new game library
<paroneayea>and he's also teaching a class on symbolic computation which goes into both combinators and generic methods
<davexunit>because I have a guardian that can garbage collect GPU-allocated objects (textures, vertex arrays, etc.)
<paroneayea>on combinators
<paroneayea>on generic methods
<davexunit>and a generic function that calls the finalizer for that type of resource
<davexunit>so I need goops classes from record types in a handful of places
<paroneayea>the result that comes out of it basically seems to be that combinators are really powerful and beautiful abstractions, though they can become fairly rigid
<paroneayea>and generic methods are kind of messy, but are super extensible and easy to keep building on
<paroneayea>so you have the classic "diamond vs mud" thing
<paroneayea>"APL is like a diamond. It has a beautiful crystal structure; all of its parts are related in a uniform and elegant way. But if you try to extend this structure in any way - even by adding another diamond - you get an ugly kludge. LISP, on the other hand, is like a ball of mud. You can add any amount of mud to it and it still looks like a ball of mud."
<davexunit>I like mud with a few diamonds in it.
<paroneayea>but replace lisp with generic methods :)
<paroneayea>davexunit: me too :)
<paroneayea>so SICM (and those problem sets linked) use a super inefficient but easy to implement generic method system
<paroneayea>using predicate checks
<paroneayea>pretty flexible, but I think it will always be O(n) per number of implementors there :)
<paroneayea>whereas GOOPS is just currently O(n) per number of implementers, but it can be fixed IIUC :)
<davexunit>sometimes linear is faster in practice
<paroneayea>that's true
<davexunit>like how alists are faster than hash tables up to a certain size
<paroneayea>though I've written some things which have a large number of generic implementors
<paroneayea>I know that some CLOS implementations try to switch strategies depending on how many implementaters there are
<davexunit>I wonder if we could speed up guile's hash tables
<paroneayea>guile's hash tables seem fast enough, if we used them
<davexunit>Ruby 2.4 apparently has a very fast implementation that we could steal
<paroneayea>oh yeah I read about that recently, pretty cool
<paroneayea>gotta shower, brb!
<davexunit>remember: you never need to shower if you don't leave your house!
<paroneayea>hm, I rarely leave my house and I disagree! :)
<davexunit>I've become 10% more productive with this lifehack!
<paroneayea>ACTION isn't really productive until finishing their morning routine, regardless of whether leaving
<davexunit>yeah I have similar things I must do
<paroneayea>but certainly, I don't judge others who can utilize that lifehack in their own homes ;)
<davexunit>creature of habit
<janneke>wow, thanks -- gives me something to think about
<paroneayea>janneke: the upside of SICM predicate check style generics is you can implement them in like 10-15 lines of code... maybe useful for mes minimalism! ;)
<paroneayea>I've been "wget --recursive"'ing the every time the class changes, looks like it's being redone a bit this year
<paroneayea> has a new assignment
<paroneayea>an adventure game!
<paroneayea>*so* relevant to my interests ;)
<paroneayea> ooh! and the adventure game new problem set is, conveniently, about type based generics!
<dmiles>probly a good usecase for minbikanren
<dmiles>(as well)
<janneke>paroneayea: great, gonna look at that!
<davexunit>"we do absolutely bananas crazy things with the guile interpreter, thus guile 2.0 sucks!"
***dje is now known as xdje
<wingo>i think with that gc thing fixed, i just need to add more prebuilt/ links, arrange to issue a warning and instructions if prebuilt/ isn't present, and figure out goops slot design issue
<paroneayea>wingo: slot design issue?
<wingo>about slots overriding initialization values, or overriding slots with the same name in subclasses, etc
<paroneayea>wingo: ah, nice :)
<paroneayea>I'll be interested to see what comes of that
<civodul>wingo: AFAICS it's fixed, indeed! thank you!
<civodul>i'll check the Guix performance
<stis>hej guilers!
<dsmith-work>ArneBab_: Thank you so much for your cheerfullness! (in responding to lilypond people)
<stis>ArneBab_: did you see what wingo and civodule came up with. Maybe really good news for lilypond with large files
<daviid>wingo: I'm scared :) that you won't look at and fix this bug before to release 2.2 would you prefer that I reopen or send a new bug report?
<daviid>hello *
<wingo>daviid: i think i just mentioned it in the channel?
<daviid>wingo: this gc bug you fixed is awsome!
<paroneayea>/query daviid
<amz3>paroneayea: interesting course you have here
<amz3>paroneayea: how do you do to know that the course changed with wget --recursive?
<paroneayea>amz3: because used to have 10 problem sets, and now it has 5, and the adventure one is new
<paroneayea>and I have the previous one wget'ed
<amz3>you check manually the difference between the page?
<paroneayea>amz3: yeah I knew he was teaching it again so I viewed the page again
<paroneayea>it looks like he's exposing the problem sets one by one as the class goes through them, and there are new or changed ones
<paroneayea>which makes sense
<paroneayea>too much mutation though! old valuable stuff isn't there right now ;)
<paroneayea>hopefully it comes back
<amz3>I just discovered that feature structures are a thing and that NLTK (the python nlp library) already has them
<amz3>I think one can define 'match' over alists using unify operator of feature structures
<paroneayea>amz3: what are those?
<amz3>paroneayea: those are nested alists
<daviid>wingo: i did read the log till the end now :) so, to clarify, my email (link above) is not a 'goops slot design' email, it is related to a bug in the expansion code for setters for slots that are _not_ being redefined, but merely inherited
<amz3>this lecture explain all of it
<daviid>wingo: it really is an annoying bug :) if you could re read this email carefully, it would be really nice!
<amz3>here's the nltk doc about features structures
<paroneayea>wingo: daviid asked me to look at the bug again, and I think I've verified their bug... it can be simplified from the example in there
<paroneayea>it doesn't have to do with module export boundaries, can be done in same module
<paroneayea>this isn't a new guile 2.2 thing
<paroneayea>this same behavior is in guile 2.0
<paroneayea>but yes it's surprising that setters seem to not be inherited with accessors, even without slots being redefined
<daviid>paroneayea: now, comment the definition of the accessor in <a>, then call the setter, and then call the accessor (which should have been defined automatically) on a <b> instance
<daviid>paroneayea: wingo wrt to this, I beleive that (1) the expansion of (setter generic-with-setter) should create an accessor if the user did not create one, then (b) the user should be able to create the accessor, in which case the setter should (also) be inherited
<daviid>brb, i have to eat some :)
<paroneayea>I suspect that a slot mechanics refactor might involve changes to this anyway though
<daviid>paroneayea: I don't uderstand that sentence
<dsmith-work>(wow 135 nicks in #guile. An new high?)
<wingo>ArneBab_: you found a nice CSE bug :)
<wingo>good evening civodul
<civodul>heya wingo
<civodul>wingo: on a quick unscientific benchmark, we have a 33% speed improvement on: ,time (guix-build "libreoffice" "ghc-shelly" "-n" "--no-substitutes")
<civodul>i like it :-)
<daviid>that is fantastic
<wingo>civodul: relative to the bad gc behavior in 2.1.8 or relative to 2.0? :)
<civodul>current master compared to 2.0
<wingo>cool :)
<wingo>i guess you picked up the gc fix
<wingo>from this afternoon
<civodul>yes, of course
<civodul>things were not as good before :-)
<wingo>a funny thing
<civodul>i think synthetic benchmarks like those ecraven runs don't capture this sort of issue
<wingo>of course not
<wingo>i mean, that has never been what i was working against
<wingo>but yeah definitely looking forward to more adoption and bugs
<wingo>i enjoy seeing ecraven's results tho :)
<cky>I'm really excited to see 2.2 come out. :-) Curious, does 2.2 (or even 2.0 for that matter) provide a secure way to create sandboxes? Or do we really have to create syscall interceptors (a la systrace) for that kind of thing?
<spk121>Cool, a fix in finalizers.c. When I get back home, I'll have to see if it helps with the strange bugs I have in Cygwin with finalizers.
<cky>I'm thinking of two scenarios: 1. some kind of IRC execution bot type thing (e.g., rudybot), or something like pasterack, and 2. this question came up on SO and I was really hoping there'd be a useful answer to it (feel free to answer yourself if you want):
<spk121>wingo: should I worry if different architectures (both x86_64) generate different .go files from the same input?
<wingo>spk121: i don't know. i think that shouldn't happen. if you can minimize a test case i'd be happy to look at it
<spk121>wingo: OK. Maybe by tomorrow.
<cky>spk121: When you say both x86_64, do you mean, say, that you get different outputs on GNU/Linux vs FreeBSD, or something different? Or different distros of the same OS?
<wingo>cky: it's complicated :) sandboxing a set of primitives is doable. sandboxing cpu time is doable if you exclude dynamic-wind.
<wingo>sandboxing memory usage is hard
<spk121>wingo: the former. GNU/Linux vs Cygwin
<wingo>i think that was for cky also
<cky>wingo: Thanks! Sounds like doing it at the OS level is probably the way to go for the moment.
<cky>spk121: Interesting. I don't have any free time at the moment but if this also differs for other Unixes, then we know it's not a Windows-specific or Cygwin-specific issue.
<spk121>wingo: Well I know that building 2.1.x on OpenBSD is a long way from working, but, maybe I still have a FreeBSD VM someplace.
<cky>spk121: Now that's piqued my interest. For many years OpenBSD was my primary OS, so I have a (sentimental if nothing else) interesting in getting modern Guile to work on OpenBSD.
<wingo>spk121: again i think that is for cky :)
<spk121>cky: three problems. Pretty simple. CPP expects that there are no flags after the filename as in guile-snarf. CPP expects no empty includes, breaking guile snarf.
<spk121>cky: and some make vs gmake stuff
<dsmith-work>spk121: Is obsd using a different compiler these days?
<cky>spk121: Thanks! Things I can hone in on in spare time. :-)
<spk121>dsmith-work: on the contrary, OpenBSD 5.9 out of the box comes with a very old gcc 4.x. I haven't moved up to OpenBSD 6.0, so this may all be moot.
<dsmith-work>spk121: Thanks
<paroneayea>wingo: I think we need to make a big deal out of the 2.2 release
<paroneayea>I'm not totally sure what that means; maybe we can commission a piece of artwork from sirgazil or etc
<dsmith-work>So what's the major changes in 2.2? Register VM instead of stack VM. Go files are now elf. What else?
<dsmith-work>Dynamic exapandable stack.
<paroneayea>dsmith-work: suspendable ports
<paroneayea>dsmith-work: goops rewrite from C -> Scheme
<paroneayea>dsmith-work: lots of optimizations
<paroneayea>dsmith-work: some types were unboxed (floats, ???)