IRC channel logs
2026-09-24.log
back to list of logs
<old>does it makes sens to have nestable heap allocation limit (per-thread) and if so, how should the nesting be handle <old>should an allocation make in a heap allocation limit percolate up to other limits that were set before, or should it only impact the current limit <lechner>old: Why does the limit percolate up rather than down? <old>(call-with-heap-allocation-limit 1000 (lambda () (call-with-heap-allocation-limit 100 (const #t)))) <old>replace (const #t) for (lambda () (make-vector 100)) <old>which limits should be impacted by this allocation <rlb>Hmm, why heap, per-thread, specifically? I could imagine wanting a limit across some related threads. <old>in other words, are limits nestable, therefore you can only reduce the heap usage as you nest the limits <rlb>(May well be a naive question.) <old>or is the limit only a current limit, which previous limits does not affect <lechner>i think it should only impact the current limit <old>see 6684f9e8c246b147249b56e2589317ba8d80c803 <rlb>If it's feasible, seems like decoupling the limit from the "actors" might be interesting, just offhand. <old>I am currently refactoring the bits to make it more in Scheme so that there are no continuation barrier in the thunk invoked with the limit <old>The limit is per-thread because you don't want your sandbox thread to impact your main thread <old>which is currently the case <old>well not anymore since 6684f9e8c246b147249b56e2589317ba8d80c803 <rlb>Ahh, OK, so it's specifically a per-single-threaded sandbox case. <rlb>(I'll recuse myself --- definitely don't have the relevant context atm.) <old>rlb: off hand do you know if glib offers compiler barrie r? <old>asm volatile ( ::: "memory") <old>I don't see it, so I think I will add that to your definition: scm_i_compiler_barrier () <old>I will add support for it for GNUC and clang <old>I don't know if other toolchain support asm statement <rlb>Ahh, OK, just wanted to make sure I hadn't just completely forgotten something :) <rlb>I suspect some flavor of inline asm is fairly common these days, but wouldn't be surprised if the precise details can vary. <rlb>...wonder if any lib we depend on has anything relevant (libgc, gnulib, lightening), guessing not. <rlb>looks like C11 has fences? <old>right signal fence is equivaelent <old>__atomic_signal_fence(__ATOMIC_SEQ_CST) <ieure>Does Guile have a syntax comment form, like #_ in Clojure? <lechner>Paredit may even work with that now. <ekaitz>you mean just a (define a 1) for example? <lechner>Yeah, I've seen them in places other than the top level. <ekaitz>lechner: you can use it in the first expressions inside of a procedure <ekaitz>(lambda (x) (define y 10) (+ x y)) <lechner>Can it go inside the think protected by with-exception-handler? <dthompson>top-level, procedure body, begin, and let forms are the general places where definitions can go. <dthompson>then of course there are macros that expand to one of these more primitive forms and thus definitions can go wherever the expansion places the definition in a body subform <old>you can also now use define in a if statement with begin <old>(if x (begin (define y 1) y) (begin (define z 2) z)) <old>that will be possible starting from 3.0.12, but is not portable I think <mwette>That seems weird. Is the scope of `y' or `z' the parent of `(if ...)'? <mwette>what about (if x (define y 1) (define z 2)) ? <old>see: 1104499ceda55d064aad865b6f88f72986949662 <old>(if x (let () (define y 1)) (let () (define z 2))) is not valid <mwette>Seems confusing. `begin' does not introduce new scope. (if x (let () (define y 1) y) ...) seems clear <old>ya I know, but the above commit will wrap branches of conditional in a (let () arm) <old>so the begin itself does not introduce a scope, but the arms of the branches does <old>and I asked about this and nobody seems to have found a case wehre this could be a problem <mwette>I think it may add burden to the optimizer. <old>well it did increase the overall module size on disk of Guile by 9.32 KiB <old>we can always reverse the commit before release if it was deem to impact performance but I suspect it should not be a big problem <mwette>It might be interesting run a timed test w/ lots of layers of `(if ...)' <old>I am working on a set of metrics for Guile <old>that could be one of them <old>my idea is to run on each commit the set of benchmark and to gather metrics that we can publish online <old>and detect regression before release <old>it's very nice. I was about to commit something in the reader and I saw that the throughput of it was down by 3 MiB/s <mwette>Match-type macros (e.g., match, sxml-match) often introduce let forms. In the past I have run into cases where compiling match or match-sxml with a large number of cases can run a long time, or even fail. The solution to the failed case was to break up a sxml-match into several. (The last case of first sxml-match was to execute the second sxml-match.) It seemed to not be the number of lets but the depth that was an issue. <mwette>^ I don't know for sure, the evidence pointed that way. <old>If you have a patholical case, I'll be happy to check <old>I had a prototype for psyntax for faster identifier resolution in some patological case (very large let) and many depth <mwette>It needs some function `time' to time how long the compile takes. You pass it a number to say how many cases to add. <old>(ice-9 time) can be used for that <mwette>You may want to modify to generate a procedure using cond or something else w/ deep `if'. <old>well the optimization I was talking about was about this issue <old>mwette: can you re-send a neew paste? Looks like TTL of debian paste is like 5 minutes <old>will copy this locally <mwette>A problem you'll have w/ guix package load is that `(export foo)' (in `(define-public foo ...)') takes a long time. <ieure>Is it faster to #:export rather than define-public? <mwette>It's faster to (export foo bar baz) than (export foo) (export bar) (export export baz) <old>well I only did measure (macroexpand) on the fiels <old>not the full toolchain <old>and I don't think that `export' are evaluated at expansion time <mwette>esp `call-with-deferred-observers' <old>eh just figured that scm_remember_upto_here_1 and friends are basically useless now with LTO <old>well I could see to make a define-public benchmark see what it gives <old>In theory, could we not accumulate the export in a private list and to a single export at the end of it ? <mwette>I was just going to suggest that. <mwette>Can export be called below the top level? <mwette>If so, you might want to only call top-level ones (incl those in begin forms). <old>you mean like in functions? <dthompson>I think it can as it operates on (current-module) <mwette>s/call top-level/collect top-level/ <old>don't know off hand how we can determine that <old>there's no way for a syntax transformer to know if it is used a top-level AFAIK <old>that would be useful on its own I think <old>rlb: sorry for the bootstrap 0 recompilation you will need to do :-) <lechner>ieure / old / mwette / define-publics are very slow. From what I remember, 6,000 of them take about three minutes on my equipment. <old>I need to know about these!!! <old>I am always looking into things to optimize <old>especially boot time stuff and compile time <dthompson>define-public ought to be discouraged but prob important to optimize because guix uses it a lot <dthompson>imo all imported/exported symbols should be specified in the define-module form <dthompson>this is how standard scheme modules work and they are much easier to understand as a result <dthompson>it's nice to not have to type a name twice, though. I get it. <lechner>dthompson / I never use the shiny extras in define-module and prefer separate use-modules or export statements. <lechner>I put the exports just above the defines or records <old>not sure but you might loose the power of declarative modules that way <old>better to follow the idiomatic way so the toolchain can better optimize your code <lechner>What is the advantage of a declarative module please? <old>many. one of which is inlining <old>both in the CU but also across modules <old>> ... for 3.0, we added the notion of declarative modules. For these modules, bindings which are defined once in a module and which are not mutated in the compilation unit are declarative bindings, which can be reasoned about lexically. We actually translate them to a form of letrec*, which then enables inlining via peval ... <rlb>old: oh, did you change something bootstrap-related? (I haven't updated in a while, but hope to get back to a utf8 rebase where I'm reworking some things in ~week.) <rlb>old: and wrt debian pastes, not sure what you saw, but "recently" they seem to occasionally "404" (don't recall the actual error, maybe 404) for a bit, and then they're fine. I think the lowest ttl is 24h. <rlb>And if remember_upto_here really doesn't work with LTO, can we actually use it? The related bugs are no fun at all. <lechner>old: Respectfully, it seems like a shortfall in Guile to me that Guile cannot tell from my top-level exports that my bindings are likewise not mutated. <old>rlb: nope I just made so that `eval' is in Scheme now <old>Idk why but `eval' was `scm_eval' so it introduced a continuation barrier <old>anyway, this requires recompiling eval.scm <old>wrt to remember_upto_here, I don't even know why it's necassary? <rlb>Ahh, that sounds "interesting" :) But no worries wrt the bootstrap, I've gotten used to them... <old>in which cases do you need that <old>lechner: How so? guile does not know what you want and needs to be backward compatible. By using `define-module' you sign a declarative contract (from which you can opt-out using #:declarative #f) <rlb>You need it after any internal references to a thing that might otherwise be GCed "too early". For example, if you have a bytevector bv, and you assign SCM_BYTEVECTOR_CONTENTS to a uint8_t *data, you need a scm_remember_upto_here_1 (bv) after the last reference to data. Otherwise the gc might free the bytevector while you're still using data. <old>compiler can not assume everything about a program <rlb>i.e. if the C optimizer drops bv earlier or hides it, or... <old>rlb: well remember_upto_here seems to be working if you ahve __GNUC__ since it's using asm statement <old>but for other toolchains, with LTO, the call to external NOP function will be DCE <rlb>It's "just" to avoid trouble from optimizer cleverness. <rlb>Perhaps we need to disable lto whenever we can't promise upto_here... <old>you can call anything outside of libguile with the object and it will work <rlb>I've had to track down related bugs when upto_here was missing, and, yeah, no fun at all. <old>well not if you statically link .. so <rlb>And I suspect civodul might have had worse from it, but don't recall for sure :) <old>actually it's very sinmple <old>I know what we can do <old>compiler barrier + store to itself or load to a dummy volatile <rlb>Hmm, would that be equivalently "cheap", i.e. no idea, but wondered if volatile might be more aggressive than we'd want wrt blocking optimizations. <rlb>Or do you just mean as a fallback? And then I guess we'd have to consider that vs disabling lto wrt perf difference, or something... <rlb>(if we thought there might be a concern) <old>hmm but then compiler barrier you either need stdatomic or asm volatile .. <rlb>wrt "can call anything outside of guile", not sure what you meant, but of course *something* has to ensure the pointer is somewhere libgc can find it for the entire lifetime. <old>I meant that if you call to an external fucntion outside of LTO realm, then you are safe because the compiler has to keep alive the variable for that call <old>but for staically linked guile, LTO is everything except system call <rlb>ahh, right, someone has to keep the top-level pointer alive, but doesn't matter who. <old>anyway, which toolchains like asm statement ? <rlb>I guess we could have one .o that's not LTO with a dummy function and call that :) <rlb>(Or is that what you meant?) <old>if your toolchain lack asm, it probably lack LTO <old>so I think in practice, we really not need to worry about this <rlb>Imagine that could well be. Though if it were easy, might as well --disable-lto when there's no asm if we can detect that. Or have it crash with an #error telling you to --disable-lto if that's easier and we think it's unlikely, or... <old>The logic should be: !lto || has-asm . so we can just check for that in configure.ac ? <old>mwette: I can't run your example <rlb>I'd guess so, one way or another. I think lto probably has a var we can check, and then we'd just need a test for asm if there's not one. <old>ERROR: Unbound variable: sx <old>so 6000 define-public and the hot path for the compiler is do_scm_equal_p <mwette>10 => 0.11, 100 => 1.49, 400 => 29.92 for guile-3.0.11 <mwette>ah, on last one (i.e., 400) gctime was 24.53 <mwette>100 had significant fraction gctime also <old>for 400 I have this on main: <old>30.19 45.72 0.27 0.00 0.00 21.84 <old>clock utime stime cutime cstime gctime <old>30.19 45.72 0.27 0.00 0.00 21.84 <old>with the reverted letify branch: <old>32.85 59.51 0.20 0.00 0.00 34.81 <old>I think the benchmark is not reproducible enough <old>why is export even using defered observer ? <mwette>IIRC to deal with some (serious) issue loading modules (in parallel?). civodul would remember I think. <old>but module does not load in parallel ? <old>I guess it can make sens when you have lots of variable in export <mwette>Maybe it could be useful to know when to recompile dependent modules when inlined variable changes. <old>geesh I should have not checkout up to 1999 to see the details <old>now I have to bootstrap again <old>I don't think observer would help with that <old>BLUE can already detect inlined exports that are used when compiling and also macros <old>so it can effectively recompile your file if something in another files has changed <old>all without observers <old>I think observers is more useful perhaps for hot-reload loop ? <rlb>old: you *might* be able to get away with "cp -a stage0 ../tmp-safe" or something and swap it back to avoid so much rebuilding. <old>rlb: too late I went with: find . -iname "*.go" -delete <rlb>(or temp worktrees, of course) <old>I did manage to shave off 15 seconds on stage0 of eval.scm btw <old>out of 2m30 so not very a big big change <rlb>Note that I've occasionally hit issues with ./cache, i.e. had to rm -rf it too. <old>yeah I always nuke the cache, epsecially with benchmark <rlb>(I think maybe with test .go files, while rebasing/debugging a test, but not sure.) <old>really wish we had an environment variable for specifying where the cache is <old>instead of just XDG_CACHE_DIR <old>I should also probably just use the --fresh-auto-compile option <rlb>(I've generally just pulled out the sledgehammer so far, instead of trying to track it down.) <rlb>I had a trivial script "clear-guile-cache PATH" that would just clobber all the relevant PATH subtrees in ~/.cache/guile, but can't find it now... And it didn't handle ./cache iirc. <rlb>Most of the time now, I just "rm -rf ~/.cache/guile" (sledgehammer again). <old>I have guile-clear-cache for that <old>otherwise I fear the rm -rf ~ <lechner>I've also seen problems with outdated cache objects. Guix tends to hide them in a development cycle involving "guix build -f" <old>btw it's not the defered observer the problem. at least not from what I can observe <lechner>I think sometimes outdated modules are loaded, especially from scripts that are marked --no-auto-compile <old>well it does add some overhead <old>so I can go from taking 24 s for compiling 6k define-public down to 15 s <old>I do wonder however if I can do better with a collect/produce approach <daviid>old, rlb how about povinding a new guild option, guild --clear-user-cache [or a better name maybe, but to avoid the confusion with guile's ccache dir ofc ...] <old>lechner: with a collect exports + single produce at the end I am down to 5 seconds compilation <old>> ieure / old / mwette / define-publics are very slow. From what I remember, 6,000 of them take about three minutes on my equipment. <lechner>old: What was the timing for the define-publics, please? <old>So for 6000 define-public <old>24 seconds is the baseline <old>I'm opening a PR that makes this down to 15 seconds <old>but in theory, we can acheive 5 seconds with a accumulate/emit-once approach <old>but it either require the user to use an extra macro to emit the accumulated exports <old>or somehow the compiler do it