***tungsten.libera.chat sets mode: +o ChanServ
<wingo>you wanted to talk inlining? <civodul>i was concerned about it being enabled by default at -O2, for a couple of reasons <civodul>for #2, it's a bit harder to explain <civodul>but i think it boils down to the fact that we/i got used to manually managing ABI issues <civodul>that is, as a module writer, i can explicitly choose what gets inlined and what doesn't <civodul>for example, when i write a macro, i can defer bits i'd rather not inline to a helper procedure <civodul>and i know that the procedure is not going to be inlined <civodul>with cross-module inlining, anything could happen <civodul>i think it'd be nice if module writer had a way to annotate things they do not ever want to be inlined <civodul>that way, they'd retain control over the ABI of their module <wingo>so firstly wrt when to enable or disable inlining -- i guess the argument for me was, are modules optimization boundaries? it sure would be nice if that linguistic boundary weren't necessarily an optimization boundary, for all the reasons that i think you are aware of <wingo>now, it could be we got it wrong :) <wingo>but, just mentioning the motivation <civodul>sure, i'm not arguing against cross-module inlining in general, but rather in favor of finer-grain control <wingo>like your argument "when i write a macro, i can defer to a helper procedure" -- this is very hard to document of course <wingo>reminds me of how lambdas used to not be inlined, and arguments about what you would see in a stack trace <wingo>not saying it's wrong, just that it rhymes <wingo>thinking about how to control it. evidently you are wanting some control here <civodul>the ABI is also part of the contract between module writer and module user <wingo>modelling the abi in your mind is hard tho <wingo>i agree it is necessary, of course <wingo>at least for experienced guile programmers <wingo>and that with cross-module inlining, when you want to guarantee an abi, perhaps you want to limit inlining. <wingo>1. define-non-inlinable; this is implementable via (define-syntax-rule (define-non-inlinable x y) (begin (define x) (define x y))) or so <wingo>but, a bit gross, and doesn't necessarily give you a guaranteed abi <wingo>2. per-module disabling of inlining <wingo>i.e. "this module is public and should have a stable abi and no inlining" <wingo>3. restrictions on inlining based on module name. e.g. "inline bindings from any module sharing a common name root" -- would make (gnu system x) be able to inline (gnu system y) but not (bar) or (ice-9 q) <wingo>(rationale would be that naming reflects conway's law division of labor and thus maintenance) <wingo>civodul: can you tell me more about times in which you have been prevented from doing something you wanted to do because of cross-module inlining? <civodul>what you propose makes sense to me, esp. #1 and #2 <civodul>wingo: i have not been prevented from doing anything yet :-) <civodul>but if we were to build all of Guix and its dependency with cross-module inlining, i think there'd be bad surprises <wingo>yeah could be. though in the guix context, don't you always recompile anyway? <civodul>yes, except in development environments <civodul>the issue is for distros other than Guix <civodul>like Debian would find themselves having to rebuild everything too <wingo>yeah very sympathetic there, predictable development environments is a big thing <wingo>ah so you are thinking of debian abi <civodul>well my main personal issue is dev environments more than Debian :-) <wingo>of course there is also the issue with explicit inlining -- like when X uses a macro from Y, and Y is updated, X.go is not re-expanded <civodul>yes, but it's part of the mental of model of "experienced programmers" i guess <civodul>sorta like putting your struct definitions and static inline functions in a C header <wingo>yeah. seems we are currently occupying a funny middle ground between ok and best <wingo>in a development environment you might want to always reexpand <wingo>for anything under your control, anyway <civodul>right, so "best" would include dependency tracking <wingo>apologies if i am not driving towards a solution here :) <wingo>yeah. best would be dependency tracking + reproducibility + abi <civodul>it's tricky because it's mostly about practices and perceptions <civodul>BTW, as an example, (guix records) has an ABI check: since we know accessors are inlined cross-module, and that's great, there's an additional trick to check for ABI incompatibilities <civodul>it proved useful for people developing Guix <civodul>if it says "recompilation needed", you know what to do :-) <civodul>with generalized cross-module inlining, there'd be a lot more "recompilation needed" cases that wouldn't be caught <wingo>we should include something like that in guile i guess... <wingo>so let's take ABI := sha256sum(module.scm) ^ fold(logxor, imports(module.scm), 0) <wingo>rather fold(logxor, map(recorded-sha256sum, imports(module.scm)), sha256sum(module.scm)) <wingo>which changes whenever any used module's source text changes, or the module's own source changes <wingo>we residualize ABI at compile time. then when loading we recompute it, and issue a "recompilation needed" warning if they differ <wingo>issue would be in systems like debian. we could exclude the warnings for files in the install tree, in those systems <wingo>though, maybe that borks the idea <wingo>backing up: two use cases, development environments and ensuring installed abi. <wingo>for development environments, this definition of ABI and ABI checking would seem to be sufficient, wdyt civodul ? <wingo>for installed abi, i think we have two options: (1) hope that module authors have an adequate conception of abi and will preserve or break it as appropriate, or (2) encourage packages and or distributors to disable cross-module inlining generally, or for specific packages <wingo>i think a mix of (1) and (2) can work; probably foundational guile module, those that aren't leaf nodes in the package graph, already have either stable abi or explicitly unstable abi <civodul>wingo: yes, allowing for #1 and #2 sounds good <civodul>as for ABI = sha256(text), it sounds too coarse grain <civodul>for development environments in particular: you'd end up having to "make clean && make" all the time <civodul>ABI could be the hash of exported macros, for instance <wingo>coarse grain or fine grain ? <wingo>so it's like, what property do you want to preserve -- do we want to print "recompile" in all cases in which recompilation might be needed, or never print "recompile" if a recompile might not be needed <civodul>right, or: would we rather have false positives or false negatives <wingo>i think we probably want the first, right? it would be preserved by hash of source + deps and not necessarily by hash of exported macros, because of cross-module inlining <wingo>in a dev environment you want no false negatives and few false positives i think <civodul>oh right, i said "macros" but that also works for "inlined bindings" in general <civodul>i think hashing the source text would give too many false positives <civodul>well dunno, OTOH that's what everyone else is doing *wingo wonders about that definition of ABI and circular module imports <wingo>would be nice to integrate with "make" or so, to determine the set of files that need recompilation <wingo>you would never need to "make clean", in theory at least <wingo>of course you could. but it wouldn't be needed <wingo>could even order them topologically <wingo>hey let's summarize though, do you agree there are the two use cases, development and installed-abi? <wingo>and in the development case, is it possible to keep inlining while improving robustness generally with dependency tracking? if that looks like it might work we should do it imo <wingo>we are left with somewhat unsatisfying results for the installed-abi case: just "be careful" <wingo>but perhaps it is sufficient <civodul>wingo: i have a sense of the two use cases, but are they distinguishable from Guile's POV? <civodul>i mean there's no notion of "installed" <wingo>there is a notion of "loaded from an installed path" tho <wingo>like loaded from /usr/whatever <civodul>but it's just one of the entries in %load-compiled-path? <civodul>but it looks like we've drifted a way from cross-module inlining :-) <civodul>though the two are closely related of course <civodul>going back to cross-module inlining, the most important things to me for 3.0.8 (?) would be (1) to give module writers more control, and possibly (2) to disable it at -O2 and below for now <civodul>the reproducibility issue is a big concern <civodul>#1 could be define-non-cross-module-inlinable or similar, as you wrote <civodul>but someone writing code for 3.0.8 and for earlier versions wouldn't be able to use that <wingo>to prevent your definitions from being inlined into other modules you can do (define-module (foo) #:declarative? #f) <wingo>or compile with -Ono-inlinable-exports <civodul>-Ono-inlinable-exports works; #:declarative? prevents local inlining too, no? <wingo>for reproducibility... i think that is essentially an in-project build system issue. one option is to turn off cross-module inlining in your project. another is, say you are compiling X, Y, and Z -- you compile them two times. one without cross-module inlining, and one with cross-module inlining, and the second time (in a separate build subdir) uses the results from the first (but not the second) <wingo>civodul: declarative? #f prevents inlining of top-level definitions within a compilation unit, yes <civodul>but yes, the build system issue is non trivial :-) <wingo>circling around. i think it would be good if guile users could program without considering modules to be optimization boundaries and the default tools should support that if possible <wingo>civodul: -flto is an interesting counterpoint -- in c++ etc there is an explicit linking step <civodul>but one also has to think about ABI concerns <wingo>it pushes most of the compile time off to the linking step <civodul>maybe we could have a "guild link" command? <wingo>what would it do? :) would it be essentially "guild compile" ? <civodul>well, "guild compile" could lower to CPS <wingo>cross-module inlining works on tree-il <civodul>that's not a serious proposal, but maybe something to keep in mind in the long run? <wingo>you would need "guild compile" to identify inlinable exports <wingo>or whatever compilation command runs first, i mean <civodul>it could keep in .go files all the definitions that are "small enough" to be potentially inlined <wingo>that is how it currently works fwiw <wingo>i mean, i think the proposal is equivalent to "guild compile -Ono-cross-module-inlining" then "guild compile" <wingo>with two builddirs and appropriately set module search paths <wingo>since #:inlinable-exports? #t is enabled at -O1, the equivalence is compiling at -O1 then -O2 *civodul can already picture the lovely makefiles :-) <wingo>so tool-wise the pieces are there <wingo>-O1 is effectively an -flto for a later -O2 compile <wingo>at least this way the equivalent of the link step is parallelizable <civodul>my arguments for keeping it off at -O2 for now (in 3.0.8) would be that otherwise (1) many packages will become non-reproducible when we switch, and (2) "experienced developers" would suddently have their handcrafted ABI change <wingo>i think you describe situations that are already the case in 3.0.7 <wingo>(which can be changed of course) <wingo>basically i hesitate to turn it off (because of effects on what kinds of guile systems get built) without a story about when/how to turn it on <wingo>speaking of the defaults, of course <civodul>wingo: "when we switch" could be Guix yes, but really any distro <civodul>re when to turn it on by default, i think ideally that'd be once module authors have finer control and there's a documented way to get reproducible builds <civodul>"documented" could be a project that can be used as an example <civodul>and we'd be adding this extra source of non-reproducibility <civodul>which is especially not good in context of Guix <civodul>"c'est le coordonnier le plus mal chaussé" and all that ;-) <wingo>i guess another point of conclusion is that there is a conflict between reproducibility and cross-module inlining. can resolve with two-stage builds, can resolve with -Ono-cross-module-inlining. but what should be the default? <wingo>civodul: fwiw you would only need two-stage builds in guix itself for those parts of guix built with -O2. afaiu that is a relatively small part. so for guix itself i guess the defaults are less important because the project is already all about reproducibility; and the problem would "just" come in when guix builds other .scm packages <wingo>i am not opposed to solving this problem, i just want it all and i want to think a little about whether i can have it all :) <wingo>like if the answer is "we can have it when we have a new build tool" that's fine imo <wingo>if we have a plan to get ther <wingo>(for completeness, can also resolve reproducibility issue by not doing cross-module inlining at -O2) <civodul>wingo: there's Guix and there's all the Guile packages Guix provides <civodul>there are also Guile modules built as part of other things in Guix <civodul>the module-imported-compiled.drv things <civodul>Guile usage is quite pervasive in Guix :-) ***ec_ is now known as ec
<tohoyn>is anybody here using Lambda the Ultimate? <civodul>is there a way from a macro to know at which optimization level you're compiled? ***dongcarl1 is now known as dongcarl
<civodul>if i do (compile exp #:optimization-level 1), am i 100% sure i'm using the baseline compiler? <civodul>just found that the baseline compiler would translate (ash x N) to (ash x (- N)) when N is a literal