IRC channel logs
2020-12-29.log
back to list of logs
<OriansJ>janneke: My understanding of that branch was that it couldn't run MesCC well enough to self-host and I spent alot of time trying to figure out why. <OriansJ>Also; anything that previously built from M2-Planet still builds with minor tweaks (moving the initialization of variables outside of loops is the only thing that really changed) <OriansJ>mihi: segfaults in mes-m2 are serious issues if you can provide examples I will address them immediately. <OriansJ>As I understand it most of the known bugs are in the macro expansion stage of the Read-expand-eval-print loop <OriansJ>if you do export MES_CORE=0; you'll skip loading all scheme code and be interacting directly with the C code of the Lisp interpreter; thus able to use cgdb to single step to exactly the root cause of any bug. <OriansJ>now mes-m2 will certainly halt with an error message if it recieves something wrong enough to create hard to debug problems. But as I routinely fuzz my programs (C and above); segfaults occuring are an issue of major interest to me. <OriansJ>Also my understanding is janneke has been clearing out all of the binaries from the Guix bootstrap save for mescc-tools (we bootstrap those all the time), mes.c and guile (these 2 are supposed to be replaced by mes-m2). So ultimately we will need an r7rs lisp interpreter to finish the bootstrap in a single step. As guile needs us to solve the psyntax.pp bootstrap problem and janneke solved the scheme to GCC bootstrap path for us a long <OriansJ>Then we have the GHC bootstrap problem to solve and the node.js bootstrap problem and etc. Then I have to build trusted hardware and all of the hard problems that comes with it. <OriansJ>In short; it'll be decade(s) before all of the pieces are finally done. <OriansJ>(or atleast done enough for me to be happy) <OriansJ>But this work is alot of fun and we all get to learn from it. <stikonas>oh, node.js also has circular dependencies? <OriansJ>stikonas: I might be remembering wrong but I thought guix had problems packaging it due to the inability to bootstrap it <OriansJ>well I guess I was wrong again. Oh well. <stikonas>well, it might very well be that there were some problems... <stikonas>e.g. how to express npm dependencies in guix or something like that (I'm not node expert, just guessing) <OriansJ>I've been unable to use guix since the 1.2.0 release as they still have not fixed the gnutls bug <stikonas>anyway, there are still plenty of non bootstrappable languages... <OriansJ>If I enable substitues, it just downloads a blob and moves on but it is a release breaking bug that is 35 days old now <OriansJ>gnutls-3.6.12 if you wish to be precise <fossy>welll, why isnt gnutls updated to .14 then <xentrac>20:49 < fossy> so we actually do have a full path from 357 bytes to gcc, abliet a mildly complex one <xentrac>congratulations! I didn't realize we'd made it this far already! <OriansJ>fossy: probably because it would force all packages down the tree to have to be rebuilt. <stikonas>what about using 1.1.0 and guix pulling to get new version? <OriansJ>stikonas: doesn't work either as gnutls-3.6.12 is needed by guix pull to successfully complete. <OriansJ>right now the only work around is enable substitutes <OriansJ>or change the system clock back a couple months <stikonas>yeah, I think something similar happened before for me (download was failing) <stikonas>(I workarounded that by editing /etc/hosts and putting that file locally <OriansJ>stikonas: guix download works on local files as well. <OriansJ>what is even weirder is if I do: guix build gnutls@3.6.12; I get: successfully built /gnu/store/91qxmwz9wm9w6jwxqmkalrlfxnkd3c0c-gnutls-3.6.14.drv <lfam>I just joined so I missed the earlier part of the discussion, but if the weird thing is the version mismatch, it's likely a consequence of the grafting mechanism. Specifically, a case where GnuTLS 3.6.12 was replaced, using grafting, with 3.6.14 <lfam>The filename of the grafted replacement must have an equal length, so it seems that, in this case, we chose to leave the version number "unchanged" <lfam>It's definitely not ideal from the UI perspective <lfam>It probably would have been better to do what we sometimes do, which is to set the replacement version as "3.6.-A" <lfam>Or something equally meaningless but equal length <lfam>Well, sorry for adding to the confusion. Of course in this case the replacement's version is the same length. <lfam>It's definitely one several UI bugs regarding grafts <siraben>OriansJ: I'm curious what the .drv files in Guix look like compated to Nix's .drv <OriansJ>lfam: thank you for the insight. I am attempting to figuring out a work around procudure for enabling guix pull to work when one installs guix v1.2.0 (stable) via following this procudure: https://paste.debian.net/1178673/ <OriansJ>currently I am trying this procedure: https://paste.debian.net/1178687/ in attempt to provide a work around that will consistently work for those wishing to actually install guix and not depend upon substitutes <OriansJ>just waiting for it to finish and then run guix pull (hopefully it'll keep going this time) <lfam>Is it a problem with the GnuTLS test suite having a test certificate that has expired? Or similar? <lfam>We have basically always "punted" on those <lfam>It's not really clear what to do except for reset the clock, as you suggest <OriansJ>lfam: or shove into a qemu vm with an arbitrary timeclock set <lfam>Yeah, somehow change the time <lfam>There are several bugs like this in our history from GnuTLS :/ <lfam>Unfortunately the software expires <lfam>I think it's happened with some other packages, too, but GnuTLS is really important for Guix <OriansJ>or maybe -rtc base="YYYY-MM-DD",clock=vm <lfam>The tricky thing is that it's part of the general class of bugs summarized as "things fall apart". When trying to use old versions of Guix without substitutes, it's basically impossible because upstream sources disappear. That is what the Software Heritage aims to address <lfam>There is not really any effective way to build a distro from source after a few years <lfam>Unless you have cached all the source code <OriansJ>Personally I am all for guix caching files as they become part of the build tree <lfam>The Software Heritage is like a cousin of Guix, also starting from people at Inria. So that's basically our solution to the "disappeared sources" problem. But it's a long-term project and doesn't cover all use cases yet <lfam>We cache sources at our build farm but it's not treated in an archival way <lfam>If there is something concrete we can do to improve the situation with this "expired GnuTLS", we want to do it. Suggestions welcome! <OriansJ>lfam: well my solution would be just to have a written procedure that people can use to work around it. <xentrac>Yes, Debian saves all the sources in the Debian source archive; it seems like a sine qua non for reproducibility <lfam>OriansJ: If you come up with a procedure that works, please let us know and we will include / distribute / implement it accordingly <OriansJ>even if it is go to this FTPS site to lookup and download the source tar; or do these funky steps to work around the system clock problem. <OriansJ>lfam: well I linked to my current plan that I am testing but I'm still waiting on the build to complete to know for certain. <lfam>I think it will work, assuming the chosen "fake time" is within the constraints of the certificate <OriansJ>when I figure out the solution I will certainly email the answer to 44559@debbugs.gnu.org so that everyone will have it <lfam>A tangential but oft-considered idea is to somehow separate "building" from "testing" of packages, but it's rather pie in the sky at this point <lfam>Usually this type of failure is in a test suite <OriansJ>well make and make test are 2 different things for a reason. <lfam>The thing is, if the tests fail, we want the build to fail. But maybe we could define packages with "test suite is expected to fail" or something <lfam>It's not a simple change in the context of functional package building <OriansJ>lfam: or we could address it more proactively. <lfam>One really appreciates the advantages of the flexibility offered by a Debian-type distro in situations like this <OriansJ>use qemu to set the time to say the year 2800 <OriansJ>then time based failures will always occur <lfam>Not to mention the filesystems ;) <lfam>We work around a huge number of bugs in upstream packages, but a solution to this one is elusive <OriansJ>lfam: we can also dictate everything inside qemu so that all builds become 100% reproducible <OriansJ>eg have guix build ext4 files, boot them with a known linux kernel to build the program in question. shutdown the vm, and copy off the built binary <OriansJ>set the memory and CPU and system too <lfam>Yes... except for the users' systems :) <OriansJ>with nothing possible to change the outcomes of the binaries <OriansJ>users' systems no longer impact the build <OriansJ>because the build is done inside of a strictly defined qemu vm <lfam>It's nice to build on a variety of hardware. It does help uncover some bugs before sending them to users <lfam>We could do this. In fact, large parts of the build farm have been virtualized at one time or another <lfam>Like I said, if your workaround recipe is effective, please let us know on the bug tracker or in #guix <OriansJ>or perhaps provide it as an option to address the sort of problems you would have previously punted <OriansJ>lfam: you know me; I always share solutions when I have them. <lfam>It would probably not be too hard to add an option to the guix-daemon that would do what you suggest. We already have QEMU integrations in some other areas <OriansJ>lfam: what can I say novelty first search once again proves useful <pabs3>(stores both binaries and sources, for everything going back to something like 2005, plus releases before that) <xentrac>yeah, but even the regular Debian archive stores all the sources, and they've been working hard these last few years on reproducible builds <pabs3>btw, has it been considered to expand the scope of bootstrappable to all FLOSS distros (Linux/BSD/Haiku/ReactOS/etc)? similar to how reproducible builds folks have. I note vagrantc uploaded some bits to Debian (mes, mescc-tools at least) <pabs3>that one contains the no-longer-supported releases (but no intermediate states) <pabs3>(snapshot.d.o is where all intermediate+final states live) <pabs3>the diff should have a debian/changelog file that lists all the prior versions <xentrac>it starts at 2.7.2.1-3; the previous two (or three?) patch levels were summarized in a "Previous changes:" section <OriansJ>pabs3: well I am more than happy to expand all of the pieces in mescc-tools-seed, mescc-tools and M2-Planet to be ported to any operating system/distro that has a dev willing to help in the process. <OriansJ>I can not however make matching claims about the higher stages of the bootstrap process as I haven't become responsible for them yet. <OriansJ>similiarly for arbitrary hardware architectures. <fossy>janneke: i will be integrating wip-m2 into live-bootstrap shortly until there is a more sutible replacement or it matures. is wip-m2 currently selfhostable? <janneke>hmm, upgrading from m2-planet 1.4.x to 1.7 gives *janneke was editing the wrong file <OriansJ>janneke: you can do arrays in struct but only if they are fixed size or you could just do type* and set them using malloc/calloc at runtime <OriansJ>eg struct foo { char sysname[65]; char* bar; }; is perfectly acceptable <siraben>I might have missed earlier discussion but is it possible to extend m2-planet to compile mescc? <OriansJ>siraben: M2-Planet is a C compiler (of sorts) and MesCC is an interpreted scheme program. <siraben>Doesn't mescc have a Scheme interpreter in C? <janneke>OriansJ: ah, great -- i can hardcode the size for m2-planet <janneke>OriansJ: it seems that all variables must now be defined at top level <janneke>(i'm not complaining, i like the strictness and new error messages) <siraben>Right, but M2-planet cannot compile htat? <OriansJ>janneke: it is taking variable initialization out of loops that where a hard to track down bug <OriansJ>siraben: it heavily depended upon C macros and pointer arithmetic. Something that M2-Planet doesn't have and something that M2-Planet needs a work around for. <siraben>Ah, so it was more worthwhile to write a new interpreter from scratch? <OriansJ>siraben: it is more like I spent 6 months trying to get it to work; had a hard time debugging it and gave up; wrote an easy to debug scheme from scratch and got stuck on how to get macros and modules to work correctly. <janneke>meanwhile, i continued on yet another route, stepwise refactoring of mes <janneke>hoping that our efforts would somehow help eachother <OriansJ>well they say only 1 scheme ever was a long term group collaborative effort (guile) <OriansJ>perhaps with effort from all parties involved we can finally create a second. <siraben>What about other schemes like Chez, CHICKEN, etc.? <mihi>OriansJ: as I fixed the bugs in symbol->string and string->symbol logic, all bugs I know of (including the segfaults) are in mes-macro.c. <mihi>Even without MES_CORE=0, setting some strategic breakpoints in gdb helped to pinpoint bugs once you had a minimal example that triggered them. As long as the bug happens during loading of e.g. srfi-13 and you already have 50000+ cons cells in memory, the hard part is reducing the bug to a smaller example (in my experience). <mihi>So the goal of mes-m2 is actually not (only) to make mescc run on M2-Planet, but to reinvent the wh^Wguile well enough so that it can run guix's bootstrap machinery? I believe one CollabNet guy once said that the most important factor for open source projects to attract and retain contributors is to clearly state what the goal of the program is and what is not. And if the goal is more complex, <mihi>provide some agreement of what steps to pursue next and what defines them to be finished. I now can see that this was probably a non negligible reason why mes-m2 landed in the situation where it is today. <mihi>I believe it is easier to get guile bootstrapped from gcc without relying on guil as early as possible and then restart the guix bootstrap with guile and the previous seed. But probably I'll have to try first where the real obstacles are in compiling guile from Debian's build-essential and nothing else :-) <mihi>In my opinion, the other bootstrap problems (node.js, scala, Kotlin, dart, etc.) can be solved indenpendently from M2-Planet by basing themselves on other already bootstrapped tools (e.g. scala bootstrap compiler could be written in Java, as the Java bootstrap is (mostly) solved) <mihi>Trusted hardware is indeed another problem to solve. And verifying the individual TTL components of that trusted hardware yet another one. <mihi>I'm not that interested in fuzzing, as long as it is easy to spot bugs by just loading modules that are already available (now that mes-load-module works). Especially since the bugs found that way are actually relevant, not potential bugs like fuzzing would find. <mihi>About the macro expansion code: I'd love to really fix that, but my problem is that I don't really see why some special forms ('if, 'let, 'case etc.) are handled differently in macro_eval from normal eval. While others like 'when are not. Having my previous scheme experience from Tinyscheme, there macro evaluation works like that: (1) Collect/bind parameters as if they were quoted (2) evaluate the <mihi>macro body as if it was a function body (3) evaluate the result of the last evaluation again. But probably Guile is different here and I just do not know about it. <mihi>About primitives: append does not match what guile does as it only handles 2 parameters, so I believe calling it append2 is more appropriate (like done in mes). <mihi>So, my current roadmap for mes-m2 is: (1) improve the README to make clear what the goals of the project are (2) fix the obvious blunders like mismatched arguments in mes-macro.c, without really understanding why it is as it is (3) wrap up everything and get rid of mes-m2 on all my hard disks (4) look at wip-m2 instead. <mihi>OriansJ: Interesting approach, when not being able to debug a program to try replicating it from scratch. THis is certainly a good learning experience. The only thing I do not like about it that it was not made clear to me (and as I interpret some other reactions on the channel recently, also others) <siraben>mihi: Did Tinyscheme have hygienic macros? <siraben>I agree with making the READMEs more clearly state what the goals of the project are <mihi>and it is no guarantee that when you have replicated the code from scratch, you won't hit the same roadblocks again <mihi>siraben, I don't think so. They have (make-uninterned-symbol) so you have to emulate it like you do it in Lisp. But my last experience with TinyScheme was ~10 years ago so perhaps they have it now. <siraben>Yeah, one of the issues with our current macro expansion is properly doing syntax-case and friends *janneke pushes a fresh wip-m2 <mihi>so mes-m2's define-macro is supposed to be hygienic macros? I don't really see that from the code, but perhaps that is the reason that it tries to hardcode some thinsg differently. But not sure why 'if and not 'when. <mihi>janneke, I'm confident without lookig at it that it is less sketchy than current mes-m2 :P <janneke>i'll be working on cleaning up the m2-planet forked bits of the mes c lib <mihi>siraben, Google suggests there is a pure TinyScheme for hygienic macros based on the non-hygienic ones they support. <janneke>note that this branch will be rewritten, as it's called wip-* <siraben>So wip-m2 is another Scheme interpreter written in C? <siraben>that should fix the issues of mes-m2? <mihi>siraben, wip-m2 is trying to be a 100% mes compatible Scheme interpreter written in C compilable with M2-Planet. <janneke>gnu mes was written with the intention of being bootstrapped in assembly <mihi>while mes-m2 is trying to also include the kitchen sink, but so far is not even crawling <janneke>m2-planet was a thing that i hadn't dare to dream of <janneke>but it took still quite some work to rewrite simple c to m2 <janneke>mihi: mes-m2 has the potential of being fast and fully guile compatible, right? <mihi>yeah, I have made some experiences with M2-Planet as well, trying to optimize hex2_linker. I would not call it c, (not even a subset of c), but it is a nice platform-independent low-level language that resembles C enough to leverage the knowledge and tooling available for C. <mihi>and especially it is bootstrapped from stage0 :) <mihi>janneke, I don't know guile enough to make a qualified statement about that. From my gut feeling, the current implementation is simplifying many things (e.g. no seperate namespaces for modules), so to be both fast and fully compatible it probably needs a few larger refactorings. <mihi>but the "potential" is there in any case, like there is potential to make mescc fully gcc compliant :P <mihi>(if you put enough person-years of effort into it) <OriansJ>mihi: if you fix something. Send a pull request and/or alert me on this channel to the commits you want me to merge. I am not a mind reader and I rarely use github. <OriansJ>Also if there is anything about mes-m2 you don't like or don't understand you can either change or ask for clarification. I am more than happy to answer any questions. <OriansJ>and the reason for the guile compatibility is to solve guile's psyntax.pp bootstrap problem and yes I certainly have done many things wrong in my mes-m2 design. <OriansJ>But I am more than happy to work with you in fixing those issues. <OriansJ>also there was a plan on expanding MesCC to the point where it will be able to directly bootstrap GCC, so we can skip the TCC steps entirely. <OriansJ>As these projects are those done out of love and not always practical reasons; So how people feel about the work matters more than the technical details involved. <bauen1>so do we finally have a complete bootstrap chain from kaem / hex0 to gcc (or tinycc) ? <bauen1>i should really get a move on with my kernel in any case <janneke>OriansJ: i have updated wip-m2 to work with m2-planet 1.7.0 and mescc-tools 1.1.0 <janneke>there are now only 10 out of ~80 libc files that need a special m2-planet file <janneke>causes are: standard prototype with an array parameter, prototype with special type name (mode_t, clockid_t, ..), prototype with an ellipsis, code that must be excluded #if SYS_gettime ... <janneke>OriansJ: i'm hitting a Floating point exception though, when using INT_MIN <janneke>it was all fine with m2-planet 1.4.x and my non-rebased wip-m2 <janneke>it's most probably on the mes side, i haven't investigated yet, but ring any bells? <OriansJ>janneke: as M2-Planet doesn't support floating point at all but x86 does create floating point exceptions for divide by zero. I'd look at modulus and divide operations <nimaje>line 63 in the readme update seems to miss a space after $ <nimaje>iirc only lines starting with one or more # are section headers in markdown, and doing a underline with some symbol doesn't create a section header <janneke>OriansJ: thanks, i'll narrow this bug down first then <mihi>nimaje, thanks, OriansJ was faster with merging :) <janneke>OriansJ: a propos collective scheme, i feel we already contributed a lot to eachother's work <janneke>eg, i used gnu indent a couple of times to backport work you did in your fork <janneke>notably for the untangling of src/*.c <mihi>and both ==== and ---- if alone on a line will create headings in Markdown <mihi>and in my experience they stand out better when reading the same file in a text editor <janneke>also, i really liked your ideas/work on mes-m2, epecially using a single, simple signature for SCM procedures *mihi got disconnected again. Seems the new wifi repeater that Santa brought is faster, but sometimes flaky... <janneke>i think we mustn't be discouraged or get impatient <mihi>(I know CommonMark is not Markdown, just like Wireshark is not Ethereal) <janneke>OriansJ: being guile-compatible from the c level, etc. <nimaje>yes, it reads better, I just didn't know that that was added (rst also allows other symbols like ~ to be used, allowing more levels) <senzilla>What is the "knight" architecture referring to in the context of M2? Haven't seen that before... <xentrac>if I understood OriansJ correctly, the Knight CPU was a product of a company (also called Knight) that went out of business in the 1970s