<mihi>OriansJ, janneke: sorry for the confusion about file.c in the bootstrap. I had worked around it (by making cc_x86 still use the old file) in mescc-tools-seed, but I did not send a separate pull request but references the commit in the pull request for M2-Planet. Next time I will make separate pull requests. Doing pull requests across submodules is presumably something you cannot get right :( <yt_>FYI this was my first bootstrap of M2-Planet on Apple Silicon, inside a qemu VM running Ubuntu :D <janneke>mihi: np, scrambling all bits together is quite a bit of work <janneke>how much attention and love all these bits constantly need... <mihi>and when you revert that commit, it works, but while being reproducable on amd64, it is not on x86 (32-bit) as the tmp variables get different names... <mihi>not sure if they are based on hashcodes or something... <mihi>probably will dig into it at some point in 2021 <janneke>OriansJ: latest m2-planet master 358b6cf Fix modulus behavior that is wrong. works! <janneke>the reason that the bootstrapped (non-gcc-built) m2-planet was so bad, is probably because m2-planet uses modulo <OriansJ>janneke: does that mean we finally got mes.c bootstrap from M2-Planet? <OriansJ>actually M2-Planet only uses division and modulo in char* numerate_number(int a); in the file functions/numerate_number.c <OriansJ>and I submitted bootstrap-binaries for inclusion in savannah. Hopefully they don't reject it. *janneke starts another build after cleaning-up wip-m2, wip-full-source-bootstrap <janneke>with mes check phase enabled...let's see <janneke>thanks a lot OriansJ for today, great progress! <OriansJ>tomorrow is going to be another interesting day <OriansJ>depending on janneke's success we might have ALOT of work in mescc-tools-seed tomorrow or I'll just have a day of making file.c get improved performance in a cc_* friendly way <OriansJ>plus I need to put some work into M2libc; so that more conventional C primitives will be available for those who expect them. <OriansJ>well I guess I need to add typedef to M2-Planet to support proper FILE as a struct <janneke>OriansJ: haha, that's one thing on my mes c library's list too <janneke>oh, building tcc-boot0 did not survive the m2-transition of mes: <janneke>libc.c:2695: error: incompatible types for redefinition of 'execve' <janneke>./pre-inst-env guix build --system=i686-linux -e '(@@ (gnu packages commencement) tcc-boot0)' <janneke>=> /gnu/store/nppws6i5vp37l4xyi9ghhrgfrwhygi4v-tcc-boot0-0.9.26-1136-g5bba73cc <OriansJ>janneke: what about mes.c self-host? can mes.c built by M2-Planet run MesCC well enough to build mes.c? <janneke>OriansJ: yes, that's actually what sh bootstrap.sh does <janneke>it runs kaem --verbose --strict to build mes-m2, then mes-m2+mescc builds mes and the mes c library <janneke>[22:38:42] <OriansJ> I'd even incorporate mes in it too if I knew an exact commit which would work <janneke>[22:39:13] <janneke> ah no, i'm keeping mes a separate package <janneke>in the guix bootstrap, i won't be using such an integration, but i imagine it could be very handy for people like fossy if mes could be built in mescc-tools-seed <janneke>so, my only requirement would be that building mes is a separate/optional phase, but i guess you already split into different phases <OriansJ>I know this sounds stupid but shouldn't the build of guix's bootstrap binary be done outside of guix to prevent circular injection attacks? <rain1>it sounds smart, I think you are right <rain1>although perhaps that only needs done once, and from then on it would be ok to do it inside guix (since it is easier and more automatic) <OriansJ>we have kaem with --init-mode (formerly known as nightmare-mode) <OriansJ>also since we can control the early stages of the bootstrap exactly why would we be dealling with compressed tarballs? waste some bandwidth <janneke>i did the simplest thing that i could imagine at the time <janneke>also, none of these patches were reviewed yet, so we'll see! <janneke>i was wondering if we could bless the bootstrap-seeds-1.0.0 <OriansJ>an endless cycle of improvement we seem to be stuck on ^_^ <janneke>iow, do multiple independent builds, add hashes, and say they are *safe* <OriansJ>janneke: well I gpg signed the tag, the sha256sums are trivial proof how much of a blessing does it need? <janneke>well, the hope is/was that once we have something that works, people will show more interest and join the endless cycle <OriansJ>well technically everyone on this channel shows that they already started showing up <janneke>well, a statement other than "Here are the bare minimal binary seeds for every host architecture <janneke>saying: these are verified by x,y,z; these are the hashes <janneke>this sounds *very* un-blessed (even if they are ;-) <OriansJ>well I do have the concern of github decides to change the contents and people get rooted because of it <janneke>yes, i don't trust microsoft for a bit <janneke>people shouldn't blindly download and run, but that goes for any download <OriansJ>that is why: git tag -as -f Release_1.0.0 Release_1.0.0 <janneke>yeah, but you also need a content hash <OriansJ>6d3fb087efe2d7cc7938cf2aff0265c6bfc86370 <janneke>rigth -- mention it on the website and check it on the packaging/bootstrapping host <janneke>it would be great to get outside help with missing documentation <janneke>that would make a great insert of the home page! <OriansJ>as clearly we have all gone insane in our quest <janneke>well, it's getting harder and harder to get any work done <OriansJ>janneke: hey I have an excuse, I like the people here. <janneke>no, but really, a statement saying these are blessed binaries, be very careful and check these, and then somewhere below (some of these) hashes, that doesn't seem too bad? <janneke>it's been almost 5 years and we only just about have our first version done <OriansJ>janneke: I could provide a simple step of steps to enable anyone to verify that the binaries correspond with the .hex0 sources <xentrac>longer if you count things like Ur-Scheme and StoneKnifeForth <janneke>*** #bootstrappable was created on 2017-06-27 10:28:29 <janneke>xentrac: but yeah, luckily there are always giant's shoulders <OriansJ>(I always forget the A though, probably because he is so nice; it creates an A hole in my mind) <janneke>in a way a parent to guix and bootstrappable builds <OriansJ>I really should start a history.org file in my talk-notes <OriansJ>So we don't forget all these really important leading pieces. <janneke>it would be good to have more completeness and detail <OriansJ>well I'll take a whack at it and probably miss a bunch in the process but hopefully patches will occur <janneke>i noticed that ARM is missing in bootstrap-seeds, how far are we with that? <OriansJ>janneke: well I have first brush assembly done (needs some refinement) and then I just need to convert to M1 and hex? <OriansJ>So it is only a week worth of work left (if I can get the time to focus on it) <janneke>from mes onwards, the ARM bootstrap now includes tcc, binutils-2.14, gcc-core-2.95.3 but is stuck on a miscompiled gawk-mesboot0 (floats?) to build glibc-2.2.5 <janneke>i planned to release mes-0.23 with ARM support months ago, but there were some unexpected hurdles <OriansJ>janneke: there is a reason why I avoid floating point. It is a massive pain in the ass for little gain. <OriansJ>an arbitrary precision library written using chars would be less trouble. <janneke>OriansJ: yes, me too -- fully agree. not necessary and a terrible "solution" <janneke>anyway, we really don't need them until tcc, but tcc needs to have them <OriansJ>janneke: well it isn't terrible if you are ok with approximations in your output (eg Physics calculations) <janneke>mes helps in makeing that jump, but it's tricksy, esp. on ARM <OriansJ>but outside of that no compiler should need floating point to work. <janneke>"lets do some calculations, and make them fast" -- i don't really care about the results <OriansJ>have you ever had dinner with a nuclear engineer? <OriansJ>don't worry you'll know because they will tell you. <janneke>a physicist cares a lot about the precision that counts and knows when more precision is a waste of time/effort <OriansJ>except when their assumptions were wrong. <OriansJ>which happens far more than they will ever admit <OriansJ>I've spent too much time talking to nuclear physicists to believe otherwise. <janneke>we could introduce imperial units in computing! <OriansJ>you had to screw up on grams in metric because of a French asshole <OriansJ>So instead of a gram being what it was supposed to be, it is 1/1000th and you have to use kilograms where grams were supposed to be. <OriansJ>meters => fine; second => imperial as fuck but who cares; liters => we are cool but what the shit you can't do grams correctly because of the French revolution and its piss match with a French scientist who was born rich?? <janneke>luckily, the times when being born rich really mattered are long gone <OriansJ>I like to think of it this way. If you had enough free money in 2008 to get into overclocking but was willing to throw away $10 on a meme; you would have $180,000,000 worth of bitcoin. <OriansJ>though odds are you would have kept them in Mount Grox and lost everything anyway <OriansJ>So life shows luck dominates every other factor. <xentrac>well, or maybe cashed out when bitcoin hit $10 <xentrac>a friend of mine once pointed out the curious fact that somehow there is real estate not owned by Sumerian investment bankers <OriansJ>xentrac: that is part of the luck bit huh. <OriansJ>some people believed it would go to a $1,000,000 per before 2021 and bet that they would eat their own dick on national TV if it didn't. <xentrac>seconds are more Sumerian than Imperial really <OriansJ>xentrac: well Sumerian seconds are a different length than modern seconds <xentrac>well, I don't think the Sumerians actually had seconds <OriansJ>thinking of bootstrappable history, I realize there are reflections in other areas too. Spore/Thrive; Open Ecology, The Knowledge, How to make Everything, Dr Stone, etc <OriansJ>Poul-Henning Kamp's FOSDEM 2014 talk NSA operation ORCHESTRA: Annual Status Report <OriansJ>iCEstorm and the FPGA work, libresilicon, etc <OriansJ>Mike Perry, Seth Schoen and Hans Steiner's #6240 Reproducible Builds talk is probably the first one I saw <OriansJ>almost like there are two common patterns for preparing for failures in the future. Plan to rebuild form scratch or plan to restore from backups (Internet Archive, libraries, etc) <akkartik>I just spotted a post by janneke on Mastodon. Looks like there's been lots of progress. Congratulations! <janneke>akkartik: yes, we had a breakthrough that was very long in the coming <deesix>OriansJ, sorry. Yesterday I said that maybe AMD64 close() is wrong. Now that I have my notes at hand I see that I was thinking about armv7l. It is missing "!0 R0 LOAD32 R0 MEMORY", right? <deesix>... wrt the x86 fix at M2-Planet 608fba306f46128fe901302f813d139c635e996a <OriansJ>deesix: it is entirely possible. We don't have a good test for close <siraben>akkartik: could you link to the post? <siraben>what was the missing piece, rewriting the scheme interpreter to support hygienic macros? <OriansJ>siraben: punting on the Guile problem for now and just getting mes.c building with M2-Planet <siraben>Nice, I look forward to replicating this work using Nix <janneke>siraben: mes->gcc has been in guix already for ~2y now; until now we started from mes <stikonas>well, I guess guix still has to use guile-bootstrap... I guess you can only get rid of it using kaem and gash manually <OriansJ>or growing mes-m2 until the point where it can run MesCC, gash, gash-utils and Guix <janneke>stikonas: guile is only used as a driver and shell/coreutils stubstitute <janneke>if you find a another way to instruct the computer, e.g., type by hand, you do not need guile; it's *not* part of the bootstrap seed <OriansJ>janneke: well all the steps guix would do could just be recorded and put into a kaem script <stikonas>hmm, I wonder how long the script would be... Might be quite long <OriansJ>Who cares if it is 1000 steps? A computer is going to be doing the work <OriansJ>stikonas: well I don't expect more than 100 commands to be executed per step <stikonas>maybe eventually that can simplify bootstrap of guix itself... <OriansJ>stikonas: just need guile and gcc to run guix <OriansJ>actually tar, xz are being replaced with bootar in the early stages and it is a scheme program <stikonas>but does that replace %bootstrap-executables from bootstrap.scm? <stikonas>it has static binaries of bash, tar, mkdir and xz <stikonas>I guess these are only used to drive guix <OriansJ>stikonas: well janneke already replaced bash with gash and gash-utils covers alot of common shell commands and they are just scheme programs too <stikonas>well, yeah, it's should be possible to use gash instead of static binaries of bash and mkdir and bootar instead of tar and xz <pder>What is the difference in functionality between mes built by M2-Planet in the wip-m2 branch and the mes-m2 project? <stikonas>mes-m2 built by m2-planet is less functional but I don't know exact details <OriansJ>pder: wip-m2 branch is janneke's original mes.c converted into something M2-Planet can compile mes-m2 was a from scratch rewrite with the goal of solving the guile bootstrap problem in guix too <OriansJ>Think of wip-m2 as a rope bridge from M2-Planet to bootstrap GCC and mes-m2 as a steel suspension bridge to move all the pieces needed to finish the city construction. <stikonas>steel suspension takes longer to build but in the end should be better <OriansJ>It needs alot more effort to be complete (full guile compatible) vs mes.c (minimal scheme to run MesCC) <pder>If I run kaem.run in the wip-m2 branch of mes, I get a resulting bin/mes. Is this scheme interpreter capable of running mescc? <OriansJ>pder: in theory yes (I haven't tested it yet) <pder>I see, so guile is still needed at this point to run the other things like gash? <OriansJ>pder: well how many of them run on mes.c is an open question as I am uncertain of what janneke has tested. <janneke>"<OriansJ> janneke: well all the steps guix would do could just be recorded and <janneke>my point exactly, guile is only a diver <janneke>we need to make that point very clear <janneke>but people like fossy are doing so, by writing another driver <stikonas>janneke: probably worth mentioning on those WIP docs... <stikonas>that gets me curious, what is fossy working on? <stikonas>oh, I guess bootstrap on some other distro... <janneke>using guix has other benefits, of course, such as containerized builds, but they also come with a price <janneke>pder: gash is no requirement, bash+coreutils also do nicely <mihi>I wanted to suggest to try busybox coreutils, but when trying to compile them with mescc, I noticed they are full of _GNU_SOURCE functions like strchrnul... So probably not the best idea :( <OriansJ>mihi: not until MesCC is able to directly compile GCC <mihi>OriansJ, or rather glibc. Not so much about language features, but about c library functions :) <mihi>on the other hand, when you have glibc+gcc available, you can compile "real" coreutils as easily <OriansJ>mihi: yeah pretty much the major problem in bootstrapping. Everything likes to leverage C features to make their lives better/easier. <mihi>tbh those features/functions also often make the code easier to review; yet harder to bootstrap <OriansJ>mihi: no one cares about bootstrapping their programs; otherwise they would be here trying desperately. <OriansJ>There are millions of professional programmers in the world. How many work on FLOSS software? Thousands? of those how many care about reproducible builds? Hundreds? Of those how many care about bootstrapping? tens? of those how many actually contribute code? 8. <mihi>and how many care about their abominations of web browsers being reviewable? The monster that used to be KHTML and is part of Webkit today needs gigabytes of temporary disk space during compilation and instead of reviewing, the browser vendors hope that fuzzing will find most bugs before badguys do. but yeah off-topic... <mihi>or let's put the sandbox into a sandbox... <mihi>then introduce high-precision timers to untrusted javascript codes, and some time later complain about Spectre/Meltdown "vulnerabilities"... <OriansJ>I'd fix all these problems if I could <OriansJ>but I am one man working in what little free time I have between a full time job, a marriage and a 10 month old baby who likes to pick locks. <OriansJ>So I am focusing on a foundation others can build upon <mihi>OriansJ, I assume it also includes you :-D <OriansJ>not to the level that xentrac achieved in dercuano but hopefully to GCC <OriansJ>mihi: in sheer lines of code the order is: OriansJ (38,277), janneke (37,776) and then fossy (1,501) <OriansJ>ooops sorry yt: you beat out fossy at 6,912 lines <janneke>when you choose to count lines of code, do no tthink of them as lines produced but as lines spent ;) <OriansJ>doing sloccount stage0/ mescc-tools/ M2-Planet/ mes-m2/ mescc-tools-seed/ mes/ we get 158,949SLOC with an estimated cost of $5,533,021 (There is some duplication in the count so it is probably wrong) <janneke>the patches i made to tinycc were *very* expensive, per loc; as are the patches i made to glibc, gcc <OriansJ>wow janneke $5Million; world class programming work there <janneke>OriansJ: irregardless of how you count it, i believe we have started something pretty amazing <janneke>it deserves much more attention, labour and love; that time will come <OriansJ>janneke: I believe deserving doesn't matter; I literally couldn't do otherwise. even if I never see anyone ever appreciate the work. I had to create it because it didn't already exist in my world. <OriansJ>The fact that people have noticed and cared enough to try to help improve it is just ice cream on top. <janneke>OriansJ: yes, i agree; it's just that i believe it is worth more attention, though <OriansJ>janneke: well we have a lot of competition on that front. It is the attention economy these days. <OriansJ>I mean world's first C compiler written in armv7l assembly basically got 2 upvotes and was buried in 5 minutes <OriansJ>xentrac: sorry I couldn't figure out where to put stone-knife forth or Ur-Scheme in it; perhaps you could help with that? <OriansJ>janneke: hopefully I didn't get your start too wrong. (feel free to fix anything you thing wrong) <OriansJ>as everyone here should be able to include what brought them in and what they are most proud of. <OriansJ>To make it more human and interesting. <fossy>stikonas: a distribution independent Linux bootstrap that relies on the least number of seeds possible with portability to other kernels <fossy>github.com/fosslinux/live-bootstrap <fossy>I've hit an error with tcc and its taking me a bit to resolve, but once I get past that I'll be back on the roaf <janneke>OriansJ: hehe, that's pretty accurate; i like your viewpoint anyway <janneke>from my pov hex0 was both an inspirational spark and at the same time it also felt like an overwhelmingly long path, a fool's undertaking -- yeah wow i want this but "i don't have time for this shit" <janneke>your m2-planet really took my by surprise, i was prepared to have to code mes.c in assembly, eventually <janneke>OriansJ: so, you actually responded/acted upon civodul's writings in the guix manual about reproducible builds and bootstrapping (that were written after the first reproducible-builds summits i believe)? <fossy>I was reading the paper "Fully Countering Trusting Trust through Diverse Double-Compiling" the other day <fossy>however the title I believe is misinformative <fossy>it dosen't really fully counter trusting trust as it requries a so-called "trusted compiler" to verify the results against, if you use two compilers that are maliciously modified in the same way your double-compiling is useless <fossy>i think that's where our work comes in :) <rain1>yes I have never understood the point of that ddc stuff at all <fossy>if you only have resources to bootstrap a compiler you can bring your other production compiler into the trusted execution space that is bootstrapped and use ddc to check your production compiler is good <fossy>however using it in the method described is useeless because your execution environment is tainted potentially <fossy>so how do you know your utilities such as sha256sum that you have to use to verify your ddc are trusted? <mihi>fossy, how do you know SHA256 can be trusted and does not have a backdoor so that your backdoored compiler has the same hash? use cmp... <OriansJ>you are right fossy; I missed David A Wheeler's DDC work. now to figure out a clever title for it <fossy>mihi: and how do you know that that's not also backdoored <fossy>my point is ddc has to be done in a trusted execution environment <fossy>which is not elaborated on at all in the disseration <mihi>by compiling it with my trusted compiler, after having reviewed it (or written from scratch) <mihi>fossy, such details are left as an exercise to the reader :P <fossy>and i believe our bootstrappable solution is the most robust trusted compiler that is possible, tbh <mihi>fossy, agreed if ~s/possible/available/ ***lfam_ is now known as lfam
<fossy>but once we get to bare metal i'm not sure how much further you can go :P <OriansJ>fossy: free hardware designs clearly <OriansJ>and janneke hopefully the new revision is a better reflection for you. <janneke>OriansJ: thanks; would you mind adding just a bit more about your own inspiration: i would like that! <Hagfish>i think the point of DDC is that it increases the cost for an attacker from "make one malicious compiler" to "make every available compiler malicious in compatible ways" <malina>I contacted edmund, of those hex bootstrapping notes many years ago and enquired vs. finding more work to get from his examples, to gcc, then ranted a bit about the US <malina>and all these years later, I see now this stage0 thing <malina>great! it is so important to have a full source based path which people can self-verify indeed. wow, great stuff, am gonna play with this and since gnu has sent me on a while goose chase , caling things are 'bootstrapping' when they are nothing of the sort really <Hagfish>stage0 seems like an idea that the universe required to be implemented at some point. probably every intelligent civilisation produces something similar at some point if it survives long enough :) <malina>anyway, ultimately, another missing piece in the puzzle is quite possibly then even mor edangerous one, firmware, esp microcode in cpu's. The free world simply should only allow products signed by some civil authority which has been handed relevant source, and not companies/countries like INtel./US, yikes signing their own shit.. unfuckibgbelievable, pardon my Inrgrih <janneke>OriansJ: come to think of it, i wouldn't mind if mescc-tools-seed's functionality would move into Stage0 <malina>you mean any intelligent civlisation which doesn't destroy itself, or anyting amazing like rare gems , would eventually digitise binary? sure. <malina>but I don't think they would necessarily, have lost the early bootstrapping processes, even if we did <malina>or so, which in turn can hit gcc and all this? <stikonas>well, at some point before everything was already bootstrapped (via hand-written assembly and I guess cathode ray lamps...) but bootstrap path was forgotten. stage0 just makes it more modern :) <rain1>unfortunately it is not a tool we have access to: the silicon stuff <stikonas>malina: yes, it can reach gcc but obviously, you still need to trust some other stuff a bit... (POSIX kernel, or your baremetal bootloader, etc...) <malina>but I htink that's a fairly good compromise <OriansJ>malina: it is a long work in progress but I promise we will get far farther than anyone expects. <malina>yes, I realise in th past years guix is my most obvious contender to ideas lol . but nvm. Although I thnk it's a GPL thing, I guess *if* I move on to some other person's distro, I might very wel do guix <janneke>malina: tbh, this is all very fresh; so while the previous "reduced binary seed bootstrap" is integrated in guix, this full source bootstrap just may need some work yet <OriansJ>janneke: as promised I added my inspiration. <malina>I thnk that is because you are trying to go beyond just the bootstrap of self compiling, but including the 'auxiliary tools' lik editor, and what not, yes? <OriansJ>malina: stage0 has a line editor if you need it <rain1>emacs is image based so there might avctually be a genuine issue there? maybe it compiles without image though <malina>I just meant OriansJ , that I see/saw you guys had been inspired by edmund's notes, and also done this work and relased it in 2017..2018 etc, so when you say something is fresh, and menbntion 'reduced seed', I am guessing you have some 'additional' milestone I just happened to coincidentally stumble upon now <malina>which is another big step forward or something <terpri>emacs isn't image-based in the same way that, say, smalltalk is. dumping is an optimization; you can run temacs that's just c and compile/load all the elisp that would normally be preloaded