IRC channel logs

2022-04-15.log

back to list of logs

<unmatched-paren>a simplified graph of the current `tree' of compilers that have been bootstrapped starting from stage0 would be nice (maybe on a `Compiler Tree' page on the wiki?)
<unmatched-paren>that way people could see at a glance what we've currently got... `unlocked'?
<unmatched-paren>we've got a graph to GCC from stage0 but we could extend that
<unmatched-paren>(and probably simplify it to only include compilers and maybe build systems)
<muurkha>tech tree
<unmatched-paren>yeah! :D
<unmatched-paren>and potential paths (does graphviz allow different colours for nodes and paths? never used it except with `guix graph')
<muurkha>yeah
<muurkha>see gallery
<unmatched-paren>muurkha: oh, you mean graphviz.org/gallery?
<unmatched-paren>oh, wow, i didn't realize it allowed that degree of customization
<muurkha>aye
<unmatched-paren>so, we could have, say, tcc -> gcc or mrustc -> rustc in green (done), hugs -> ghc (or whatever the most viable path is looking like right now) in orange (potential path), APC -> FPC -> nim in blue (solution is being worked on but not done yet) and lone SML/NJ and MLton nodes in red (not bootstrappable, and nobody has yet found or created a path)
<unmatched-paren>i'm kind of surprised that nothing like that exists yet
<unmatched-paren>at least, i can't find anything
<unmatched-paren> https://bootstrapping.miraheze.org/wiki/File:M1_diagram.png <- this is the diagram i'm thinking of
<unmatched-paren>but it only goes to GCC
<unmatched-paren>an exhaustive graph should probably have only one node per project instead of per program
<unmatched-paren>so [hex0...cc_*.S] would be replaced with just `stage0'
<oriansj>unmatched-paren: the reason it doesn't exist yet is rather simple. No one was willing to put in the effort to create it yet.
<oriansj>A more complete graph is here: https://github.com/oriansj/talk-notes/blob/master/live-bootstrap.pdf (its source is here: https://github.com/oriansj/talk-notes/blob/master/live-bootstrap.dot )
<oriansj>it is a good bit out of date
<oriansj>but if you make a pull request with an update, it'll get merged.
***janus__ is now known as janus
<unmatched-paren>oriansj: ok, i'll read up on using graphviz and send a PR to add a full graph :) then we can add it to the wiki
<unmatche1-paren>ok, my graph now shows the paths for:
<unmatche1-paren>C, Make, Perl, Autotools, C++, Python, CMake, Ninja, Meson, Guile, Chez, MIT/GNU Scheme, Racket, CLISP, ECL, SBCL, Raku, D, Pascal, Nim, Go, Rust, Haskell, Idris, OCaml, Zig, C#, F#, SML, Fortran, Ada, Java, Groovy, Scala, Kotlin, Gradle, and Clojure.
<unmatche1-paren>What have i missed?
<unmatche1-paren>(Of course, not everything there is bootstrappable yet)
<unmatche1-paren>the graph should have every notable language or build system that we know of in existence, even if it's trivial to bootstrap
<unmatche1-paren>oh yeah, maven
<unmatche1-paren>there's a lot of red 'circular dependency' arrows around gradle...
<unmatche1-paren>it's done! i think... https://static.miraheze.org/bootstrappingwiki/7/74/Tech_Tree.svg
<unmatche1-paren>it's intentionally simplified, especially at the live-bootstrap parts
<unmatche1-paren>since there's already graphs of that bit, and it's quite large and complex (e.g. it shows four musls)
<unmatche1-paren>please let me kow if there's any mistakes.
<unmatche1-paren>muurkha: ^
<bauen1>i'm a bit surprised the linux kernel doesn't depend on perl, python or something like that, surely there are some scripts that get executed during build ?
<bauen1>or is it all bash + awk + grep, coreutils ?
<unmatche1-paren>bauen1: i think you're right actually, i'll fix that
<unmatche1-paren>let me check...
<unmatche1-paren>bauen1: you are apparently correct, according to gnu/packages/linux.scm in guix
<unmatche1-paren>i'll upload a new version
<unmatche1-paren>the whole graph completely changes its layout when you make a tiny modification like this :)
<unmatche1-paren>there's probably many languages i'm missing that i can't think of off the top of my head
<unmatche1-paren>muurkha, bauen1: fixed version: https://static.miraheze.org/bootstrappingwiki/7/74/Tech_Tree.svg
<unmatche1-paren>odd, it's not fixed
<unmatche1-paren>the online one, that is
<bauen1>unmatche1-paren: for ninja, there is a C implementation, with only minor differences https://github.com/michaelforney/samurai which can be used to put it a lot further down in the tree
<unmatche1-paren>bauen1: samurai is in that graph :)
<unmatche1-paren>maybe i should replace all references to ninja with samurai
<unmatche1-paren>and meson with muon
<unmatche1-paren>oh, i think it's just qute caching the image
<bauen1>is there any plan on making a more detailed graph ? e.g. where each node describes a "build recipe" including versioned dependencies and a link to the repo where the build is implemented ? that would probably be massive
<unmatche1-paren>and... now i get a not found when trying to click to the large version?
<unmatche1-paren>bauen1: i'm not THAT crazy :)
<unmatche1-paren>ok, it seems to be working now. you'll probably need to clear your browser's cache to see the new version
<bauen1>unmatche1-paren: nope, no problem here no cache cleaning necessary
<unmatche1-paren>hm, never mind then :P
*unmatche1-paren looks at https://en.wikipedia.org/wiki/List_of_programming_languages_by_type
*unmatche1-paren sobs quietly
<unmatche1-paren>probably only about ~15% of them actually matter though
<unmatche1-paren>i don't think bootstrappable cares about swift, for example
<unmatche1-paren>since its sole purpose is to write apps for Apple(R)(TM)(C)(C) devices
***unmatched-paren is now known as Guest8610
<oriansj>unmatche1-paren: stage0 doens't depend on linux, only stage0-posix does (which is just a port to simplify scripting the bootstap process)
<muurkha>bauen1: that sounds like a thing that could be scripted
<unmatched-paren>oriansj: guess i'll correct that node to say `stage0-posix' then
<unmatched-paren>or, actually, change the `linux' node to point at `m2-mesoplanet' instead, which doesn't work without posix, right?
<unmatched-paren>so is the idea that we build m2-planet to work on bare metal, then write a tiny kernel in m2 that can run mes etc?
<unmatched-paren>i've uploaded a new version now :)
<oriansj>unmatched-paren: that is correct M2-Mesoplanet requires an operating system for the Environment and spawn behavior (which could be disabled with a custom cc.c file) and a minor detail Mes-m2 can be built by M2-Planet right now. The M2-Mesoplanet is about building mainline Mes (instead of running MesCC on Mes-M2)
<unmatched-paren>oriansj: ok, i'll change that in a moment :)
<unmatched-paren>oriansj: so i suppose mes-m2 requires a kernel?
<stikonas>unmatched-paren: mes-m2 doesn't
<stikonas>mescc probably does
<unmatched-paren>how does it interpret .scm files if there's no such thing as a file?
<unmatched-paren>unless it runs under EFI...
<stikonas>no, you would have to feed everything in one go then
<stikonas>if you want file input
<unmatched-paren>but then by using the EFI implementation from the CPU vendor you get a whole lot of other trust issues :)
<unmatched-paren>ah, i see
<stikonas>well, same as m2-planet vs m2-mesoplanet
<stikonas>m2-planet needs a single c file
<stikonas>and actuall without kernel it's not even a file
<stikonas>you need so use simplified main() without command line arguments
<unmatched-paren>so you modify them to read from a hardcoded constant string if you want them to run on bare metal?
<stikonas>anyway, it's probablu not too important because why would you need to run it without kernel
<stikonas>just build kernel with m2-planet
<unmatched-paren>yeah
<oriansj>unmatched-paren: or from a serial port or paper tap ereader or etc
<unmatched-paren>oriansj: ahhhh
<stikonas>it's hardware dependent
<stikonas>and hence not easily automatable
<stikonas>that's why we have stage0-posix
<unmatched-paren>right
<stikonas>so that people can run nice bootstrap chains without worrying too much about non-portable issues
<oriansj>extremely hardware dependent but done in such a way stage0-posix and stage0 on bare metal should produce identical binaries given the same sources
<stikonas>it's a bit of a cheat
<stikonas>but a lot of people were interested in it
<unmatched-paren>basically a shortcut until we can get it going from bare metal, if i understand right?
<stikonas>so it was worth having it
<stikonas>even after that
<stikonas>some might not want to do manual steps
<stikonas>e.g. let's say some distro wants to do bootstrapping from stage0
<oriansj>unmatched-paren: Less of a shortcut and more of a second root path to use
<stikonas>they probably would just start with stage0-posix and build on top of that
<stikonas>rather than each time manually kick off someting on baremetal
<muurkha>right, reproducing the same binary means you don't need to worry your kernel intervened in your build process to introduce a backdoor
<stikonas>well, we do have checksums
<muurkha>although until you can build the kernel itself with m2-planet you still have to worry that it intervened *after* your build process
<stikonas>so either all kernels are backdoored in the same way
<stikonas>or they are not
<muurkha>right, of course
<oriansj>but a hardcore bootstrapper may wish to do the steps manually to provide a second root check against the stage0-posix binaries produced.
<unmatched-paren>oh, yes, it's safe to use prebuilt binaries if the sha256sum is the same as a totally bootstrapped version
<muurkha>right, although you need to not run the sha256sum on a potentially backdoored kernel
<unmatched-paren>of course... once we get stage0 from bare metal, we'd need to start worrying about the CPU and BIOS itself, which is a whole other can of worms...
<stikonas>unmatched-paren: the question is how do you trust your sha256sum then
<unmatched-paren>yeah
<oriansj>and then we need to do libre-silicon and other fun
<stikonas>implementing trusting trust attack on sha256sum is probably harder than in the compiler
<stikonas>actually it's not even relevant, since sha256sum does not produce output...
<stikonas>so any backdoor would be non-propagating
<oriansj>also we can dump binaries to paper tape or CDs or other media for inspection from offline validators and use those same dumped binaries for running
<unmatched-paren>even if we built a completely audited CPU, 99% of the world will probably STILL go on using their intel or AMD chips with Microsoft Windows 11... :(
<oriansj>unmatched-paren: one can not save a man who wishes to drink poison
<muurkha>I think less than 30% of the world uses intel and AMD chips with Microsoft Windows 11
<muurkha>most people use hand computers now
<oriansj>ARM+Android/iOS
<unmatched-paren>muurkha: oh, i forgot that laptops are apparently considered to be outdated technology now by most people
<stikonas>didn't look like that during covid lockdowns...
<stikonas>everybody was buying pcs/laptops