IRC channel logs
2025-01-17.log
back to list of logs
<mid-kid>As in, the live-bootstrap setup already has a script that takes care of building everything, wouldn't you be able to make a much smaller containerfile if you just used that? <lrvick>Because various steps fail, or need investigation a lot, when we are customizing things <lrvick>and then you have to start from 0 <lrvick>by having every step on a dedicated RUN line, we can always pick up from where we failed clean <lrvick>it is a massive development/experimentation boon, but also much easier to audit without all the plumbing in the way that we have to reason about <mid-kid>Though, wouldn't the ADD instruction near the top invalidate all other layers, every time anything in the steps changes? <lrvick>Yes it would, though for now I am not touching the steps themselves and just overriding things in place at lower levels with env vars and heredocs to override things in place at the steps I need to experiment <lrvick>and then when a pattern works, I can go adjust the base for a clean end result <mid-kid>It'd be really cool to get something like this useful for development, the current setup with packing and extracting packages works but is a bit finnicky. <mid-kid>ah, that makes sense for a workflow yeah <mid-kid>I guess you could also insert an ADD to update a specific step's files in the middle there <lrvick>yeah exactly. done that in a couple spots too. like just-in-time patches <lrvick>what I ultimately hope to do is one step beyond that example, and just export a folder of scripts and a Containerfile that are standalone and do not require live-bootstrap, configurator, or script-generator at all that bases on an existing stage0 build like we have in stagex <lrvick>at that point we end up with a super easy to audit/debug setup, but we can regenerate it if there are any significant developments in live-bootstrap upstream <lrvick>a container-native full-source-bootstrapped deterministic and multi-signed toolchain to build all the things <lrvick>based on stage0, and then partially live-bootsrap for our stage0 and stage1 container images, then we pivot to a cross compiler container in stage2, and native in stage3, and build everything else all the way up to rustc and beyond from there, which is "stageax" <lrvick>and then deriving a new "stage1" based on live-bootstrap right now <lrvick>whole distro works on our previously super hacky one, but making a leaner and easier to audit replacement now <lrvick>since OCI images do not even need docker to build these days, and can jsut use shell scripts, they make for a portable standard for staged builds like bootstrapping <lrvick>but you get nice output caching when you happen to be using dev toolchains like docker <lrvick>or kaniko, or buildkit, or podman, etc <lrvick>whole goal here is toolchain diversity <mid-kid>was reading the stagex code a little <lrvick>Thanks! We are basically a drop-in replacement for debian, alpine, etc in basically every containerized build process deployed in the wild today. <mid-kid>I'm just wondering what the approach is towards dependency resolution, unless there isn't any? <lrvick>Well classically in containerized builds, people would just "apt install gcc" or something, which is non deterministic and means your builds will vary from one day to the next. <mid-kid>I mean how do you decide what is built in what order <lrvick>oh. Our thinking there was just use the tools people already have installed <lrvick>so the current solution is just make <lrvick>targets.py generates makefiles, then make can handle build order <lrvick>using docker, or a shell script based OCI runtime, or buildah, etc <lrvick>we feel the more options we can implement the better <mid-kid>ah it's interpreting the COPY commands <lrvick>most decisions we made were about auditability over elegance <lrvick>and we also are bootstrapping a binary "sxctl" which we plan to be a standalone builder that can read from the packages directory and build anything in it, with only a basic kernel as the only dependency. Then we can build the whole tree baremetal in addition to being compatible with all containerized build systems. <mid-kid>I generally tend to forgo containerfiles/dockerfiles in favor of just using shell scripts, keeping each step "manageable" time-wise so I don't have to think about layers as much. <mid-kid>But you really went all the way and that's really cool. <lrvick>very explicit, but that is the point. <lrvick>you can build and get all the same digests we do at the top of the file, and rapidly form confidence we did not tamper with the final airgapos iso, which in turn is used for high trust key generation ceremonies etc <lrvick>we want to add a "lock" feature to sxctl that grabs the hashes from the lates release, verifies they have been signed/reproduced by 2+ maintainer keys you trust, and then automatically edit the Containerfile to pin the new hashes <mid-kid>wait, for some reason I thought hashes wouldn't be reproducible because you don't preserve the entire graph of images used to build one <lrvick>all our digests are deterministic, and match what is in the "digests" folder in the stagex tree <lrvick>so you can run "make" in the stagex tree in main, and you will get all the same digests from your own local build of the packages directory as what is pinned in this airgapos file <lrvick>Docker does not allow this as they have weird metadata nonsense, but if you force docker to build OCI spec compliant images (recently supported) or use any other OCI compatible builder, they match <mid-kid>yeah I also don't expect the hashes to *always* change when the hash of an input changes <mid-kid>that's what nixos does and for some reason that confused me a bit <lrvick>you just tell docker to use the "containerd-snapshotter" backend (OCI) and then all the intermediates are deterministic as long as your actual build commands are deterministic. <lrvick>sloci-image would be easy to support very very early in bootstrapping <lrvick>there is no reason live-bootstrap could not support it, and then live-bootstrap images could in theory build stagex directly too <lrvick>Thus live-bootstrap -could- build all the way up to rustc by pivoting to sloci-image to bridge to stagex <lrvick>But yeah, nixos and guix did not fit our threat model, so were kind of forced to start over <lrvick>and thankfully we did it at a time when the OCI standard can well cover all the package management needs so we don't have to roll yet another new packaging standard <mid-kid>that makes a lot of sense, especially to roll small container images, as most of the distros, even the reproducible ones, tend to include a lot of stuff only expected on a desktop system <lrvick>Yes, and all of them have points of trust in single maintainers, or things that are not actually reproducible. <lrvick>like they -say- qemu is reproducible in some of these distros, but they do not actually build the firmware etc from source, which we do. <mid-kid>yeah qemu is a bit of an "eh" problem <lrvick>we actually full source bootstrapped it with all firmware, because we need it in our build nodes <lrvick>since we run all builds in throw-away VMs in our CI <lrvick>next stop we want to be able to prove the CI systems have not been tampered with, so we need a remote attestation solution that will work on many different hardware vendors <lrvick>TPM2 attestation fits the bill, but that requires trust in the system firmware to not lie about PCRs <mid-kid>so how do you ensure full reproducibility <mid-kid>because as far as I'm aware you kinda have the same problem as guix where it's a best effort based on compiling everything on as many machines as possible <lrvick>and it is currently not possible to reproduce coreboot without an ada compiler... so currently working on bootstrapping ada, then we are finally able to go all the way down to firmware and tpm2 trust <lrvick>mkid-kid: We actually can not release unless multiple maintiners build and get identical results <lrvick>but our "repros" OS, also bootstrapped from stagex, is a potential answer to this <mid-kid>that's an insane amount of effort, really impressive <lrvick>where lots of people deploy ReprOS nodes that automatically build sign reproductions of every PR <lrvick>and we cannot merge until N signatures from maintainer-run ReprOS machines with remote attestation proofs all land <aggi>docker, virtual machines, etc... all this implies huge dependency graphs and a circular dependency with regards to bootstrapping; may not be a hindrance to elevate trust and reproducibility, but it is a hindrance to bootstrapping <lrvick>that is our plan to get to rolling releases without trusting any single maintainer or hardware vendor <lrvick>aggi: docker is optional, but a major quaility of live improvement for development <lrvick>aggi: you can build anything in the stagex tree with a posix shell script too <lrvick>that is why compatibility is a win <lrvick>standards mean people can build and develop the same core packages under different threat models. Some contributors can use docker on macs, while others will build only from a baremetal microkernel. <lrvick>once we have more people helping with reproduction, we can even mandate certian diversity charitaristics. Some building from baremetal microkernels, some in remote attested cloud VMs using ReprOS, some building on workstations, and in a diversity of legal jurisdictions <lrvick>we want to make it easy to keep stacking trust <lrvick>must be more env magic I did not account for <lrvick>The fact it is failing this deep makes me think this some sort of a bug that is being swallowed today some other way <lrvick>because chroot build works from the same head <lrvick>though 2.sh -does- run with "set -e". hmm <lrvick>What about autogen-5.18.16 is unlike anything that came before it pattern wise... <lrvick>which failes at autogen-5.18.16 as commented <lrvick>Maybe someone knows of a hack or two going on there that may be burning me <lrvick>looks like append-reverse is a feature that was added in mes-0.27 <lrvick>But yet it is complaining about append-reverse missing. Hrm <stikonas>if it somehow picked up mes, then failure is not unexpected <lrvick>Oh. so it is always injecting mes stuff, and this just happens to be the first spot it burned me <lrvick>at what point should I be unsetting that? <lrvick>Must have missed that since my setup forces me to bypass update_env.sh <lrvick>You probably just saved me several hours <lrvick>it seems to have made it further. Good sign <lrvick>Once this builds I get to run it again and find out why hashes do not match what is stored in tree sometimes. Probably other env var goofs <lrvick>we made it to python. very good sign. <stikonas>live-bootstrap does try to keep environment reasonably clean <stikonas>it's not really possible in early stages before bash <stikonas>but often later we clean environment by using exec env -i bash... <lrvick>Well and in a Container every line is a standalone shell that only inherits whatever env was explicitly above it <lrvick>which will make it very easy to reason about the state of the env once I get this working <lrvick>but currently it is hard to reason about all the script-based env mutation <lrvick>Maybe I inject an "env" command early and have it print the state of the env after every step and when it differs from a previous step so I can adjust the env in my containerfile at that step ^_^ <lrvick>Also is there a reason FINAL_JOBS is hardcoded to 2? Is there some determinism factor here? <lrvick>I know nproc is probably not available, but no reason I can't do JOBS=$((${$(</sys/devices/system/cpu/possible)/0-/} + 1)) right? <aggi>depending on when this is done, fiwix (or linux2.4) haven't got sysfs <aggi>minor side note, included JFS support with linux2.4; it's working, it's stable, it's easily done, and give a little more headroom until year2106 with posix32 kernel <aggi>it's just, i cannot backport nilfs2 to linux2.4, too much hazzle, and i need to migrate all my storage to another filesystem since i intend to move my development host back onto linux2.4 for extensive testing <lrvick>worst case we get a super bare bones "nproc" in mescc-tools-extra or something <lrvick>that would probably be the most portable solution to allow making use of more cores earlier in the build process <stikonas>live-bootstrap I think can override it using config file <stikonas>and the reason to keep it a bit more controlled manually is to be able to run in low memory requirements <stikonas>you can stil set it to high core count manually <lrvick>It seems it does not even switch above 1 core until the build of musl 1.2.4 <lrvick>but is there any reason in a container or chroot to not let it use all available cores for the whole build? <lrvick>even with rootfs.py chroot setup, it uses 1 core until musl 1.2.4. Thought maybe that was a determinisim workaround <lrvick>comment says SMP support, but not relevant if you are already running from a modern kernel <stikonas>well, at least I don't remember the reason <stikonas>maybe fossy or Googulator know something <lrvick>But okay, so I hardcode to 1, and switch to 2 as soon as I have bash and can run that one liner above to parse /sys/devices/system/cpu/possible <lrvick>like 99% of the time we are working on this we are building in containers or chroots with 20+ cores and waiting for half a day so this would be a major productivity win <aggi>apropos, SMP seems stable now too with linux2.4/tinycc, and this system can bootstrap full GNU wherever an i586-tcc-linux-musl.iso can boot <aggi>but i want to do extensive testing before, hence i'm glad i got a production filesystem with linux-2.4 which may not need any update soon <aggi>the devdrop ISO is 450MiB in size currently with the complete package set supported (can be easily reduced in size but i want to keep test-coverage for tinycc) <lrvick>now that I think about it, even if I did have JOBS > 1, until we have GCC, nothing can really take advantage of multiple cores anyway right? <lrvick>and we have coreutils 5.0 before we get gcc <lrvick>so could just switch JOBS to whatever nproc --all gives us after coreutils is built <aggi>well, tinycc is very fast, although it too benefits from parallel make <aggi>didn't include coreutils myself, and chose busybox plus extra utils (jfs, iptables whatever) <aggi>keeping in mind i rather avoided autotools with as many packages as possible, ideally a system profile can be tailored to bootstrap a complete system with development utilities without relying on autotools too early during boostrapping <aggi>i think it's even possible to avoid binutils, if i rewrite the bootcode assembly parts of linux-2.4 (it isn't many) <aggi>to assemble those with as86 instead <aggi>as86 itself may be easier to bootstrap too <stikonas>lrvick: GCC is single threaded, it's nothing to do with gcc <aggi>some piece is missing still, since i need some networking filesystem; linux-2.4 got some shfs patchset which i'll try tomorrow <aggi>the beauty of linux2.4 in this case, it can be fully bootstrapped with tinycc, and provide rather useful features early in the dependency chains <aggi>and tccboot is another decent approach to fill a gap with kernel bootstrapping (that might rely on the mentioned bootcode rewrite for as86) <aggi>the fiwix-kexec approach got an advantage over my approach still... it's ready and working, while i could not fully clarify yet, how much of the userspace stuff could be salvaged _before_ fiwix abi is required while bootstrapping <lrvick>stikonas: ah right, so actually it -would- make sense to have the total number of cores in a var before make is built in the early stages. Fair <lrvick>Okay so then calculating the number of cores dynamically with mes or tinycc before the first version of make is built might make sense. <lrvick>a tiny bit of c or scheme to parse /sys/devices/system/cpu/possible seems like the shortest path at that stage <lrvick>unless anyone else has better ideas <stikonas>hmm, maybe I'm wrong, maybe only --jobs work <lrvick>as best I can tell make has no built in way to detect cores on the fly <stikonas>it's configuration on invocation either with -j or -l <lrvick>Yeah okay, so shortest path sounds like writing cpunum binary in .c that can be built with M2Libc/tcc ? <stikonas>well, but steps between make and coreutils are fast <stikonas>you'll be just wasting your dev effor for little benefit <lrvick>Yeah, fair enough. What matters most is that we have it before gcc, which is expensive <lrvick>I mean the most expensive thing is mes, but it seems there is no helping that noe <stikonas>lrvick: you could speed up mes in another wya <stikonas>(which can be upstreamed to live-bootstrap too) <stikonas>create unified-libc file rather than building each little .c file one by one <lrvick>Yeah we bootstrap from hex0 to mescc-tools-extra in our "stage0" container, then we have our own Containerfile runner for live-bootstrap as stage1 <lrvick>but we can override anything in live-bootstrap easily now <stikonas>and so lots of time is spend loading mescc over and over again <lrvick>Okay so you are thinking there is no reason to not do that same hack for the mes build which might save us some signifiant time on that step <stikonas>well, there might be increase in memory usage <stikonas>hard to tell how much exactly without trying <lrvick>great call. left a comment to try that on my next pass <lrvick>will see how much JOBS=$(nproc) saves us <homo>hi, I have some progress trying to bootstrap microhs from hugs, currently compile-time errors are fixed, but runtime is broken, I hope someone can continue from here, because this is as far as I can go https://0.vern.cc/Mm.patch <lrvick>Got almost all of live-bootstrap to build with auto-detected JOBS=20 <lrvick>This setup fully builds live-bootstrap master, skipping all plumbing and scripts not relevant for containers, with caching for each layer to hack on any step https://termbin.com/iemv <lrvick>script-generator and configurator are bypassed <aggi>good to know a recent live-bootstrap is confirmed to succeed <lrvick>live-bootstrap dies almost immediately with no error w/ latest M2libc though <stikonas>not sure if it will affect x86 build though <lrvick>Error 02 trying to configure musl-1.1.24 <lrvick>but no actual output beyond that <lrvick>I am not using stage0-posix though. Just live-bootstap with no submodules, dropping in my own hex0-mescc-tools-extra toolchain (which was built with latest m2libc) <lrvick>and then I just checkout M2libc in place and skip straight to build steps <lrvick>Only thing I changed to get my build past the first musl pass was downgrade M2libc from 27ab5de4247e8b00459b2b21109109db8eb4d643 to 3a700010872697c4be9e3fab3cf707fce706741e <lrvick>that means no tar symlink support, but nothing in the live-bootstrap source actually needs it, so not a blocker, but probably needs some integration testing. <lrvick>With my updated containerfile approach it can build in under 2h, so a good candidate for free CI service <stikonas>strange M2libc should have no effect so late... <lrvick>well musl is built with tcc, so maybe m2libc impacted tcc in some way that broke the musl build <lrvick>I assume tcc must build with m2libc <lrvick>May try swapping it back and debug deeper once I have this PR landed/working/verified <stikonas>and rebuilt quite a few times during bootstrap <lrvick>FYI PJ on the stagex team was able to get 386 go 1.23 built against musl and tcc very early in the live-bootstrap flow <lrvick>That might unlock the removal of a lot of other steps <rkeene>I'll probably make some additional changes to tar, based on lrvick's note that it has problem untar'ing some files