<doras>I'm thinking about a plugin that fetches the live-bootstrap git repo, then parses a manifest which has a standard format and contains all the required files to be fetched and their checksums. Then those will be converted to metadata for BuildStream, which can then automatically do the caching and mirroring necessary to ensure this works reliably. <stikonas>some of the files are fetched from obscure places <stikonas>most of the stuff is from standard gnu ftp server/mirrors but not everything <doras>BuildStream also supports parallel fetching to ensure it maximizes the available bandwidth. <stikonas>e.g. it was quite hard to find flex 2.5.11 <doras>Oh, and maybe the destination directory too if different ones are needed. <doras>I wonder if we can use something more standard for all of this... <doras>One way which I've seen being used before is to host all of your dependencies in git repositories, and combine it with git submodules. <doras>Basically create a git mirror of sorts in GitHub or similar. <stikonas>well, live-bootstrap downloads everything into a single directory (sources/) <stikonas>and submodules are bad at having multiple versions <stikonas>so you have to get two copies of the same thing <stikonas>we did use submodules a bit more until oriansj added untar to stage0-posix <stikonas>but even then only until GNU tar was built <stikonas>and gnu tar is built very early in live-bootstrap <doras>I think `git submodule update --depth 1` would suffice, no? <stikonas>oh, I was proably thinking about some earlier times... <doras>Another option is to switch your repo to BuildStream, if you're interested. <stikonas>yes, new gcc and guile increased it a lot <stikonas>well, it's up to fossy, but I think we tried to keep depedencies minimal <doras>How do we know that everything we bootstrapped actually works fine? I guess it's not trivial to know this. <doras>I think this is mes-m2 building... something. <stikonas>hmm, these are warnings from mescc, janneke might tell you more about those <doras>Is there any chance that we'd end up with some "bad" binaries that are broken in slightly and unexpected ways? <stikonas>and then tcc is rebuilt 5 or 6 times with itself <stikonas>there are some issues with floats in early binaries that are built with tcc and mes C library <stikonas>those are gone after we rebuild tcc with musl twice <doras>Do we verify final binaries somehow? <doras>Also, mknod is not allowed either. <stikonas>also there are a couple of intermediate non-redistributable binaries during bootstrap but they are not present in the final image <doras>I'm basically running a chroot bootstrap locally without root. <stikonas>heirloom-yacc and heirloom-lex are CDDL licensed and it is incompatible with GPLv3 <doras>The first thing that failed was the tmpfs mount, the second one is the mknod. <doras>I can't really tell, but it seems to be part of bootstrap process itself (?) <doras>I think it's from sysb/run.sh, but I can't say for certain. <stikonas>well, it's probably from pupulate_device_nodes function <doras>It happens right after perl finishes building. <stikonas>there is some bug somewhere (bash?) and musl configure script creates an empty /dev/null file <stikonas>even if /dev/null was proper device node <oriansj>deblob-check is in the checksums but not downloaded <stikonas>we used deblob mode where it completely removes files with blobs <stikonas>rather than surgically removing the blogs <stikonas>and we don't need that much from the kernel anyway <fossy>I have no interest in switching to BuildStream <fossy>since really the primary target is metal/qemu <fossy>and ideally any preparation system can be trivially removed <fossy>however, conversely, I have no problem with BuildStream being an option <fossy>if people qanr to use that as a preparation system <fossy>basically I dont want a preparation system to be completely nessecary, should always be able to be built by hand <oriansj>doras: just a question, if the tars to download was just a json or s-expression list with download location and checksums; would that be something BuildStream could handle? <fossy>it should be trivial to externally download sources, however <fossy>stikonas: thoughts on shaing sources within live bootstrap? <fossy>oh, doras, those warnings are normal, they are just mescc compiler warnings (similar to GCC warnings) <stikonas>after sha256sum just run a single command <stikonas>oh, actually sources are put in individual subdirectories <doras>oriansj: we'll need it to write code that handles it, but it should be possible. <stikonas>fossy: or do you think just shain'g internally? <stikonas>if something is misdownloaded, you want to find asap <stikonas>both is fine but it would be nice if we could share checksum file <stikonas>maybe inside live-bootstrap we should just have /sources directory <stikonas>rather than putting all sources in subdirectories of each source package? <oriansj>fossy: my thought: a source.json file and the current SHA256SUMS.sources <oriansj>that way we have a standard file all build/setup tools could use and a single file making it trivial to verify that the sources have been downloaded separately in the after.kaem hook <stikonas>maybe then rootfs.py can read json (or maybe yml) file instead of it being hardcoded in .py files <fossy>yes, as long as the config file is human readable <oriansj>fossy: absolutely, without question. <doras>Another thing to consider is that our repo should be fully buildable in any of the architectures we support. We also have aarch64 and ppc64le CI runners in addition to x86_64 that constantly build it, potentially from scratch if nothing is cached (which is rare, but must be possible). This means that as far as I can tell, we can't make live-bootstrap an element of the main repo, as it can only produce an x86 system at the moment. Once <doras>that system is produced, we are expected to use a x86 or x86_64 hardware to cross-compile the other architectures. <doras>So I think we'd have to put the entire live-bootstrap build process it in a separate repo which would eventually build a Docker/OSI/OSTree image (I'll need to see what would work best) for each architecture, that we could then use in the main repo as a binary base to continue the bootstrap process. <doras>That separate repo would be buildable only in x86 or x86_64 hardware. <doras>Also, regarding /dev/null and similar, it seems that BuildStream can create those in its sandbox without any special permissions. <doras>I suspect it uses bubblewrap to expose a few /dev nodes from the host so that it doesn't have to create them on its own. It's not a full chroot. <doras>I'm trying to add an option to use a `bwrap` sandbox instead of a chroot. Since it relies upon user namespaces (or a SUID binary) it can mount tmpfs, device nodes and probably everything live-bootstrap requires without having to get explicit root permissions. <doras>This will still prove problematic when built using BuildStream, since nested sandboxes aren't going to work. <doras>So far it seems to build. I exposed the expected /dev nodes from the host's to the sandbox, so hopefully autotools won't complain. <doras>Also, "as" crashed during the build. It didn't stop the build though. <doras>It crashed with SIGSEGV when this command was executed: "/usr/bin/as --fatal-warnings -o conftest.o conftest.s" <doras>Got this now, much later than before: <doras>mknod: `/dev/null': Operation not permitted <doras>Seems to be after the chroot to sysc, so at least sysa was successful. <doras>That nested chroot is a bit tricky. Why do we do it? <stikonas[m]>Ideally file system state should be similar in qemu/chroot mode <doras>I tried creating the /dev nodes inside sysa/tmp/sysc/dev/* too. <doras>So now we have them in both sysa/tmp/dev/ and sysa/tmp/sysc/dev/ <doras>I started the build again, but unfortunately as you know I have to go through the sysa build again. <doras>I think the project may need some better structuring. There's a lot of copying, jumping around, execution of elevated commands at arbitrary places, and lack of clear sandbox-specific execution paths (QEMU, chroot, etc.) <doras>sysc has a tmp directory but it's actually build under sysa's tmp directory is also a bit strange. <stikonas[m]>Well, we had no downstream users yet... It's hard to adjust when you don't know requirements <fossy>stikonas[m]: remind me the original reason for why we did chroot_transition under sysa for chroot, was it just consistency? <doras>You didn't want to user to enter a password again mid-build. <stikonas[m]>Else you just need rootfs.py to copy things and start new chroot... <fossy>w/r/t structuring, i generally agree <fossy>but any restructing needs to keep in mind that the first-class platform is bare metal, possible without rootfs.py <stikonas[m]>Password mid-build means you can't leave it running unsupervised <fossy>because, at the end of the day, we are trying to eliminate seeds <doras>But it's only because sudo is used in the first place, and also for the build itself which is very bad in general. <fossy>that is ~not possible to eliminate <fossy>there is no concept of users or groups for quite some time in the build <fossy>any tool eliminating root access would have to be external <doras>I'm using "unshare ---map-root-user" instead of fakeroot, since it's much more standard and simple. <doras>And yeah, I don't actually need root to build anything as far as I can tell. <fossy>yeah, i wouldn't say fakeroot is the best for this application, but i see what you're saying <doras>sysa completed successfully without any root permissions, and I'm assuming sysc can too. <fossy>doras: no build per se requires it but mknod, etc <fossy>well, i am assuming that if you aren't wanting to use root, you are not working with filesystems <doras>mknod and mount is not required either if you use user namespaces. <fossy>so really all this requires is a bit of logic within live-bootstrap and a couple new options to rootfs.py <stikonas[m]>Well, let's see if sysc builds, then maybe you can create a patch <fossy>doras: can you elaborate on "lack of clear sandbox-specific execution paths" <doras>As I mentioned, I'm currently adding a bwrap (https://github.com/containers/bubblewrap) mode as an alternative to the existing chroot mode, since relies upon user namespaces (or SUID as fallback) behind the scenes for mounting /dev nodes, tmpfs the initial chroot, etc. <doras>So it allows me to build without root privileges at all, but it also reveals that we're not preparing things ahead of time enough for the sandbox-specific execution paths. <fossy>one of the design goals of live-bootstrap is that as little external preperation is completed as possible, so i like to avoid too much preparing things ahead of time <doras>For example, the /dev nodes. They are created on-the-fly during bootstrap at the moment. <doras>This is something we can prepare before entering bootstrap. <fossy>we can but we shouldn't, as little as possible should be done outside of the bootstrap. if it's needed for a specific usecase we can add that, but it shouldn't be a default <doras>I'm guessing that I'll find the same about sysc's tmpfs, but I haven't gotten there yet. <stikonas[m]>We can add an option to create them before if you want to run rootless <stikonas[m]>Adding some optional paths is easy once we have bash <fossy>it's odd that mknod requires root tbh though <stikonas[m]>Actually these days even before bash, kaem supports "if" too <doras>It requires the CAP_MKNOD capability, yeah. <doras>Which means it requires elevated permissions. <doras>Gah, I got "mount: permission denied" <doras>I guess I got to the on-the-fly creation of sysc's tmpfs <fossy>(note that, on-the-fly creation is required, whether within or outside of isolation mechanism, as binaries from sysa need to be transferred to sysc) <fossy>tl;dr is that chroot/bwrap/etc don't use sysb, as sysb prepares a *real disk* <fossy>you want to go with chroot as a starting point <doras>I did set CHROOT to True, yes. <stikonas[m]>Well run.sh should choose the path based on CHROOT env variable <fossy>depends on how CHROOT was set to true <fossy>doras: did you just set that in the host environment, or some other way <fossy>oh, so you got to sysc without problems? <doras>With many solved problems :) <fossy>well, problems with the transition :P <fossy>they are required for something (don't remember what) <doras>chroot is allowed in the bwrap sandbox. <fossy>but they can deffo be externally put in <doras>I just ensured that we had /dev nodes prepared for the eventual chroot ahead of time. <fossy>yeah, chroot should be as close of a mirror to qemu as possible IMO <stikonas>hopefully the whole thing is not hard to adjust <stikonas>once we have bash, scripting is quite easy there <fossy>shouldn't be, bootstrap.cfg makes this pretty simple <fossy>well, nothing rooted is required pre-bash <stikonas>first time device nodes (/dev/null) is used is for musl's configure script <stikonas>although like I said before something is broken there <stikonas>and /dev/null node gets replaces by a file during configure run <fossy>configure script has some incompatibility i think <fossy>hmm, no i think it is simpler <fossy>we dont have /dev/null node afaict <stikonas>no, even if we have /dev/null it still does that <stikonas>(in subsequent musl rebuilds after we have /dev/null) <stikonas>but maybe we still used that old bash then <stikonas>hence I was blaming bash built with mes libc <doras>By the way, the current /dev we create is a normal directory, while it should usually be devtmpfs. <doras>I'm not sure what can affect, but just FYI. <doras>Anyway, one way to make the entire preparation part more consistent is to always do it using bash. <doras>So both the sysa and sysc preparation is similar in syntax and reuses the same logic. ***ChanServ sets mode: +o janneke
<doras>Is there a trick I can do to start building sysc directly when I already have sysa built? <doras>I think I need a few iterations here and it's getting annoying rebuilding everything each time. <stikonas>doras: well, you can probably copy sysc somewhere else after preparations... <stikonas>but it's not easy to recreate rest of the environment... <stikonas>migth be easier to stop the build before it fails (comment out failing code and anything later) <stikonas>that's what I used to do for development <stikonas>although, you'll have to manually copy in static build of busybox <stikonas>the first bash that we build is non-interactive <stikonas>only bash 5.1 (currently step 75) is built with readline <doras>So the interactive one is built only as part of sysc? <doras>I'll try a trick, maybe it would work. <doras>Huh. The sysa "ls" seems very minimal. <doras>Or maybe I'm not doing something correctly. <doras>Or maybe I did something wrong. <stikonas>but the very first bash is built with mes libc <stikonas>e.g. we don't have bashrc to setup --colour alias or something like that <stikonas>I'll be away from computer for some time but I'll read the logs later <doras>stikonas: it's actually surprisingly easy to run sysc's build once sysa finishes. <doras>This command seems to work for me: `unshare --map-root-user ./build_sysc.sh -- chroot /sysc /init` <doras>"./build_sysc.sh" is a confusing name, it actually only executes bwrap with the right things set up. Basically the exact same command I added to rootfs.py. <doras>Now it's called "./bwrap-exec.sh" :) <doras>And I tested bwrap inside BuildStream, and it seems to be allowed. At least my basic test was fine. <doras>So maybe we could use the bwrap-based mode in BuildStream after all. <doras>I'm part-way building sysc with the command above. Everything seems fine so far. <doras>Alright, so how do we get from gcc 4.7 to 11.*? :) <oriansj>basic #include functionality is now in M2-Mesoplanet <oriansj>there is a minor bug in cc_macro.c I haven't sorted out yet, stikonas could you take a look to see what I am missing <doras>So as far as I can tell, the bash I'm thrown at when bootstrapping completes is running in a sysroot that includes both the products of sysa and the products of sysc. Is this intentional? <doras>It also includes all the sources used by sysc under /usr/src. <doras>I think a good chunk of the size is basically not something we should be keeping around as it serves no purpose. <doras>things we shouldn't keep around* <doras>Also, it seems we may have some confusion around the library prefix. A few executables were installed under /usr/i386-unknown-linux-gnu. <doras>There's also a mysterious /usr/bin/bin/ symlink which points at the relative usr/bin which doesn't exist. <doras>Hmmm... and everything is statically linked? <doras>And there's no dynamic linker? <oriansj>doras: well if you define a subset list of outputs, it should be rather straight forward process to reduce the output to that set. (literally just a simple copy to the target locations) <doras>oriansj: admittedly I'm less familiar with musl-based systems, so I'm not entirely sure what should be kept. <doras>But I agree, it's fairly simple to filter out only the output one needs. <oriansj>doras: perhaps a better idea is to just build exactly you want as a base from those tools available <oriansj>as then you could use any libc you desire and have only exactly the tools you want in the final output <doras>oriansj: I think we would want to do that anyway, but we'll also need to keep sysc in image form (Docker/OCI/OSTree) because it doesn't support aarch64, ppc64le and risc-v bootstrap yet, so it can't be part of the main repo. <doras>So it would be best to have this image as small as possible. <oriansj>well the image doesn't have to be the fully built result if one is worried about size. <oriansj>rather just the sources and the seeds <oriansj>hmm actually we end up having to build a great deal of sources <oriansj>so a hook script to strip things down probably is what you want. <doras>oriansj: can you point me at an existing hook script? <doras>With bubblewrap installed, `./rootfs.py --bwrap` is enough to bootstrap rootless and without elevated permissions. <doras>It still requires some documentation updates, better commit descriptions and a decision on how to handle the tmpfs mounts. <doras>I'm still not familiar enough with the project to tell what each temporary directory is meant to hold, so if anyone wants to work on the last part I would appreciate it. <doras>I think bwrap's own tmpfs mounts are automatically unmounted when the command it executes exit, so maybe in that case we never need unmounting too. <doras>But yeah, definitely not familiar enough to be certain of any specific approach. <doras>Actually, I may need one more change around environment variables to ensure it works on every system. <doras>sysa's /bin not being a symlink to /usr/bin is a bit strange, by the way. It's a normal directory that contains 3 symlinks for bash, pwd and sh. <doras>I'm not sure if it's intentional, but it's... unusual. <doras>So here's my bootstrapping challenge for you: I want to use live-bootstrap to create a base system, but live-bootstrap depends on a system with Python and bwrap (with my new mode). However, that would be bootstrapping using an existing system, which means I need a foreign image. <doras>This is true for every mode, even for bare metal, right? <doras>For bare metal we basically need live-bootstrap to produce an initramfs. <doras>So every build mode of live-bootstrap still requires a foreign system to act as an initial environment for running the Python logic. <doras>After the Python scripts do their thing, we try to sandbox stuff in a way that convinces us enough that no foreign image is involved in the bootstrap process, but evident ally it did at some point. <doras>All of this may sound obvious to you, but now my challenge remains that I need a system with Python, bwrap and a working shell to build live-bootstrap, which kind of defeats the purpose of having to pull no foreign systems at all to get sysc built. <doras>So my thoughts are as follows: the bootstrap part of live-bootstrap doesn't actually need Python, bwrap or a shell. All it needs is a kernel to execute the initial seed within a chroot. <doras>So if I got this right, if I create all of the directories, download all the sources and put everything where the seed and bootstrap steps expect things to be, and then chroot and execute the seed, it should, in theory, bootstrap successfully. In other words, if I skip all the Python stuff, my dependencies are reduced to a kernel and a chroot. <oriansj>and the chroot isn't even technically needed <oriansj>as kaem-optional-seed is the init and the driver that starts the whole bootstrap process <doras>Well, I'm assuming we require an isolated environment at some point so build products can be put in directories and all the /dev and /proc expectations are met. <doras>So a chroot is one tool to provide that initial requirement. <oriansj>doras: we can just use a blank environment <doras>But anyway, a kernel is something I have regardless of where our repo would be built, and BuildStream should in theory provide me with a chroot without any external image dependency because it runs the chroot from using host tools. <doras>How do you achieve a blank environment without chroot on a running system? <oriansj>qemu or just do mkfs on a disk and boot that disk <doras>QEMU suggests a foreign image again because it's a non-trivial requirement. <doras>chroot requires only a kernel as far as I know. <doras>mkfs and boot means a reboot, which is a no-go for CI. <oriansj>QEMU isn't the requirement, just a way to approximate the formated disk booting in a blank physical system <doras>But anyway, if a reboot is out of the question, the most minimal requirements we can reach is a kernel and a chroot. <doras>And those can in theory be provided by BuildStream without any external dependencies. <doras>But... it also means that I have to basically skip most of live-bootstrap. <doras>So my point is: I will try to do this, but if it works, it seems to me like the scope of live-bootstrap is a bit too wide for our use case. <oriansj>well a blank docker image shouldn't even need a chroot <doras>Docker uses a chroot too, I think. <oriansj>but it shouldn't also need one internally <doras>Well, currently sysa does use the chroot command explicitly to switch to sysc. <doras>But the initial requirements do not require it, correct. Any means by which you could get a chroot-like environment are good enough. It's just that chroot provides a chroot-like environment :) <doras>I will need to be able to avoid that second chroot through, I think. <doras>Because I can't provide /dev, /proc, etc. arbitrarily with BuildStream. They are only provided at the base of the chroot. <oriansj>at its core live-bootstrap is a chain started with kaem.x86 that builds up to the after.kaem which live-bootstrap replaces to start the chain until completion. <oriansj>The only binary dependencies are the kernel and those in bootstrap-seeds