IRC channel logs
2022-05-21.log
back to list of logs
<doras>freedesktop-sdk would still have the rust binary seeds unfortunately, but it would be a huge step towards becoming fully bootstrappable. <doras>We used to have a go binary seed until recently, but that was removed by building go using gcc. <doras>stikonas: do you have an example of the conditional block syntax in kaem? <stikonas>but yes, if and else should be sufficient <doras>stikonas: it seems everything works now. I'll run a few bootstraps to make sure, including one where I update mes and expect checksums to change, and then open a PR. <doras>28 files changed, 311 insertions(+), 130 deletions(-) <doras>Hagfish: good question. No idea :) <doras>But it sounds like GCC 12 (+ some patches) is needed, which live-bootstrap can't build at the moment. <stikonas>Hagfish: I don't think rustc_codegen can bootstrap rust <stikonas>I haven't looked in detail, but I would guess that rustc_codegen_gcc replaces llvm backend <stikonas>Hagfish: and mrustc does not work on x86 <Hagfish>i did find this intriguing comment in a readme though: <Hagfish>"Note: llvm13 is needed until there is a release of gcc12.1. As soon as gcc12.1 is in the freedesktop sdk, gcc can be used instead of clang." <stikonas>so at the moment live-bootstrap can't bootstrap rust <Hagfish>"Fails to build under x86 (32bit): Excessive memory usage" <Hagfish>interesting that it's a memory issue <stikonas>indeed, 32-bit systems are limited to 3 GB <Hagfish>hmm, i knew there was a limit, but i never would have guessed that gcc would be enough to hit it <stikonas>compilers are one of the most memory hungry programs <Hagfish>but yeah, putting the complexity at build time rather than runtime does make sense <stikonas>it's those optimization passes that are complicated <Hagfish>having to use gigabytes of memory just to compile "hello world" is hard to imagine, but presumably most of gcc isn't used for trivial code <stikonas>single-pass stuff (like what M2-Planet does) without optimizations is one thing <stikonas>it's to compile C files whose size is e.g. 100MiB <Hagfish>yeah, using gigabytes for that is way more reasonable <Hagfish>are there nasty polynomial scaling problems that compilers encounter? <stikonas>but it's not just mrustc, try to compile e.g. any of that chromium/webengine stuff <Hagfish>like if you double the source code, it quadruples the memory? <Hagfish>hmm, i guess there's a tradeoff of compile time and runtime optimisation that you can configure <Hagfish>but there's no simple flag for splitting up a 100 MiB file into smaller ones, i'm guessing <Hagfish>the only other tradeoff i can think of is RAM vs disk, but presumably that's not a way to work around the 32-bit limitation for gcc <Hagfish>the problem is address space, not physical availability <muurkha>oriansj: yeah, Pascal has less primitive types than C, but its typing discipline is significantly stricter <Hagfish>bootstrapping really does expose all the dirty secrets of big important projects <Hagfish>(another commenter says "Chrome stopped being able to compile on 32 bit systems years ago, as a C++ programs. I think Firefox is in the same situations.") <stikonas>not really a secret... Those big projects are well known to be memory hungry <muurkha>I'm used to "memory hungry" meaning a process needs over 128 MiB by itself, and this is another 16 times worse <stikonas>muurkha: try running source based distro <muurkha>when Linux moved from a 2:2 virtual address space split to 3:1, it was motivated by supercomputer applications, weather forecasting and nuclear explosion simulations and things, which can suck up as much compute as you can throw at them <muurkha>but I guess unconstrained resource use grows unconstrainedly <pabs3>OOM is one thing, worse is when the compiler needs more memory than the available virtual address space. IIRC Debian hits that a lot on 32-bit, especially on MIPS IIRC <pabs3>I think I read an LWN article about getting rid of that artificial split btw <muurkha>wouldn't be surprising but most of the possible gains were obtained with the move from 2:2 to 3:1 <muurkha>even if you can reserve the entire memory space for userspace (say by switching mappings on kernel entry) you can never get more than another 33% increase <oriansj>there are very *GOOD* reason why x86 is generally only supported these days with PAE <oriansj>but yeah, in this day and age, 32bit is dead for the Desktop. You can build everything essential just fine on 32bit but once you go video or web browser, just forget it. <oriansj>my x200 is maxed out at 8GB of RAM but if I don't give it another 16GB of swap, there is no way to build rust to the level required to build mpv <oriansj>I just wish the 65% keyboard form factor was available for the x200 <fossy>stikonas[m]: yeah i'm doing python after my disk/curl PR, which should be done this weekend assuming e2fsprogs cooperates <midgardian[m]><janneke> "goes to try an aarch64/arm..." <- I am curious about this, do you have bare metal hardware for these arches? <oriansj>midgardian[m]: well ARM chips are pretty cheap and available but most likely he is going to be doing the work ontop of Linux on said chips. <oriansj>possibly GNU Hurd if they finally got that port working <oriansj>and if for some reason real hardware isn't available QEMU usually does a pretty good job <oriansj>although sometimes there are important differences between real machines and QEMU behavior (like real hardware doesn't zero registers on exec *stares at RISC-V* for stupid reasons) <midgardian[m]><doras> "freedesktop-sdk would still have..." <- What exactly is bootstrapping freedesktop-sdk useful for, the toolchain used to build binaries on Flathub? <oriansj>well bootstrapping Flatpack binaries seems the obvious answer <midgardian[m]><muurkha> "when Linux moved from a 2:2..." <- Where can I read about this 2:2 vs. 3:1 virtual address space dichotomy, I am uninformed... <midgardian[m]><oriansj> "I just wish the 65% keyboard..." <- What does it mean "65% keyboard"? <oriansj>midgardian[m]: 65% keyboard is a keyboard with 67 or 68 keys <midgardian[m]><oriansj> "possibly GNU Hurd if they..." <- I only ever heard of i686 Hurd, there is work on armhf/aarch64? <midgardian[m]><oriansj> "well bootstrapping Flatpack..." <- Wouldn't using something like Guix to build them be potentially more secure? <oriansj>midgardian[m]: all x200 computers have 94 keys <oriansj>midgardian[m]: yes and so? if someone wants to bootstrap something here that is fine; no judgement if they are doing the work. <oriansj>people are encouraged to bootstrap anything they *feel* should be bootstrapped. That doesn't mean other people will agree with it, nor does it mean that other people will use your work. Only that we encourage progress on the shared goal of bootstrapping the world. <muurkha>oriansj: my girlfriend's 32-bit Raspberry Pi 400 seems to do okay with video and web browsers most of the time <oriansj>muurkha: well given 4 CPU cores and the Broadcom VideoCore VI which supports native 4K; not surprising. But I was speaking more of the requirements for building said software rather than just compiling it. <muurkha>yeah, I don't think she can compile Chromium or Firefox on it :( <oriansj>midgardian[m]: that being said, please note you are free to point out why doing X is superior to doing Y and argue your case but the ultimate decision is made by the people doing the work. Because "Who does decides" and the best way to make sure the things you want done are done in the way you want them done is to do the work. <oriansj>muurkha: or RUST (which rules out things like MPV) <muurkha>ultimately if X and Y are means for doing the real objective Z then ultimately you don't know whether X or Y is better until you've done both <oriansj>muurkha: or if the answer is obvious upon inspection. <muurkha>sometimes obvious things are wrong ;) <muurkha>midgardian[m]: might be hard to find info these days, it's been 20 years or so <oriansj>and sometimes obvious things are so obviously the right solution, the other solutions seem like a huge waste of time. <muurkha>yeah. and often they actually are; you can find lots of examples in the literature of the 60s and 70s <muurkha>I think you can *run* rustc on a 32-bit machine? <midgardian[m]><oriansj> "midgardian: that being said..." <- Okay this is fine, I did not make a statement about superiority though, I asked a question. <oriansj>midgardian[m]: thank you for clarifying <midgardian[m]><oriansj> "muurkha: well given 4 CPU..." <- What is the difference between building the software and "just compiling" it? This is worded as if they are different <oriansj>midgardian[m]: not building and compiling; running and compiling <muurkha>midgardian[m]: I don't think rustc needs 3 GiB of address space to compile and link hello.rs <midgardian[m]>oriansj This is what your message said : "But I was speaking more of the requirements for building said software rather than just compiling it." <oriansj>midgardian[m]: as in to build the rust compiler vs running the rust compiler to build rust programs <midgardian[m]>Ohhh okay so "But I was speaking more of the requirements for building the Rust compiler rather than just using it to compile Rust code." <oriansj>midgardian[m]: yes that could be a way of better phrasing my statement <doras>midgardian: among other things, yes. Beyond its use in Flatpak, it's also used as the base of GNOME OS, has Docker images publicly available and some additional use cases. <doras>freedesktop-sdk currently uses its previous stable version as the bootstrap seed of its current version. We've actually had cases where we lost the previous image due to hardware and software failures. We managed to salvage it using local backups that a maintainer happened to have had, but it's hardly a great backup plan for a project at this scale. <doras>In one case we had to rebuild a few versions until we reached the latest one, because each depends on the previous. <doras>Becoming bootstrappable would allow it to fully bootstrap every version even if we lose the image, as long as all the bootstrap seeds and sources are properly backed up. <doras>Aside from the practical reasons, it's also a fun challenge :) *midgardian[m] grins evilly <midgardian[m]>Actually they don't even make a way to download an archive pre-2000 I can find.... <oriansj>midgardian[m]: they only enabled single downloads <doras>stikonas: I'll have a few conflicts with this <stikonas>I thought it's quite small and not likely to conflict :( <doras>I'm running a final bootstrap to verify everything works. <doras>I feel that our mkdir calls, transfer between sysa/sysb/sysc and directory structure in general is done in a very freestyle way that makes it a bit hard to extend without creating more mess. <doras>It improved a bit with your last MR, stikonas @stikonas:libera.chat, but it's still not great. <doras>We also have many environment variables set all over that either don't have to be exported at all, or are simply being inherited from kaem that exports everything. Some are set but are actually unused. <doras>So while I tried to keep my changes to the minimum, I feel like I made more mess. <bauen1>it's also a bit hard to reenter the sysc chroot with the correct variables exported because of that <stikonas>bauen1: last point can probably be fixed by writing out .bashrc <stikonas>but yes, with packaging stuff we have more env variables now <stikonas>doras: well, some things can definitely be simplified <stikonas>not much we can do about kaem exporting variables right now <stikonas>and if you want, you can create clean environment once you build coreutils and env <doras>stikonas: I was thinking of clearing out the environment variables at different stages. <doras>But haven't had a chance to experiment with this yet. <doras>So maybe we could do the same when running bash for the first time in sysa. <doras>I'm guessing kexec cleans the environment as well, considering a new kernel and initramfs are loaded. <stikonas[m]>doras: bash is built before env if I remember correctly <oriansj>well kaem with --init-mode will clear out any and all environmental variables if you need that fresh shell feeling in your bootstrap. <stikonas>though one can do osme tricks to achieve that <stikonas>i.e. once bash is built, save wanted env variables to .bashrc, run kaem with --init-mode that runs another bash <doras>I'll upload the MR updating mes once its bootstrap completes. <stikonas>doras: I spotted a couple of unrelated documentation issues. Do you want me to merge it and for me to fix them? Or can you fix them <doras>stikonas: however you prefer. <doras>I'm less likely to spot documentation issues being less familiar with the project :) <doras>I only updated the parts that changed with my PR. <doras>stikonas: I think we could improve upon my checksum changes in the future to reach a point that we can use "cp -r" from a single directory in sysc and into the git worktree to overwrite all checksums in live-bootstrap. <stikonas>and only very minor merge conflict in my other PR <doras>I think it would be nice to also split the large sha256 into smaller ones per-package at that point. <doras>Then we wouldn't need to grep it <oriansj>doras: actually you are more likely to spot documentation issues related to user performed steps because you are less familiar with the project. <doras>oriansj: in theory, this can be true. However, I'm not usually the type to read documentation when source code is available :) <oriansj>aka the documents say do a b d but forgot to include step c and here is the error one would see because they missed that step in the documentation. <oriansj>but anything that was a gotcha for you should be documented. <stikonas>hmm, maybe I'll move checksums out of distfiles into sysc*/ directory <doras>stikonas: probably a good idea. <doras>And with that, rootless bootstrap works again. At least for me, locally. <doras>The last open issue is to figure out what to do with the fact that we can't mount the tmp and sysc/tmp directories as tmpfs without root. Or actually can mount them as tmpfs using mount namespaces, but then the host can't access them. <stikonas>doras: can you not use normal directories without tmpfs for rootless? <doras>I could, and I'm doing it locally. <stikonas>doras: actually, why do we even have to set chmod there? <stikonas>but yes, +x wouldn't work on chmod from stage0-posix <stikonas>that one only understands octal permissions <doras>stikonas: it may work, I can try it. Though it seems like a good practice to set proper permissions for scripts that we actually execute instead of relying on catm's default. <doras>Which I think is u+r only, or something like this. <stikonas>well, we can set it but executable permission is not necessary for scripts that are not run as ./ <stikonas>but if you think it's better to set them, then ok... <doras>I sometimes chroot into the sysa's or sysc's sysroots for testing purposes, or to resume a build that failed. It may be useful to execute run2.kaem without having to change its permissions. 