<bauen1>approximately how much free space do i need for a run of live-bootstrap at this point ? <bauen1>hm, i never actually made a mr with my addition for using unshare i think <bauen1>stikonas: alright, guess in the worst case my laptop will just refuse to let me login tomorrow <stikonas>tcc -I . -DHAVE_DECL_GETENV -DHAVE_DECL_MALLOC -DHAVE_DIRENT_H -Dsize_t="unsigned long" -DHAVE_LIMITS_H -DHAVE_GETEUID -DHAVE_MKTEMP -DPACKAGE_BUGREPORT= -Ded_PROGRAM=\"/nullop\" -Dmbstate_t=void -DRETSIGTYPE=int -DHAVE_MKDIR -DHAVE_RMDIR -DHAVE_FCNTL_H -DPACKAGE_NAME=\"patch\" -DPACKAGE_VERSION=\"2.5.9\" -DHAVE_MALLOC -DHAVE_REALLOC -DSTDC_HEADERS -DHAVE_STRING_H -DHAVE_STDLIB_H -c -o error.o error.c <stikonas>hmm, I think something from CPPFLAGS is causing it <fossy>make shells out to /bin/sh for some substitutions oddly <stikonas>now I've got patch building using makefile rather than kaem script <unmatched-paren>ok, last-minute change of plan: if i can get https://harelang.org packaged in Guix, i'll use that for my compiler instead of C (they went public today, the website no longer tells you to keep it a secret) <unmatched-paren>the lexer is kind of half-done, but it's small and should be easy to port to hare <bauen1>can somebody upload their `gawk-3.0.4_0.tar.gz` with correct checksums somewhere so I can compare why my (unshare) chroot build has a different result ? <bauen1>11:18 ^D8/<^Dg ^Dgmid-kid^Dg^D8/>^Dg ^Deyeah it's the file order that messes up the checksum for gawk-3.0.4_0 <bauen1>looks like me not using a tmpfs is the issue here <stikonas[m]>yeah, we don't have any sorting early in the bootstrap, so probably get sorting from fs <stikonas>in principle this should be fixed and we shouldn't rely on order from kernel <stikonas>hmm, gawk is actually built after coreutils and sort <stikonas>bauen1: maybe try patching tarball creation to use sort if it's available <stikonas>hmm, it's not particularly clear how to do that though <bauen1>stikonas: yes there was some talk about that here already <bauen1>but we are building coreutils just before gawk ... <stikonas>yes, but we then need to somehow wire in sort into tar <bauen1>oh we're not piping filenames to tar ... <bauen1>then patching tar is probably easier <stikonas>somehow wrong automake version is picked <bauen1>actually there's a bash somewhere segfaulting 6 times around the time the configure script was probably running, so probably missing some unshare setup to make bash run properly <bauen1>hm, does bash-2.05b even support interactive usage or does the `sysa/bash-2.05b/patches/dev-tty.patch` make that impossible ? <bauen1>and bash is stripped so figuring out where / why it segfaults is a bit harder <bauen1>sysa/tmp/usr/bin/automake is indeed version 1.4p6 <bauen1>but automake-1.6 is also version 1.4p6 that looks very wrong <bauen1>'run.sh: line 340: [: x: integer expression expected', may have something to do with it i guess, time for another run <bauen1>i think the syntax error i made caused (a few) bash subshells to segfault and messed up the unpacking or something like that <stikonas>yeah, that first bash is a bit crashy and quite a few things don't work (e.g. traps) <bauen1>well, bash is still segfaulting and automake-1.6 is still version 1.4-p6 ... <bauen1>hm, so the automake <-> automake-1.6 are actually made by creating a hard link it seems: `ln /tmp/destdir/usr/bin/automake /tmp/destdir/usr/bin/automake-1.6` <bauen1>If i had to guess that (probably combined with not using a tmpfs) is part of the reasn why I end up with all 2 versions (or 3 files) with the same content: <bauen1>12e46b4fba59637e2f46e040fa9268b5a7798a2135338090d37a8e117bd72e0c /usr/bin/automake <bauen1>12e46b4fba59637e2f46e040fa9268b5a7798a2135338090d37a8e117bd72e0c /usr/bin/automake-1.6 <bauen1>12e46b4fba59637e2f46e040fa9268b5a7798a2135338090d37a8e117bd72e0c /usr/bin/automake-1.4 <bauen1>yup, the tar extraction just doesn't care about hardlinks, and seems to just override /usr/bin/automake (or one of the other versions) changing every other version <unmatched-paren>is there any reason we couldn't use dash instead of bash in live-bootstrap? (or maybe even mrsh?) <bauen1>unmatched-paren: at the point wher we need the shell we only have tcc ***unmatche1-paren is now known as unmatched-paren
<stikonas>I think fossy and I looked at dash, it was not easy to build <stikonas>unmatched-paren: keep in mind that we need to be able to build shell without any support for interactive features <stikonas>if mrsh can be used to run musl's configure script that might also fix yacc licensing issues <stikonas>fossy: just tried to run live-bootstrap without rootfs.py but with chroot <stikonas>so I 1. coppied stage0-posix into new directory <stikonas>2. on top of stage0-posix I added files from live-bootstrap <stikonas>3. I had to move sysa/after.kaem to ./after.kaem <stikonas>and then 4. run "sudo chroot . ./bootstrap-seeds/POSIX/x86/kaem-optional-seed" <stikonas>we can eliminate step number 3 if we move that file in the repo <unmatched-paren>stikonas: even if mrsh's default frontend doesn't allow you to disable interactive mode, it's mostly just a library <stikonas>and hope that we don't hit any mes libc stsubs <unmatched-paren>stikonas: promising: mrsh's ./configure allows a --without-readline option <stikonas>yeah, that would lower requirements on libc <stikonas>first we can try to build with configure <unmatched-paren>ah, looks like if you use --without-readline it uses a simplistic getline() loop <stikonas>bauen1: when building without tmpfs I also get gawk checksum error <bauen1>stikonas: and if you ignore checksums, do you also get the automake version 1.6.3 vs 1.4-p6 conflict ? <stikonas>yeah, checksums seem to be due to different order in tar <stikonas>so I think I agree that patching tar makes most sense <stikonas>basically we need to backport --sort name optoin <stikonas>or rebuild newer tar with custom makefile once we have musl <stikonas>hmm, that requries fairly new tar version 1.28 <samplet>After some morning hacking, it looks like MesCC is running on Mes with a proper module system. It’s compiled about 50 files so far, but is going very slowly. We’ll have to wait and see if the compiled code actually works (I’m hopeful). <samplet>Then, on to porting Gash – and some major cleanup of the changes I made to Mes. :) <bauen1>stikonas: does the log indicate that /usr/bin/automake is also hardlinked ? <stikonas>samplet: nice progress! And we can worry about speed later (if ever) <bauen1>stikonas: grep for `ln /tmp/destdir/usr/bin/automake /tmp/destdir/usr/bin/automake-1.6` <bauen1>having a faster mes-m2 would be quite nice, it feels like that's half the time of live-bootstrap ... <stikonas>-rwxr-xr-x 3 root root 195K 2022-04-24 19:09 automake-1.4 <stikonas>-rwxr-xr-x 3 root root 195K 2022-04-24 19:09 automake-1.6 <stikonas>bauen1: I think that slow down is even more than what we currently have <bauen1>interesting, so it does and it's not some configure script messing around <samplet>stikonas: Thanks! The new version has to call into Scheme to lookup every variable. The old version of Guile (1.8) has a fast path for the default case written in C. That’s always an option for Mes too (as a later improvement). <stikonas>bauen1: perhaps for additional debugging you can remove code that cleans up build directories <stikonas>then check what actually happens in automake-1.6 build tree <bauen1>stikonas: the problem seems to be that tar isn't removing the hardlinked file, it seems to literally overwrite it, causing all other versions to be changed too instead of just automake-1.6 <stikonas>as for gash'es speed, we might not need it that much. Especially if faster shell can be built soonish (bash or even mrsh) <bauen1>stikonas: what i did was run `/usr/bin/unshare --pid --mount-proc --fork --map-root-user --mount --net --uts --ipc --cgroup ./unshare_setup.sh "$PWD/sysa/tmp" "/usr/bin/tar" "-v" "-C" "/" "-xzpf" "/usr/src/repo/automake-1.6.3_2.tar.gz"` and confirmed that all 3 files are still hardlinked, which isn't what i would expect <stikonas>well, my steps are just run with chroot but without tmpfs <bauen1>stikonas: so it's the tmpfs messing with how tar extracts files then ? <stikonas>I'll paste diffoscope output to that bug <stikonas>or maybe not having tmpfs is messing up with the order <bauen1>how would the order affect tar to seemingly not unlink a file ? <bauen1>hm, tar has a `--unlink-first` option, perhaps that will help <bauen1>looking at tar-1.12/src/create.c there is some logic to find out if a file has multiple links <bauen1>running a build with `--unlink-first` added now ***Noisytoot is now known as [
<bauen1>stikonas: seems to proceed past autoconf-2.55 now <stikonas>bauen1: please do, I'll review and merge it <bauen1>still need to fix some things up, just letting ci run once <stikonas>bauen1: and please rebase on top of master <bauen1>stikonas: i got to make-3.82, after that it fails for me because of the chroot (and probably mount trying to do some things it's prohibited from) <bauen1>so i only got to the end of sysa <stikonas>bauen1: well, try running ./rootfs.py -c <bauen1>but yes, i guess that is a problem <stikonas>if that is fixed, then we can merge MR and then see what make does <bauen1>stikonas: that's caused by my own changes <bauen1>stikonas: i still don't have sudo permissions on my laptop :) <stikonas>anyway, if you fixed make-3.82, I think you'll hit an error when applying tar-1.34 archive <bauen1>stikonas: make succeeds, the run.sh of sysc then fails due to trying to do root things <stikonas>oh yes, switching to sysc does some root things <bauen1>is /sys actually required, or can i just not mount it and don't care if i'm not doing disk things ? <stikonas>I guess 'exec chroot "${SYSC}" /init' fails <bauen1>stikonas: no the `mount -t sysfs sysfs /sys` fails due to insufficient permissions <bauen1>stikonas: and populate_device_nodes won't work, i'll make a few hacks and try again <bauen1>stikonas: yes, if you do `unshare --mount-proc --pid` it should work <stikonas>well, try to remove /sys and see if it helps <stikonas>then I can double check that qemu mode is not broken <bauen1>stikonas: I'll have to make some adjustments to rootfs.py and the helper scripts so I can use unshare, I'll see if I can make a MR of that in the near future <bauen1>as in, if i can get it cleaned up enough that it becomes possible <stikonas>yeah, getting rid of root requirement would be nice <samplet>\o/ All Mes (and MesCC) tests are passing with the new module system Now time to clean up the changes. <bauen1>stikonas: if you still want to use a tmpfs + unshare you would need to unshare before setting up $gitdir/tmp (and it will disappear as soon as all processes disappear) so that might be a bit annoying, I'll probably start with an option to disable the tmpfs mounting ? *bauen1 hands samplet the keys to a bulldozer <samplet>Woah! I’m messy but /that/ messy! :) <stikonas>bauen1: hmm, if you disable tmpfs mounting, we should probably still run things in a new non-tmpfs folder <stikonas>hmm, I guess in live-bootstrap we'll have to reinstate mes-m2->mes step then <bauen1>stikonas: what exactly do you mean "run things in a new non-tmpfs folder" if there wasn't a tmpfs mounted ? <bauen1>you can have a tmpfs, e.g. by using `unshare --map-root-user --mount ; then running mount -t tmpfs tmpfs mydir` but you would need to do that before rootfs.py runs, or do some magic to get rootfs.py into the mount namespace or in some way rexecute rootfs.py <stikonas>bauen1: I mean a copy of live-bootstrap files should be made (basically what we do now but just remove mounting tmpfs) <stikonas>if you run without a copy, it's a bit destructive <bauen1>yeah, you can copy all you want :) <stikonas>but e.g. when I ran without rootfs.py (copy stage0-posix and overlay live-bootstrap on top) then I could kick the run with just chroot but I can only do that once