<fossy>when using git it is significantly easier for it to be MITMd and data changed within <stikonas>but bootstrap can check validity of checksums <bauen1>stikonas: that depends on the hosts tar being decent enough <fossy>we have to do that either way? <fossy>(download million of gnulib snapshots) <fossy>as a submodule points to a specific commit <bauen1>stikonas: another option discussed here was using the software heritage archive as backup mirror <stikonas>well, git downloads all snapshots, but it compresses them better <fossy>yes but downloading a tar.gz snapshot is like depth=1 <fossy>so it's always going to be faster/smaller than full git submodule <stikonas>but yes, I guess we depend on tar being decent enough <stikonas>but once we have more, like 20, I'm not sure what is faster... *fossy would eventually like to remove the need for submodules entirely and preferably git on the host as well <stikonas>well, Ideally we create a release at some point <fossy>how does having more change how fast it is? cause you still need a submodule for every single gnulib checkout, and same for the tarball <fossy>don't remember with which package <stikonas>that's the only early package that we build twice <fossy>the main one and then -patched <stikonas>ok, then I agree, submodules wouldn't be faster <stikonas>hmm, slightly annoyingly though, get_file function only supports 1 download <fossy>we need to look at how to 'fix' that <stikonas>get_files function that is similar to get_file wrt to the first argument but also downloads rest of the files? <OriansJ`>well git submodules only make sense if you have various pieces all in development and you can't depend upon tar, wget and/or sha256sum being available <stikonas>well, fossy convinced me that snapshot is better anyway.. <OriansJ`>which is fine for the really low level pieces like stage0-posix (which you can manually check for correctness) <stikonas>I had an impression that there would be more caching, but apparently that's wrong <OriansJ`>yeah git submodule really hogs bandwidth <OriansJ`>which doesn't matter if the repo is 10KB but becomes a bitch when you have something like 100MB repo <OriansJ`>that you would have to download perhaps a dozen times <OriansJ`>right now live-bootstrap can *NOT* be downloaded using git on a flaky network connection <fossy>it has a lot of submodules rn, yes <stikonas>oh, yeah, that might be the problem. luckily my connections is quite reliable <fossy>i wonder how downloading over tor would go <OriansJ`>and because git pull/fetch doesn't resume downloads it tries to download the 100+MB pack file from scratch again <OriansJ`>So the only way it can be downloaded for those people is to setup a VPS and do the git clone --recursive and then tar it up and use sftp to download it (it supports resume) <stikonas>but in any case I was thinking that at some point once we get a bit further, we should make a release <fossy>we should do that once we get to modern toolchain <fossy>that's when i think it has fufilled its main purpose <fossy>then we need to go back and make various improvements <OriansJ`>absolutely traditionally in bootstrapping projects once you bootstrap GCC, you call it version 1.0 <fossy>i think OriansJ` means modern GCC <fossy>i don't think it's release-ready <OriansJ`>and 1.0 releases don't have to be amazing just something to point to and say hey look <fossy>mm, kinda, i prefer saying 1.0 is stable-ish and known to work well <OriansJ`>with the promise of something better at version 1.1 or 2.0 (depending how fast you are making progress) <stikonas>although, I was thinking we should write to bootstrappable mailing list that live-bootstrap can now get to GCC 4.04 <OriansJ`>fossy: It is your project, do what makes you happest ^_^ <stikonas>by the way, a few more tests showed that we might already win a minute or two if we don't rebuild mes <OriansJ`>stikonas: public announcments in bootstrapping is one area we have always been traditionally weak at but I hope you can help address that weakness. <stikonas>well, I can write there, but I first want to solve pregen files in findutils <stikonas>(should be similar to what I'm trying with tar now) <fossy>stikonas: i'm happy to draft an email and we can co-sign it? i'll post a draft in a second <fossy>i'm waiting for a run of live-bootstrap rn anyway <stikonas>but maybe let's send if once findutils issue is resolved <stikonas>it should be fairly similar to tar, so once tar is done we can copy/paste the same solution to there <stikonas>by the way, other than submodule, that PR looks alright? <fossy>if you push with submodule i'll give it green tick <fossy>also, perhaps 4-space indenting in import-gnulib.sh (dosen't matter too much tho) <stikonas>I'm now writing some fucntion to get multiple files <stikonas>2-space indenting was inherited from copy-pasted file <stikonas>but manpage says remote magnetic tape server <OriansJ`>rmt: "Using this utility as a general-purpose remote file access tool is discouraged." <stikonas>hmm, I have a slightly different version <stikonas>since I haven't unpacked multiple tarballs yet <fossy>i think yours is /slightly/ cleaner but also mine does positional arguments <stikonas>we now have two functions doing similar things... <fossy>another advantage of mine is that it has a supprot for all features <fossy>i can invisinge usecases for that <stikonas>fossy: maybe yours can be a bit cleaner if you can split into smaller functions? <OriansJ`>just have to duplicate for AMD64 and AArch64 to finish off but that will take a while to spot check for everything. So what did you want me to review stikonas? <stikonas>although, I think fossy's has more features <fossy>note more features != good lol <fossy>but in this case i think it is more versatile <OriansJ`>I like the more explicit options in fossy's that is definitely a good idea <stikonas>yeah, I didn't like those positional arguments that we had before <OriansJ`>but stikonas: your seperate _get_file function is a good idea <stikonas>yeah, that's why I asked fossy if that big function can be split into smaller <stikonas>then we might end up with a better option than both initial versions <stikonas>anyway, I'm going to bed now, will review in the morning <OriansJ`>so here is my suggestion choose fossy's for merge and then break up that big function (steal stikonas' good idea of a seperate _get_file function) and it should be in better shape long term. <OriansJ`>(I really need to find time to work on mes-m2 this week as in its current form it can't run on AArch64 and the AMD64 isn't native but legacy) <OriansJ`>janneke: if you get a minute free to look at the current mes-m2; I'd like it to be able to actually run the copy of MesCC in it (as well as have guile be able to run that copy of MesCC) so that when I do M2libc conversion work I can make sure to keep it working every step of the way. <OriansJ`>that way mes-m2 will have a native AMD64 build, a working AArch64 and a working armv7l build as well. <OriansJ`>and mes-m2 (mes.c) MesCC will have native M2-Planet binaries for all M2-Planet supported architectures <fossy>siraben: i have split it into a couple of smaller functions and am testing full run now then ill push <janneke>OriansJ`: that's very good news. MesCC is mostly "just another" scheme program. <janneke>./pre-inst-env mescc scaffold/main.c <janneke>and it runs scripts/mescc (from scripts/mescc.in), which runs <janneke>scripts/mescc.scm (from scripts/mescc.scm.in) <janneke>to run mescc on guile, just do MES=GUILE ./pre-inst-env mescc ... ***ChanServ sets mode: +o rekado_
<gforce_de1977>4 out of 15 runs of live-bootstrap in qemu, have bad bash checksums: my base64 seems not to work, i will upload some binaries and logs in ~12 hours <gforce_de1977>here are only logs of failed bash-builts: search for 'bash: FAILED', using base64-encoding, i can "screenscrape" the bash-binaries, and it seems in cause if "fail" they all have the hash df8251032006c8ef3f173f3c8859414ad0c9fe2cf4c5df5bcf5afbaf34ded1f5 <stikonas>gforce_de1977: ok, this time difference from good is very small <stikonas>A is from good hash and B is from your bash binary <stikonas>gforce_de1977: that looks like pipesize issue <gforce_de1977>very good...but remember, there was also a version with another size an totally different binary layout. but first things first <gforce_de1977>just for saying it: your output if "gcc -dM -E -xc /dev/null" is the same like in my CI (also on case of a "bad" bash binary) <gforce_de1977>stikonas: can you say me the exact pathname of the generated pipesize.h (or similar) <stikonas>gforce_de1977: well, that might be one option... <stikonas>let's see what fossy and pder think about it... <stikonas>gforce_de1977: path is builtins/pipesize.h <stikonas>or I guess absolute would be /after/bash-5.1/build/bash-5.1/builtins/pipesize.h <pder>stikonas: nice to see you got tar working- I started on that until I noticed the gnulib-tool issue. <pder>There are many places where we are doing make MAKEINFO=true. Could we set MAKEINFO=true when running configure instead? <pder>It's not clear to me what the bash pipesize.h value is for gforce_de1977. I guess the sleep 3; sync is not working? <stikonas>pder: hmm, not sure about MAKEINFO, haven't tied <stikonas>pder: maybe we can just try building makeinfo... <stikonas>even some old version might be better than nothing <stikonas>pder: as for tar, it's almost working, but we need to download gnulib tarball <stikonas>fossy and me tried yesterday, got something working <stikonas>but it might be that this bash script is just getting too big <stikonas>I'm not looking at whether to rewrite it in Python <gforce_de1977>stikonas: pder: it is a qemu run. always with the same kernel and the same initrd. mostly on runs without issues, but around 5% has at least this pipesize variation. there are also other variations already spotted, i'am preparing another massive parallel run (50 qemu-invokations each with 3gb ram) <stikonas>gforce_de1977: pipesize issue is intermittent <stikonas>but adding sync command after sleep 3 in psize.sh seemed to help <stikonas>(you can also try replacing sync with sleep 1) <stikonas>well, it's probably some kind of race, since it is intermittent <stikonas>(and not that it only happens in bash built with mes libc) <stikonas>i.e. you would have correct pipesize.h if you rebuild bash 5.1 with bash 5.1 <bauen1>gforce_de1977: are compute resources an issue for you at all ? <stikonas>I'm preparing rootfs change that will allow any number of runs too... <bauen1>i'm not sure if why 8gb ram laptop is gonna fancy parallel runs lol <fossy><stikonas> fossy and me tried yesterday, got something working <fossy>yeah I finished that last night <fossy>after bashing out an annoying bug <fossy>(remember to declare variables local kiddies)