IRC channel logs

<mihi>in the spirit of announcing wip projects instead of having them forgotten: In the last weeks I've been (very slowly) working on bootstrapping GNU Autogen. It's not finished yet (I know of 8 pre-generated files that are still used, and I may have missed some), but I have the feeling that I'm on the right way. My WIP code is at https://github.com/schierlm/gnu-autogen-bootstrapping/tree/wip, and for some reason I

<mihi>don't know yet, it does not finish on GitHub Actions...

<mihi>an open question: What is the proper way to deal with "half-generated" files? There are two files (in CGI support and in copyright-updater) where the template uses the output file as additional input and copies a few lines from them. My approach would be to remove all non-needed lines from the files via sed, and regenerate them. But I can also patch the CGI support and the copyright-remover script out if that is

<mihi>preferred.

*mihi doubt anyone uses Autogen CGI support to builds his website...

<stikonas>mihi: I think I once dealt with half-generated files by using cat

<stikonas>though I don't remember where...

<stikonas>oh probably here https://github.com/fosslinux/live-bootstrap/blob/9b5c6ca3a19d656caf389689b72e8ce291a02903/sysa/autoconf-2.53/autoconf-2.53.sh#L15

<stikonas>anyway, great work!

<stikonas>Autogen looked fairly scary to me so I did not try to bootstrap it

<stikonas>and I think it was the same for fossy

<mihi>I don't really want to alter the way the files are generated. The template currently does something like 'open OUTFILE for reading, read a few lines into template variable, close it, open for writing/truncate, apply template'

<stikonas>and I tend to announce wip projects too...

<mihi>I know that this is risky (what if the script dies? then also the non-generated stuff is lost!) but that's how it is.

<stikonas>I see, so it's a bit different

<mihi>also, I have no idea what the point is of having all format strings in a long (generated) string constant and then have them index by offset into that constant (except that gcc will not warn about format string issues).

<mihi>but yeah, autogen is a weird beast. Only bad that it is a GNU project and therefore ubiquitous.

<stikonas>indeed...

<stikonas>and especially it is used for binutils and gcc to generated some templates...

<stikonas>they are not critical but still annoying

<stikonas>but if we can get it in live-bootstrap, I guess it still goes after guile...

<mihi>yeah, I needed to patch latest release slightly to work with guile-3.0.7 (but only version number checks)

<stikonas>well, at least we'll finally have a good reason to have guile in live-bootstrap

<mihi>seems that author did not believe guile can at some point no longer be major version 2 :)

<stikonas>dangerous assumption to make...

<mihi>yeah. Recently some got bitten by Firefox / Chrome 100...

<stikonas>but a fairly common mistake... There is quite a lot of breakeage with linux versions

<stikonas>especially 2 ->3

<blockhead>gtk3, python3 :p

<mihi>also I did not find out what version of gnulib creates the #defines in the order that the script expects, so I just patched the script :)

<stikonas>how hard could it be to properly deal with versions...

<stikonas>I mean if there is a number, assume that it can change

<mihi>and if the software is TeX, assume that its version can change to \pi

<stikonas>:D

<stikonas>that will happen when time t \to \infty

<mihi>I believe it will happen by definition when Donald Knuth dies.

<mihi>but maybe that's an urban myth

<unmatched-paren>mihi: not an urban myth: https://texfaq.org/FAQ-TeXfuture

<unmatched-paren>metafont will be changed to version `e' :)

<stikonas>fossy: ok, so looks like we might want to add git to live-bootstrap

<stikonas>for eventual autogen bootstrap https://github.com/schierlm/gnu-autogen-bootstrapping/blob/wip/bootstrap.sh

<oriansj>mihi: glad to see you working on another really hard project. I know you'll do a great job as always

<stikonas>yes, having briefly looked at it, I'll be very impressed if autogen is bootstrapped

<stikonas>and speaking of progress, I have been updating stage0-uefi binaries with some improvements (properly release all resources, use new style M1 defines, more robust argument parsing), done with hex0, hex1, hex2 and catm, still need to do kaem-minimal

<stikonas>and after that I can start working on M0.efi

<stikonas>(with these changes hex0 unfortunately got slightly bigger, though hex1 is actually smaller)

<oriansj>sweet

<Hagfish>fantastic

<Hagfish>interesting that the sizes changed in opposite directions :)

<stikonas>that's because hex0 was already slightly more optimized in terms of size

<stikonas>so my changes had some size saving (but less so in hex0). But another change added some bytes (I had to add some calls to close UEFI protocol)

<Hagfish>ah, that makes sense, thank you

<stikonas>though when you look at hex0, smallest size is not always the best

<stikonas>it's the jumps/calls (especially long ones) that are hard to analyze

<Hagfish>what's the alternative to jumps/calls?

<stikonas>though perhaps for kaem-minimal I should split some stuff into functions (despite long jumps), i.e. we have 3 calls to open protocols

<stikonas>Hagfish: sometimes inlining stuff can avoid jump

<Hagfish>the "rule of 3" suggests that having 3 calls is the right moment to make that change (but it's a very rough heuristic)

<stikonas>but of course at the expense of number of bytes

<Hagfish>hmm, okay

<stikonas>for some stuff you could try branchless programming

<Hagfish>wow

<stikonas>but we don't use it in stage0-posix

<Hagfish>yeah, i've never tried it, but it seems like an advanced technique

<stikonas>yeah, so that might be the reason why we don't use it

<Hagfish>fair enough

<stikonas>e.g. say you want to find smallest number

<stikonas>you can write a function with if/else (compare a < b) and return the smallest one

<stikonas>or with branchless programming you could always do both calculations, e.g. a * (a < b) + b * (b <= a)

<Hagfish>wow. i don't think i've seen an example like that before

<Hagfish>it makes sense, i guess

<stikonas>there might be yet another thing one can try to avoid manual calculation of jumps, but again not used in stage0

<stikonas>one could try to save address of the label on the fly as it is passed during execution and store it in the register

<Hagfish>wow, so making the code it's own interpreter, kinda?

<stikonas>kind of

<Hagfish>dynamic programming? is that what it's called?

<stikonas>but again that might be a bit confusing

<stikonas>not sure

<Hagfish>sure

<stikonas>well, something like "call 5" after label can push address of current label onto stack

<stikonas>but as you can see all of these come with tradeoff of more bytes

<oriansj>also by ensuring what we create is easy to learn and understand, it provides root for people later to come and make major improvements

<Hagfish>at this level of bytes, i feel like auditability is the deciding factor (unless there is some critical piece of small hardware you want to support where these bytes make the difference)

<Hagfish>but yeah, if you start with something that is understandable, other people can port it for different requirements

<pabs3>"Compiling Rust with GCC: an update" https://lwn.net/SubscriberLink/907405/8844655df94a7886/

<Hagfish>nice! thanks for sharing

<stikonas>hmm, not sure if this can be used to bootstrap rust

<stikonas>they don't seem to target cargo at all

<stikonas>unlike mrustc which has minicargo

<stikonas>but maybe I'm missing something

<Hagfish>"It is also now possible to bootstrap rustc with rust_codegen_gcc, which is a big milestone"

<Hagfish>but yeah, not sure about cargo

<oriansj>stikonas: well it is potential not yet explored but as we already have a working bootstrap path for Rust, it is more like a nice extra

<stikonas>indeed

<stikonas>and at least Rust bootstrap path is well maintained (for now)

<stikonas>so bootstrap compiler is eventually targetting newer and newer versions of rust

<stikonas>unlike openjdk where we are stuck with version 7

<stikonas>and then have to build all intermediate versions

<oriansj>well that is because there are not multiple people willing to work for years to push the Java Bootstrap forward

<oriansj>and I feel like I am so close to finally cracking the gentoo encrypted / and /boot procedure

<oriansj>now it is just a matter of finding what modules are actually needed

<oriansj>I have figured out that I did build the dm-crypt.ko module but I didn't include it in my initramfs

<oriansj>so now I am going to test if just copying all of it will work

<oriansj>(which will really ballon the initramfs)

<oriansj>which if that works, I just need to figure out the subset

<oriansj>yes (ish, needs a little tweaking but it boots)

<oriansj>well it works; is ugly as sin and needs some basic enhancements: https://paste.debian.net/1253444

<oriansj>ok, down to 21MB in size

<oriansj>(still seems too large)

<oriansj> https://paste.debian.net/1253446

<oriansj>well the clean up is going to need several iterations of testing before it'll work all major bugs out in the procedure but now I can better document what gentoo's wiki should have

<midnight>I was wondering if anyone here happens to have an mbox of the bootstrappable ML they'd be willing to share? looks like the freelists ML site doesn't collect stuff in downloadable form.

<midnight>(rather not use an unfriendly scraper)

<oriansj>midnight: well I could compile one for you if you are willing to wait until monday

<midnight>oriansj: I am in absolutely no rush whatsoever. :-)

<oriansj>I could even include it in the wiki's git repo so that it'll be just a git clone away for anyone who wants it and it becomes trivial for anyone who wants to keep it up to date

<midnight>oh that would be ideal.

<midnight>it also solves the programmably-accessible ML archive thing permanently then.

<oriansj>well as long as someone bothers to append new messages and commit them to the repo (which I can give to anyone who wants to be reponsible for the health of our wiki)

<midnight>:-)

<oriansj>and if enough people think the git wiki is the way forward for wiki updates; then we just need to point to the git wiki by default.

<midnight>I would offer to perform the mbox append but I'm still working out how to privately subscribe

<oriansj>midnight: well if I remember correctly, the steps were to go to: https://www.freelists.org/list/bootstrappable put in your desired email address and select subscribe from the drop down and then click next

<oriansj>which could be done via tor

<oriansj>or if you want I can setup an email forward for you

<midnight>Oh; no I can do that. I'm just working out which email to subscribe. Sorry. :)

<midnight>In another life I run my own MX..

<oriansj>well right now it is kinda low volume and mostly just big announcements

<midnight>cool

<oriansj>sam_: my revised initramfs script: https://git.sr.ht/~oriansj/System_setup/blob/main/files/build_initramfs.sh should be generally useful for the gentoo community

<oriansj>and unlike debian's default failure on the luks unlock will get you into a shell where you can safely successfully fix things up and properly boot

<oriansj>the gentoo install procedure still has a bunch of errors but I should have it clean by monday

<oriansj>(assuming I don't hit too many issues around the kernel build)

<oriansj>the full gui and related bits probably will take a bit longer as compile times grow and I need to figure out how to add stikonas's bootstrappable gentoo extension

<doras>PR for musl dynamic linking support in live-bootstrap: https://github.com/fosslinux/live-bootstrap/pull/194

<stikonas>doras: by the way, you can make pull requests that depend on other PRs (just need to pick a different base branch and later rebase once base branch is merged)

<mihi>stikonas, fossy, is it hard to bootstrap git? It should be feasible to bootstrap autogen without it (if there is no .git directory, the bootstrap scripts pull changelogs and version info from ZZJUNK file; and I assume you can clean the tarball manually so that it matches the git repo)

<mihi>once bootstrap is finished, I might add a bootstrap-from-tarball script for that.

<mihi>I just started from the git repo as it seemed to be prudent to make the first bootstrap as easy as possible, considering the big ball of mud that autogen is.

<stikonas>mihi: no, not hard, it should just build

<stikonas>so if it's more convenient it's fine to add git as dependency

<stikonas>mihi: though it would be nice if there was an option to bootstrap for autogen tarball rather than download on the fly (even if local git is still required for some steps)

<stikonas>anyway, this can be sorted out later

<mihi>stikonas, if you touch build/tagged, it will skip the download.

<mihi>build/tagged-src, that is

<mihi>assuming that you already "downloaded" and tagged the source :)

<mihi>however, a tarball will not help as it usually does not contain .git directory

<mihi>the build-deps on https://packages.debian.org/source/bullseye/git looked a bit scary to me, but probably most of them are optional...

<stikonas>yeah, I think most is optional...

<stikonas>looking at gentoo ebuild, I think only zlib and openssl are needed

<stikonas>rest are optional...

<sam_>oriansj: nice, I'll take a look later today!

<oriansj>tell me if this sounds stupid but couldn't the script I wrote for building initramfs files also be used to solve the guix double password prompt problem? (which a couple minor tweaks of course)

<oriansj>^which^with^

<janus>who is working on haskell bootstrapping?

<janus>i got nhc98 running, working on compiling ghc-0.29 with it, i'd like to pair on it though

<oriansj>janus: if I remember correctly that was rekado and I think they got stuck on something: https://elephly.net/posts/2017-01-09-bootstrapping-haskell-part-1.html

<janus>right, i have been reading that. but i'd like to get the bootstrapping working with minimal patching first. so i am just compiling nhc98 on redhat6.2 and it works almost out of the box

<janus>that blog post is really about nhc98, so it is kinda orthogonal. i mean, sure it would be nice if the nhc98 runtime didn't depend on some specific memory layout

<janus>rekado isn't in here, how do i reach them?

<oriansj>they also are on #guix if I remember correctly

<oriansj>and I believe they do respond to emails as well

<janus>ok thanks

<oriansj>sam_: well I found a trap in gentoo that I can't find a solution to in the gentoo documentation. If one sets PORTAGE_BINHOST in /etc/portage/make.conf and does an emerge; then goes back and comments out or deletes the line, all following emerges fail with: "!!! /etc/portage/binrepos.conf is missing (or PORTAGE_BINHOST is unset), but use is requested"

<sam_>i suspect you have an EMERGE_DEFAULT_OPTS set with getbinpkg or something

<oriansj>echo -e "\nEMERGE_DEFAULT_OPTS=\"--keep-going=y --autounmask-write=y --jobs=2 -G\""

<oriansj>would that be the -G?

<oriansj>yes

<oriansj> thank you for the sanity check sam_

<sam_>:D

<sam_>np

<doras>So apparently glibc doesn't like being built for i386 and our GCC in live-bootstrap can't handle anything else.

<doras>Wait, perhaps not. Hmmm...

<doras>Oh I see, it has a default `-mtune=i386` `-march=i386`.

<stikonas>it has to build. There are still distros supporting i386...

<stikonas>maybe some configure argument or env variable is wrong

<stikonas>guix also builds glibc for i386

<oriansj>worst case we do i686

<oriansj>and that should cover every chip made after the Pentium Pro

<sam_>define "doesn't like"

<sam_>(full log please)

<oriansj>sam_: I put logger "cpp_match for ${CATEGORY}/${PN}: ${cxx_files}" in post_src_prepare() would I be right in that stage3 doesn't contain what logger needs to write to the logs and thus would need to add something like emerge -av syslog to the procedure?

<oriansj>because journald isn't picking up anything and /var/log isn't getting any logger command

<oriansj>or maybe I'm missing extra mount commands???

<sam_>right, I don't think a stage3 contains virtual/logger

<sam_>(which depends on any of the various packages providing 'logger')

<sam_>so yes, I think your conclusion is right

<oriansj>sam_: well I guess figuring out that detail will have to wait for the next iteration of testing the setup procedure (by testing the mount option change) and hopefully that'll be that and I'll finally have some logging of how many cpp files are in the packages built in the process of following the procedure

IRC channel logs

2022-09-11.log