IRC channel logs

<stikonas>fossy: oh, I think I found tcc patch to deal with static inside []

<stikonas> https://repo.or.cz/tinycc.git/commit/ef668aae1ee2b8bc904c50a13bf58df613b2f0b0

<OriansJ>up to step 39 has been documented.

<OriansJ> https://github.com/oriansj/talk-notes/blob/master/live-bootstrap.dot ( and for those without graphiz installed https://github.com/oriansj/talk-notes/blob/master/live-bootstrap.pdf )

<fossy>stikonas[m]: oh, nice

<fossy>feel free to add to tcc-patched

<Hagfish>a work of art!

<Hagfish>OriansJ: you should sell the NFT of that ;)

<Hagfish>seriously though, did you have any more inspiration about picking subgraphs of it?

<Hagfish>maybe you could use coloured background regions to represent the "build environment" that a build occurs on (if that can be meaningfully separated from the idea of a dependency)

<OriansJ>Hagfish: yes however I'm keeping it simple until I get all the steps documented.

<Hagfish>good thinking

<Hagfish>i look forward to seeing what you come up with next

<OriansJ>up next is the next autotools/autoconf cluster

<OriansJ>at this point I am probably missing dependencies and have a few false dependencies but I am close enough to being done that, I'll fix it later unless someone beats me to it.

<OriansJ>and finally all 66 steps are recorded (with some possible errors but should be easy to fix)

<OriansJ>darn near 1/2 the graph is just dealing with autoconf and automake

<OriansJ>Something tells me when I get M3 working, that graph is going to be ripped down into something much smaller.

<Hagfish>66 steps, like the books of the bible

<Hagfish>but yeah, M3 will have an interesting effect on that graph

<OriansJ>stikonas[m]: sorry for the delay (I have merged your pull request)

<stikonas>ok, xz builds fine with patched tcc

<stikonas>(that patch was useful anyway, since it allows to drop musl patch)

<bauen1>I've also wondered if (maybe once you have stow) it would make sense to build components in their own chroot ? that would require some copying or bind mounts and /sbin/chroot but then you have a clear list of dependencies not just "everything so far"

<stikonas>bauen1: yeah, possibly...

<stikonas>chroot is probably easy to add to 2nd build of coreutils

<stikonas>(the one that is built with musl rather than meslibc)

<bauen1>that would also put additional requirements into the kernel, but from my experience building a proper vfs (with mount support) should make adding chroot support comparatively easy

<bauen1>and you can always copy everything over (but at some point memory will become an issue if you continue down that road)

<stikonas>yeah, we are already using a fair bit of memory

<stikonas>we can reclaim some though

<stikonas>by cleaning build directories

<stikonas>also by not doing debug builds...

<bauen1>combine that with stow and basically have a package manager with manual dependency resolution lol

<bauen1>*and you

<bauen1>is there a way i can rerun rootfs and have it restart at the last failed step ?

<stikonas>bauen1: not at the moment

<bauen1>or at least skip the part before "after" ?

<stikonas>there is a way to skip that part though

<stikonas>you have to cherry-pick pder/overlay commit

<stikonas> https://github.com/pder/live-bootstrap/tree/overlay

<stikonas>it basically starts with tcc

<stikonas>and if step fails, I often chroot inside and do some manual runs of that step

<stikonas>something like bash -c 'set -e; export PREFIX=/after; . helpers.sh; build ...'

<bauen1>yes that's what i'm trying to do with musls configure

<bauen1>but it seems to succeed even if the step inside run.sh fails :D

<bauen1>maybe i had my chroot setup in a slightly different way

<stikonas>is that with user ns?

<bauen1>yes

<bauen1>i should try qemu or normal chroot probably

<stikonas>well, that will work

<stikonas>at least if you use 64 bit kernel

<stikonas>at some point it would be good to check if hashes match with 32 bit kernel

<bauen1>does rootfs.sh currently target i386 or x86_64 ?

<stikonas>i386

<stikonas>but we test on 64 bit kernels...

<stikonas>well, it shouldn't matter until at least autotools

<bauen1>that's basically my setup

<bauen1>does the chroot require /proc or /sys ?

<stikonas>there with configure I tend to specify --build=i386-unknown-linux-gnu but it's possible that I missed it somewhere

<stikonas>bauen1: no, it does not

<stikonas>bauen1: well, "./rootfs chroot" can set up correct chroot

<stikonas>there isn't even /proc or /sys folder inside there

<stikonas>even /dev is missing, but we mkdir /dev just before musl

<bauen1>right, currently i do that before even reaching musl because you can't mknod inside a user ns

<bauen1>so that could be part of my issue

<bauen1>also doesn't the chroot "leak" the hosts PATH ?

<stikonas>I don't think musl actually needs /dev/null

<stikonas>it only fails configure step if /dev is missing altogether

<stikonas>bauen1: no, it shouldn't leak

<bauen1>it does pipe /dev/null into temporary files to create them

<stikonas>yeah...

<stikonas>so that fails if /dev is missing

<bauen1>it's probably gonna be something really simple and stupid

<stikonas>fossy: ok, xz should be ready for review https://github.com/fosslinux/live-bootstrap/pull/77

<bauen1>and i think musl 1.1.24 just worked

<bauen1>wtf

<bauen1>OriansJ: perhaps you could also make an svg of the .dot graph available, that is better if you want to zoom in

<bauen1>or not

<bauen1>i think it's actually failing `build musl-1.1.24 binutils-rebuild.sh checksums/pass3 patches-pass3`

<bauen1>but running that step from my manual chroot compiles successfully ._.

<OriansJ>bauen1: https://github.com/oriansj/talk-notes/blob/master/live-bootstrap.svg

<bauen1>huh doesn't look like github likes svgs ? https://raw.githubusercontent.com/oriansj/talk-notes/master/live-bootstrap.svg gives me xml errors ...

<bauen1>in any case i also have it locally so no big issue

<OriansJ>built with: cat live-bootstrap.dot | dot -Tsvgz >| live-bootstrap.svg

<OriansJ>sxiv doesn't like it either but mirage does

<gforce_de1977>stikonas: bauen1: /proc or /sys is not needed, see ./rootfs.sh minikernel

<stikonas>even locally, somehow I find it easier to zoom into PDFs with pdf viewer than into svg with image viewers

<gforce_de1977>at the very moment live-bootstrap fails for me: http://intercity-vpn.de/live-bootstrap-3bf5d4152d4d59ee4d420381d2c7d03fda8db339.txt

<stikonas>OriansJ: by the way, all perl's are miniperl up to 5.005

<gforce_de1977>but i will investigate deeper, maybe again a RAM issue

<stikonas>In 5.6 I started statically building some modules (I think not all, only those that are needed by automake)

<gforce_de1977>stikonas: you can zoom into SVG with every browser?

<stikonas>hmm, maybe I didn't try browsers, just normal image viewers

<stikonas>and okular with pdf worked better for me...

<OriansJ>stikonas: thank you, I'll update accordingly

<stikonas>gforce_de1977: yeah, it might be lack of space...

<stikonas>I just didn't bother with perl modules until we actually needed them

<stikonas>they are not too hard to build, but still makefile is significantly longer

<stikonas>also, I'm not sure we actually use coreutils 6.10 that much...

<stikonas>it's mostly 5.0

<stikonas>coreutils 6 only has very few things built, sha256sum, date and mktemp

<stikonas>mktemp is probably used during building of bash...

<stikonas>date is probably not used, otherwise it would see build hashes change

<stikonas>flex 2.5.11 also has bootstrap stage if I recall correctly

<stikonas>OriansJ: any reason for bash to be in eliptic box?

<stikonas>instead of rectangular?

<stikonas>in general that live-bootstrap.dot is now fairly accurate

<stikonas>oh, a few definitely unnecessary dependencies

<stikonas>musl 1.1.24 (binutils self-host) and tcc-0.9.27 (musl self-host) don't use autoconf 2.13

<stikonas>binutils 2.14 needs both autoconf 2.13 and 2.12

<stikonas>anyway, accurate dependencies later in the bootsrap are probably too complicated to plot

<OriansJ>stikonas: probably because I forgot to give it a box

<OriansJ>but accurate dependencies in the later bootstrap are just a question of taking the time to trace them out

<bauen1>OriansJ: or building things in chroots then you know exactly what you need

<stikonas>yeah, that's true, but it might still look too complicated in the plot

<bauen1>oh ffs

<stikonas>if everything depends on 30 things

<bauen1>it took **too long** for me to release that not musl-1.1.24.sh is failing but binutils-rebuild.sh

<stikonas>oh, that's tcc :D

<stikonas>and maybe build script is a bit confusingly named

<bauen1>yes, just slightly

<bauen1>stikonas: but with a chroot it becomes much easier to figure out what depends on what, though direct and indirect dependencies will overlap

<stikonas>we don't have many indirect dependencies...

<stikonas>for now everything is built statically

<stikonas>(just because we can't build libc.so)

<bauen1>then things are even easier

<stikonas>probably perl is the most common indirect dependency (due to autotools)

<OriansJ>clarity through bruteforce is sometimes the best approach.

<gforce_de1977>stikonas: i'am sorry, i have double checked, that we have a 32bit kernel + host and 4G RAM, but 'automake-1.6.3' still fails: http://intercity-vpn.de/live-bootstrap-3bf5d4152d4d59ee4d420381d2c7d03fda8db339.txt

<stikonas>let's see, that's automake stage1.sh

<gforce_de1977>stikonas: i do another build now on qemu with 486 cpu enforced and another kernel and will report

<stikonas>we are removing pre-generated files from tarball in that step

<stikonas>I don't see why would it not work for you...

<stikonas>unless some how it was not unpacked properly

<gforce_de1977>i will debug that and report

<stikonas>try rerunning once...

<bauen1>so while i'm waiting, would the following be useful: a "upkg" utility that is both a rudimentary package manager and chroot build tool ? it maintains the symlinks in / and a /upkg directory using stow and can be used to build more "packages" in a chroot when given a list of packages (with specified version), a source tree and a build script ?

<bauen1>that would require chroot support from the kernel, and maybe bind mounts

<bauen1>in theory it could also be used to skip various steps in the bootstrap during development more easily

<stikonas>well, I guess you can try and see if it works

<bauen1>true, getting live-bootstrap to work for me is a bit of a requirement lol

<stikonas>where is it's repo?

<bauen1>stikonas: it's not yet written

<stikonas>so what failed in binutils-rebuild.sh? is that still with user-ns

<stikonas>oh, I see...

<bauen1>stikonas: i'm not sure it could just be the line that tests / removes /dev/null i'm waiting for the build right now

<stikonas>I thought maybe there is something...

<stikonas>well, nobody tried running live-bootstrap without root before

<stikonas>only in qemu or full chroot...

<stikonas>I guess running it in rootless podman would hit the same problems that you see

<OriansJ>sourceforge is not returning heirloom-devtools-070527.tar.bz2

<stikonas>hmm, occasionally that happens...

<stikonas>maybe we should think about mirrors...

<stikonas>I have a copy locally

<stikonas>I can make them available

<stikonas>OriansJ: https://stikonas.eu/files/bootstrap/

<bauen1> https://archive.softwareheritage.org/browse/content/c283431474fa714e1176897f4ec221d4aee6407b/ should also be it

<OriansJ>stikonas: thank you but I agree with bauen1 that using software heritage as a fallback mirror if the checksums fail seem like a good idea.

<bauen1>they also have an api https://archive.softwareheritage.org/api/1/content/sha256:9f233d8b78e4351fe9dd2d50d83958a0e5af36f54e9818521458a08e058691ba/

<bauen1>OriansJ: i'm not sure if they particularely like that idea, might be worth asking first

<bauen1>direct download link would then be https://archive.softwareheritage.org/api/1/content/sha256:9f233d8b78e4351fe9dd2d50d83958a0e5af36f54e9818521458a08e058691ba/raw/

<OriansJ>That way if someone finds something in the download, we can't be blamed because it isn't something we are hosting.

<OriansJ>bauen1: well doing it that way is certainly an option (I know guix was thinking of doing something similiar themselves a while ago)

<OriansJ>hmm 498M for everything downloaded (98.2MB for just the downloaded tars; I'm betting IPFS, torrents and other distribution channels are also options

<bauen1>so i've figured out that it gets up to the musl rebuild after binutils steps successfully, so next rebuild

<stikonas>well, at this stage for distribution we need something where we can easily change the list of downloads

<stikonas>oops, I managed to build xz with tcc but it segfaults

<stikonas>hopefully fixable by tweaking configure options

<gforce_de1977>stikonas: i'am stuck somehow with debugging: i have an interactive BASH session and can run e.g.

<gforce_de1977>cd after && . ./helpers

<gforce_de1977>build automake-1.6.3 stage1.sh

<stikonas>and it's fails?

<stikonas>*it

<gforce_de1977>but the error is...another?

<gforce_de1977>+ make -f Makefile

<gforce_de1977>make: Makefile: error 02

<gforce_de1977>make: *** No rule to make target `Makefile'. Stop.

<gforce_de1977>please wait, will share copy/paste

<stikonas>are you running it with bash or busybox?

<gforce_de1977>bash

<gforce_de1977> https://paste.debian.net/1191381/

<stikonas>that's probably busybox

<stikonas>+ default_src_compile

<stikonas>it shouldn't run that function

<stikonas>there is src_compile function defined in stage1.sh

<gforce_de1977>yeah.../after/bin/bash = 347208 bytes and when running it it just says "exit"

<gforce_de1977>dumb question: why do we use bashisms? the script really should be posix

<stikonas>we only have bash shell there

<stikonas>but you didn't start bash

<gforce_de1977>do you have an idea why it says 'exit'?

<stikonas>because we don't have interactive bash at that stage yet

<gforce_de1977>ah!

<stikonas>first bash is compiled against meslibc and so can't have libreadline

<gforce_de1977>ok...arrggggg! 8-) ok, will include my own static bash5.1

<stikonas>libreadline needs more advanced libc

<stikonas>just run it non-interactively

<stikonas>bash -c 'export PREFIX=/after; set -e; . helpers.sh; build automake-1.6.3 stage1.sh'

<gforce_de1977>ok, so /after/bin/bash -c "my command" ???

<stikonas>yes

<gforce_de1977>ok, thanks

<gforce_de1977>here is the same with bash: https://paste.debian.net/1191382/

<stikonas>strange...

<gforce_de1977>ok, when running without -e it seems to work:

<gforce_de1977> https://paste.debian.net/1191383/

<stikonas>without -e it ignores errors...

<stikonas>I don't think we should run it like that

<stikonas>you should check why those files are missing

<stikonas>are they not in your tarball

<stikonas>or ar they not there after unpacking

<gforce_de1977>good question, will check and report (~30mins, have to do family stuff)

<stikonas>yeah, I'll be away to

<stikonas>too

<bauen1>what's the easiest way to create a patch for a source tree ?

<bauen1>or rather, is there a better way than git init, git add ., git commit, git format-patch ?

<bauen1>also i just found a shiny (incomplete ?) list of files generated by autotools https://www.gnu.org/software/automake/faq/autotools-faq.html#Which-files-are-hand_002dwritten-and-which-generated-_0028and-how_0029_003f

<bauen1>oh amazing gnu stow is basically one perl file with autotools used to generate documentation and replace a massive 3 macros inside stow.in before installing it lol

<gforce_de1977>stikonas: here is *my* automake-1.6.3: https://paste.debian.net/1191393/

<gforce_de1977>does it differs from your?

<gforce_de1977>stikonas: why are you using asterisks when deleting? why not just the real full path?

<gforce_de1977>which files do you want to catch with: */Makefile.in */*/Makefile.in

<gforce_de1977>is it: lib/Makefile.in tests/Makefile.in lib/Automake/Makefile.in lib/am/Makefile.in ?

<bauen1>so building the newest gnu stow is dead simple, 3 sed statements + install commands ; but it uses Getopt::Long GetOptionsFromArray which is from 2007 https://github.com/Perl/perl5/commit/8de02997ac38b64dd5d3d654b85f7d29a1fc56af so just 4 years after perl 5.6.2 was released

<bauen1>but version 2.2.2 should work and doesn't miss too many important things, let me try that

<stikonas>gforce_de1977: all of these

<stikonas>all Makefile.in's

<gforce_de1977>ok, tomorrow i will tinker why these are missing after untar

<gforce_de1977>(maybe the globbing just does not work)

<gforce_de1977>bauen1: yes, git format-patch is the easiest way

<bauen1>so now that i have `user namespaces` instead of `sudo` working, would it be ok to PR that to completely replace to usage of sudo ?

<bauen1>it requires unprivileged users to have access user namespaces but i think most major distributions have enabled it by default (debian will switch the default to enabled for the 11 release), maybe arch has yet to do that

<stikonas>but how does it affect qemu?

<bauen1>hm, right

<stikonas>don't you have to pre-create device node with user namespaces

<stikonas>and also baremetal

<stikonas>which is the more important case

<stikonas>if those cases keep working, I guess that's fine

<bauen1>for baremetal linux could be made to auto mount /dev as devtmpfs ?

<bauen1>but yeah i suppose it might break those

<bauen1>i'll do a bit more testing

<stikonas>how can we make it automount?

<stikonas>I thought it has to be mounted manually... hmm

<bauen1>stikonas: DEVTMPFS_MOUNT

<stikonas>hmm, I guess depends on when we can use too

<stikonas>bootstrap kernel might not have it

<stikonas>and actually, I'm not sure how well this would work

<stikonas>because we don't mount rootfs

<stikonas>"Automount devtmpfs at /dev, after the kernel mounted the rootfs"

<bauen1>stikonas: rootfs is special in the kernel sense

<stikonas>oh ok

<bauen1> https://www.kernel.org/doc/html/latest/filesystems/ramfs-rootfs-initramfs.html

<bauen1>but yeah bare metal is fragile

<bauen1>we might not even have a linux as kernel :D

<stikonas>This option does not affect initramfs based booting, here the devtmpfs filesystem always needs to be mounted manually after the rootfs is mounted.

<stikonas>so my reading is that it won't work

<bauen1>oh so i didn't read far enough

<stikonas>anyway, I'm not opposed to replace sudo...

<stikonas>just need to make sure it doesn't break other stuff

IRC channel logs

2021-03-28.log