IRC channel logs

<fossy>normally, when something won't boot with an older kernel, its often missing device drivers, particularly on laptops

<fossy>although sometimes it is't too

<stikonas>hmm, something is messed up with qemu mode with my local changes to live-bootstrpa

<stikonas>I don't see any serial output

<stikonas>but if I start the same qemu command line manually, it seems to work

<fossy>that's odd... is -append console=ttyS0 and -nographic there?

<stikonas>yes, it was there...

<stikonas>I'll probably have to debug it at some point later

<stikonas>will be away for a week

<fossy>ok! holiday?

<stikonas>yes

<fossy>have fun and a good break

<stikonas>today still testing bzip2 packages to see if that saves enough space

<stikonas>although at some point new tarballs in /sources will become too big...

<stikonas>if we add more packages to sysc

<stikonas>possibly need to move sysc /sources to preformatted disk

<stikonas>fossy: I have a small PR which is independent of refactoring https://github.com/fosslinux/live-bootstrap/pull/148

<stikonas>(and possibly one can replace patch kaem build script with makefile but that's not part of this PR)

<fossy>stikonas: yeah, that looks fine

<fossy>stikonas: what refactoring have you done since that last PR?

<stikonas>fossy: basically just having /sysa and /sysc from repo directly copied into initramfs

<stikonas>and no /sources copying into /sysa/pkg-ver/src/

<stikonas>instead each build script grabs required packages from /sources in initramfs and this is all done with kaem scripts or helpers.sh function

<stikonas>fossy: I can push my WIP commit if you want to take a look

<stikonas>chroot mode works already

<fossy>stikonas: that would be good

<fossy>I have separately done merging /sources but I don't really mind which impl we use, it was not much work at all

<stikonas>fossy: https://github.com/stikonas/live-bootstrap/tree/rootfs_refactor

<stikonas>well, take a look at my implementation while I'm away

<stikonas>we can decide later

<stikonas>ok, qemu is still broken even with bzip'ed packages...

<fossy>looks alright, but i'm not really a fan of having to put SRCS= for every package

<fossy>i was considering extracting ${pkg}.tar* by default and then extra_srcs= for anything extra

<stikonas>yeah, that can be changed

<stikonas>feel free to take over those changes

<stikonas>default ${pkg}.tar* can be added to helpers.sh extract function

<stikonas>and that takes care of most of the packages

<fossy>yep

<stikonas>it's mostly gnulib stuff left

<stikonas>and bziping /usr/src/repo packages can save about 10%

<fossy>but still not enough?

<stikonas>no

<stikonas>but maybe I messed up something else

<stikonas>hard to debug in qemu

<fossy>well, the added space usage is from copying in all tarballs

<stikonas>did I copy them?

<stikonas>I thought I extract most of them on the spot from /sources

<stikonas>only early kaem tarballs

<stikonas>and those can be removed once we build coreutils and rm

<fossy> https://github.com/stikonas/live-bootstrap/commit/ffd4c7ab483ff381de6b5f482a0b5da203567b9f#diff-d6568c0c3a316afaf1731ed250447f2f80f4490a2e5aee9baf0b35934a593414R65

<fossy>won't this copy in the sysc tarballs into sysa as well?

<stikonas>oh yes

<stikonas>not sure how to best fix this

<stikonas>if we want to have easy way to run everything without rootfs.py

<fossy>neither, cause we need those tarballs for sysc regardless...

<fossy>i'll have a look at where we can shave off a bit more space

<oriansj>dongcarl: No worries. Internet trolls are just part of internet life.

<stikonas>well, for chroot mode that doesn't matter

<fossy>where does it fail at the moment

<fossy>(in qemu)

<stikonas>extracting linux kernel

<fossy>ah...

<stikonas>but adding more and more tarballs to sysa is not scalable anyway

<stikonas>if we start adding gcc 10, gcc 11

<stikonas>that will quickly use up all the RAM

<fossy>yeah, true

<stikonas>perhaps this:

<stikonas>for chroot everything can be in /sources

<stikonas>for qemu we create sda disk formatted with ext4 (which we already do) and have /sources filled in with sysc tarballs

<fossy>that sounds reasonable, given that we already copy in the sysc scripts

<stikonas>scripts themselves are not big

<stikonas>in principle they can go to initramfs

<fossy>they could, but if we have to copy sources in, makes sense to copy them in too (as we currently do)

<fossy>ultimately, we can network download tarballs, but that is still a way to go (need proper net support)

<fossy>gio's way of doing it in asmc is pretty nice, he uses ipxe to get early tarballs, but i think that still relies on having the appropriate kernel opts enabled

<stikonas>in principle proper net is easy in sysc

<stikonas>just need to install ip program

<stikonas>I think kernel is configured with networking

<fossy>i don't remember if i disabled networking or not

<stikonas>I think no

<fossy>oh, i didn't, you're right

<fossy>CONFIG_E1000=y

<stikonas>so I think we are only missing net-tools

<fossy>and curl, or the like

<stikonas>well, yes...

<stikonas>although that will pull in openssl

<stikonas>for https support

<fossy>can't curl be built without ssl support?

<fossy>at least until we have openssl

<stikonas>it can but most of the stuff is not server via https

<oriansj>wouldn't downloading allow cheating in the bootstrap?

<stikonas>openssl can probably be built earlier

<fossy>oriansj: how so?

<stikonas>well, it allows downloading binaries

<oriansj>curl $url:/binary_blob -o my_totally_legit_build

<stikonas>but so is puting binary blob into /sources

<fossy>sure... but you could just inject binaries anyway right now

<stikonas>I mean if you want to cheat, why even bother with bootstrapping

<oriansj>true

<stikonas>and this is only an issue with 32-bit kernels

<stikonas>(which is the only bootstrap path we support right now)

<stikonas>on 64 bit we have no issues with RAM

<fossy>ideally we would keep the ram usage under 4G for older coreboot/MEless machines but if that's not possible oh well

<stikonas>kernel is a bit of a cheat anyway

<fossy>yeah, kernel is unsolved for now

<stikonas>well, it's probably possible if we keep tarballs in /sysc

<fossy>yep, most likely

<stikonas>so maybe split /sources into /sysa/sources and /sysc/sources

<stikonas>(using exactly these paths)

<stikonas>then copying /sysc onto qemu hard drive will automatically copy sources

<stikonas>but will keep sysa sources in initramfs

<oriansj>well x86-pae would up the limit to 64 GB of RAM but would eliminate anything older than a Pentium Pro from 1995

<stikonas>and what about qemu?

<stikonas>not sure if qemu supports pae

<oriansj>I don't know, lets look it up

<stikonas>that might be kernel build option rather than qemu option

<stikonas>qemu with kvm will just pass though your host CPU

<stikonas>and in most cases that would be a modern CPU

<oriansj>and qemu does seem to support the pae cpu extensions if you do: qemu-system-x86_64 -cpu ?

<oriansj>and yes the kernel would need to be 64bit or built to support pae

<stikonas>so maybe we can enable this...

<stikonas>4GB limit only means that you can't automate the whole chain in 1 go and need more manual steps

<stikonas>it's not really a fundamental blocker for bootsrapping

<oriansj>also expecting a kernel to be able to write to disk isn't a bad thing either

<muurkha>disks are so last millennium, bro

<oriansj>muurkha: stares at 16TB drive on desk

<oriansj>but it was made in 2022....

<oriansj>you cna't expect everyone to just buy a 100TB SSD (https://www.techradar.com/news/at-100tb-the-worlds-biggest-ssd-gets-an-eye-watering-price-tag)

<littlebobeep>Yeah i saw that 100TB SSD but it's not even NVMe

<muurkha>oriansj: that disk is the Loch Ard of storage devices

<muurkha>(I mean the ship, not the lake)

<mid-kid><stikonas> but in any case we don't have many options, it's either backporting hw support to older kernel, switching to different hardware or try to upgrade early toolchain in live-bootstrap <- or have the option to never replace the kernel ;)

<mid-kid>I figured the chroot option already kinda does that but eh

<oriansj>littlebobeep: completely fair but having the latest and great shouldn't be a requirement to being able to bootstrap.

<fossy>never replace the kernel is not going to be a long term option

<fossy>appart from deving in chroot mode

<littlebobeep>oriansj: lol I don't think you need 100TB to bootstrap I am just saying to pay that much and get sub-SATAIII speeds on AHCI sounds lame you are just paying for the rediculous capacity

<stikonas[m]>fossy: i sorted out some qemu problems while on the train

<stikonas[m]>Removing almost all stuff from /sources before unpacking linux helps

<stikonas[m]>It removes a few hundred MiB

IRC channel logs

2022-04-13.log