IRC channel logs

<vagrantc>stikonas: good luck with the mouse/upgrade :)

<matrix_bridge><Andrius Štikonas> Its not mouse upgrade, its Qt upgrade

<matrix_bridge><Andrius Štikonas> But now with Wayland I guess input devices can be affected too

<vagrantc>right, i just meant good luck with the mouse && good luck with the upgrade :)

<vagrantc>the other "release" key for mescc-tools i have is for oriansj but it has expired ... and looks to be intentional

<jcowan>The PDP-8 has no fixed calling convention except the JMS instruction itself, which places the current PC at the target address and jumps to the target + 1. Sometimes what follows is pointers to the arguments, sometimes the arguments themselves.

<matrix_bridge><Andrius Štikonas> vagrantc: yeah, that's the first release made by me...

<vagrantc>overall, mescc-tools 1.5.0 (for reals) seems to be building fine on debian: https://salsa.debian.org/debian/mescc-tools/-/pipelines/601198

<vagrantc>will upload to debian shortly.

<vagrantc>i guess i should find and subscribe to the bootstrappable list...

<vagrantc>i ought to be able to handle the 10-20 mails per year :)

<vagrantc>stikonas: although reviewing 2023's list archive, i do not see mescc-tools announced specifically ... maybe part of some other release announcement for mes?

<vagrantc>ok, on to mes itself....

<mihi>vagrantc, https://www.freelists.org/post/bootstrappable/stage0posix-release,1

<vagrantc>mihi: e.g. part of some other release announcement :)

<mihi>vagrantc, yeah, I gueesed you might have been interested which one without looking into every one :)

<vagrantc>mihi: thanks :)

<stikonas>yeah, stage0-posix came with all other releases too...

<stikonas>we also have stage0-posix tarball too which contains everything

<vagrantc>well, i've got enough to get a new mes into debian, i think

<stikonas>bootstrap-seeds/mescc-tools/M2-Planet/mescc-tools-extra/M2-Mesoplanet

<vagrantc>oh, i need to enable riscv64

<stikonas>yep, riscv now works

<mihi>stikonas, vagrantc, mescc-tools is somehow special in the stage0 project that it is not only consumed by stage0-posix but also by mes...

<vagrantc>i haven't actually poked at stage0 much ... only go as deep as mes so far. someday. although the prospects of Debian ever adopting a full source bootstrap seem ... unlikely at best :/

<stikonas>yeah, mes is based on the same assembler/linker

<stikonas>vagrantc: 1.5.2 now uploaded to savannah

<stikonas>still need to do github

<vagrantc>doh. i guess i should have waited :)

<vagrantc>some fun for another day....

<vagrantc>stikonas: thanks though! that will be less confusing in the end. :)

<vagrantc>ACTION waves

<stikonas>yep, and will help janneke a bit too

<stikonas>as previous release had a broken shebang in the test script

<stikonas>ok, pushed to github too

<oriansj>vagrantc: well I am open to removing submodules iff such that it would result in a simpler and easier to get correct system for people who are not familiar with the tools in question.

<tmg1|michelson>for what it's worth, my rescued-form-github fork of ninja will try to stay only dependent on python just for you as long as i can keep it https://git.freecumextremist.com/themusicgod1/ninja-build

<matrix_bridge><Christoph> vagrantc: What keeps debian from adopting full source bootstrap?

<oriansj>Christoph: their current dependency graph probably, it'll take a while to unwind all of it: https://wiki.debian.org/DebianBootstrap

<oriansj>so a clean bootstrap will probably come easiest from the outside via a functional package manager (like guix or nix) building deb files

<ztrawhcse>tmg1|michelson: there is no bootstrap problem with ninja at all, as your own link already points out

<jcowan>oriansj: And after that it is a political decision. If someone gets a Thompson attack into an important distribution, adopting source bootstraps will be quick.

<jcowan>or if someone important is willing to spend political capital on getting them in.

<oriansj>jcowan: yeah, solving human organizational problems is so much harder than the technical problems involved in bootstrapping

<ekaitz>jcowan: so the best thing we can do to make our work relevant is to trust-attack a major distro??

<ekaitz>;)

<jcowan>In the sense that if you want to raise the GDP of your country, you should go around breaking as many windows as possible, thus triggering the economic activities of making and installing replacements, yes.

<ekaitz>jcowan: :)

<oriansj>ekaitz: we don't even need to do that, we just need to point out trusting trust attacks that already exist

<Googulator>Is 2.19 really the latest version of util-linux we're able to use in live-bootstrap?

<Googulator>It would be nice to use a version that isn't obsessed with CHS geometries and starting/ending partitions on cylinder boundary.

<oriansj>Googulator: nothing is the latest we can use, improvements are always possible.

<oriansj>it is just a matter of effort and doing the work required to improve the situation.

<Googulator>I did found a workaround to get 2.19 to use proper alignment (echo "2048;" | sfdisk -uS -S32 -H64 "/dev/${DISK}" instead of just echo ";" | sfdisk "/dev/${DISK}" in sysb/run.sh), but 2.26 and up will just do the right thing by default

<Googulator>without this, the ext4 fs will start at offset 0x200, which is misaligned (if you have an AF HDD or an SSD), and also leaves no space for grub (that's another thing I'm working on, so after bootstrap, we can boot back into sysc at any time and do more compilation or auditing in a trusted environment)

<Googulator>I already solved bringing the Linux kernel image into sysc's file system, will post the code for that soon

<oriansj>thank you Googulator, we look forward to your improvements.

<matrix_bridge><Andrius Štikonas> Googulator: no, its likely that newer util-linux will work

<matrix_bridge><cosinusoidally> Thought this might be of interest. I've been working on an alternative bootstrap path for tcc https://github.com/cosinusoidally/tcc_bootstrap_alt . I've still got work to do, but I have now got to the point where I can generate tcc 0.9.10 using only cc_x86 and M2 (and technically a JS dependency atm, but that can be fixed by porting a ~120 line JS program to C). Process is essentially this:...

<matrix_bridge>... ./mk_js_to_c_cc_x86 ; mv tcc_js/tcc_boot.o tcc_10/ ; ./mk_loader_m2 . This process generates an unlinked copy of tcc 0.9.10 and deposits it in tcc_10/tcc.o . Next step is to write an elf loader/linker in the cc_x86 dialect of C, plus enought of a libc to allow tcc 0.9.10 to run and build tcc 0.9.23.

<matrix_bridge><Andrius Štikonas> Especially since we updated binutils since then

<Googulator>matrix_bridge: "The latest version is not used because of autotools/GCC incompatibilities." - I wonder what version introduces those incompatibilities.

<Googulator>If it's 2.27 or later, then we can just trivially switch to 2.26.

<stikonas>Googulator: well, you'll have to try

<stikonas>by the way, pinging matrix_bridge wouldn't work... it's a bridge

<stikonas>Googulator: generally we have reasonably new autotools by then, though not the latest

<stikonas>gcc so far is a bit old (4.0.4)

<stikonas>most likely that one can also be upgraded

<stikonas>4.6 migth be of particular interest as that is what riscv64 port will target

<Googulator>About that alternative TCC bootstrap path: do we actually use the Scheme side of mes for anything other than running mescc?

<Googulator>If not, we can drop mes altogether from the bootstrap path, and just go straight from m2 to tcc

<stikonas>Googulator: we will

<stikonas>but we are not doing it yet

<stikonas>at least in live-bootstrap we'll switch to gash for some early steps

<Googulator>what does gash get us?

<stikonas>it will solve one licensing issue that we have

<stikonas>right now we build heirloom tools (yacc)->bash->musl

<stikonas>heirloom tools ins not redistributable

<stikonas>if we do

<stikonas>gash->musl->heirloom tools (yacc)->bash then we are fine

<stikonas>bash/gash are needed to run (non-autotools) configure script of musl

<muurkha>heirloom tools is not redistributable?

<muurkha>I thought they'd been properly open-sourced at last. is that wrong?

<Googulator>Looks like they're 4-clasuse BSD

<stikonas>muurkha, Googulator, not it's not heirloom tools, it's heirloom tools + meslibc

<stikonas>heirloom tools are CDDL

<Googulator>& meslibc is GPL I guess, which is a problem when they get linked

<stikonas>exactly

<stikonas>and bash has some parser .y file that would be nice to rebuild

<stikonas>well, rebuild .c file generated from .y file

<Googulator>And I guess Bison has some dependency that prevents its use at that stage

<stikonas>yeah, bison bootstrap is far more complicated

<stikonas>it requires at the very least more capable libc

<stikonas>so we need musl

<stikonas>also I think some shell scripts are used there

<stikonas>Googulator: oh, and heirloom tools themselves

<stikonas>that's how we bootstrap bison

<stikonas>well, both flex and bison

<Googulator>& I guess we need a more capable shell than kaem for musl

<Googulator>musl should be linkable with CDDL code, right?

<stikonas>kind of

<stikonas>musl is linkable, yes

<stikonas>there is anoher workaround, not to use ./configure and create config file manually

<stikonas>but given that gash will be coming anyway

<stikonas>then we can just use it

<stikonas>also gash might be usable for building mes/tcc too

<Googulator>problem is, gash is written in Scheme

<Googulator>and right now, running Scheme in mes is d o g s l o w

<stikonas>yes...

<stikonas>oriansj might be able to make it a bit faster

<stikonas>though musl's configure script is fairly short

<Googulator>which is why I was hoping for this alternative tcc bootstrap path with no Scheme dependency

<stikonas>I don't think it will add signigicant time for bootstrap

<muurkha>oh, CDDL is fine on its own, just not in combination with GPL, sure

<stikonas>Googulator: alternative path is interesting, but I think it's disadvantage would be it's x86 only

<stikonas>s/it's/its/

<Googulator>with strategically no /g at the end :)

<stikonas>:)

<stikonas>anyway, mes might be a bit faster once M2-Planet supports switch

<stikonas>and mes.c restores it's switch statements that were once replaced with if/elseif

<Googulator>BTW, in the Guile bootstrap, is mes still being used?

<Googulator>(happens quite late in sysc)

<stikonas>no, mes is not used at all after musl

<stikonas>and after tcc, only meslibc is used

<stikonas>mes is not even transferred from sysa to sysc

<Googulator>Because right now, it's the single slowest part of bootstrap, specifically when it starts touching Scheme code for the first time

<stikonas>Googulator: well, slowest part early on

<Googulator>I mean the Guile build

<stikonas>isn't GCC13 far slower?

<stikonas>but yes, guile build takes a while

<Googulator>could be - haven't had a successful GCC13 build yet...

<stikonas>Googulator: try running all of this on riscv64 (we only reached first tcc right now) and all those x86 builds will feel blazingly fast

<Googulator>compiling the very first Scheme file took 30 min for me on a very modern AMD machine

<Googulator>slower than the whole Linux kernel

<Googulator>for that one file

<stikonas>Googulator: with kernel bootstrap?

<stikonas>and it's probably a second scheme

<Googulator>yes, building Linux in Fiwix takes less time than compiling a single Scheme file during the Guile bootstrap

<stikonas>mes-m2 is the first one that is build with M2-Planet

<stikonas>M2-Planet compiles very quickly, though it's binaries are slow

<stikonas>btu yes, I know what you mean...

<stikonas>part of it is builder-hex0 being also slow

<stikonas>bootstrap in chroot/bubblewrap is a bit faster

<Googulator>and that's with Linux building on a single thread, while Guile compiling that one single Scheme file (within sysc, way after builder-hex0 or anything could be relevant) spins all 8 Zen 4 cores at 100% for half an hour

<stikonas>my 8 year old laptop can build to tcc in about 30 minutes

<Googulator>I'm not talking about the mes build

<stikonas>oh, that guile bootstrap thingy tha mihi wrote

<stikonas>that's trying to solve some precompiled files in guile source issue

<Googulator>GCC 10 also builds faster than a single Scheme file during the guile bootstrap

<stikonas>somethign to do with preexpanded macros

<stikonas>GCC13 will be slower as it is rebuilt with itself (--with-bootstrap)

<Googulator>don't all GCCs do that?

<stikonas>still, I'm a bit surprised that that Guile file takes 30 minutes

<stikonas>Googulator: no, it's an option in ./configure

<stikonas>most distros do that

<Googulator>I'm guessing it's hitting some kind of worst case on AMD

<Googulator>related to Spectre or similar

<stikonas>still, that is a bit strange

<Googulator>& beyond that point, everything feels slower than before, even after Guile is done building

<Googulator>Until gcc13 runs out of memory and I'm dropped to a Bash shell

<stikonas>hmm, how much memory do you have?

<stikonas>it shouldn't take too much

<stikonas>although in 32-bit you are limitted to 3 GiB anyway

<Googulator>I'm giving 8G to qemu, but /proc/meminfo only shows 3G free

<Googulator>3G total I mean

<stikonas>yeah, you won't get more on 32-bit machine

<Googulator>probably 32-bit restriction, although it's weird since the kernel's built with PAE

<Googulator>Right now I'm doing another bootstrap, this time with a swap file early in sysc/run.sh

<Googulator>PAE should support 64G

<Googulator>With only a 4G/process restriction

<Googulator>(or 3G/process because of the user/kernel split)

<stikonas>well, that only helps with parallel builds

<Googulator>I'm building on 8 cores / 16 threads right now, so maybe that's why it's running out of memory there

<stikonas>perhaps PAE is misbehaving?

<Googulator>I'm guessing either qemu isn't reporting PAE support to the guest

<Googulator>Or maybe the 4.9 kernel doesn't properly detect PAE support on AMD in 32-bit mode

<Googulator>tbh contemporary Intel CPUs aren't even officially supported in any 32-bit environment, although AMD is supposed to be a bit more lenient here

<Googulator>How does the Linux kernel even receive the memory map in our setup with kexec-linux from Fiwix?

<jcowan>What about using Solaris rather than GNU toolkit programs?

<jcowan>it is CDDL

<Googulator>Toolkit isn't the issue

<Googulator>meslibc is

<Googulator>which isn't easily replaced

<Googulator>& it's strictly GPL with no linking exception, unlike other libc implementations

<muurkha>yeah, I don't think Solaris has a libc that can be compiled with toolchains from 40 years ago

<Googulator>not even "from 40 years ago" at that point, but rather minimal toolchains specifically made for bootstrapping

<muurkha>well, I was thinking of heirloom-tools

<muurkha>aren't those from 40 years ago?

<stikonas>well, using gash/musl is the easiest way to replace meslibc for heirloom-tools

<janneke>Christoph: i'd say a debian developer who wants to attempt it

<janneke>it should be possible to patch dpkg so that it can build a package that ignores build-essential and instead specifies all of its build dependencies

<janneke>then, as a first step, all packages that are part of build-essential could be rewritten and be bootstrapped

<janneke>for the rest of debian, i defer to oriansj's answer :)

<stikonas>janneke: by the way, I've released mescc-tools 1.5.2 with that fix to test.sh

<stikonas>(the one where shebang was broken)

<janneke>stikonas: lovely, thanks

<oriansj>although the meslibc CDDL issue could be resolved by moving musl earlier and building flex/bison with it instead. But stikonas would know better if I missed something obvious

<stikonas>oriansj: yeah, if we move musl before heirloom tools then it would be resolved

<stikonas>though we need at least heirloom tools for bash

<stikonas>which is why gash would help us with this issue

<Googulator>Looks like the extreme Guile slowness was memory pressure - adding swap seems to have helped.

<oriansj>which means it would be an ugly kaem script for the tcc build of musl but it could potentially be done instead of waiting for gash

<Googulator>With 8G swap, less than 2 hours after starting the VM, I'm already bootstrapping Python

<Googulator>Python bootstrap done just before the 2-hour mark, now building gcc 10

<Mikaku>Googulator: the memory map is passed to Linux using the Linux boot protocol: <https://github.com/rick-masters/Fiwix/blob/fiwix-1.4.0-lb3/kernel/kexec.c#L915-L928>

<Googulator>(with the slight blemish that the trick I did to bring the Linux kernel into sysc failed due to an incorrectly called tar command, so we saved some time there by not creating that tarball)

<Googulator>Mikaku: That explains it, since Fiwix censors the BIOS memory map, dropping anything in PAE address space (https://github.com/rick-masters/Fiwix/blob/fiwix-1.4.0-lb3/mm/bios_map.c#L126)

<Mikaku>Googulator: yes, Fiwix doesn't support PAE

<Googulator>worse than just "doesn't support", it blocks anything it kexecs from using its own PAE support either

<stikonas>Googulator: how about if we kexec again into linux?

<stikonas>does that block remain?

<Googulator>Yes, because now Linux gets the BIOS memory map censored by Fiwix, rather than the original one

<stikonas>I see

<stikonas>so only cold reboot would help...

<stikonas>or maybe warm reboot but not kexec reboot

<stikonas>hmm, still if we have memory issue, then it's probably not worth building with so many threads

<stikonas>even -j1 only takes 3h or so, so maybe just -j2 would work better for you

<Googulator>Seems like Fiwix's map censoring is quite easy to fix

<Googulator> https://github.com/Googulator/Fiwix/commit/42f8af6c221e3c6c49739e5cea0325d2f161941b first attempt

<Googulator>Fiwix did boot with this patch, but unfortunately it's not the only place where we're losing memory

<Googulator>kexec-fiwix.c actually constructs a fake memory map, and that's the one Fiwix receives, and eventually passes on to Linux

<Googulator>For reference, this is the current diff to live-bootstrap I'm running with: https://paste.linux.chat/?06da0f3839d325fe#Dmo5MoXKWcUgRkZm3VhZ4qewrr6u2jkXHnrfxT2v3YLB

<Googulator>(the Fiwix URL there is not valid yet, I just put the modified tar.gz in distfiles)

<stikonas>Googulator: openssl fails to download?

<Googulator>yes, redirects to HTTPS URL

<Googulator>I've made an issue about this on GitHub

<Googulator> https://github.com/fosslinux/live-bootstrap/issues/327

<Googulator>[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000001fffffffe] usable

<Googulator>YES!!!

<Googulator>Mikaku: https://github.com/rick-masters/Fiwix/pull/1

<Googulator>(submitted to Rick's repo, but should also apply cleanly to yours, except for the bits related to Linux kexec targets)

<Googulator>this will also need a fix to kexec-fiwix to use the actual BIOS-provided memory map instead of its own fake one - for now, I band-aided over this by hardcoding 4GB of available high memory right above the non-PAE address space

<vagrantc>stikonas, oriansj: the clean target in mescc-tools is missing a couple things: https://salsa.debian.org/debian/mescc-tools/-/blob/debian/latest/debian/patches/0001-makefile-Call-cleanup.sh-for-test12-and-test13.patch

<stikonas>vagrantc: merged

<stikonas>hopefully you don't need a new release urgently...

<vagrantc> stikonas: yeah, no release needed. just wanted to cleanup the tag/tarball confusion :)

<vagrantc>i am fine for applying a few patches usually, but i did not want anyone else to go through the labrynthe of waht happened with 1.5.0 and 1.5.1 again, if it could be avoided. :)

<vagrantc>stikonas: and thanks!

<vagrantc>probably have a similar patch i need to work on for mes

IRC channel logs

2023-11-12.log