IRC channel logs

<stikonas>well, stage0 is machine code (not interpreter) and it's not an optimizing compiler

<oriansj>stikonas: I was more thinking of how grateful I am that you, fossy and janneke (with help from all of our friends) managed to pull together the whole world of software together just using my little hobby. That rickmasters and Mikaku have and are doing beautiful work solving the kernel bootstrapping problem. That civodul, rekado and the whole guix team have and continue to map out the path for new and more software than one can ever

<oriansj>need.

<janneke>oriansj: likewise, well said

<avih>oriansj: so what's up with the "strict" hex0 spec? i want to fix the hex0 sh script, but first i wanted to understand what's up with the spec. if you wanna drop it that's fine too, just let me know.

<avih>also, even if you don't take the strict spec too, i think current spec is worth clarification in the form of more formal spec (which i have too)

<oriansj>avih: sorry, yes. I like it and will be adding it and will be clarifying the hex0 standard this weekend.

<stikonas>oh I'm not complaining about mes, just saying that it's probably expected that stage0-posix is much faster than mes

<stikonas>interpreters are slow by nature unless they are JIT (but then they are essentially compiling stuff at runtime)

<avih>oriansj: no worries. you mean you'll take the strict spec? or only the more formal current spec?

<stikonas>and JIT is unnecessary complication for bootstrap

<avih>gforce_de1977: what are the various tcc versions here? http://intercity-vpn.de/bootstrap/memplot-memhack-24-999M.png

<avih>(specific commit hashes of tcc-mob? or from someplace else?)

<oriansj>avih: well yes and I was thinking of writing hex0 up as a formal grammar

<stikonas>avih: no, we don't use tcc-mob

<stikonas>it's tcc-0.9.26 for the first build and then tcc 0.9.27 for the rest

<stikonas>well, strictly speaking we build tcc 0.9.26 five times

<stikonas>but it's all in one build script, so you don't see it in that graph

<avih>stikonas: so how is tcc-0.9.27 different from tcc-0.9.27-7 for instance? where are these diffs?

<avih>i mean, i presume there are diffs, and it's not just the same source recompiled, right?

<stikonas>not sure why it's called -7 there (probably gforce_de1977 added that number to distinguish between different builds but his numbering is not sequential)

<stikonas>avih: there are some patches

<stikonas>but also often it is recompiling the same source

<avih>oh

<stikonas>the toolchain does matter

<avih>what's the value in that? (recompiling)

<stikonas>to fix bugs/issues

<stikonas>e.g. we build musl then tcc then musl then tcc again

<stikonas>if you just build musl and tcc, lots of things will still be somewhat buggy (maybe floats?)

<avih>so the 1st tcc is with whatever libs, and a later one is with musl?

<avih>libc*

<stikonas>at least parts.rst claims that https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst

<stikonas>well, let's start with mescc

<stikonas>we use it to build patched tcc-0.9.26 with lots of features disabled (using -D flags)

<avih>right

<stikonas>then we rebuild it 5 more times

<stikonas>each time building it with more features

<stikonas>e.g. addings flags like (-D HAVE_FLOAT=1 -D HAVE_LONG_LONG=1)

<stikonas>and also rebuilding mes libc each time

<avih>fwiw, i maintain a (currently non public for no particular reason) a script which builds a reproducible tcc for windows, where you get an identical set of files on any system (win/linux/bsd/etc) starting with any compiler (gcc/clang/msvc/tcc/etc)

<stikonas>then we build first version of tcc 0.9.27

<avih>the gist is probably not unlike parts of bootstrap, where 1st it builds a native tcc which targets windows, and then uses this tcc to build a native windows tcc

<stikonas>this first version is actually not patched at all

<stikonas>(then we build some oldish versions of common POSIX tools

<stikonas>then we build a patched tcc 0.9.27

<stikonas>so we apply some patches that can help us build musl

<avih>right

<avih>what's the gist of the tcc patches? can they be upstreamed?

<stikonas>no

<avih>is it mainly removal of features?

<stikonas>e.g. one of these patches removes an error like tcc_error_noabort("'%s' defined twice", name);

<stikonas>so tcc does not error on dublicate symbols

<stikonas>this is a big of a hack

<avih>hmm

<stikonas>and then we patch musl to convert weak symbols into normal symbols (thus we end up with duplicate symbols)

<avih>i'm guessing that if done properly-ish, upstream will not reject a configuration option to help with bootstrap

<stikonas>to do it properly, you should add weak symbols to tcc

<stikonas>anyway, now we have tcc and musl, but for some reason it's quite buggy

<stikonas>not sure what causes it

<avih>is it part of some spec?

<stikonas>possibly some bugs in mes libc

<stikonas>weak symbols, yes, I think so

<avih>c99?

<stikonas> https://en.wikipedia.org/wiki/Weak_symbol

<stikonas>no, that's for linkers

<stikonas>in C99 it's attribute

<avih>also, possibly musl would accept patches to make it easier to use in bootstrap

<stikonas>again, even if they accept, these are older versions

<stikonas>that are already released

<stikonas>it's harder to build latest musl

<stikonas>we build 1.1.24 (which is not too old, maybe 5 years old)

<stikonas>(but much newer then bootstrap glibc that guix uses, that one is probably over 20 years old)

<avih>right, i guess a bootstrap system does have to rely on older versions of compiler/libc. so basically you maintain these older versions?

<stikonas>we don't have to maintain them anymore...

<stikonas>once we build them once

<stikonas>(unless we want to add e.g. riscv support of something like that)

<stikonas>(but musl 1.1.24 is new enough to support riscv)

<avih>or a bug is discovered which affects some change in the process which you deem useful

<stikonas>anyway, after that we bootstrap bison/perl and autotools

<stikonas>avih: well, then we apply patches

<avih>yeah

<stikonas>if we figure out what causes some bad behaviour

<stikonas>once we have autotools, we can build binutils much easier

<stikonas>and binutils comes with better linker and assembler than what tcc has

<stikonas>so we finally get rid of that hacky weak symbols patch

<stikonas>and build tcc and musl properly

<stikonas>(also removes some other assembly patches that caused tcc to fail)

<avih>"Weak symbols are not mentioned by the C or C++ language standards; as such, inserting them into code is not very portable"

<avih>so i don't think it's standardized?

<stikonas>well, perhaps it isn't...

<stikonas>anyway, there are other patches, not just weak symbols

<avih>stikonas: right, so basically the various tcc-0.9.27-X are the same source, possibly with different patches and/or libc, etc?

<stikonas>e.g. tcc errors out on the following inline assembly "jecxz 1f"

<stikonas>avih: indeed

<avih>gotcha, thx.

<stikonas>so I replaced it with "cmp %ecx,0" followed by "je 1f"

<avih>hmm

<avih>so the assembly is in musl, and it's something which tcc does not (yet?) support?

<stikonas>yes, there is assembly in musl

<stikonas>which is unavoidable in C library

<avih>yeah

<stikonas>it has to do some stack work on initialization. Also call syscalls

<stikonas>which are not part of C

<avih>right. i'm guessing set/longjmp too?

<stikonas>don't remember this one...

<avih>or maybe those are compiler things...

<stikonas>musl patches are here https://github.com/stikonas/live-bootstrap/tree/master/sysa/musl-1.1.24/patches

<avih>the replacement you mentioned specifically is actually part of sigsetjmp :)

<avih>nice, the patches are fairly minimal.

<stikonas>wek symbolsone isthe ugliest...

<avih>i wasn't sure if setjmp is a libc or a compiler thing.

<stikonas>and tcc patches: https://github.com/stikonas/live-bootstrap/tree/master/sysa/tcc-0.9.27/patches

<stikonas>evne fewer

<stikonas>static link patch is optional, we can just call tcc -static everywhere

<avih>so that's for the 5 years old 0.9.27?

<stikonas>yes

<stikonas>2nd patch is probably backport from tcc-mob

<stikonas>and then we use it all to build gcc 4.0.4

<stikonas>but maybe we can now jump to slightly newer gcc

<avih>new tcc version is really needed. 0.9.27 no longer builds out of the box due to usage of some gcc builtin thing which was removed since

<stikonas>when we started we bootstrapped binutils 2.14

<stikonas>avih: but does it matter for bootstrap?

<stikonas>we don't use gcc to build it

<stikonas>or do you say they'll release new version soon

<stikonas>because it can't be built anymore on normal system

<stikonas>having newer tcc might be useful then

<avih>i say i hope they release a new version soon. i don't know how it affects you. if the version you have is enough then maybe not.

<stikonas>anyway, I was saying we used to have binutils 2.14 but now we go straight to 2.30

<avih>right

<stikonas>so that might unblock newer kernels and gcc

<stikonas>still, we should wait for rickmasters work to be merged first

<avih>my point about tcc version was that i think it's important to be able to build newer tcc using older tcc, but currently older tcc doesn't build normally. so possibly a minor patch to work around that for 0.9.27.1 or some such, and then 0.9.28 (or 1.0) which can be built with 0.9.27[.1]

<stikonas>well, I guess this might force tcc team to make a new release

<stikonas>right now fossy is somewhat avoiding random tcc commits

<stikonas>as anybody can push to there without any code review

<avih>i sent an email about that to the tcc list, didn't really get a useful response...

<stikonas>hmm...

<avih>yes, everyone can push, but the main maintainer does clean up occasionally, reverts stuff, etc. so it shouldn't be overlooked

<avih>(and others report issues etc)

<stikonas>yes, so in the worst case we could switch to some checkout

<avih>i think currently mob is quite stable

<stikonas>but then 0.9.27 worked for us for now

<avih>sure

<stikonas>maybe it would be useful if we need riscv support

<avih>where does msvc get into the picture?

<stikonas>but then we need to patch tcc 0.9.26 anyway

<avih>sorrt, i misread. riscv.

<avih>y*

<stikonas>well, if you want to bootstrap on risc-v

<stikonas>live-bootstrap for now is x86 only

<avih>right, iirc riscv was added after 0.9.27

<stikonas>(not even amd64)

<stikonas>indeed

<stikonas>but amd64 bootstrap would probably be easy to fix once mes supports it

<stikonas>for riscv, we only have support in stage0-posix for now

<stikonas>but there is some work in mes too

<avih>mes is gnu, yes?

<stikonas>it is

<stikonas>though it doesn't really matter too much here

<stikonas>most of the components are GNU

<avih>sue

<stikonas>GCC, binutils, bash, bison/flex, autotools, tar, gzip

<stikonas>oh flex might not be

<avih>it doesn't seem like you need to, but many tools might be easier to build from alternative sources, like suckless ubase and/or sbase

<stikonas>but we also build many non-gnu tools

<stikonas>avih: strangely it's often harder to build

<stikonas>you can try...

<avih>some of the implementations there are wickedly clean (and posix compliant), like grep.

<stikonas>but they are often minimalistic tools and rely on full blown libc

<avih>yes

<stikonas>whereas gnu tools often have in tree implementations as they are meant to be more portable

<stikonas>so we can build them with meslibc

<avih>right

<avih>and yes, from my experience typically suckless tools do rely heavily on useful libc

<stikonas>we added a few small tools to mescc-tools-extra (part of stage0-posix)

<stikonas>we found a few simple ones that were easy to simplify further

<avih>well s/useful/compliant/

<stikonas>so we have e.g. untar, ungz, unbz2 even before mes

<stikonas>(also sha256sum)

<avih>right

<stikonas>they are often slower and more limitted than GNU tools

<stikonas>so we replace our old tools with old gnu tools later

<avih>i have a reasonably small base64 if you need (not public yet, but i plan to publish it). it's typically faster than coreutils base64, and much smaller

<avih>(but equally compliant, and even supports more strict spec - canonical, which coreutils base64 does not)

<stikonas>hmm, not sure if we need base64 right now...

<stikonas>we used injected base64 a bit before in the early days of live-bootstrap

<stikonas>initially we only had qemu mode

<stikonas>so it was hard to debug stuff there

<stikonas>e.g. if something is not reproducible

<stikonas>we had to dump it out

<stikonas>but by now, we have an easier way

<avih>that's the source https://0x0.st/Hou9.txt but the url is 404 for now, till i push it.

<stikonas>(just let it run further till it builds linux kernel and mounts disk image)

<avih>(i didn't even add the license/copyright block, but it would be MIT)

<stikonas>this is something that I guess would build with tcc. Wouldn't build with M2-Planet right now but might be easy to fix it (get rid of switch/case and ++ -- stuff, possibly a few other things)

<stikonas>still, I don't recall any use of base64 in bootstrapping right now

<avih>sure

<stikonas>we could do with something simple to build than bash

<stikonas>but that might come to us in the form of gash

<stikonas>there is some work going on to get it to run on mes: https://git.savannah.nongnu.org/cgit/gash.git/log/?h=wip-modular-mes

<avih>you mean shell?

<avih>(not familiar with gasg)

<avih>gash*

<stikonas>yes, shell

<stikonas>gash is some implementation of shell in Scheme

<stikonas>guix uses it quite a bit

<avih>right. is it posix compliant? or some minimal thing?

<stikonas>I think it's good enough to run ./configure scripts

<avih>(i think i have connectivity issues maybe? i can't reach that git url)

<stikonas>so must be reasonably good

<stikonas>(though also very slow)

<stikonas>in principle, we could solve our license compatibility issues if we can run musl configure script (and this one is smaller handwritten script, not generated by autoconf)

<avih>yeah, it's probably useful, though autotools configure is known to cater also for older shells, so it doesn't strictly require posix

<stikonas> https://github.com/bminor/musl/blob/master/configure

<stikonas>but this is far beyond capabilities of our minimal kaem shell

<avih>yeah, this configure would need a fairly good shell

<avih>is this gash? https://github.com/szos/gash

<stikonas>looks like it. but an old fork

<avih>damn, for some reason i just can't reach https://savannah.nongnu.org/ does it work for you?

<stikonas>actually I'm not sure if it's the same gash

<stikonas>let me see if they have another mirror

<stikonas>nongnu.org works for me

<stikonas>avih: I can fork it somewhere for you

<avih>nah, no need, thanks. it's probably this https://savannah.nongnu.org/projects/gash/

<avih>not sure what's up with the connectivity though...

<avih>whatever.

<stikonas>avih: https://git.stikonas.eu/andrius/gash

<avih>thanks. now i can see that :)

<avih>very cool. "Gash is a POSIX-compatible shell written in Guile Scheme"

<avih>i wouldn't call guile light i think. what's the value of using it over dash or some such?

<stikonas>tsample is sometimes here on this channel but very rarely

<gforce_de1977>avih: @oriansj: interesting catch: yes, the numbers behind the package-name is in fact sequential over all packages. for some reasing (?) it starts with flex-2.5.11-X where X is 1 and each package after it raises the number. I once had it hacked into a giant script: https://github.com/bittorf/kritis-linux/blob/main/minilinux.sh#L689 and till bash-2.05b it was without number, after that with

<gforce_de1977>a numbers: ugly - sorry for that 8-)

<rickmasters>gforce_de1977: re: your question a few days ago. I've made changes to support a 1400MB (MB=1024*1024) ram drive on Fiwix.

<rickmasters>However, I'm working on moving to using a disk drive for the Fiwix phase so its probably not relevant.

<rickmasters>The changes to Fiwix for very large ram drives were something I'd rather avoid and we can reduce memory usage by using a disk.

<rickmasters>Mikaku: regarding the kexec for Fiwix, I'm not sure Linux will work with the mechanism you're building.

<rickmasters>Mikaku: But it will probably be a good starting point to learn from and hopefully adapt to Linux.

<rickmasters>For everyone the kernel bootstrap is on hold until I debug problems I ran into when trying to change the Fiwix phase from ram drive to disk drive.

<aggi>greetings

<Mikaku>rickmasters: I'm now consolidating to boot Fiwix using kexec, then I'll try with Linux. I've already made some testings already trying to boot Linux but all them were unsuccessful

<aggi>apropros kernel bootstrap... i digged out some old linux-2.4, and try to revive tcc-compiler with it

<Mikaku>rickmasters: it looks like Linux wants some type of information be passed in some way that I need to learn

<rickmasters>aggi: I have to admit the thought of that has crossed my mind but I've gone so far with Fiwix that I'd really rather not go back.

<aggi>newer kernel version, got too many issues

<Mikaku>rickmasters: I'm a bit concerned with the problems you say you have with disk drives in Fiwix, I'm using it daily and it's pretty stable

<rickmasters>aggi: The state of building Linux with tcc was bad. Real bad. So I just gave up and took a different path.

<aggi>with linux-2.4/tcc-compiler: it compiles (except for tiny bootsect.S parts fed into binutils for 16 bit realmode asm), and it boots, and it crashes... currently

<aggi>yes, that's what i am working on, it's nasty, to debug

<Mikaku>rickmasters: I'll do all my best on Fiwix to help you to not regret this path :-)

<rickmasters>Mikaku: I'm trying to narrow down the problem enough and/or give you a way to reproduce it.

<Mikaku>rickmasters: ok, thanks

<rickmasters>Mikaku: My fork still has a lot of stuff not in your tree and live-bootstrap is huge so it would be a lot to dump on you.

<rickmasters>Mikaku: Basically, qemu exits when it gets to unpacking binutils-2.30 although I've seen qemu exit in other places.

<Mikaku>I hope God will help me when you make this dump on me

<rickmasters>Mikaku: Yeah it would be totally different from what you're use to with FiwixOS and all the tools you use.

<rickmasters>But I should say the recent few weeks of delay are related to me not putting as much time into this. It's not that I'm stuck.

<Mikaku>good to know

<rickmasters>This work of moving to a hard drive is just not as fun because I'm reworking existing functionality rather than blazing new bootstrapping territory.

IRC channel logs

2023-03-25.log