IRC channel logs

<fossy>stikonas, is there a reason apart from "it didnt work" why we manually go through the autoreconf steps for each subdirectory in gcc 4.7.4?

<fossy>i remember that there was a reason, but i can't remember what

<stikonas[m]>GNU autogen

<stikonas[m]>Same in binutils

<stikonas[m]>Top level Makefile.in is autogen template

<fossy>oh- yeah; (i also forgot autoreconf does subdirectories automatically :P)

<stikonas[m]>It was a bit annoying to regenerate all those subdirectories but it is what it is...

<stikonas[m]>Maybe it is better in newer GCC

<fossy>it seems we should be able to replace 95% of it with the one autoreconf invocation which is lovely

<stikonas[m]>That's good

<stikonas>ok, so far bwrap mode seems to work

<stikonas>but still progressing to build newer python versions

<fossy>good

<stikonas>we do have a lot of old python stuff left on sysc filesystem...

<stikonas>but it's not a new problem

<fossy>yeah, same thing applies to other packages

<fossy>its not too hard to remove if we wish though

<stikonas>well, yeah, we could always extend build function to remove some previous package

<stikonas>before installing new one

<stikonas>but this is not urgent

<fossy>yes, disk space is cheap and we use little of it comparatively

<fossy>i think sysc is only about 4G at the current end

<fossy>i guess it uses more in intermediate stages, but that's irrelevant

<stikonas>fossy: hmm, I'm a bit confused python 3.3.7 does not launch if I try to run it mid-build

<stikonas>yeah Python 3.4 build seems to be progressing

<stikonas>s/yeah/yet/

<stikonas>Fatal Python error: Failed to read bytes from /dev/urandom

<stikonas>Aborted (core dumped)

<stikonas>well, we'll see once 3.4 finishes...

<fossy>wth

<fossy>it should work, i ahve invoked all of them manually before

<fossy>that failed to read bytes may be a bwrap thing, given its using a device

<stikonas>yes, but I'm surprised that it 3.4 built

<stikonas>and with the correct hash

<stikonas>we should probably understand this before merging

<stikonas>(and I need to do a bit more reviewing anyway, perhaps over weekend there will be more time)

<stikonas>fossy: it must be bwrap...

<stikonas>I unpacked tarball to my Gentoo system and ir runs fine

<stikonas>perhaps bwrap parameters need tweaking

<fossy>how strange

<stikonas>well, on Gentoo I do have /dev/urandom

<stikonas>so no more bwrap restrictions

<stikonas>but bwrap does bind-mount /dev/urandom inside filesystem namespace

<fossy>hmm, i'll try running a bwrap build too

<fossy>what happens if you cat /dev/urandom > /dev/null

<stikonas>fossy: stuck

<stikonas>(not the build, the build just finished, cat is stuck)

<stikonas>oh, but perhaps that is expected

<stikonas>and I'm still not sure if we should keep it uppercase Python

<stikonas>fossy: even stranger...

<stikonas>I unpacked python3.3 package after bootstrap is complete

<stikonas>and it does run fine

<stikonas>(just unpacked package tarball into random subdir

<stikonas>and HOME=$PWD usr/bin/python3

<stikonas>so really wth...

<stikonas>oh, I know

<stikonas>I chrooted into the system while it was running

<stikonas>that must be it

<stikonas>instead I must be doing bwrap with bind mounts

<stikonas>indeed, that's the reason

<stikonas>so I think we are fine on this front

<fossy>ah, that's good to hear

<fossy>oh, yeah, uppercase python - i'm ok with changing it. i'll make that change now

<ekaitz>fossy: wanna talk via IRC better?

<fossy>ekaitz: oh hey! i do not know how i missed you in this channel, i was looking for your IRC handle

<ekaitz>fossy: I'm not always connected, that's why you might have issues to find me

<stikonas>yes, ekaitz has been here for a while, probably close to 2 years

<fossy>i recognised the name

<ekaitz>the name is not a common one lol

<stikonas>ekaitz originally wrote hex0 for riscv

<stikonas>(before I did the rest of stage0-posix steps)

<fossy>i will be tuning into your fosdem talk, ekaitz

<fossy>one day i'll go to an in person fosdem, but it would be very expensive for me haha

<fossy>(australia flights to europe are Not cheap)

<ekaitz>fossy: it's a virtual one, I won't visit fosdem either

<ekaitz>you are located in australia! OMG

<fossy>oh that's cool, didn't realise they opened up to virtual talks, although i don't follow fosdem very closely apart from the talks

<fossy>yea mate (:P)

<ekaitz>anyway, as a summary stage0-posix -> mes -> tinycc (mes) -> tinycc -> gcc 4.6.4 -> gcc 7.5 -> modern gcc

<ekaitz>that's the goal, more or less

<ekaitz>janneke can give us more detail

<fossy>7.5 is first gcc w/ upstream riscv?

<ekaitz>yes 7.5 is the first

<fossy>right

<ekaitz>i backported the 4.6.4 and the tinycc (mes)

<ekaitz>but we need to package everything

<stikonas>and tinycc can build gcc 4.6.4?

<fossy>reading through your blog, i was unsuprised to see you spent lots of time bashing with build systems

<stikonas>somehow I had trouble getting anything newer than gcc 4.0.4 in live-bootstrap

<fossy>build systems often cause more strife than the actual code in my experience :P

<ekaitz>stikonas: i've been told it can but i still don't believe it

<fossy>stikonas: wasn't there something binutils related?

<stikonas>I don't remember

<stikonas>one would have to try again

<ekaitz>build systems are my limit here, that's why I'm involving more people in the process to help make the chain

<ekaitz>also in gcc 4.6.4 there are some things missing surely

<ekaitz>but we'll find out

<stikonas>well gcc 4.6.4 -> gcc 7.5 -> modern gcc should be simple enough

<ekaitz>yes

<stikonas>it's tinycc -> gcc 4.6.4 step that I'm worried about

<ekaitz>me too

<ekaitz>we had 2.95 in the middle before

<janneke>ekaitz: that's right, but we aren't there yet, notably building the first glibc-2.16 using gcc-4.6.4 still needs work

<ekaitz>but we want to remove that one

<ekaitz>janneke: these guys in live-bootstrap went with musl maybe that is interesting for us too?

<janneke>ekaitz: yes, i've been using musl for debugging mostly

<janneke>iwbn if we could do without musl, we need glibc anyways

<stikonas>in the worst case you could build glibc later

<janneke>sure

<janneke>and as a prototype bootstrap, certainly

<fossy>irc logs indicate that 4.1.2+ was discounted as an option for *some* reason but that reason wasn't elaborated at the time

<janneke>that's nice to know

<janneke>yeah, before settling on 2.9.3 and 4.6.4, i tried *many* gcc-3/gcc-4 and glibc versions

<stikonas>the 2 reason why we tried to use musl is 1. very simple build system that does not need autoconf/automake (live-bootstrap tries not to use pre-generated files including configure before autoconf/automake are available) 2. You immediately get to resonably new libc and don't need long chain of ancinet->old->new versions

<janneke>yes, esp. 2. is very attractive

<stikonas>indeed, makes bootstrapping simpler

<janneke>we would want a minimum of "ancient" packages in the bootstrap

<stikonas>though it adds a jump from musl to glibc later on but in guix that should be fairly simple

<stikonas>due to installation into /gnu/store prefixes

<fossy>yeah, retracing historical steps is not particularly ideal, but is often much faster than custom solutions

<stikonas>on a simimar note about ancient packages, at some point I briefly looked at whether we could update some other packages. In particular I looked at tar 1.12 -> 1.13. meslibc is currently unable to build 1.13 though it is only missing a couple of functions

<stikonas>but then I got busy with other things, so haven't implemented those

<janneke>stikonas: right

<fossy>i think there are many versions that are a bit arbitarily chosen because the time/benefit ratio is quite low to determining the best versions

<janneke>there's possibly some patches on wip-gcc4 on my gitlab that could be useful

<janneke>stikonas: what benefit does 1.13 bring?

<fossy>eg. linux kernel, i chose 4.9.10 out of the 4.9 series almost arbitarily - i knew 4.9.10 worked, 4.9.200 didn't, choose one that works

<fossy>could bisect it but it would take a looong time

<stikonas>probably nothing critical but let me look at changelog

<stikonas>(I possibly wanted to reach the version that understands bz2 without --use-compress-program=bzip2

<janneke>right

<fossy>hrm, has compiling tcc mob been investigated before?

<fossy>i wonder if that would unlock GCC versions

<fossy>although, tcc code is kinda gross, not sure if i would want that can of worms

<janneke>not recently that i know of

<stikonas>hmm, yes, perhaps 1.13 brings better bzip2 support but anyway, that's a minor thing, it works with --use-compress-program anyway

<stikonas>and now even stage0-posix has unbz2

<janneke>a couple of years ago, i tried the bootstrappable-tcc patches on latest tcc git but couldn't get it to work

<stikonas>tcc mob might be better for building gcc

<fossy>by bootstrappable-tcc patches, the ones applied to 0.9.26?

<janneke>yes -- ah i misunderstood

<janneke>you prolly mean bootstrappable-tcc -> tcc-mob (instead of tcc-0.9.27)

<fossy>yea or even bootstrappable-tcc -> tcc-0.9.27 -> tcc-mob

<stikonas>and it would also be good to long term improve amd64 support

<stikonas>otherwise we'll have a problem in 15 years

<ekaitz>didn't you guys experience tcc-mob being a little bit... unreliable

<stikonas>some ancient stuff is gets broken when clock resets

<janneke>yeah, iwbn though to have bootstrappable-tcc patches on latest tcc...

<fossy>ekaitz: how do you mean?

<fossy>stikonas: including musl 1.1 series

<ekaitz>fossy: lately it didn't pass its own tests in guix

<stikonas>fossy: yes, e.g. perl crashes

<ekaitz>and I have to choose another older commit

<fossy>ekaitz: oh. yes.

<fossy>unsuprising

<fossy>it's the weirdest development model constantly breaking

<fossy>that is why i am not sure if i like the option very much

<fossy>anyway, i am off to bed, thanks for chatting everyone

<ekaitz>i've also had issues with the main dev there being an asshole to me

<ekaitz>i wish we had another reasonable c compiler to use instead of tcc tbh

<stikonas[m]>ekaitz: somebody here suggested another path

<stikonas[m]>Oh but perhaps still uses tcc

<ekaitz>sad

<stikonas[m]>Has anybody looked at https://github.com/rui314/chibicc

<stikonas[m]>Though the problem is of course building the compiler with mescc

<ekaitz>i know the project but never payed too much attention to it

<ekaitz>also cproc looks promising

<ekaitz>> I wrote this code not for me but for first-time readers.

<ekaitz>duuuuude this is awesome

<ekaitz>i'll take a look to the code to see if this is actually true :)

<ekaitz>it also says the book was supposed to be published in 2021

<ekaitz>maybe it's published?

<ekaitz>lcc is also interesting but probably not for our goals. I bought the book a year ago or so and it's cool but it's focused only in the compilation step (no preprocessor or stuff like that) and it's also focused in speed more than anything else

<stikonas>no idea about the book or chibicc, I just found it online

<stikonas>you are more familiar with mescc capabilities than me

<stikonas>and even if we can go from mescc to chibicc, we still need to find bootstrap path to gcc

<ekaitz>probably we won't be able to build gcc with it

<ekaitz>unless we go for a 2.95 or something like that

<ekaitz>i think i found the book: https://www.sigbus.info/compilerbook

<doras>oriansj, stikonas: is it possible that this logic doesn't return the error code correctly? https://github.com/oriansj/stage0-posix-x86/blob/562b24c0267a0403224fbfecb45604e9c2653921/kaem-minimal.hex0#L173

<stikonas>doras: yes, I think on x86 there is a bug

<stikonas>possibly crashes are not handled correctly

<stikonas>might need to investigate it when i have some time

<stikonas>doras: could you describe your issue a bit more?

<doras>stikonas: we see that build failures in live-bootstrap are detected since "ABORTING HARD" is printed, but we exit code 0 from `kaem-optional-seed`.

<doras>stikonas: here we leaked an environment variable by accident: https://gitlab.gnome.org/-/snippets/5205/raw/main/bootstrap-failure

<doras>but we get*

<stikonas>doras: I could try to take a look but might not have time this weekend

<stikonas>(hex0 code is not easy to edit)

<stikonas>hmm, at least /bin/false case works

<stikonas>doras: any chance you can find out what's the exit status of full kaem?

<doras>You can also see it here: https://gitlab.com/freedesktop-sdk/freedesktop-sdk-binary-seed/-/jobs/3622657139#L3698

<doras>stikonas: what do we need to use it?

<doras>Well, what do we need to change to use it?*

<stikonas>perhaps I should explain a bit more

<stikonas>so right now we have a proces tree in live-bootstrap

<stikonas>there is kaem-optional-seed, that at some point in bootstrap runs full

<stikonas>though I think full kaem later execs directly into bash

<stikonas>basically what I want, is to get the exit status of child process

<stikonas>that kaem-optional processed incorrectly

<stikonas>I could try to reproduce it with exec -w

<stikonas>(that is in the error in the pipeline)

<doras>Well...

<doras>You can see "ABORTING HARD" twice in the outputs.

<stikonas>doras: ok, I can reproduce it

<doras>The first after "Subprocess error 2", which I would guess means the exit status is 2. Though I'm not certain about this one.

<doras>The second says "Subprocess error" without 2, which means we lost the exit code somehow?

<stikonas>yes, that is because kaem-optional-seed launches full kaem as child process that then launches other stuff

<stikonas>"./bootstrap-seeds/POSIX/x86/kaem-optional-seed test.kaem" exits with 0 where test.kaem contains exec -w

<stikonas>that is good enough for testcase

<stikonas>AMD64 version works fine

<stikonas>so x86 and AArch64 are broken

<stikonas>amd64, riscv32 and riscv64 works

<stikonas>doras: running live-bootstrap with "./bootstrap-seeds/POSIX/AMD64/kaem-optional-seed kaem.x86" is likely to workaround your issue

<doras>stikonas: we'll try it

<stikonas>anyway, I'll try to fix x86 version

<stikonas>probably not AArch64 version

<stikonas>I'm still not that familiar with aarch64

<doras>stikonas: will running with the AMD64 seed change the hashes of live-bootstrap build products?

<stikonas>doras: no

<stikonas>not a single output hash will be changed, not even intermediate x86 artifacts

<doras>stikonas: great. We'll try it. Thanks! :)

<doras>Finding bugs in seeds is always fun.

<stikonas>oh I was already aware that there is something wrong in x86 result parsing, but I thought it's only triggered on crashes

<stikonas>we briefly discussed it with oriansj a couple of weeks ago

<pder>fossy, stikonas: I created a pull request for live-bootstrap that eliminates the need for make 3.80 and uses make 3.82 which we use later instead

<stikonas>pder: thanks!

<stikonas>that was on my todo list for some time but never got to it

<stikonas>actually make is one of those things that really benefit from newer versions earlier

<stikonas>but 3.82 will for for now

<pder>i havent investigated much yet, but make 3.82 created a slightly different util-linux, which appears to be more correct that with 3.80

<pder>make 3.80 generated an extra file in /usr/share/man/man8 named simply .8

<pder>with make 3.82 that strange file is no longer created

<stikonas>hopefully make 3.82 wouldn't cause any new problem for fiwix

<stikonas>but given that we run it at the end of bootstrap, it should be fine

<stikonas>(at the end of sysa)

<pder>is any of the code that builds fiwix pushed anywhere?

<stikonas>pder: I don't think so

<stikonas>it's more of a case of live-bootstrap running on fiwix rather than building fiwix at the moment

<stikonas>fiwix build is still wip

<stikonas>and the whole thing needs patches

<stikonas>so at this point probably don't worry about fiwix

<stikonas>pder: just out of curiousity, have you tried removing second build of make?

<stikonas>most likely we want to keep it, but I'm wondering how accurate is that second description in parts.rst

<pder>I didn't try eliminating it. The first make build would be with tcc and mes libc, whereas the next make build is with gcc and musl

<stikonas>yes, I know, which is it's best to keep it

<stikonas>it's also built with autotools and configure, so likely to have more features

<stikonas>anyway, I merged it

<stikonas>and also merged python PR

<pder>Not sure what the random segfaults with make would be, but could be related to memory leaks, especially building something as large as the linux kernel

<stikonas>could be bugs in mes libc too

<pder>cool, thanks. now that we have python, what does that unlock?

<stikonas[m]>Possibly meson

<stikonas[m]>Though there is C implementation called muon

<stikonas[m]>Also might help bootstrap portage

<fossy>stikonas, pder: we do need second make 3.82 build for Linux kernel, for some reason mes version segfaults in dependency resolution

<stikonas[m]>OK, so it was mes causing segfault

<stikonas[m]>Well that was my guess too

<stikonas[m]>fossy: and do you remember what was preventing newer kernel? Binutils?

<fossy>stikonas[m]: tbh, i'm not completely sure; i think it was binutils, but it may have been something else

<fossy> https://nullprogram.com/blog/2023/01/18/ claims pkg-config is not bootstrappable.

<fossy>i am having a lot of difficulty verifying this claim, does anyone know what the story is here?

<fossy>we don't bootstrap it in live-bootstrap

<sam_>fd.o one is crap anyway

<sam_>(and needed glib, or bundled it)

<sam_>pkgconf should be fine

<sam_>it has no dependencies other than for tests

<sam_>i honestly am very confused by that post. he wrote one a few weeks ago about SDL, where he encouraged using sdl2-config. didn't mention pkg-config except in a throwaway line. now he writes a pkg-config impl?

<sam_>i'm not convinced

<fossy>hm, perhaps we replace pkg-config with pkgconf in live-bootstrap, i'm not a fan of the bundled glib either

<sam_>we dropped fd.o pkg-config entirely from gentoo maybe 2 years ago. nobody complained. fixed a bunch of bugs in the process.

<sam_>i think debian is finally moving now too

<sam_>(we defaulted to pkgconf for years before)

<fossy>well, if debian is moving, then i think that's a sign that something's time is finalyl up :P

<fossy>hm, do we enable pie/ssp for GCC 10 in live-bootstrap? that would move it in line with modern standards, but i have a feeling mixing pie/non-pie has a fair chance to break things, and 32-bit takes a significant performance hit from ssp - maybe ssp/pie is something we leave for later/to distributions?

<stikonas_>pkgconf did not work in live-bootstrap

<stikonas>at least without further fixes/patches

<stikonas>some packages needed pkg-config

<stikonas>still, I don't understand that blog

<stikonas>pkg-config can be built without pkg-config as proven in live-bootstrap

<stikonas>we just had to use embedded glib as proper one does need pkg-config

<stikonas>fossy: why does mixing pie/non-pie cause problems?

<stikonas>I guess the final binary is non position independent

<stikonas>even if parts are position independent

<stikonas>anyway, fell free to not enable it

<stikonas>as live-bootstrap is not a end-user distro

<stikonas>but it only a tool to bootstrap other stuff

<sam_>stikonas: yes, the page is nonsense

<sam_>i don't think they know much about pkg-config

<sam_>my impression is they learned about it in the last week

<sam_>(I'm not saying that to be mean, but given they barely mentioned it in the post last week (and promoted an inferior tool) and now this..)

IRC channel logs

2023-01-18.log