IRC channel logs

<gbrlwck>so we now have a full source bootstrap for riscv64 (at least up to mes)? next step would be compiling guile with mes? and then we can bootstrap a full guix system on top of that?

<gbrlwck>stikonas: how did you test c compilation? just `./scripts/mescc foo.c` ?

<stikonas[m]>I've used mescc.scm from live-bootstrap

<stikonas[m]>The one in scripts is almost identical but templatefmd values were nit filled not filled

<stikonas[m]>Also grabbed config.h from there but that can probably be replaced with -D define

<stikonas[m]>You'll also have to get nyacc and define GUILE_LOAD_PATH

<stikonas[m]>gbrlwck: next step is porting mescc

<stikonas[m]>Guile is far far away

<stikonas[m]>Once mescc is ported, you can 1) rebuild mes with it (optional but easy) or try to port tinycc to RISC-V (hard)

<stikonas[m]>Guile has quite a bit of dependencies

<stikonas[m]>And is best left after GCC is bootsrapped

<stikonas>gbrlwck: basically we'll have to more or less follow https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst

<stikonas>so tinycc will be the first tricky step (right now it doesn't work even on amd64)

<stikonas>then after that it would probably be not too hard until binutils (although, old autotools versions might not like risc-v too but probably fixable)

<gbrlwck>tcc is a synonym for tinycc, right?

<stikonas>I think so

<stikonas>but we have 2 versions of it

<stikonas>one is very old 0.9.26 patched by janneke

<stikonas>so that it can be built with mescc

<stikonas>but patches only work on x86

<stikonas>tcc is actually quite capable compiler

<stikonas>at leats that newer tcc 0.9.27

<Hagfish> https://utcc.utoronto.ca/~cks/space/blog/programming/GoVersionOfYourSource

<Hagfish>"The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present."

<civodul>argh

<Hagfish>it would be nice if it also indicated that every commit was signed by a PGP key

<Hagfish>civodul: do you think that will make it non-reproducible?

<civodul>Hagfish: i think commit IDs are the new timestamp :-)

<civodul>the change less often than timestamps, so they're less of a problem from a reproducibility viewpoint

<civodul>but still, they're "silent metadata"

<civodul>but i don't know, there are also advantages to embedding provenance metadata like this (as --save-provenance in Guix)

<Hagfish>hmm, nice

<Hagfish>i think that commit IDs should (ideally) change exactly as often as the code itself changes

<Hagfish>i guess someone might create version 1, then version 1.1, then remove that code and call it version 1, even though it is later in the revision history

<Hagfish>in other news, the Qubes-OS people have tried rebuilding the packages in Debian buster and have confirmed that 92.1% of them are reproducible:

<Hagfish> https://debian.notset.fr/rebuild/results/bullseye_full.amd64%2Ball.png

<Hagfish>a little less than the number that Debian's own reproducible builds effort calculated

<civodul>yes, nice!

<civodul>actually Debian was building slightly different packages than those people actually use

<civodul>the numbers were about those

<civodul>kinda confusing

<civodul>also, the Qubes OS folks are starting from pre-built binaries from snapshot.*

<civodul>so my understanding is that they're not rebuilding "recursively"

<Hagfish>ah, interesting

<Hagfish>it's slightly worrying if "recursiveness" is what causes the difference

<Hagfish>that would mean that a non-reproducible change is infecting builds further down the line, i.e. the output of a system isn't dependent on its source + input

<gbrlwck>stikonas: https://gitlab.com/janneke/tinycc shows version 0.9.27 ?

<stikonas>that's not the branch that you can build with mescc

<stikonas>I guess you might want https://gitlab.com/janneke/tinycc/-/tree/mes-0.23.8

<vagrantc>/29/6

<stikonas>gbrlwck: there is also this snapshot that is known to work on x86 https://lilypond.org/janneke/tcc/tcc-0.9.26-1136-g5bba73cc.tar.gz

<stikonas>I guess that's commit g5bba73cc..

<vagrantc>Hagfish: the difference with the builds from debian.notset.fr and the builds at tests.reproducible-builds.org are that debian.notset.fr builds the packages with the same toolchains that the packages in the debian.org archive used, whereas the tests.r-b.org builds use whatever happens to be currently in the relevent debian release

<gbrlwck>`git checkout g5bba73cc` does not seem to work (on that source tree)

<vagrantc>both of which are basically using binary seeds of "what was once in the debian.org archive" or "what is currently in the debian.org archive"

<stikonas>gbrlwck: oh, remove first g

<stikonas>it's 5bba73cc

<vagrantc>Hagfish: also the tests.r-b.org builds intentionally vary a bunch of arbitrary things to trigger reproducibility, the debian.notset.fr tries to minimize differences ... neither project really does any bootstrappable sort of stuff

<gbrlwck>:) thanks

<gbrlwck>they all show version 0.9.27, btw

<gbrlwck>is mescc more than invoking mes with a mescc.scm script?

<stikonas>gbrlwck: well, mescc.scm script doesn't have much logic in the file itself

<stikonas>it loads mescc module from modules dir

<stikonas>so depends on what you mean by "more than"

<stikonas>if you mean do you need to run anything else than no

<stikonas>it's just mescc.scm that you need to run

<stikonas>but under the hood it loads much more code

<stikonas>mes is just scheme interpreter

<stikonas>and all the C code is split between parser (nyacc) and mescc module

<gbrlwck>makes sense :) i was wondering if there was another repo involved which i couldn't find

<stikonas>well, nyacc

<stikonas>and you need specific commit

<stikonas>I think it's not backwards compatible

<stikonas>1.00.2 from https://download.savannah.gnu.org/releases/nyacc/

<gbrlwck>so, since we can not /compile/ scheme yet, i simply have to make the nyacc modules available in the load path of mes, right?

<stikonas>yes

<stikonas>well, mes is an interperter

<stikonas>there is nothing to compile

<stikonas>mes itself is a C program and can be compiled

<stikonas>but at the moment you can only recompile mes for x86

<gbrlwck>yeah, just bc nyacc's make actually compiles the modules (and install probably copies them to $GUILE_LOAD_COMPILED_PATH)

<stikonas>if you want to rebuild it on riscv64 you need to first port mescc to it

<stikonas>there is some initial work done in mes branch

<stikonas>oh, make probably just uses guile

<gbrlwck>well, guild, but yes

<stikonas>guile does some processing from scm files to guile objects, but I'm not very familiar with it

<stikonas>yes, probably guild...

<Hagfish>vagrantc: thanks for the extra context

<stikonas>Hagfish: well, we also rebuild some stuff twice in live-bootstrap

<vagrantc>it's a project i'm following with much excitement :)

<stikonas>so there are some bugs that propagate a bit

<Hagfish>it's interesting that t.r-b.org deliberately tries to trigger non-reproducibility, and yet i think they also achieve a higher percentage of reproducible packages

<stikonas>(most likely due to bugs in C library)

<Hagfish>stikonas: intriguing

<Hagfish>ah yes, that makes sense

<stikonas> https://github.com/fosslinux/live-bootstrap/blob/master/parts.rst e.g. see 20 and 22

<stikonas>first tcc has float handling messed up

<stikonas>so if binary snapshot used for building is older, having to rebuild everything twice is not entirely surprising

<stikonas>but yes, it does look a bit dodgy

<stikonas>at least when you do that in bootstrapping it's a bit less dodgy

<stikonas>since everything in the end is built from source

<Hagfish>right

<vagrantc>Hagfish: possibly due to toolchain fixes introduced since the original package build

<stikonas>especially considering that toolchains often does source optimization

<stikonas>so if you have newer toolchain, it will compile packages differently

<vagrantc>well, debian.notset.fr is one build with old toolchain compared against another build with old toolchain, and tests.r-b.org is one build with current toolchain compared against another build with current toolchain

<vagrantc>but the current toolchains have fixes that fix reproducibility issues ... nobody's expecting bit-for-bit identical with different toolchains (other than with projects where that's a design goal like mes)

<Hagfish>yeah, good point

<stikonas>well, even different versions of mes and M2-Planet are not guaranteed to produce the same bit-for-bit identical binary

<stikonas>if we need to fix codegen bugs

<stikonas>and adjust assembly output, then binaries will change

<vagrantc>(i meant mes itself)

<stikonas>Mutabah seems to be doing good job with mrustc. Almost done with rust 1.54. Which is still fairly new

<fossy>yes, tcc is synonym of tinycc

<gbrlwck>yeah, it's also written in README

<gbrlwck>does linux-libre build reproducibly?

<fossy>yes

<stikonas>fossy: we don't do checksums for it

<stikonas>so how do we know?

<stikonas>at least by default kernel does have timestamps (compile date) but I don't remember if you removed that

<stikonas>and initramfs is definitely not fully reproducible bit-by-bit because we don't strip timestamps from it

<fossy>oh wait I meant to add that

<fossy>it is reproducible, I checked

<fossy>initramfs is not

<stikonas>but the contents of initramfs should reproducible, I think it's only metadata that differs

<stikonas>at the very least all binaries and sources in it are checksumed

<gbrlwck>nice

<oriansj>stikonas: to the question of what is the future of mes-m2; well that is hard to answer. Ideally porting to multiple architectures but I doubt it will be backported into Mes as M2-Planet builds don't comply with the GNU build standards. (lack of autotools and all that jazz)

<stikonas>well, I guess it's fine keeping it separate...

<stikonas>although then the question is how to merge mes updates back

<stikonas>from upstream into mes-m2...

<oriansj>as for what go is doing. seems like just minor metadata additions. Personally if I were to go that route, I'd just add a source section to the ELF binaries themselves (trivial to do in M2-Planet) and distribute the source code used in making the binary in the binary itself.

<stikonas>by the way, that diff I posted has #if defined at the end...

<oriansj>well mes.c doesn't get updated much, but MesCC bit certainly does.

<oriansj>odd ctrl-f search didn't find the word defined at all

<stikonas>e.g. https://github.com/stikonas/mes-m2/commit/740da2816ac1aa7c46c64c574f0a4e69427b7e91#diff-d1913f7376be639b02e96c98895f96726df7d5f8abcd4e9331ed4f6c63f95c25R24

<stikonas>maybe I gave the wrong branch

<stikonas>well, mes-m2 actually doesn't have to deal with mescc much

<stikonas>as long as we can build mes-m2, we can then switch to upstream mes

<stikonas>and use mescc from there

<stikonas>I think mescc is somewhat compatible across mes versions

IRC channel logs

2021-10-27.log