IRC channel logs

<ztrawhcse>jackdk: speaking as meson upstream, knowing about hardcoded lists of compilers is not a rule we are unwilling to bend. It is slightly awkward since a lot of the more advanced features that people want from a build system do in fact require knowing compiler-specific details (actual POSIX requirements on a compiler interface are quite sparse), but probably doable. Unfortunately, to date

<ztrawhcse>everyone interested in implementing a generic interface has flaked out.

<lanodan>Did anyone at meson pinged austingroup (POSIX) about those features? Would make sense to at least standardize some of them.

<ztrawhcse>lanodan: at least POSIX issue 8 standardized how to create a shared library :_

<ztrawhcse>at any rate, we have officially noted that it seems quite reasonable and we will accept patches for, a) refactoring meson's compiler classes to collect knowledge of "a POSIX compiler" in a single interface, that other (hard-coded) compilers can inherit from, b) using that as a basis for defining a "compiler detected as just-use-POSIX"

<ztrawhcse>it is quite possible that a "just-use-POSIX" compiler won't necessarily support all meson features, but if you're using a POSIX compiler perhaps you *do not care*

<ztrawhcse>like

<ztrawhcse>if you're bootstrapping a system with a POSIX compiler I nearly guarantee that you don't care about using meson's sanitizers option, or enabling meson's LTO option

<ztrawhcse>stuff like checking a get_define or has_function may not be reliable, you'll have to check that out, but not all projects need that anyway...

<ztrawhcse>I'd love to see your thoughts on compiler.symbols_have_underscore_prefix(): Returns true if the C symbol mangling is one underscore (_) prefixed to the symbol.

<stikonas>well, even guix and live-bootstrap bootstrap paths don't really care too much about POSIX

<stikonas>well, POSIX is basically a portability standard, so good for some things but when you bootstrap from a very tiny seed, it's simpler to just use all available tools

<stikonas>after all M2-Planet or mescc are also not even C99 compatible...

<fossy>surely not ansi c compatible either

<lanodan>ztrawhcse: Yeah, although one I noticed as missing is -Wl,-soname

<fossy>hi! shortly, https://github.com/fosslinux/live-bootstrap/pull/493 is going to be merged in. this is the beginning of a live-bootstrap distfiles mirror network. if you have a web server with ~2GB free, it would be appreciated if you would consider running a mirror. this is a bit different from a traditional "mirror" as you don't rsync another primary mirror, instead each mirror is

<fossy>individually collating upstream sources. `mirror.sh` in the PR is responsible for doing that, but you could write your own script too. only dependencies are something to get HTTP files with, and git.

<fossy>it's really helpful if the mirror supports plain HTTP as well

<fossy>there is basic instructions in DEVEL.md

<lanodan>Wait… you're sure about `git config tar.tar.gz.command gzip` ? Because git internal gzip command is deterministic, gzip(1) isn't

<lanodan>Heh "curl: (33) HTTP server does not seem to support byte ranges. Cannot resume." thanks git.savannah.gnu.org

<lanodan>Huh gnulib-8f4538a5.tar.gz curl: (22) The requested URL returned error: 400

<lanodan>Which is git.savannah.gnu.org as well…

<stikonas>fossy: I'll try to set up a mirror but probably not today

<matrix_bridge><Andrius Štikonas> lanodan: when is gzip non-deterministic?

<matrix_bridge><Andrius Štikonas> is that across different versions of gzip?

<lanodan>yeah, forges have been bitten by this many times

<lanodan>I'm not exactly sure why but pretty sure it's the reason why git got it's own gzip

<matrix_bridge><Andrius Štikonas> hmm, maybe it is

<matrix_bridge><Andrius Štikonas> I've found some blog

<matrix_bridge><Andrius Štikonas> https://medium.com/@mpreziuso/is-gzip-deterministic-26c81bfd0a49

<matrix_bridge><Andrius Štikonas> replacing the same commands with bzip2 fixes it though

<matrix_bridge><Andrius Štikonas> "gzip -n" seems to work though

<lanodan>Nah this blog article just skips a whole lot of problems

<lanodan>Effectively because reproducible in the same environment doesn't means it's deterministic across different versions.

<matrix_bridge><Andrius Štikonas> yeah, but where else can non-determinism come in?

<matrix_bridge><Andrius Štikonas> it's either metadata

<matrix_bridge><Andrius Štikonas> or compression algorithm changes

<lanodan>Heh well on trying to find it again also leads to re-discovering that the change from gzip(1) to git archive gzip also broke checksums https://github.com/git/git/commit/4f4be00d302bc52d0d9d5a3d4738bb525066c710

<stikonas>hmm, but in principle we don't need to use gzip if we do compression ourselves...

<stikonas>can as well use something more stable, e.g. bzip2

<fossy>lanodan: the reason we do that is because forges generate *their* archives using gzip(1) instead of internal gzip

<fossy>and it is best if the tarballs we generate match upstream snapshots, where they exist

<fossy>in theory, we do not need to do this

<fossy>but practically, until we have more mirrors, i would prefer we do this

<stikonas>fossy: so longer term do we plan to stop using upstream snapshots?

<fossy>stikonas: as in, use our own snapshots that don't match upstream?

<stikonas>yeah... I'm not saying we should do, jus tasking

<fossy>i think we need to discuss that more, i'm not personally opposed to it when we have more people verifying, but i haven't thought much about it

<stikonas>ok, let's leave it to future

<stikonas>I haven't though through it either

<stikonas>fossy: and can you remind me how mirroring works wrt to live-bootstrap commits?

<stikonas>we need to rebuild it for each new commit if tarballs change?

<stikonas>hmm, should just look at that PR again

<fossy>stikonas: so, ./mirror.sh needs to be rerun on each commit, yes. but it's incremental update; it keeps anything around that still matches checksum

<fossy>also, theres an internal "state" directory that holds git repositories to avoid recloning them

<stikonas>yeah ok. Not perfect I guess but at least better than nothing

<stikonas>we need to start with something

<stikonas>it's not completely clear how to make older commits work for longer anyway

<fossy>Hmm, i am confused how git archive is currently deterministic lol

<fossy>stikonas: well, mirror.sh also doesn't delete old distfiles, so any given mirror should work for any commit mirror.sh is run against

<stikonas>I guess it's deterministic from 2023?

<stikonas>oh yeah, that's probably helpful

<fossy>sorry, i mean how mirror.sh's invocation is deterministic

<fossy>because we don't pass -n

<fossy>is -n default?

<stikonas>maybe git archive passes -n

<stikonas>it's not default for gzip

<stikonas>but maybe if you run gzip on pipes, there is no metadata to store

<stikonas>as opposed to running on files

<fossy>oh - that is probably the case

<stikonas>well, maybe in the future we'll still implement p2p solution for mirroring, then older commits might still be covered by some of the mirrors

<stikonas>though having http mirror is good too in certain cases

<stikonas>e.g. if you are behind some firewall

<fossy>yeah, i would love for the p2p idea to be implemented some day, but i think this is a reasonable starting point

<fossy>mainly want to increase the robustness of obtaining distfiles currently, which i think this does well

<stikonas>yeah, and we'll have more copies

<stikonas>in case some upstreams disappear

<stikonas>or move urls...

<stikonas>e.g. openssl was often breaking

<stikonas>ok, running mirror.sh now

<stikonas>maybe you can test my mirror once it is setup

<fossy>mine is at http://samuelt.me/pub/live-bootstrap/

<stikonas>fossy: http://live-bootstrap.stikonas.eu/ (https is also there)