<OriansJ`>The closer you get to GCC, the more LFS documentation becomes a better and better reference. <OriansJ`>Although I wonder if the LFS community might be interested in a FULL bootstrap book <stikonas>well, we basically have everything to build gcc now (once we merge what we have in local git repos) <stikonas>well, maybe some minor things will be needed... <stikonas>but tcc with binutils and musl should be quite good <stikonas>and once we have gcc, like you say, it's just LFS <stikonas>although, it might be that binutils will introduce some non-reproducibility... <Hagfish>i like the idea of "linux from scratch from scratch" :) <Hagfish>i also think that each distro should create an appendix to LFS which documents how to build an ISO image of their minimal install image <Hagfish>apparently a debian install image is "generally 150-300 MB" <Hagfish>it should be possible to build that image from source, bit for bit, from an LFS system <Hagfish>i think there is a contradiction, though: "This is the last version buildable with 5.004." vs "5.6.2 is the last version buildable with 5.004." <stikonas>I think you have just volunteered to do a PR <Hagfish>i want to say i don't have an account, but it's more true to say that i shouldn't use my work account <Hagfish>i could download the file and create a diff for you, if that would be acceptable :) <Hagfish>i'm happy for you to receive the credit for that epic patch :) <pabs3>Hagfish: I would have thought distro ISO images would require tools from the distro, so the path to build the ISO should be hex0 -> distro build-essential -> ISO build-deps -> ISO <pabs3>for Debian, the debian-cd package is how the ISOs are built <stikonas>and for binary distros the point is moot anyway <Hagfish>pabs3: yeah, i guess there are 2 complementary philosophies: distros should be self-hosting, and also the distro's build essentials should be buildable from the simplest possible binaries <pabs3>Debian already has a cross-bootstrapping project, so I think would be amenable to full bootstrap <stikonas>then there is at least hope of reproducing exactly the same binaries <Hagfish>in the back of my mind i have this worry that trying to build, say, the Debian package of gcc, or the kernel, from an LFS system will fail in some subtle way because the system isn't "Debian" enough <stikonas>well, official packages are built in Debian chroot... *pabs3 would like to see hex0 etc in every Debian apt mirror so you could bootstrap Debian solely from a source-only apt mirror <Hagfish>stikonas: i don't think it's documented how to build a Debian chroot from source on a non-debian system, right? <Hagfish>i guess you are supposed to run debootstrap or something, in a Debian machine? <pabs3>debootstrap doesn't require Debian machines and is available in Fedora/etc <stikonas>debootstrap runs anywhere, but it downloads binary debian packages <pabs3>its very hacky though, it hardcodes ordering information, the apt/dpkg folks are working on a spec to eliminate that <Hagfish>maybe there needs to be a debootstrapbootstrap which downloads the debian source package and turns them into binaries using the non-debian host system <pabs3>but yes, getting from LFS to the first .deb would be the hard bit, I think Daniel Schepler has been doing work on that <Hagfish>thinks keep moving in the right direction, though, i'm pleased to see <stikonas>that's why I was thinking we should first pick easier targets... <Hagfish>stikonas: something intermediate between LFS and Debian? <stikonas>source based distros rae probably as easy as it can get... <pabs3>what is the status of hex0 -> GuixSD? <stikonas>hmm, not sure if we can produce guix bootstrap binaries... <stikonas>and guile also has some bootstrap problems if I recall correctly <fossy>stikonas: what do we need 2.13 for <fossy>stikonas: and there are no binary or binary like files for any of these yeah? <stikonas>fossy: 2.13 and 2.12 is needed for binutils <fossy>odd, two different versions lol <stikonas>original binutils configure scripts are done with mix of 2.13 and 2.12.1 <fossy>and none are generated by the build system? (I mean fully, not trivial sed) <stikonas>fossy: well, autoconf automake and friends are produced <fossy>Acrually ill wait a couple for the ci <stikonas>fossy: it might need more work to make binutils create reproducible executables, in case they insert time or date... <stikonas>but I think we can at least build them first and then iterate from there <stikonas>and binutils 2.14 seems the be the limit for tcc... <Hagfish>(although i say that based on my ignorance of the details of both processes) <stikonas>well, maybe at some point later in live-bootstrap we can have distro "plugins" <stikonas>where live-bootstrap branches to bootstrap chosen distro <fossy>that is basically my plan from the start <fossy>why does ur tar file have to use all these gnu extensions which we dont yet support <stikonas>well, some other tar files also used them but those were optional <stikonas>so just unnecessary file in extracted dir <fossy>stikonas: yeah, these are key source files :| <fossy>however, the git snapshot is fine <stikonas>fossy: although, I suspect some of them might have problems with reproducibility <stikonas>then I'll rebase on top of it and make PR for reviewing <fossy>stikonas: yep, merged, ygot that tick <fossy>stikonas: also, gentoo bootstrap is trivial (although slightly messy); <fossy>you just have to use gentoo prefix <fossy>create a prefix, ./bootstrap-prefix.sh, then use the prefix to create a stage3 and extract that in a real root <fossy>alterntaively we can attempt to use catalyst <fossy>well catalyst does need a seed tarball <fossy>but one can probably use a prefix tarballed as the seed <fossy>identation is off in the for loops <fossy>where does pder deserve attribution? <stikonas>fossy: pder wrote configure command and those alloca hacks <fossy>yeah could you add pder to SPDX-FileCopyrightText <fossy>nope, not in binutils-2.14.sh <fossy>i assume the touch command makes them be regenerated? <fossy>and the absence of the files <stikonas>in principle touch should be enough even without removing <stikonas>fossy: maybe you can also clean up some old branches? <stikonas>number of branches in the main repo is getting quite large <stikonas>and what about those 4 sed lines. I guess they are acceptable? <fossy>the TODO makes it even finer <stikonas>we have used similar hacks in other places <fossy>stikonas: also, your checksums need full paths, preferably <stikonas>fossy: hmm, looks like the build is not fully reproducible <stikonas> /after/bin/as: FAILED and /after/bin/objdump: FAILED <stikonas>well, I'll push full path change for now to checksums <stikonas>fossy: oh, I'm missing a few other files <stikonas>there is also /after/lib/libopcodes.{a,la} <Hagfish>if there were a way of trading shares in developers, i'd be fully invested in stonksinas <Hagfish>well, i'd have a portfolio consisting of all the developers here :) <OriansJ`>Hagfish: there is a way of trading shares in developers. It is called a vested co-op. <OriansJ`>or for those more sports minded: fantasy software development <OriansJ`>in mescc-tools news; debug for ppc64le is now working <plasma41>Recursively vendored libraries are driving me nuts. All I wanted to do was play the new release of 0AD after making sure I could properly compile it using only system libraries. <fossy>plasma41: i know right, i hate them *fossy shakes fist at go and rust <plasma41>fossy: I haven't even gotten to the flying circus that is Spidermonkey yet. I'm still only wrestling with premake. <plasma41>premake5 comes bundled with curl, libzip, lua, mbedtls, zlib. <plasma41>I'm using Devuan, which doesn't have premake5 packaged yet, so as a starting point I took a look at the Guix premake5 package definition. What I found was that Guix just uses the bundled libraries instead of properly bootstrapping them. (╯°□°）╯︵ ┻━┻ <OriansJ`>well yes, Guix cheats alot because it is less work for them. <OriansJ`>But when we unpack all that shit for them and wrap it up nicely for them, they will use it. <plasma41>OriansJ`: Which is really sad to see when at the same time by contrast Guix provides a full bootstrap chain for the ouroboros that is rust. <OriansJ`>plasma41: where do you think that bootstrap chain came from? Developers here did al the leg work. Because if it wasn't for us, it would be just like the GHC situation in Guix. Just one big fat binary blob <plasma41>Indeed, and I'm trying to unwind the madness, but it's so frustrating when I just keep finding layer upon layer. <OriansJ`>all of the Guix developers who care about bootstrapping are here. <OriansJ`>and sadly, we are outnumbered by those who just want their toy or app or feature. <OriansJ`>So there is eternally churn to keep things bootstrappable <plasma41>OriansJ`: Don't get me wrong, I fully credit you and janneke and fossy and the others in this channel for Guix being at the level of bootrappability that it is. Y'all have done and continue to do amazing work <OriansJ`>plasma41: and I am sorry that despite everything we are slowly getting further behind. <OriansJ`>substitues are too easy of a default and hide all of these bootstrapping details from guix users. <OriansJ`>So most guix developers skip bootstrap chain work that would have been easy but instead punt it to those of us who care and I am sorry that you are stuck working on it plasma41. I just hope you upstream the work into Guix so that others don't deal with that same pain too. <plasma41>It's just a bit disheartening to look in Guix and see amazing achievements in bootstrapping like Rust, Java, and Haskell right next to packages like fpc (Free Pascal) and premake5. <fossy>plasma41: bundled binaries, or bundled source? <fossy>bundled source gets me annoyed slightly but bundled binaries make me very annoyed <fossy>plasma41: free pascal is very hard <OriansJ`>which honestly shouldn't even be called source <plasma41>fossy: bundled sources in premake; bundled binaries in fpc <fossy>plasma41: someone here was looking into pascal <fossy>the best route seems to be to go back to the good old days of GCC Pascal <fossy>and attemtp that to bootstrap fpc <fossy>OriansJ`: i agree 100% on generated source <OriansJ`>I vote we call "generated source" cheat-blobs <plasma41>Regenerating generated source is far too often much less straightforward than it should be. <OriansJ`>plasma41: especially because sometimes it doesn't actually correspond with the cheat-blob at all <OriansJ`>or generated years ago and never updated to match the "source code" <OriansJ`>or generated based on source that isn't included at all <OriansJ`>or generated from malicious sources that are never shown but we are told it matches these sources over here <OriansJ`>only for us to later discover: that too was a lie <plasma41>"It's reproducible though. I can use the cp command to prove it." /s <Hagfish>OriansJ`: perhaps there is a term like "compiler larping", based on your funny comment from before <OriansJ`>plasma41: or It's reproducible (on my system) because we just copy the system copy of GCC <OriansJ`>Hagfish: well traditionally the phrase around here would be: Blowing smoke out of one ass. In regards to pretending to do something you are actually not. <OriansJ`>But somehow I don't see that as being catchy enough <Hagfish>it would be nicer to have a one-word term to describe the fakeness of other approaches, which is why i liked your choice of "larping" so much <Hagfish>i don't want to get political, but there are terms like "alternative facts" and "fake news" and "sharpiegate" <Hagfish>"phantom code" kinda works, because you're not sure whether the source code really exists or not <Hagfish>i'm trying to think of a type of monster that convinces you to trust it, and then reveals it was never really there, or something <Hagfish>" Čert is trying to bring evil characters to hell, he often helps or befriends heroes in this process and gives them various magical items and treasures. " ***efraim_ is now known as efraim
***mephista is now known as spicy_icecream
***terpri_ is now known as terpri
<stikonas>fossy: oh, that objdump and as non-reproducibility issue is actually just sha256sum choking on 2MB files <stikonas>Hagfish: it breaks gforce_de1977's sha256sum program <stikonas>since we are not using sha256sum from binutils yet <bauen1>sha256 could also be used for source tar files, so a non-existant limit would probably be best ? <stikonas>bauen1: that's not what we use sha256sum for... <stikonas>at the moment we only checksum tars outside live-bootstrap <stikonas>and inside it we checksum binaries/libraries <stikonas>but yeah, maybe eventually we can checksum source tar files too <stikonas>thers is no reason why not even though we don't do it right now <bauen1>stikonas: i mean, having sha256 available (for any amount of data you need) gives you one cryptographic primitive that can be used to "skip" reviewing code if you've already reviewed it once, and it is a lot less code to audit than coreutils <Hagfish>thanks for the link. it seems that it doesn't actually check that the file fits within 2MB, and nor does it output an error code/message to the user if that assumption breaks <Hagfish>i can't work out what would need to change for it to read an arbitrary amount of data in <bauen1>also using fgetc when you have a fixed buffer is quite not as efficient as it could be <stikonas>it might be simple to split executables that we want to hash into chunks using "split" <Hagfish>would the hashing code call that directly, or would the split happen first and then the parts specified in order when calling the hash function? <Hagfish>the latter would sort of be backwards compatible, if you wanted to preserve the behaviour of it giving the wrong answer on big files (but that would be weird) <stikonas>another option is to backport sha256sum from coreutils 6 <stikonas>so hopefully patch applies without too much trouble <bauen1>stikonas: but you can have sha-2 a lot sooner than coreutils ? <stikonas>well, we can use current broke implementation from amosnier <bauen1>i'm wondering why the sha2 single file implementation doesn't expose "sha2_init", "sha2_dowork", "sha2_gethash" or something like that, might be a bit more complicated but then you've covered basically every usecase <nckx>‘all of the Guix developers who care about bootstrapping are here’ is unprovoked nonsense, OriansJ`. <nckx>Conversely, you've made 0 documented effort to report the problem with premake5 anywhere I can find it, yet find the time to rant about it here. <nckx>Colour me unimpressed with your ‘leg work’ here. <nckx>I hope you find the time to do something more productive. As you say: it's easy work! <pder>stikonas: great work on autotools and binutils. I'll have bash cleaned up shortly and I'll rebase it on top of binutils for now <OriansJ`>nckx: you are absolutely right. I don't document problems for guix anymore. I just document solutions when I find time to make them. Which I figure is more productive use of my time. <OriansJ`>stikonas: if you look at my last paste, it is a tweak to gforce_de1977's sha256sum program to read the files before allocating a buffer to hold their contents (and added support for sha256sum's -c file behavior) <OriansJ`>nckx: as for the existence of Guix developers who care about bootstrapping that are not here; you are right it might be entirely unprovoked nonsense. But it is just an observation from my limited experiennce; which very much may be wrong. <civodul>again, i don't see the need for an us/them kind of discourse ***Noisytoot is now known as [[
***[[ is now known as Noisytoot
<stikonas>OriansJ`: thanks, although I also got sha256sum from coreutils now <OriansJ`>civodul: exactly we have shared goals and benefit from our mutual cooperation. <OriansJ`>stikonas: well that is even better. Good job ^_^ <stikonas>well, wasn't too hard, since I am only building one single binary from coreutils 6.3... and was able to reuse most of the stuff from 5.0 <fossy><stikonas> maybe I can just bump the limit... <fossy>regarding guix, idk how many ppl here are guix devs <stikonas>just bumping the limit somehow didn't work for me... <fossy>my local diffoscope said they were identical <stikonas>doesn't matter... I've got coreutils sha256sum now <fossy>stikonas: i wonder if theres an overflow of an int or smth <stikonas>that's why I'm more keen on using old gnu tools <fossy>anyway idk how many ppl are guix Devs but I extremely doubt any bootstrapping project would have got any traction without lots of support from all devs <stikonas>yeah, although strictly speaking those guix devs that are in this channel can't take that comment personally, since by definition they are in this channel and thus excluded <stikonas>but yeah, I think bootstrapping wouldn't have been that far without support from all devs... <OriansJ`>well Guix devs are the majority, followed by nix and Debian. <OriansJ`>But the majority people here are not Distro Devs <stikonas>and hopefully I've got all generated files... <stikonas>hopefully will just work, but we have a lot of .l files <nckx>OriansJ`: Hi! <I don't document problems for guix anymore. I just document solutions [...] more productive use of my time> That's reasonable. Your time is yours alone. <nckx>I'm just pointing out you spent it doing the former, not the latter. <nckx>Complaining about the problem here to make a point, without making anyone who could actually fix it aware of it. <nckx>Like we're not on the same team. *vagrantc hedges bets and plays on several teams :) <stikonas>fossy: ok, fixed binutils build now, luckily everything just worked, pushed PR now <nckx>vagrantc: And, weirdly, they all seem to be winning. How can this be? 🌈 magic. <stikonas>nckx: anyway, if you want to be informed, live-bootstrap now has some useful tricks that might be nice to adopt in guix (at least in longer term) <stikonas>but yes, without guix it would have been much harder to do that. live-bootstrap borrowed a lot from guix bootstrap <nckx>From my (super limited) understanding it seems more orthogonal, but that's good too. <stikonas>but we are also bootstrapping gcc and friends <stikonas>and tried to do it in a bit more strict way <stikonas>e.g. no pregenerated files (e.g. bison or lex generated files) <stikonas>(actually not even using configure scripts until we manage to regenerate them) <nckx>We all agree that should be nice to do in Guix as well, it's merely a lot of work. <stikonas>although, there is an example to follow now. Just as we followed Guix too :) <nckx>So I'm very happy that someone (you?) is demonstrating that it's feasible. <stikonas>well, not just me, and we built on others work... <nckx>It's certainly interesting. <stikonas>pder: before you rebuild bash, we probably need to figure out where binutils inserts non-reproducibility