<OriansJ>fossy: "well, finish your exams first :)\nperl can wait" <= this <Hagfish>human civilisation has been waiting thousands of years for this bootstrap, and, with any luck, it will have thousands more years to be grateful <Hagfish>i'm sure they won't mind you taking some time off every now and then :) <pabs3>the post claims a need for RISC-V membership, but I think bootstrappable could easily bypass that, Debian folks are <OriansJ>the question is who is willing to be responsible for the RISC-V porting work (I'll provide help but I lack the time to do the later stages) <stikonas>later stages жill be particularly tricky for risc-v... <OriansJ>well there is already someone working on porting MesCC to risc-v; Hopefully someone form the TCC community steps up and saves us that work. <OriansJ>musl libc, binutils and GCC would be the big pieces of work. <OriansJ>but one can crib/steal from the later risc-v port to GCC/binutils <stikonas>hmm, that's actually one advantage of going via musl rather than old glibc <stikonas>although, maybe we'll need old glibc too later... <stikonas>but maybe C++ is patchable to build with musl <OriansJ>stikonas: well what was the oldest version of GCC with risc-v support? <vagrantc>you can bootstrap straight to a recent musl version? <OriansJ>hmm risc-v support isn't in FSF GCC but a seperate fork <vagrantc>pretty sure that's a mostly abandoned fork <OriansJ>correction it does not appear that that they backported to 4.7.4 <stikonas>unless somebody can build gcc 4.7.4 directly <OriansJ>So RISC-V backporting to 4.0.4 and/or 4.7.4 would be potentially a full time job worth of effort <OriansJ>So we can with the resources available get from stage0 to TCC; then we depend upon the TCC team to get us to the GCC/binutils and then we will need sponsorship to fund the backporting work for GCC <OriansJ>not ideal but a task that would have been considered impossible just 5 years ago when we started. <stikonas>well, risc-v is a new arch, and in practive even fairly recently bootstrap dependended (and to a large extent still depends) on a lot of old GNU software <stikonas>TCC might need some backporting to to 0.9.26... <stikonas>but that should be much easier than GCC backporting <OriansJ>not if MesCC expands to be able compile TCC LATEST directly <OriansJ>I wonder if the RISC-V porter for MesCC would be willing to help add those features; then when TCC adds RISC-V support, we win for free. ***mephista is now known as spicy_icecream
***ChanServ sets mode: +o rekado
<bauen1>OriansJ: doesn't tcc mob already have some risc-v support ? i've never tried it but occasionally the mailing list gets a few emails regarding risc-v support <bauen1>also if tcc could actually do a new release at some point and cleanup their code, that'd be awesome ... ***sm2n_ is now known as sm2n
<OriansJ>bauen1: The best way to help ensure something gets cleaned up or fixed is to send a patch (or dozen depending on the mess) <OriansJ>But risc-v is just one of the many items on our ever growing list of bootstrapping tasks <OriansJ>So pick your priorities and start hammering on what you want to work on first. <melg8>OriansJ hi, i've just recently learned about live-bootstrap project, and and really exited by the idea. But can you maybe point out where can i find those "ever growing list of bootstrapping tasks" or general milestones waiting to be achieved by the project? I'm just learning for now, but maybe could be involved some time later. <stikonas>melg8: I don't think there is a formal list <stikonas>also it's not finished (although goes quite far to GCC) <melg8>i'm from nix world, so for me it's interesting to get relative new gcc and try to rebuild nix world with it, but is there some solid point in mind - where this live-bootstrap should connect with other distros? or it should grow into it's own distro? <stikonas>well, at the moment it's just gcc 4.0.4 (C only) <stikonas>but idea is that it will connect to distros <stikonas>once we have new toolchain, can start writing distro bootstraps... <stikonas>some should be easy (especially source based like Gentoo) <melg8>is there a plans for building kernel from source and not using precompiled one? is it even possible? <stikonas>at some point kernel will be rebuilt in the middle of live-bootstrap <stikonas>but we also need smaller bootstrap kernel <stikonas>although, stuff up to M2-Planet can also run on baremetal <melg8>is different kernels affecting hashes of what is build somehow? <stikonas>anyway, for now live-bootstrap was focussing on userspace bootstrapping <stikonas>there might be some intermittent (sync?) errors <stikonas>ok, I'm going offline, no more answers for now... <melg8>okay, until now i've just tried to build reproducible nixos docker images for my pet project, but maybe i'll try this next level game with live-bootstrap, But i think i'll have much more questions in future about project. Thanks! <OriansJ>melg8: in regards to the kernel question. Yes kernel bootstrapping is part of the future work. A minimal kernel in the M2-Planet subset of C could be built on bare-metal and then used to do all the steps until we are able to build Linux and use that instead. <OriansJ>So any work that looks undone and like fun to do. do it and share with us ^_^ <OriansJ>we encourage questions and expanding of understanding. Improvements to documentation especially from a fresh perspective are very helpful. <melg8>okay, for now i'll just look how it all setup, works, builds and so on) In regards of c++ compilation what was last results or what problems occurred? <OriansJ>melg8: as I understand it g++ has not been built yet as for why not, I am not certain as I tend to work lower in the stack but fossy and stikonas would be the ones to know why. <gef>oriansj: I am in favor, in lieu of documentation that would quickly drift behind at this stage, to setup gitlab ci/cd processes (a service on top of the existing repos, no need to move anything), so that it is easy and flexible to bring all efforts into a visible state (I find gitlab more welcoming for the exploration). Also, about C++ and bootstrapping into other distros: EasyBuild project might help with providing tens of different compiler versions, <gef> it's kind of a solved problem in the EB community. <OriansJ>as for the likely reasons: needing glibc to be built first or newer version of bison to parse the grammer or something not right with autotools would be my guesses. <gef>I'm afk for a few hours, but I'm very interested in your comments and plan to be back in ~5hours, especially in case anyone feels like experimenting with gitlab workflows for software building. (tbc) <OriansJ>gef: I've always encouraged the learning from and sharing with other communities. <OriansJ>and CI will certainly help improve testing but it is in no way a replacement for proper documentation; which as the goal of providing understanding to people who might not be familiar with the technicals. <OriansJ>it isn't enough to figure out how to do something yourself. That understanding needs to shared so that others can build upon the work, fix it or even replace it when a better understanding is obtained. There is no done, only good enough for now. ***Server sets mode: +cnt
<stikonas>melg8: thee ae a few reasons why g++ was not built yet <stikonas>first of all for GCC we couldn't yet use top level configure (or actually Makefile.in) as it is autogenerated and we have not yet built all the dependencies to regenerate it (it needs autogen and hence guile) <stikonas>secondly, it did not just build out of bux with musl libc <stikonas>so we either need to patch musl incompatibilities or build glibc <stikonas>bison should be fine, we have a not too old bison and newer bison can always be built if we need it <stikonas>gef: if you work on Gitlab CI, I think do that in chroot mode (./rootfs --chroot) <stikonas>should be easier, no problems with running kvm inside kvm... ***stikonas_ is now known as stikonas
<melg8>stikonas what is current development workflow? is there any? like - as i can see default script rebuilds everything from scratch and that takes some time. Is there a way to skip some parts/use previous build result? so if i try to poke around - it would not require to wait full rebuild for 1 line change? <stikonas>melg8: at some point pder/overlay branch allowed to skip slow stuff in the beginning (unpacked pre-built tarball with eveything up to tcc) but since then we changed some stuff in the beginning, so that's broken <stikonas>you can create something like that again though <melg8>And if i inside of final result shell - is there a way to reproduce step? for example i've tried . helpers.sh && build gcc-4.0.4 pass2.sh checksums/pass2 - but that did failed to reconfigure and still returned 0 - because binaries are with right hashes <stikonas>but generally you have to rebuilt everything... <stikonas>bash -c 'set -e; . helpers.sh; export PREFIX=/after; build gcc-4.0.4 pass2.sh checksums/pass2' <stikonas>then you'll get set -e which means shell script exits on error <stikonas>(don't just run set -e in result shell because then on error you'll be completely kicked out of result shell) <melg8>i see, prefix was my missing part of puzzle <stikonas>melg8: although, there is no guarantee that if you rebuild final step, hashes will remain the same <stikonas>I don't know when gcc hashes reach fixed point <melg8>it started building now, nice <stikonas>that's what I usually do for development <stikonas>although, this does not always result in correct hash <melg8>i've never build gcc in my life) even though i've developed some c++ programs, is it because of rm-s of config - not building g++? is three some easy way to recreate build failures of g++ on my machine? <melg8>i've got "sha256sum: WARNING: 6 of 13 computed checksums did NOT match" <stikonas>well, just unpack gcc tarball and try to run configure there <stikonas>melg8: yeah, sha mismatch for gcc is kind of expected <melg8>but it should at some point be self replica? so constant hashes? <stikonas>gcc pass2 is built with musl 1.2.4, so we an expect some slight differences <stikonas>yeah, I think after pass 3 it is very likely to be fixed <stikonas>this mismatch is probably because musl got updated between gcc pass1 and pass2 <stikonas>melg8: oh, you need to inject full gcc tarball into live-bootstrap <melg8>okay, i see, btw - is there a easy way to move files around from my host linux environment inside this quemu and back? i've seen it mounts tmpfs - but can i get for example build results from inside to host? <stikonas>so either full gcc or gcc-core AND gcc-g++ <stikonas>that's why we have chroot mode for development <stikonas>there is hard way to get stuff out of qemu <melg8>it failed on me with my nix setup - i'll figure out then how to fix it <stikonas>run qemu inside tmux, print base64 of binary, and then use tmux copy/paste feature <stikonas>some distros put chroot into sbin rather than bin <stikonas>I am starting chroot with env -i (clean environment) <melg8>btw, thanks for your time answering my questions, that's very welcoming to see such attitude to new people <stikonas>melg8: you can update README with info that you found was missing <stikonas>sometimes it's easier for new people to see what would be useful additions <stikonas>we indeed don't have much regarding suggested development workflows <melg8>from my initial perspective on project - there is lack of some kind of roadmap to get sense what is missing and already planned, readme + parts.rst done a good job providing what has been implemented so far. <bauen1>OriansJ: iirc tinycc doesn't accept (or doesn't like) style changes, so i doubt they'd accept me makeing the code more readable (and i don't want to spend the time to test that right now) <stikonas>well, later in the bootstrap it's just modern GCC and we can try to bootstrap some distros <stikonas>and lower in the bootstrap I guess it's kernel work <melg8>are resulting filesystem reproducible as well? meaning - if i tar all up - ci tar it all up - will those tars have same sha256sum? <stikonas>oh one other thing, I think bauen1 is working on some minimal package manager <stikonas>and then building individual packages in chroot <bauen1>yes, i just kicked off another build since the last rebase broke something <bauen1>the putting builds into pseudo packages part is mostly done, the build in chroot part not really <stikonas>well, at least now live-bootstrap is a bit more stable <bauen1>melg8: with the right flags you should be able to strip dates etc.. <bauen1>i also plan on using tar to extract the "pseudo package" from the chroot and doing checksums or something like that <stikonas>but other than timestamps, there shouldn't be much <stikonas>although, some are generated in the build process (e.g. autoconf/automake) <bauen1>in fact you could already use my stow work to skip build steps if the pseudo package already exists (and checksums match) <bauen1>i don't actually want to do that yet, as i want to include all inputs into one checksum, and then that + the result into another one and base the decision to build on that <bauen1>i'm not really sure about using tar as format, since well you'll need tar, also how exactly pseudo packages should look like and what meta information would be useful to include <stikonas>OriansJ: now has untar that is buildable with M2-Planet <bauen1>can tar also do fully reproducible archives or are there still some bugs ? <stikonas>you can just open tarball in text editor <melg8>@bauen1 what would be goal of this pseudo-packages? contain state of some part of build? <stikonas>bauen1: tar is basically just concatenation of files with a bit of metadata <bauen1>^ and allow one to easily skip single components, and have a clear and very explicit dependency graph <stikonas>e.g. if we eventually build in chroot (not to be confused with outer chroot of ./rootfs.py --chroot) then we need to know inputs <bauen1>and actually enforce that dependency craph <stikonas>we have rough drawing, but it's not very accurate later <bauen1>which then allows one to more easily reorder components to make it smaller <melg8>thats sounds like soo similar to what nix is doing) they have like set of tools to even visualize <melg8>but maybe it is not easy to build it's build closure, idk <bauen1>melg8: and a more advanced goal i have is to allow the pseudo packages to be reused by bootstrapped distributions, so their dependency graph can be made non-cyclic <bauen1>as in you no longer build gcc by using the gcc package, but e.g. live-bootstrap-upkg-gcc-8.0 <bauen1>which you could also do by simply taring up everything at the end, but i like to be explicit <bauen1>oh and you can use it to easily answer the quesition where a resulting file came from <melg8>for me that project is interesting from building for the first time - trusted(auditable) binary of compiler and applying diverse double-compiling to check what's up with the rest of compilers <stikonas>also "removing" stuff is harder without packages <bauen1>^ currently e.g. /after/bin/bash is just replaced, which isn't particularely nice <stikonas>yes, so e.g. when we update package with newer version, we just rely on make install overwriting old files <stikonas>but e.g. what if newer version installs fewer files <bauen1>oh and if by some accidents parts of a Makefile don't respect DESTDIR or have some other bug, they can just write all over / <stikonas>I think there are still some mes libc files in final fs <bauen1>btw isn't there that one component that can't be distributed in binary form ? we should probably try to replace / rewrite it <stikonas>some files are overwritten, some are removed <bauen1>i've discovered most of those (after perl where i have stow) and removed them now too in favour of unlinking the old "package" <melg8>for me personally would be really reassuring if i could easily see what is exact final product of each of "stages" - what files goes where and what are theirs hashes <melg8>btw is there any automation involved - for example - is there a way now to check - if all binaries and libs are covered with hash check? or for now it's just with eyes? <stikonas>well, at least as soon as we have checksuming program <stikonas>initially it's very simple hash (fletcher16) <stikonas>initially some simpler implementation, later from GNU coreutils (which works with files larger than 2MB) <stikonas>melg8: oh, but we don't have automation to check that all binaries are checksummed <stikonas>early it's usually just 1 binary per step that is installed <bauen1>i'd really like to hash *all* files generated, upkg-build (or some of its simpler forms) will make that straight forward <stikonas>later we have autotools, so one can try to use DESTDIR to see what files it installs <stikonas>yeah, hashing all files automatically woud be nice <melg8>I can imagine that current implementation of sha256sum and self-check algo could have some bugs or imperfections - which could lead to not caught problems, propagating to later stages, it would be nice to have check from "outside" perspective for the whole product of build - ideally - with one to one relationship from git commit to the whole result <stikonas>hmm, I though inside implementation is actually better <stikonas>since it's bootstrapped, so less likely to be "backdoored" <stikonas>well, we had that issue where first initial sha256sum was calculating hashes incorrectly <stikonas>well, I usually run external sha256sum to calculate checksum file <stikonas>and then once live-bootstrap runs, internal sha256sum checks it <stikonas>well, self-check algo is just test -e "${checksum_f}" && sha256sum -c "${checksum_f}" <stikonas>but maybe bauen1's work will include that for free <melg8>but, do checksum_files are the same the whole time?) (hello, that"s my paranoia) <stikonas>well, that will be resolved once bauen1 has "packages" <stikonas>and before autotools it's definitely the same <stikonas>autotools makes things a bit harder to track <stikonas>but also gives more tools (eg. DESTDIR installation into staged dir) <melg8>how you correctly exit from qemu? i've tried exit from inside of shell - it spits out qemu messages, but doesn't return to the parent shell from which i started <stikonas>(pkill probably supports tab completion too)