<bauen1>fossy: ironic as in: you guys are the first to start complaining once something achieves "self hosting" <bauen1>as long as the documentation is readable without processing it really shouldn't be an issue, just when you can't read documentation without additional tools does it become an issue <OriansJ`>bauen1: well we are probably the only people actually truly self-hosting anything. <fossy>stikonas: opinions on splitting out the parts to a seperate file <fossy>i honestly think that the "part" descriptions probably dont belong in the readme anyway <stikonas>yeah, we can split them into a separate file <stikonas>I think with restructured text you can do that <stikonas>or at least with sphinx (that uses rst) you can do that <stikonas>fossy: so interactive bash seems to work when chrooting... <stikonas>although, I still need to try it on qemu <OriansJ`>I find org-mode allows for fast and easy restructuring. <OriansJ`>alt-up, alt-down, alt-left and alt-right <stikonas>yeah, github also supports org-mode too... <stikonas>although, what github can render maybe shouldn't be deciding factor... ***remexre_ is now known as remexre
<stikonas>oh we already have /dev/console in qemu live-bootstrap <stikonas>maybe I should conditionally create device nodes <stikonas>fossy: ok, so I've got shell in qemu now <Hagfish>getting to the stage of having an interactive shell, and versions of perl, really feels like civilisation hitting the industrial revolution <Hagfish>there are some exciting things still to be built, but it will hopefully require less sweat :) <stikonas>and don't forget that we have a modern C library <Hagfish>(i don't know enough about C to understand all the implications of that, but it does feel essential) <stikonas>well, C library is all those C functions that you take for granted... <stikonas>string manipulation, maths, file operations, etc... <Hagfish>doesn't every C program rely on those? <stikonas>baremetal ones don't, or something that directly calls kernel syscalls does not have to use those <Hagfish>a pre-modern C library, or directly calling stuff? <stikonas>well, early enough direct syscalls, M2-Planet has very minimal C library, OriansJ` is expanding it <stikonas>later for quite some time we use much bigger mes libc <stikonas>it's still much smaller than glibc or musl <Hagfish>so how does musl make things easier? <stikonas>it has more functions that some programs use <stikonas>so C library is the next most important thing after compiler <OriansJ`>Hagfish: direct syscalls or in the NATIVE ports, directly talking to hardware devices <Hagfish>yeah, i guess it's like saying that to compile a piece of source code you also need to compile all its dependent libraries, except this library is a bit more special because it is used by so much <stikonas>but I guess direct syscalls or directly talking to hardware devices is less portable <stikonas>well, it's more like to compile a piece of source code you need to compile c library first <OriansJ`>stikonas: indeed, the C standard libraries (and syscalls) hide a boatload of details and make porting fast and efficient <stikonas>yeah, so you only have to do porting once.. <Hagfish>is it meaningful to ask about the relative complexity of compiling, say, musl versus something like bash? <OriansJ`>well M2-Planet doesn't need precompiled libraries and neither does MesCC due to the nature of M1+hex2 <stikonas>Hagfish: depends on the stage. fossy wrote our own makefiles for bash <OriansJ`>Hagfish: yes entirely if one is thinking of C compiler features required. <stikonas>Hagfish: for musl, we could already use their build system, but had to patch musl a bit to build <stikonas>so musl was more tricky in terms of things we had to change <stikonas>but I'm not sure about work hours spend... <stikonas>rebuilding bash is now quite easy though <Hagfish>sounds like slightly different approaches to each piece of software <Hagfish>at such a low level, i guess everything is a special case in some way <Hagfish>(assuming the .tar file itself has been audited, and the checksum checked) <stikonas>although we don't check whole tar source code... <OriansJ`>bootstrapping is a very different game when you have a C compiler able to build GCC (TCC) and a reasonably well tested libc (musl) to work with. <stikonas>yeah, it's now much easier than early stages <stikonas>much closer to distro packaging at this point <OriansJ`>plus more people to help address problems (TCC and msul developers) <stikonas>At least easier to find stuff online if you hit some problem <stikonas>even if we don't directly contact their developers <Hagfish>i hadn't even thought about how "googleable" these problems are <OriansJ`>Hagfish: It is kind of this is water sort of perspective shift <Hagfish>are you hitting issues that came up 20 years ago, with solutions from back then? <OriansJ`>Low level stuff -> No help anywhere and you have to figure it out all by yourself. Current stages -> bet there is a solution on stackoverflow <OriansJ`>So once M2libc reaches feature parity with Mes libc; it'll stop growing <Hagfish>yeah, i was having a hard time visualising how all these libs telescope into each other <stikonas>is there any chance M2-Planet can build tcc at some point? <stikonas>(questions since mes seems to be hard to build from m2-planet) <OriansJ`>stikonas: Well M3 was a plan to turn TCC into something M2-Planet could compile directly <OriansJ`>breaking the C preprocessor and Assembly and Linking into seperate programs to help MesCC. <stikonas>Hagfish: although, I don't think I found many problems I hit on SO or google or ddg... <OriansJ`>That way MesCC could leverage binutils for development and standard operations but M3-* pieces for bootstrapping <stikonas>I found exact problem on mailing list a few years ago (building musl with tcc was failing) but not answers. But I managed to solve it myself <OriansJ`>Hagfish: short version is up until M2-Planet, there is no Libc. Then M2-Planet bootstraps M2libc to become faster. and then M2libc is supposed to become the libc for everything until musl libc is bootstrapped. <OriansJ`>which reminds me, we probably need to add testing to M2libc (GCC+M2-Planet) <stikonas>well, tests are always good. Not the most fun thing to write though <OriansJ`>stikonas: yeah is it kinda a problem in bootstrapping. <OriansJ`>tests keep things functional while features are grown or architectural changes are done. <OriansJ`>but it is just work and hard to inspire people to do it. <stikonas>in this sense scripting work (live-bootstrap) is much easier to test. It basically is its own test... On the other hand if you tweak compiler to fix a problem, something else might break without being noticed <OriansJ`>We honestly got lucky with Matt enjoying nyacc and janneke wanting to do a C compiler in scheme. Otherwise we would have been stuck with M2-Planet (mescc-tools missing blood-elf and janneke's %hex2>enhancement) <OriansJ`>but yes testing needs to become a more continuous activity as we grow into a more mature group. In all of our pieces. <OriansJ`>perhaps siraben can help with CI like they did with blynn-compiler for live-bootstrap <Hagfish>are there existing test suites out there that can be reused? (either standalone projects, or tests in other projects that could be run against the code from this project) <OriansJ`>Hagfish: well the various pieces should be 100% reproducible so in theory one needs only to test and record the SHA256SUM (or any checksum really) of the functional binaries <OriansJ`>So if the binary checksum changes, it just needs to be tested (probably against its own test suite) <stikonas>well, that's what we do in live-bootstrap now <Hagfish>yeah, good point, each stage that is built should already have its own tests <stikonas>stuff that is built is hashed and compared <stikonas>so far I only had to tweak perl a bit to restore reproducibility, everything else was reproducible out of box <stikonas>although, the hashing is more manual in live-bootstrap... Guix basically hashes everything <OriansJ`>stikonas: perhaps reach out to the reproducible side to perhaps find other possible reproducible issues with it? <stikonas>hmm, what else can go wrong with reproducibility? <OriansJ`>well the short list is usually: time stamps, locales, floating point, build paths, optimizations (Usually build machine architecture related), user IDs, system names and probably more that I am forgetting at this moment. <stikonas>if binary hash matches from time to time we should be good... <stikonas>hmm, locales might matter on normal system, but I guess in bootstrap we don't even have them <OriansJ`>stikonas: well yes you have the advantage of absolute control of exactly the state of the system building the program. <fossy>ok i think we need to make the seed linux support kexec for now <fossy>i have not found any viable replacement <fossy>right, i have a new plan for kernels <fossy>so we compile tccboot with linux 2.4.26 <fossy>lol i thought you were asleep <pder>I think there is a problem with the sha256 implementation in live-linux. I get a different result for /after/bin/bash compared to my system sha256sum <pder>hmm MAX_FILE_SIZE is 2*1024*1024? so only files up to 2MB work properly ***janneke_ is now known as janneke
<fossy>pder: hm, you are correct; that nerds fixing <siraben>OriansJ`: yeah, when this semester is over I hope to update myself with the progress on this project and replicate it in Nix, with CI as well <siraben>I'm glad to see everything coming together! very exciting. <siraben>heh especially with the new Apple laptops with an M1 chip, bootstrap discussion has become widely discussed about (at least in Homebrew/Nix/Mac Ports) <gforce_de1977>(sorry, that MAX_FILE_SIZE is my fault and was just a guessing that "it will never happen") ***coldtom_ is now known as coldtom
<fossy>gforce_de1977: only bash binary <fossy>Its fine, i thought none of our binaries were over 2 megs too <OriansJ`>gforce_de1977: use size = lseek(f, 0, SEEK_END); to get the file size and lseek(f, 0, SEEK_SET) to reset the pointer to the start of the file. <OriansJ`>There is probably a C function that does it faster that I probably forgot about. <OriansJ`>for example in M2libc f->buflen would give you the size of the file being read. There is probably a matching feature in the FILE struct for the libc you are using. (or there is a standard C function for providing that value) <OriansJ`>Many of the C function names are terrible to guess but seem reasonable in reflection. <pder>gforce_de1977: bash is the only executable currently over 2MB which was making me very confused why all of the other hashes were ok <stikonas>pder: probably the only executable over 2 MB that we'll have... <pder>I think the static binutils utilities might have been larger <stikonas>after that I think we can build shared musl <gforce_de1977>OriansJ: thanks for the hint, will do: fossy: can i send a PR? <pder>gforce_de1977: I was looking at the patch for sha256sum and thought there might be an issue in hash_to_string. the line with hash_string[65] = '\0' seems both unnecessary and potentially dangerous. the hash_string pointer was incremented in that function so you would effectively be writing a null character 65 bytes after the end of the string. <pder>Wont sprintf do the null termination for you? <gforce_de1977>pder: interesting, if sprintf will do that, but it does not hurt, the hash is *always* 64 bytes long, is'nt it? so the root cause was the static buffer with 2mb. i will read into sprintf, i'am not good at C, sorry 8-) <pder>The issue I pointed out in hash_to_string is unrelated to the 2MB file limitation. Just something I noticed and thought that you don't need (or want) that hash_string[65] = '\0' line <gforce_de1977>pder: you are right, this is maybe not a problem, but at least unneeded and should be fixed, i will to that together with the 'read-the-real-filesize-and-reserve-a-buffer-for-that-patch' <pder>great, thanks for fixing that! <stikonas>pder: by the way, for your bash rebuild, I think we should also rebuild bison file <pder>stikonas: I agree, I added that last night <pder>Before didnt we use yacc? <stikonas>and should we build with ./configure --prefix=/after? <pder>Oh yeah, I tried make install and it wanted to install man pages <stikonas>otherwise, thigns would have been so much more annoying <stikonas>we used bash quite a bit before we were able to build bison <pder>Should I just install the binary to /after/bin <stikonas>well, I'm fine with installing man pages... <stikonas>although, maybe there is configure flag to disable that <stikonas>anyway, installing manpages should be fine <pder>I was confused about the sha256sum for bash failing last night but that problem is now understood <stikonas>well, maybe we can get newer coreutils soon... <stikonas>although, I think we might need to beef up autotools before that... <pder>So one approach would be to rebuild musl without the patches, but we might also be able to jump to a gcc toolchain as well <stikonas>although, I'm hoping we can jump to newer gcc... <stikonas>that might be helpful with porting to new architectures... <stikonas>most of the trouble is probaby in compiler... <stikonas>pder: some patches to musl will still be necessary <stikonas>I think we can only skip those weak symbols <stikonas>well, and "AR" makefile patch and tcc static patch... <pder>Do you know if tcc 0.9.27 can build newer versions of gcc, or do you think we will need to build tcc from the mob branch? <stikonas>guix has much simpler C library, so they can't try <stikonas>(at least when paired with musl, not mes libc) <stikonas>and having some binutils should help us a lot too <pder>In any case we will need be careful about using compatible versions of binutils, gcc, and glibc <stikonas>at least I hope binutils shoulnd't matter too much <stikonas>maybe gcc and glibc should match more... <pder>Probably need to at least be fairly close on the time period they were released <stikonas>so I think fairly close allows at least a few years... <stikonas>pder: also for bash keep in mind that hash depends on arch... <stikonas>GNU bash, version 2.05b.0(1)-release (x86_64-unknown-linux-gnu) <stikonas>--target --host --build options might help.... <pder>ah yes, thats a good point <stikonas>at some point I guess we'll have to build x86_64 compiler if we are on x86_64... <stikonas>or maybe I should say middle of bootstrap... <stikonas>we are not really in early stages anymore <pder>Any thoughts on what you think the end goal of live-bootstrap should be? My thought was possibly having a system that included all of the prerequesites for a host system for Linux from scratch <pder>Maybe a bit more since our requirements are to use autotools instead of generated configure scripts <stikonas>pder: I thought maybe even something further... <stikonas>i was hoping we can even bootstrap one or two simpler distros... <stikonas>at some point we might have to give user a choice which path to take... <stikonas>e.g. one option might be to install portage... <stikonas>another would be to just leave in standard state for LFS like install <stikonas>(we already have most of that list from LFS) <stikonas>only gcc/glibc/findutils/kernel/xz/texinfo and python are missing <stikonas>findutils is probably not too hard to build even now <stikonas>hmm, how come that LFS list does not include anything to download stuff with (wget/curl/rsync)... <gforce_de1977>stikonas: pder: my ultimate goal would be to bootstrap from a mini-OS (singletasking, minimal support for files in ramdisk, minimal syscall support) and bootstrap till we can kexec a linux kernel and after that build an automated-linux-from-scratch (ALFS) till desktop <gforce_de1977>the early parts are hard work, the other things is just playing with scripts ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<stikonas>pder: by the way, should we still build ls in the first pass of coreutils? <stikonas>in principle we can build ls later and drop the patch <stikonas>but if you think it's worth keeping it, I'm find with it too <pder>Are you talking about the patch to mes-libc involving qsort? <pder>Oh, there is also the strcmp patch <pder>I think keeping the qsort patch is useful, since it fixes a potential segfault, but as long as ls is not needed, I can see skipping it until coreutils pass 2 <pder>So the ls-strcmp.patch could be removed <pder>It might make sense to keep the first pass of coreutils to the absolute minimum needed to get through the next stages <fossy>gforce_de1977: go right ahead for the pr <stikonas>pder: yeah, I was talking about ls-strcmp... <stikonas>fossy, pder: do you think we should deal with autoconf/automake stuff in src_prepare or src_configure? <stikonas>because e.g. before we dealt with bison stuff in src_prepare <stikonas>(just minor style question but better to deal with it before we have too much)