<stikonas>I also wanted it to work for both .tar and .tar.anything <stikonas>maybe something simpler can be written but oh well... <vagrantc>i see a lot of buzz going on, is mes likely to see a new release soon? <vagrantc>trying to get mescc-tools uploaded now for debian's upcoming bullseye release (freeze starts soon) <OriansJ>anyone seen this bug before in make: make -j 2 results in the same step being done at the exact same time twice. <vagrantc>my best test for mescc-tools is, of course, building mes :) <deesix>OriansJ, didn't happen here. That sounds nasty. <OriansJ>deesix: happens with 42+ cores but only if the cores are slow enough <OriansJ>eg on lower end arm servers but not higher end AMD64 servers <deesix>Maybe filesystem/arch related, instead of speed. <OriansJ>so I sent a message to help-make mailing list with the hope that they might find us a better work around than having to type make clean M2-Planet test -j42 <OriansJ>as that always is successful in the testing I have done <OriansJ>well I'll double check to see if choosing ext4, zfs, btrfs, ext2 or xfs results in different behavior <deesix>And even flags for those, the FS matrix is kind of infinite. <stikonas>fossy: argh, tar submodule does not work with my kaem file... <deesix>I'm trying a bit on tmpfs, OriansJ, but this machine is small ;) I really don't understand how make decides to re-build, I guess it's timestamp based, but seems a bit inconsistent, as sometimes it only recompiles M2-Planet and sometimes everything (without clean in-between). <stikonas>fossy: shall I go back to wgeting and unpacking tarball? <fossy>yes go back to that and I'll figure it out <stikonas>in git history lib folder was adding at around tar 1.13.80... <deesix>OriansJ, no... sorry... Not just M2-Planet, but sometimes it rebuilds everything (due to M2-Planet rebuild, as expected) and sometimes it decides to not do it. <deesix>Hmm, but this happens without -j too. <OriansJ>stikonas: what happens when you do: git submodule update --checkout <stikonas>and alloca.c and other files in lib/ folder are present in tar-1.12 release tarball <deesix>--checkout is the default, given no explicit config. <fossy>once i merge your PR i'm going to try bare metal <deesix>That's not even in this manpage. <fossy>running the initramfs on bare metal <fossy>putting it on a usb and then using grub to load it <stikonas>although, would be nice to get bash too... <fossy>yeah i'll do bash while you sleep <stikonas>once we have bash, we can add it at the end of kaem script <stikonas>so that we get shell instead of kernel panic <fossy>but it dosen't really matter, we need both anyway <stikonas>but the question is do we want to write kaem script for bash, or write manual makefile <stikonas>make is quite a bit more powerful than kaem <fossy>yeah i think i'll go with make first <fossy>we can probably even do makefiles for everything then up to gcc.... <fossy>because makefiles rules are super useful <stikonas>you can have same rule for all .c files... <stikonas>fossy: ok, I think my PR is almost ready <fossy>yeah just push and i'll test and merge <stikonas>anyway, we are getting quite a few tools <stikonas>now just need better driver instead of kaem <stikonas>I wonder if we are still reproducible at this point <stikonas>i.e. whether binaries are the same on different runs <stikonas>mescc is the slowest part of bootstrap... <stikonas>well, for now at least, later I guess gcc will take longer <fossy>at least it dosent take hours like it used to <stikonas>oh yeah, I think I've read that it was that slow, I haven't used it back then <stikonas>oh, now with kaem escaping we can also fix mes.kaem (no need to use pre-written config.h) <OriansJ>well I guess I found a solution deesix <OriansJ>but it works for all tested systems up to 256cores <OriansJ>(I can't find a VPS provider with more than that many cores for rent) <OriansJ>with all of the tests copied to a seperate file named tests <OriansJ>I'll do it in the merge and push. Not ideal of a solution but it works and still enables parallel builds <deesix>Is your mail to the list waiting moderation, maybe? I don't see it in the web archives (that seems stuck at Dec) <deesix>Maybe it's better to wait for a reply that introducing that thing <deesix>I'm wondering if the workaround noise complicates further unification and so... <OriansJ>well all it does is breakup the makefile into 2 files and serializes the first where M2-Planet is built <deesix>Oh, you mean "test copied" as in the rules, got it (for a moment I read it as copies of the scripts). <pder>Tried live-bootstrap and got a segfault very early on with "cc_x86 hold M2.M1" Any ideas? <deesix>pder, dis you fix the submodule thing of earlier? <pder>I can post the output of git submodule status --recursive if that helps <OriansJ>pder: ok and all of the submodules are on the expected commits (git submodule update --checkout will force that) <OriansJ>pder: could you please update to commit: 0f549aee03a6d97dd441ec17ea832aa5229f3e77 <deesix>OriansJ, --checkout is the default. We may want --force instead, for safity. <pder>Weird, I am back to the behavior where it looks like M2-Planet is no longer a submodule <pder>I tried checking out the commit you mentioned in mescc-tools-seed then running git submodule update <pder>Now M2-Planet appears to no longer have an independant git history <fossy>pder: git submodule update --recursive --init? <fossy>because there are submodules inside submodules <OriansJ>pder: cd into the mescc-tools-seed folder then try the checkout <pder>fossy, that did the trick. I guess I havent used nested submodules like this too often <fossy>need to finish that README .../ <OriansJ>well cc_* doesn't like to complain just bail hard <OriansJ>so generally looking at what it was given and doing M2-Planet --architecture $arch -f $file will provide the answer to that question. <OriansJ>but based on the fix fossy did, the submodules didn't show up and cc_* choked on an empty input file <OriansJ>as reversing a NULL list is going to generate a segfault <pder>Things are running well with the original mescc-tools-seed commit. When I tried 0f549, there was a failure in M2-Planet with missing FILE. That must be related to the --bootstrap-mode flag <OriansJ>pder: as the kaem files include that change; how could that even be possible? <pder>im not sure, but possibly the kaem files in question are not in mescc-tools-seed in its submodules <OriansJ>every kaen file run in mescc-tools-seed is checked into the main repo to prevent such issues <pder>im talking about kaem files in live-bootstrap <siraben>OriansJ: is blynn-compiler fixed with the latest commit of mescc-tools-seed? <fossy>pder: i have not updated live-botostrap to a version of mescc-tools-seed with bootstrap-mode yet...... <pder>ok, that makes sense then why I got that error trying a newer version of mescc-tools-seed <OriansJ>siraben: I haven't had a chance to check as I was buried with the M2libc work <pder>siraben: what is the issue? <fossy>i missed "Things are running well with the original mescc-tools-seed commit." <fossy>pder: i need to update mescc-tools-extra and wip-m2 eneds to be updated for bootstrap-mode before i can use that, which probably means waiting for a release so janneke will include it <OriansJ>pder: mescc-tools-seed commit 05d8bced32f7ba16597ba46d9cd98a83eaf9e336 broke blynn-compiler's bootstrap build but I haven't had a chance to look too deeply into it <OriansJ>I amhoping the latest mescc-tools-seed commit fixes the issue but I haven't had a chance to do the testing and find the proper fix <pder>does it hang somewhere? I remember siraben mentioned increasing the timeout <pder>this live-bootstrap project is very cool, seeing all the pieces come together <OriansJ>and thanks to deesix's hard work and brilliant engineering M2-Planet now supports massively parallel builds <OriansJ>(just don't try to mix architecture virtualization with it eg x86/armv7l at the same time) <deesix>Thank you for your support, testing the testing :D and finding a workaround for that make thing you saw, OriansJ! <deesix>My part was just a bit of scripting and some love :) <OriansJ>pder: well janneke finally made it possible for our rope bridge to GCC to finally jump from M2-Planet to Mescc (Which was surprisingly hard) <OriansJ>soon we will have a path that universally works for all distros to reproduce the work. <OriansJ>then we go after harder and harder bootstrap problems like guile and ghc <OriansJ>with some fun enhancements if you are still up for the work (use blynn-compiler to get Gnu Mes a proper scheme interpreter written in Haskell) <OriansJ>Then we get to hit the bare metal hard and drive out all trust attacks there too. <OriansJ>Now no one will be able to claim bootstrapping the world is an impossible problem. <OriansJ>It'll be a *SOLVED* problem laying bloody on the floor begging for mercy. <OriansJ>now for a stupid question not related to bootstrapping. does there exist a number n such that doesn't become prime prefixed by $a digits of pi and postfixed by $b digits of pi (for values of $a and $b >= 0) <OriansJ>if so does it hold that properly against digits of e <OriansJ>such that if one wrote p123e44 => prefix a file with 123 digits of pi and postfix 44 digits of e and the file becomes a prime number. <deesix>I'll pass that to someone I guess would find it interesting. <xentrac>OriansJ: it seems unlikely but I don't think enough is known about the digits of pi to answer that ***xentrac is now known as xentrac_
***xentrac_ is now known as xentrac
<bauen1>OriansJ: true, a formal verification of the bootstrap kernel (and perhaps the editor used to review things) would be very good to have, essentially because when you apply formal verification to later stages you can then be sure that what you've verified will actually get executed correctly (unless there's some loophole you can use to ensure that without proofing that the kernel does what it's <bauen1>C also doesn't really work too well with formal verification :/ (there is frama-c i suppose) <bauen1>i should give the work on seL4 a closer look too <stikonas>sudo shouldn't really eat PATH but I guess that's still configurable <stikonas>gforce_d11977: I think fossy broke run with last commit <stikonas>need to set prefix in after.kaem.run too <stikonas>creating a docker container might also be interesting, something between qemu and chroot... <fossy>stikonas: ugh how did i do that? <fossy>maybe i didn't apply the change when i testran it LOL <fossy>anyway, working on make and bash now <fossy>i was wondering why my builds were failing <stikonas>oh, my get_file function in rootfs will have to be fixed for different extentions (now it only looks for file.kaem) <stikonas>probably need to check for file.mk and file.sh too <fossy>stikonas: well i was missing a few defines but acidentally unmounted the chroot so rebuilding now lol <stikonas>fossy: it might be easier to set prefix in after.kaem.run... <stikonas>and maybe bindir, libdir, includedir too <fossy>stikonas: hm, i think you're probably right, i was thinking the same thing, i'll include that with the make and bash commits <gforce_d11977>stikonas: fossy: commited your proposal (sudo/path), but build to *the end* still does not work... <stikonas>have you rebased on top of latest master? <fossy>gforce_d11977: yeah i know, tcc broke <fossy>stikonas: up to bash now, but i'll do that tomrrow morning <fossy>tomorrow, i am uncertain it works and there's a linker warning that has me confused <fossy>stikonas: also, fwiw, i was trying to make tcc patched and sed -i 's/blah/asdf/' was not working... <stikonas>gforce_d11977: for future, don't open new PR when rebasing <stikonas>gforce_d11977: maybe even now, reopen #7 and force push into that branch <gforce_d11977>stikonas: sorry about that, will do so in the future - you are right <siraben>OriansJ: Oh I didn't tell you that the PR wasn't ready for merge >.< <siraben>I can make another one fixing it anyhow <siraben>provided mescc-tools-seed's HEAD can bootstrap blynn-compiler now <siraben>yeah I know that, I forgot to for this one <OriansJ>bauen1: the problem with proofs of correctness for a bootstrap kernel is we only have one compiler with a language usuable to have proofs (blynn-compiler) but it is still a bit of work away from being bare metal bootstrap ready. <siraben>I've been somewhat busy so haven't been able to follow everything here, but OriansJ, what's the current status of blynn-compiler from mescc-tools-seed? <OriansJ>siraben: probably but I need to figure out what is causing it. Probably a regression that slipped in <pder>siraben: it appears the hang you encountered was with mescc-tools-seed commit 05d8bced32f7ba1659. Try updating to 0f549aee03a6d97dd, and it shouldnt hang. Also you should be building commit a918fb34777662ee35c of blynn-compiler. <pder>OriansJ: it appears to force pushed your master branch in blynn-compiler and we lost commit a918fb34777662ee35c. Is that intentional? <pder>"Changing the mescc-tools-seed commit to hopefully fix the build" <pder>siraben: I dont think it will work. You need commit a918fb34777662ee35 of blynn-compiler <pder>oh ok, interesting. So i guess the changes to marginally.hs on dont need the change to reorder the #define and // CONSTANT? <siraben>It doesn't look like it, why was the reordering needed? <pder>Not sure, probably a question for OriansJ- something related to the preprocessor changes <gforce_d11977>stikonas: fossy: including my latest fix 'live-bootstrap' has successfully built tcc + sed,tar,gzip,gunzip,zcat,cmp,diff - wow! ***stikonas_ is now known as stikonas
<stikonas>gforce_d11977: well, gunzip and zcat is the same binary as gzip <stikonas>usually it's all links... but I jused made a copy for now <stikonas>although, gunzip is probably all that we need in the bootstrap <gforce_d11977>stikonas: is there a special reason why the tools are built in this order? sed,tar,gzip,gunzip,zcat,cmp,diff ? why dont we just build m4+make to bootstrap e.g. busybox or toybox? <stikonas>gforce_d11977: sed can be moved after tar <stikonas>so that we ship sed.tar instead of using git submodule of sed but we can talk to fossy about that <stikonas>gforce_d11977: gzip needs a small patch (a few lines removed) which I do with sed <stikonas>gunzip and zcat are gzip, it's the same thing <stikonas>I did diffutils (cmp and diff) because they were easy, but it doesn't have to be first thing after gzip... <stikonas>in fact it might have been even easier if we had make (which fossy is working on) <stikonas>gforce_d11977: anyway, now we need to bootstrap autotools build system <stikonas>so need make/bash/autoconf/automake and maybe some others... <stikonas>after that it's just more or less standard build <gforce_d11977>stikonas: yes, but m4 + make should be first, everything after that is easy <stikonas>well, fossy doesn't want to rely on pregenerated configure files <stikonas>so we might need to write some makefiles <stikonas>with make/bash we are already out of kaem land <stikonas>a lot of things could be written much simpler in POSIX sh <stikonas>but it's definitly very useful at the beginning <stikonas>hmm, maybe later I should try to create dockerfile for live-bootstrap, a bit more isolation than chroot... <gforce_d11977>stikonas: fossy: i did another full run in chroot-mode: took 1893 seconds here 8-) maybe a faster CPU can do it in 10 minutes... <stikonas>comment out blynn-compiler and it will be faster <stikonas>gforce_d11977: it was added before m2-planet->mes-m2 bridge was done <stikonas>I guess there was some expectation that haskell will be used to run mes <gforce_d11977>stikonas: ok, will try to comment out and test if it works without <stikonas>then you'll see that most of the time is spent running mescc to build mes and tcc ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<OriansJ>siraben: the reason for the reordering was both #define and //CONSTANT worked and the definition that came last "won"; which resulted in not sizeof but 1; which caused it to hang infinitely <OriansJ>fortunately yt fixed my screw up and now --bootstrap-mode disables #define entirely and removes the need for the change entirely. As my commit caused a merge conflict with your pull request and didn't actually provide anything of value; Hence my dropping of the commit <OriansJ>gforce_d11977: think of kaem as the worst possible build tool, that you could shove into 737bytes <OriansJ>that is designed to error hard and fast if everything isn't perfect <stikonas>you can probably cut bit more things out :) like comments and print to screen <OriansJ>stikonas: there is still hope with blynn-compiler to write a proper scheme in its haskell subset; so that scheme and Haskell are needed to bootstrap C <stikonas>well, yeah, that's why haven't removed blynn-compiler from live-bootstrap <OriansJ>stikonas: Line comment support is always required in stage0. If something doesn't support comments it doesn't belong at all <OriansJ>the print to screen could be dropped if someone wanted to show me up by making a smaller shell for bootstrap-seeds <siraben>blynn-compiler has quite a bit of potential IMO, if the VM were written in assembly then we skip the C phase entirely <stikonas>well, I'm not really suggesting to drop print, just saying theoretically you can probably go to something like 600 byte shell... <siraben>I even had some musings about a shell written in Haskell <stikonas>but it's better to have a few more bytes that make it significantly clearer what happens <OriansJ>siraben: actually no as a C compiler is needed for the later stages. <stikonas>siraben: well, kaem is in principle good enough to go all the way to bash... <stikonas>and hex0 is definitely smaller than what VM assembly would be <siraben>Hm, then a C compiler written in Haskell! <stikonas>siraben: if you can build tcc with a C compiler written in Haskell, then why not <OriansJ>hence Scheme in Haskell to run MesCC ^_^ <OriansJ>but one small miracle at a time right <siraben>I do hope with a bit of exposure this attracts some programmers from the haskell community <OriansJ>or atleast shame them enough to help with the GHC bootstrap work <gforce_d11977>OriansJ: i really like the kaem approach, in the future it can maybe has even less code... <gforce_d11977>stikonas: fossy: i did a live-build without 'blynn' and saved >600 seconds... should a add an PR for that? (rip out blynn) <stikonas>gforce_d11977: probably no... it might be useful later. See discussion above <OriansJ>gforce_d11977: I always look forward to people doing a better job than me; besides we still need to shave 70bytes off the x86 bootstrap seeds to be a 1KB bootstrap <OriansJ>as the bare metal bootstrap is fortunately at 512bytes (x86) and 342bytes (Knight) <bauen1>OriansJ: there's already a bare-metal bootstrap for x86 ? <bauen1>i thought that was still some ways off (unless you're talking about a mbr with a hex editor) <pder>Another C compiler that can compile tcc is the one written by gio in the "g" language. Maybe the g compiler could be ported to M2-Planet so the path would be M2-Planet->G compiler->tcc? <OriansJ>bauen1: I am talking a MBR hex0 for x86 and there unfortunately is a great deal of work remaining there <OriansJ>pder: gio did an M2-Planet bootstrap in G <OriansJ>an old version if I remember correctly but good enough to build the current M2-planet <bauen1>that's interesting, isn't G this pseudo-C language bootstrapped from such an mbr ? <pder>Are we talking about the same thing? I mean a g compiler that can compile c_compile.g. I dont see source for the g compiler in anything except asm <pder>in the asmc repo, the file I am talking about is asmg/asmg.asm <gio>Yes, I never wrote a G compiler other than the one in x86 assembly. <siraben>By the way, is it a strict requirement that we have C compiler, what about a C interpreter? <bauen1>for bootstrapping from bare-metal you eventually want to compile a kernel, so you can have a vague concept of a VFS and processes, hopefully enough to bootstrap all the way to a functioning linux kernel <bauen1>i don't think a c interpreter would work too well for that task <siraben>Yeah, the overhead would be too much, even a naive C compiler would do way better <bauen1>siraben: it depends, if the interpreter is fast enough and some critical functions are written in assembly, then you could get away with it until you have another c compiler bootstrapped <bauen1>depends on how much time you have really <bauen1>if it means that you have to review less code at a lower level (with inferior tools) and it only takes a few hours longer, then that might be a worthwile trade-off <bauen1>couldn't you do some really heavy trade-offs in that direction ? the thing that will take the longest would usually be the code review by a human anyway (limited by the human and maybe the i/o bandwidth of the computer <-> human) <bauen1>i should talk less and start working on my kernel again <gio>BTW, let me add that the G language is basically a way to write sort of understandable assembly code. It was never meant to be portable. So it makes little sense to use it other than in the context of ASMC: in a bootloader, as the very first stage to have a compiler out of assembly. <gio>It makes little sense to implement another G compiler, from my point of view. The G code is not portable, and you will probably have to modify the G code from ASMC if you want to compile it with another compiler. <pder>gio: thanks for the info. I was just brainstorming other paths from M2-Planet to tcc. <pder>I am not sure what is missing in M2-Planet that your C compiler written in G has <gforce_d11977>academic question: is there a measurement-method how long it takes to understand/proof lines of code in language XY? in other words, can one compare the MES and the ASMC approach in such or similar base? <stikonas>gforce_d11977: well, you can set up two courses, have exam and measure performance of students... but even then it's not very precise... <fossy>gforce_d11977: I still dont like the echo statements and the verbose, it already outputs everything it foes using set -x <fossy>im still writitng a more complex readme too <stikonas>for non-debugging purposes set -x is actually more than necessary... <stikonas>fossy: by the way, should we move sed bootstrap after tar? <fossy>stikonas: well we can, I dont see any reason not to <stikonas>although, we have to prepare tar ourselves <fossy>Im not a fan of repackaging upstreams <stikonas>but I can submit another small patch that might be useful in the future... <fossy>stikonas: we will have to build patch before make... the make binary is broken but i figured out where in the source code the probelm is, but it woudl require a 's/a/b' expression which is broken with our sed <fossy>luckily according to gcc-seed it dosen't require patching <fossy>(patch dosen't require patching, that is) <stikonas>I think there are less than 10 files to build <fossy>stikonas: i believe it should work, just that it is broken under tinycc/mes c library... <stikonas>hmm, I just picked some 4.x version at random <fossy>but older versions dont have -i... <fossy>except for the basic use we require it for <stikonas>but anyway, patch is better for patching <senzilla>Does anyone in here feel like you're reinventing the wheel? <bauen1>senzilla: if you're not reinventing the wheel when working on bootstrapping you're doing something wrong <bauen1>first commit of the year to myunix and ci has already decided to blow up, off to a good start <senzilla>Just for the record, I was referring to this ^ <malina>for the same reason 70 years old sigh when 20 years old talk about their first shag <mihi>gforce_d11977, not sure about toybox, but busybox uses lots of glibc'isms to stay small (strchrnul, fnmatch, ...) which don't exist in our mes libc. Probably possible to rip them off e.g. musl or bionic, but for what? At the end everybody wants to have original bash/coreutils/etc. anyway <malina>my anecdote , ye nvm people dont get it. it's nothing to do with a bad day though. Anyway, toybox is old skool and perusing at his talks I am guessing the sigh could have been the fact he has bee doin this a log time. Although form a good while ago, one DOES see he mentions the cc lacks still some (alternative?) toolchain. <mihi>By the way, any plans how to bootstrap coreutils? It seems to me that very many basic syscalls (e.g. rename) and functions (ctime) are missing from our libc, and I do not think (but may be surprised) that coreutils can live without them... Or are there already plans how to get to a better libc? <malina>mihi, well most of the gnu tools was ripped off from the original, and ironically, the bsd's/posix's were left int he dust. <malina>parastic but for sure, not everyone therefore wants only gnu. however, most want the kernel, which itself, despite movelle claiming elsewise was also totally 'inspired' by the *nix. <malina>the sighing though was nothing to bother about. This is a great project in itself. I think you guys are doing a bottom up approach, not sure if the toybox to some degree was a top down.