IRC channel logs
2022-05-03.log
back to list of logs
<theruran>huh, does M1 need M2-Planet to build? I wasn't expecting that given the numbering scheme <oriansj>achaninja: the difference between M2-Planet and M2-Mesoplanet is the difference between a C compiler and a C macro preprocessor. (aka M2-Mesoplanet does macro expansion and does fork to make compiling a single step instead of separate compile, assemble and link steps) <oriansj>littlebobeep: the reason they are depreciated is because we standardized the flags to be more consistent and reduce the numbers of issues the inconsistent capitialization cased <oriansj>we gave 2 years notice and a whole 5 releases after our change to the new flags to enable people to switch to them. 2 releases warning of the depreciation before their final removal. <oriansj>janneke: I'll look into it but be warned macro expansion in kaem is primitive <oriansj>theruran: well stage2 FORTH is just the Turing Core of FORTH, there is still a good bit of FORTH to write before it contains the more advanced features of FORTH. <oriansj>theruran: well M1 is a C program and we need it (or M0) to use M2-Planet <oriansj>and originally cc_x86 could build it, so it did come before M2-Planet <stikonas[m]>yes, kaem only supports ${var} and that's it, not even $var <oriansj>and echo ${foo:-bacon} will work just fine but echo ${foo:-${bacon}} will just echo ${bacon} without evaluation <oriansj>so you can do CPATH=${CPATH:-/bin/mescc} and it'll do the correct thing <oriansj>for example: foo=${foo:-/usr/bin/M2-Planet}; ${foo} -f bar -o bleh; will spawn M2-Planet and compile the file bar into the output bleh <oriansj>but foo=${foo:-$HOME/bin/M2-Planet} or foo=${foo:-${HOME}/bin/M2-Planet} currently does not work <oriansj>but you can work around that with if/else statements using match <oriansj>I admit it isn't pretty and kaem variable expansion could be enhanced to enable cleaner constructs <littlebobeep>Wow I did not know stage0 was in comencement.scm I have not been paying attention clearly <littlebobeep>It looks like it depends on tar and coreutils binaries though? <stikonas>those things are not used in the build itself but more like automation driver <stikonas>but it's not mandatory and we have a kaem based automation (but that's obviously outside guix) <littlebobeep>Right I was confused by adding stage0-posix in guix but not go from scratch like in live-bootstrap <stikonas>well, two different ways of doing bootstrap... <stikonas>one (guix) is kind of started from the top down <stikonas>guix is first of all a distro with lots of packages and the goal was to reduce bootstrap seed <stikonas>live-bootstrap started with just stage0-posix and built stuff on top of that <stikonas>live-bootstrap is stricter in not allowing pre-generated files <stikonas>but it's not meant to be end-product for user like Guix is <stikonas>live-bootstrap is more of bottom-up approach <stikonas>it started with very little and eventually built more stuff up to gcc and even guile <littlebobeep>I'd need to check live-bootstrap history to know what even the point was originally if it could not build up to GCC <oriansj>littlebobeep: well when live-bootstrap started there was a gap between M2-Planet and Mes.c <oriansj>but it was a gap that slowly was closing (and ultimately closed with mes-m2) <oriansj>so about 5 years *AFTER* stage0 got started <stikonas>but once m2-planet and mes gap was closed, gcc was reached fairly quickly in live-bootstrap <oriansj>so it was a considerable amount of work to get the point where we are today <oriansj>so a good deal more work is needed to iron out the details and flush out ports to different architectures <stikonas>oriansj: we have a smaller seed than 357 bytes :) <oriansj>well, I didn't want to distract too much from janneke's announcement on such a small point <stikonas>littlebobeep: stage0-posix is multi-platform <stikonas>then we have some gaps to mes->mescc->tcc <oriansj>littlebobeep: because it is the pieces after stage0-posix that have bootstrapping problems for different architectures <stikonas>stage0-posix to mes is not particularly hard, just a bit of work and was done in some other cases <oriansj>for example MesCC doesn't support RISC-V yet <stikonas>because old version of tcc was patched to build with mescc <stikonas>so quite a few gaps and not enough volunteers to close them <littlebobeep>But isn't tcc written in C so why is this not ISA-agnostic? <oriansj>littlebobeep: there is no such thing as any serious program being ISA-agnostic <oriansj>even cat has different behavior on x86 than AMD64 <littlebobeep>haha really aren't LISP and Python programs ISA-agnostic <stikonas>well, it's a bit easier with interpreters <oriansj>littlebobeep: nope, try to run Python or any LISP on both x86 and S390X <littlebobeep>I know literally nothing about S390X, can we pick something easier?? :P how about x86 and ARM.... you just need to compile the intepreter for either ISA and your same LISP or Python "serious program" should work, no? <oriansj>it is easier to port a C compiler to a new architecture than to get an interpretted language to behave the same on a new architecture. <stikonas>but at least some people would find writing compilers much easier <oriansj>but M2-Planet works on more architectures than Mes.c/MesCC does <oriansj>and GCC works on more architectures than Guile does <oriansj>now that might be a matter of programmer effort (janneke is probably a better programmer than I am) <littlebobeep>So basically if I want to bootstrap on ARM I need x86 hardware (in 32-bit mode only?) and get up to tcc (or gcc?) then I have to cross-compile stuff for ARM? What is the utility of the stage0/stage1/stage2 ARM code if I cannot use it? <oriansj>or perhaps level of complexity (MesCC is much more advanced than M2-Planet) <oriansj>littlebobeep: the 32bit arm bootstrapping work hasn't been done yet. <littlebobeep>oriansj: I am honestly more interested in ARMv8-x/aarch64 <oriansj>so a couple days of refinement work and conversion and it could be done if one was so interested. <littlebobeep>Yeah I have owned armhf hardware in the past but honestly am more interested in aarch64 <oriansj>but janneke certainly could indicate the state exactly <oriansj>so with some work (not certain to the exact amount) live-bootstrap could be expanded to support your target architecture as well <littlebobeep>I don't know what the stage0 code can be used for if the next step is completely missing <littlebobeep>stikonas[m]: Oh that's nice about M2-Planet.... as for Mes, is it the Scheme interpreter in C that does not work or the Scheme code in MesCC that cannot compile armhf/aarch64 assembly yet? <littlebobeep>I see.... so how do I bootstrap ARM systems?? >_< I need x86 hardware then to get up the GCC and just cross-compile Guix bootstrap binaries and use Guix (because stage0 on ARM seems useless without later stages) <stikonas[m]>I guess but you'll get different guix bootstrap binaries <littlebobeep>stikonas[m]: Different..... well does this mean every subsequent package I compile will be different hashes because my bootstrap binaries were different? <littlebobeep>Why would substitutes not work once I had base system installed? <littlebobeep>If Guile is running Guix should be able to download substitutes, no? <stikonas[m]>Cause all your hashes won't match what build servers use <stikonas[m]>Hash of bootstrap binaries affects all other binaries in guix <stikonas[m]>So every single guix output will be different than on CI server <stikonas[m]>There is no easy way of replacing binaries from bootstrap.scm <oriansj>but doing a double bootstrap might get you what you want <oriansj>and we can hope that Guix uses live-bootstrap as a second path to the bootstrap binaries <oriansj>but ultimately possible (even if it requires a bit of funny work) <stikonas[m]>pabs3: in favour of cc_x86 approach (C compiler in assembly) <stikonas[m]>Mostly due to lack of interest in bootstrapping via lisp/forth <pabs3>I see. do you want to respond to the post? <stikonas[m]>But some posts there feel as if people are suggesting possible paths <achaninja>my new perspective is that the repositories are not exactly the clearest to follow <achaninja>they are filled with failed experiments and old code <achaninja>the stage0-posix repo is quite confusing but at least running it was simple <achaninja>e.g. multiple copies of the m2 libc as different nested submodules <pabs3>I'll leave responding to someone who knows the details <achaninja>I'm not trying to be critical, I greatly appreciate it all and think its fantastic <pabs3>like the repro builds project, there always be regressions and new arches/software things to bootstrap, so I definitely wouldn't call it done <achaninja>it just takes a lot of jumping around repositories and rereading to understand what is going on <pabs3>(also things like shortening bootstrap chains to avoid zillions of ancient versions) <littlebobeep>stikonas[m]: I don't understand why you cannot have different hashes of bootstrap binaries then just use Guix to download substitutes, you are not compiling the substitutes so why does it matter that locally compiled packages would be different hashes than substitutes built with different bootstrap path? If the substitutes support your ISA they should run, even if your bootstrap binaries are <littlebobeep>different, no? I have not studied substitutes much I never used them so correct me if I am wrong <stikonas[m]>Guix ci server simply won't have substitutes that your guix is requesting <stikonas[m]>gcc-hash1 and gcc-hash2 are different unrelated things for guix <achaninja>i dunno about guix, but nix has more than one type of package, some are 'content hashed', others are 'build script hashed' <achaninja>content hashed packages in nix don't care about where they came from <littlebobeep>stikonas[m]: Okay I will have to study how substitutes work to understand as I never used them before, but it does not make logical sense to me once you have a base OS installed that binaries for the correct ISA cannot simply be downloaded and executed, even if the bootstrap path was different... a different bootstrap path just means the hashes should be different if you compile yourself, it <littlebobeep>should not cause binary incompatibility with substitutes built using a different bootstrap <oriansj>pabs3: well the lisp and the FORTh approaches were discarded because writing a useful C compiler in Assembly was literally done in less than 24 hours. I challenge *ANYONE* to make a cc_x86 in Lisp or FORTH or any real language in that time frame before they claim their language is better for bootstrapping. Then I want to see them build their language in assembly because no one wants to do that shit in hex <oriansj>achaninja: from your fresh perspective what could be done to aid in making things more clear and easier to understand? <oriansj>pabs3: I can't respond but if I were to respond, I'd merely suggest if those languages were actually any good at bootstrapping, why when they had decades to prove that point that a single inexperienced C programmer did what they couldn't in less than 24 hours and then started to do bootstrapping speed runs to show how trivial of a problem it really was. <achaninja>oriansj: as one example - I don't think knight is explained once. My understanding now is that knight is a custom virtual architecture you have made to simulate bootstrapping. I guess I inferred stage0-posix is a port away from the knight architecture <pabs3>I can paste these answers to LWN if you want <theruran>oriansj: I am interested in cutting out C from the bootstrap path to Ada, that's why I was looking at lisp.s and forth.s <oriansj>pabs3: give me a couple minutes to compile a well framed reponse for you to paste <oriansj>achaninja: stage0 is bare metal (no operating systems) stage0-posix is the stage0 steps on top of a POSIX kernel. Knight was an ancient CPU architecture that was rare enough for me to say that probably wasn't backdoor'd and used it to learn how to program in assembly <oriansj>theruran: well you would either have to find an Ada compiler written in a language other than C or Ada or create one using a language that wasn't bootstrapped in C or written in C and repeat until you hit assembly. <achaninja>The goal of this project is to provide a minimally-adapted GNU Mes which can be compiled by M2-Planet and is able to compile the real GNU Mes (and thereby run mescc). <oriansj>achaninja: we don't actually compile the real gnu mes right now and just run MesCC on mes-m2 <oriansj>and use that to compile TCC and bootstrap the rest <achaninja>right, that snippet is from the readme which confused me <theruran>oriansj: I was originally looking at portable Schemes, but I want to explore other paths too. my impression of Forth is that it's easier to read and write than assembly <theruran>well, and this seedForth-interactive has exceptions, for example that would be handy <achaninja>I attempted to run m2-mes but got a few segfaults <oriansj>theruran: depends on how comfortable you are dealing with stacks honestly <oriansj>achaninja: well mes-m2 is *very* touchy unfortunately <achaninja>fwiw, im trying to bootstrap a linux package tree from stage0-posix <oriansj>so it is better to just blame me for its bugginess <oriansj>achaninja: did you look at live-bootstrap yet? <achaninja>I guess live bootstrap would be the best example though <oriansj>achaninja: you don't need qemu, just write directly to a hard disk and boot <achaninja>it looks like live-bootstrap uses python and qemu - i guess its ok <achaninja>I was trying to make a system that could be bootstrapped on a host linux that only had curl and shell basically <achaninja>I suppose python and qemu are quite common anyway <achaninja>thanks for the help :) I will study live-bootstrap more <achaninja>in a way livebootstrap is better than what I was thinking because it also controls the kernel <oriansj>achaninja: well in live-bootstrap the python and the qemu bit can be disposed of if you are willing to do manual work as once you have it, you only need a kernel and to use kaem-optional-seed as your init <pabs3>I think that last para will need to be edited, that sort of language isn't appropriate on LWN :) <pabs3>replaced with "stop talking" :) <oriansj>pabs3: thank you for the correction in language <muurkha>you could probably find a way to phrase it that makes it sound like you're calmly amused by the boasting of the Forth and Lisp enthusiasts <muurkha>rather than angrily trying to... out-boast them <oriansj>muurkha: good point; not sure how to do that but would be interesting to see <vagrantc>reproducible builds doesn't mean much without bootstrappability ... <vagrantc>and bootstrapping is a lot stronger with reproducible builds... <muurkha>well, when you say, "In the decades for which Lisp and FORTH existed, why didn't they solve such a trivial problem?" it sounds like you're attacking Lisp and Forth and think they're worthless <vagrantc>this whole "one of the other argument" is silly. <vagrantc>it's not which is more important, because they are more meaningful together <oriansj>as the combination of M2-Planet+mescc-tools+MesCC proved <muurkha>oriansj: imagine what an elderly, respected professor who had written cc_x86 would write about people claiming Forth and Lisp were better <muurkha>calmly amused and secure in his accomplishment instead of defensive and counterattacking <muurkha>because you deserve to be secure in your accomplishment <muurkha>and I think what you're doing is doctorate-quality work <muurkha>I think it's possible that the Lisp/Forth/whatever approach will ultimately yield a shorter, simpler path, but that has yet to be shown <muurkha>it might turn out that it does, and that would be a good thing, because then we'd have a shorter, simpler path <muurkha>but for now the best one we've found is cc_x86 <muurkha>the other thing is that if someone *did* find that shorter, simpler path, we'd want them to expect a warm welcome here <achaninja>personally I think if bootstrap must be done via tcc, there is not that much point in using lisp/forth <muurkha>achaninja: it might turn out that something lispy or forthy turns out to be a shorter path to tcc <muurkha>or even a shorter alternative to tcc for building gcc <oriansj>muurkha: I'd honestly love to see myself get proven wrong <muurkha>oriansj: that anticipated delight isn't coming through yet <oriansj>a shorter and better path in *ANY* language is quite welcome <littlebobeep>Sorry LISP/Forth path is SUPER interesting topic, but can someone please tell me how to use stage0 without a kernel binary, I thought it was impossible for live-bootstrap to function without Linux image at present? <muurkha>qfitzah is 731 lines of assembly right now, and will probably be close to 1000 lines when it's done, which is significantly less than the 3783 SLOC of cc_x86. but qfitzah is just an interpreter <oriansj>littlebobeep: live-bootstrap does need Linux right now but stage0 pieces don't require a kernel at all <oriansj>in fact every piece can run on bare metal with a single input and a single output (say serial ports or paper tape or etc) <littlebobeep>oriansj: Okay I don't understand how to use stage0 without a kernel.... obviously this is on x86 only so U-boot is out so can you load init steps diretly from GRUB? <littlebobeep>OKay I don't have serial or paper tape on my x86 hardware haha <oriansj>that is why one idea is to get a POSIX kernel M2-Planet can build which is good enough to run live-bootstrap up to the Linux bootstrap bit <oriansj>littlebobeep: well on x86, you fit everything in an MBR (510 bytes) and work off of that <muurkha>the hypothesis behind qfitzah is that you can write a C compiler in the language it interprets in a lot less than 3000 lines of code, because otccn.c is 524 SLOC and otccelfn.c is 730, and I think the Qfitzah language will be easier to write compilers in than C <muurkha>yeah, the MBR is a thing that the BIOS imposes on your disk <littlebobeep>muurkha: BIOS? I have either proprietary UEFI or coreboot <oriansj>muurkha: I look forward to seeing your program, it looks awesome <oriansj>littlebobeep: mbr behavior is pretty standard for all x86 firmware <oriansj>littlebobeep: There is still an MBR loaded on boot with the standard signature <muurkha>littlebobeep: btw your nickname is awesome <littlebobeep>oriansj: You are saying still MBR loaded on SSDs with GPT schemes? <oriansj>littlebobeep: you can prove it to yourself if you like <littlebobeep>oriansj: So would I not even need GRUB? I could load stage0 binaries directly from UEFI or coreboot? <oriansj>in fact GRUB installs in the MBR its loader to boot <oriansj>that is how GRUB even gets loaded at all <muurkha>UEFI in general is pretty good at loading binaries directly, it can load something like a PE executable <littlebobeep>oriansj: Can't you install GRUB elsewhere and avoid putting anything in MBR? <muurkha>you don't need to go through the traditional 512-byte MBR → second stage bootloader → real kernel chain like with BIOS <muurkha>I don't know much about coreboot except that I wish I was using it <oriansj>littlebobeep: yeah with coreboot, you can ignore the MBR entirely by using GRUB instead of seabios <littlebobeep>oriansj: Ahhh well GRUB does not support LUKS2 IIRC which was problematic for choosing a payload (I haven't chosen a payload yet) <vagrantc>what is especially cool is if you can bit-for-bit reproducibly arrive at the same point from two different bootstrapping paths <vagrantc>more bootstrapping paths are surely welcome <achaninja>btw, I found a C compiler that might offer an alternative to tcc - <achaninja>though i guess theres no rush to replace tcc <oriansj>littlebobeep: well there are some serious restrictions on luks2 use right now but remember we are all able to learn from each other and no one is expected to know everything. <littlebobeep>But I might still need seabios to choose between multiple SSDs, USB flash, CD/DVD, etc. boot devices with nice menu, I don't think GRUB supports this, so I might need something messy like coreboot --> seabios --> GRUB <littlebobeep>But I don't have a good OS to compile these on as of yet <oriansj>achaninja: no license file means absolutely no go <oriansj>if it had an FSF approved license, it might be a different story but right off the bat the lack of a license file is a serious red flag <littlebobeep>I don't know why I would prefer one KDF over another, I need to read more... <muurkha>achaninja: maybe you could ask the author about a license <oriansj>littlebobeep: yes that would be an example of a restriction you need to be careful about if wanting to leverage grub to boot a luks2 partition <muurkha>littlebobeep: PBKDF2 is fine, but scrypt is memory-hard and therefore possibly less vulnerable to ASIC attacks <muurkha>littlebobeep: yeah, I don't mean you can't accelerate it with ASICs, but that the speedup is smaller than for non-memory-hard KDFs <oriansj>littlebobeep: you need to use GRUB 2.06 or newer. The GRUB EFI binary does not support LUKS2 yet <littlebobeep>oriansj: Yeah I don't see a problem using a new version if it supports the features I want, but what do you mean "GRUB EFI" don't all GRUB 2.x versions support EFI/UEFI? Anyway I don't like UEFI much so I will avoid Tianocore and use Seabios or something I guess <littlebobeep>I have one machine where coreboot is incompatible so I might just for example use GRUB 2.06+ but again here I don't understand your dichotomous contrast between GRUB and GRUB EFI <littlebobeep>I think with UEFI I need unencrypted FAT32 efi partition which sounds lame <oriansj>as I remember is GRUB EFI 2.06 had an issue which resulted it being disabled for EFI builds. I am however fuzz on the exact details <littlebobeep>oriansj: But aren't all GRUB 2.x versions UEFI compatible? <oriansj>ALL UEFI binaries are PE binaries, not all GRUB builds are PE binaries <oriansj>hmmm, I wonder if s-expression C would still count as C code <muurkha>if it doesn't all you need is a parser <muurkha>which I guess might sound sort of like "now draw the rest of the owl" <muurkha>in general I think of parsers for programming languages as being a bit easier than the other parts of a compiler (type checking, code generation, the operations on primitive types) but I guess C is a simpleminded enough language that parsing might actually be the majority of, say, otcc <oriansj>well I was thinking of how much could cc_* be trimed down if given a blank sheet of paper <muurkha>it's a little hard to tell in otcc because parsing is interspersed with code generation <muurkha>skip(')'); unary(0); if (tok == '=') {next(); o(0x50); /* push %eax */ expr(); o(0x59); /* pop %ecx */ o(0x0188 + (t == TOK_INT)); /* movl %eax/%al, (%ecx) */} else if (t) { <muurkha>certainly there's nothing wrong with writing your bootstrap C compiler in S-expression syntax if you want to avoid writing a precedence parser <muurkha>there are even cases where S-expression syntax is clearer, like if (x & 0x03 == 0x02) { ... } <muurkha>which (as is probably obvious to you but not to everybody) actually means if (false) { ... } <muurkha>because 0x03 == 0x02 is 0 and x & 0 is always 0 <muurkha>I think it's pretty common for S-expression syntax to be less clear than infix syntax, though <muurkha>Forth syntax is even simpler: "words are separated by spaces". but I never got to the point of finding Forth syntax as readable as infix or S-expression syntax <muurkha>you know how Neal Stephenson says that every man cherishes a dream that, given the right circumstances, he could spend a few years studying martial arts at a Shaolin temple or something and become an invincible fighter? I don't <oriansj>well in C I can get away without +/-/*/\//% or &&/&/||/|/^ primitives <muurkha>I cherish the dream that if I spent a few years programming in Forth it would become readable to me <muurkha>as readable as S-expressions, at least, if not as readable as infix syntax <muurkha>fuck, I'd be satisfied with "no more bug-prone than assembly" <muurkha>I'll never know because I'll never spend years programming in Forth because it's not 01989 <oriansj>well FORTH just makes me wince at another off by one stack bug <muurkha>yeah, although I did finally learn how to solve *that* problem <muurkha>the problem is that I was trying to use the operand stack for variables, and that's bad <muurkha>it's good for maybe one variable, if that <muurkha>the other problem is that I was trying to put multiple lines of code into a single function <muurkha>if you limit your Forth subroutines to one or maybe two lines of code you can use the text interpreter as a single-stepping debugger <muurkha>but instead of typing "step" in GDB you type the name of the next line of code you want to run <muurkha>that makes it pretty easy to track down your off-by-one stack bugs <muurkha>at that point Forth becomes just an untyped C with obfuscated RPN syntax <muurkha>but an interpreter, and if you're running on some kind of poorly characterized hardware an interpreter is worth the hardware's weight in gold <muurkha>but at that point you can pretty much compile C to Forth one to one <muurkha>instead of while(c != '\n') c = fgetc(input); you write something like begin c @ lf <> while input fgetc c ! repeat <muurkha>and you give it a name so you can call it from the text interpreter <muurkha>the problem I have with that kind of thing is that because I am never going to spend a few years programming in Forth I am just never going to be able to read begin c @ lf <> while as easily as while (c != '\n') <muurkha>or even (while (!= c '\n') or whatever <muurkha>but the readability difference between while(c != '\n') c = fgetc(input); and (while (!= c '\n') (= c (fgetc input))) is just not that major to me <muurkha>btw sizeof(char) is 1 by definition, it doesn't give you any extra portability <muurkha>I'd factor out that definition of strdup into a separate function <oriansj>ok I stripped out a few things from C that aren't actually required and it does a ok subset of C <oriansj>needs refinement but it is solidly in the right direction for minimal C subset implementation <oriansj>achaninja: well it was a 20 minute hack to cut down M2-Planet so not yet <oriansj>also I need to figure out a hack to put in gcc_req.h that would enable # line comments in C such that I don't have to add /**/ support back in <achaninja>have you seen c4 btw? I guess you might have <muurkha>hmm, isn't /**/ support something like if (nextchar('/') && nextchar('*')) while (!(nextchar('*') && nextchar('/'))) discard_char(); ? <muurkha>the intended semantics there are that nextchar(c) advances the input pointer by 1 if the next char is c, returning true, and otherwise returns false and leaves the input pointer unchanged <muurkha>hmm, I guess that's not quite right because you actually need to leave the input pointer before the '/2' or '* ' instead of in the middle of it <oriansj>but I am wondering if there was a way to skip it while remaining GCC compatible <muurkha>hmm, how does that handle the "/2" case? <muurkha>I don't see where the byte consumed on line 270 gets unconsumed if the following thing wasn't a '*' or '/' or '=' <oriansj>/2 just becomes / and 2 as separate tokens <muurkha>sorry if this is a super naive question <oriansj>we just pass the char back in the second time we hit that function <muurkha>oh, because it falls out the bottom and returns '/' <oriansj>so as 2 doesn't match anything else in that if statement and it just falls out as the return char to the calling function who just shoves it back in <oriansj>the token "/" is just placed in the list <oriansj>as we build the token list in a push down fashion and just do a reverse at the end to save time and work <muurkha>haha, "The compiler in qscm (17.qscm ~7k) and in C (qscm.c ~5k)." <muurkha>D(6,((L(2)==')')?qnil:B(Z4(qcomp,L(3),L(2),A),B(Z1(qnxt),Z3(q,L(2),Z4(qapp,L(5),A,qtok)))))) <achaninja>yeah, i misremembered, I remember a version that emit binary similar to tcc <oriansj>well for a second there I thought an APL programmer was using long variables names and was confused. <oriansj>and torture c does work with int foo(int a, char* b) {...} <achaninja>(I've sort of been collecting these projects over time) <oriansj>achaninja: have you seen sectorlisp or sectorbasic or sectorForth yet? <achaninja>oh nice - I had forgotten about sectorlisp - had not seen the others <muurkha>I was trying to find a list of what language features torture_c does or doesn't have <oriansj>well it supports structs (with -> access), arrays, functions, variables, inline assembly, 'chars', "strings", if and while statements <muurkha>integer constants? sequences of statements? parameter type declarations? <muurkha>structs passed and returned by value? struct assignment? <oriansj>well = is supported; it does support integer constants. compound statements and parameter type declarations and custom compond types <muurkha>achaninja: I'm not saying those things *should* be supported, just asking if they are <oriansj>but a = b + c; would need to be rewritten as a = add(b, c); int add(int b, int c) { asm("load eax, [esp+4]" "add eax, ebx");} <achaninja>I think the first version of C basically had int and arrays only. <achaninja>I had a link to the first C compiler ... let me see <achaninja>I just found it humorous since the first C compiler might have been a worse torture to use than torture C <achaninja>oriansj: are you working on making the bootstrap more portable? <oriansj>achaninja: well I will work to help get stage0-posix, mescc-tools and M2-Planet ported to more architectures; my main goal is to push the limits of bootstrapping to new heights <achaninja>I was exploring alternative paths other than tcc <achaninja>i gotta make sure they work with mescc I guess <achaninja>I like tcc, but i am not a fan of its code at all tbh <oriansj>as if you could build qbe with your C compiler <achaninja>i have been trying to get a version of musl that proc can build <oriansj>we could skip over TCC and mescc entirely and go straight to GCC building <achaninja>but yeah, just lots of pieces but nothing that all works <muurkha>achaninja: I think the first versions of C didn't have arrays, just pointers <muurkha>some people describe bc as being "an early version of C" and I think that's sort of true <oriansj>achaninja: well it takes a good deal of work to get all of the pieces working together well <muurkha>(but of course early B or C was never arbitrary precision) <muurkha>achaninja: you don't like t = "++#m--%am*@R<^1c/@%[_[H3c%@%[_[H3c+@.B#d-@%:_^BKd<<Z/03e>>`/03e<=0f>=/f<@.f>@1f==&g!=\'g&&k||#l&@.BCh^@.BSi|@.B+j~@/%Yd! <oriansj>which is why I am so tempted to repick up the M3 work and build my own binutils compatible linker and assembler <achaninja>I spent some time refactoring tcc into just a linker and assembler, but it wasn't fun at all <oriansj>what if I solved that linker and assembler problem for you? <achaninja>well my original plan before i found mes/stage0 was to make a binary seed that contained cproc - qbe - some assembler - some linker . <oriansj>achaninja: well what would you find most fun? <muurkha>oriansj: I promise not to argue with achaninja about the nature of truth <oriansj>achaninja: the most important bit is you enjoy yourself and have fun in your bootstrapping work. <oriansj>we encourage people to solve the problems they want solved in the way they want to solve them. <achaninja>nice - my main goal is bootstrapping my distro from stage0 <oriansj>we are here to provide encouragement, support, help and friendship <fossy>stikonas[m]: oh, ok, i'll add those in my PR <achaninja>haha - well its a niche topic and not everyone enjoys it :P <fossy>stikonas[m]: the out of space problem was my fault, i had got my directories in some weird state triggering something weird in qemu/linux kernel, a fresh clone and it's all gone <achaninja>but I can appreciate wanted to shorten all these bootstrap chains <oriansj>well right now the longest part of live-bootstrap is MesCC building TCC if I remember correctly <muurkha>my thought with Qfitzah was that it ought to be reasonably straightforward to implement an interpreter for a very-high-level language in assembly <achaninja>oriansj: how long does it take mesCC to build tcc? <oriansj>hmmm, stikonas or fossy would know better than me but if I recall correctly it is in the 30 minute range <oriansj>so there is room for serious improvement <achaninja>I am not too worried though - for my package tree I was gonna have an optional bootstrap that you can toggle <fossy>achaninja, we are getting closer and closer to being able to run live-bootstrap without python and qemu as well <fossy>you can run live-bootstrap right now in a chroot without python <muurkha>leaping directly from machine code to a higher-order programming language with pattern matching, flexible parametrically-polymorphic data containers, and dynamically dispatched method calls with multiple dispatch <oriansj>muurkha: yes your Qfitzah looks like it could be quite interesting <achaninja>can I get a link to Qfitzah ? I might have missed it <fossy>oriansj: on my system, mes libc is about 15 minutes, and tcc is a further ~10? <fossy>i'm just up to that part in alive-bootstrap right now, will report back when it finishes <oriansj>cc_x86 is 14,479bytes in size so 1184 bytes of executable is rather tightly written assembly <oriansj>even M0 is 1613bytes in size and that just supports DEFINEs, Raw strings, hex literals and %immediates <fossy>littlebobeep: RE: arm. there is quite a bit that would still need to be done to get arm working. I think mes has proper arm code generation now so that's fine, mes libc might be incomplete though. I think mes-m2 is also fine. TinyCC is def not fine, 0.9.26 (mes built version) i think has no arm support at all. 0.9.27/mob are ok i think. then it would be a matter of ironing out <fossy>aarch64 may be unsupported by some part of the early triplet we use (binutils?) <fossy>although, we arent really up to additional architectures in live-bootstrap yet <oriansj>one could say we need more developers to help flush out those issues before we can really say we support those architectures <littlebobeep>fossy: literally one minor point release of tcc magically has arm support? <janneke>fossy, littlebobeep : on the guix' experimental wip-arm-bootstrap branch we build until glibc-mesboot0 <janneke>much work has been done in bootstrappable-tinycc <muurkha>oriansj: yeah, but Qfitzah is doing a great deal less than cc_x86. it's an interpreter in which you could conceivably almost write something like cc_x86 <oriansj>well cc_x86 is a pretty low bar for a high level language; literally just pattern matching and basic stat tracking <littlebobeep>fossy: At what point are python and qemu required in the process currently? <muurkha>oriansj: yeah but Qfitzah doesn't have, like, arithmetic. just enough logic that you can define arithmetic in terms of the Peano axioms. but I've come to the conclusion that it at least needs integer subtraction and probably bitwise AND or OR, and most of the code is in there for that <muurkha>well, if you're using a program written in Qfitzah to output a program in assembly, you can certainly say what to include in it <oriansj>and I have to get up for work in 4 hours so off to sleep for me <janneke>the initial bootstrap phase is 32bit for x86 and arm <littlebobeep>janneke: Okay well stage0 has aarch64 support so what is it Mes and TCC that are armhf-only? <janneke>littlebobeep: 64 bit in the initial bootstrap is not essential, i.e., "just for fun" <muurkha>with the predictable response that HN assholes saw someone being defensive and went on the attack :( <janneke>littlebobeep: it's not about goals, it's about priorites <janneke>for arm and x86, 64bit is unnecessary work <janneke>the guix bootstrap performs a gcc cross build anyway, so we automagically go from 32bit to 65 there, if necessary <achaninja>oriansj: theres another C compiler that now has inline assembly <achaninja>however, cosmopolitan is its own can of worms lol <littlebobeep>janneke: You mean it cross builds from armhf to aarch64...? Not from x86 to arm, right? <janneke>littlebobeep: right, and it builds from x86 to x86_64 <janneke>note that the guix bootsrtap for arm does not reach that gcc-cross stage just yet <achaninja>cosmopolitan libc might be perfect for bootstrapping though <littlebobeep>janneke: If this is true, what is the purpose of aarch64 and x86_64 bootstrap binaries like bash, mkdir, tar, xz, and guile and glibc ld? <littlebobeep>I see Mes is 32-bit only but other stuff has 64-bit versions <janneke>littlebobeep: why not use native versions if they are available? <janneke>or to put it differently, before we implemented the reduced binary seed bootstraps, each arch had its own binaries <littlebo1eep>janneke: I am just confused because you just said it is minimal priority to have 64-bit native support but I see all these 64-bit binaries in boostrap.scm and also early on, maybe only TCC and Mes are missing? I know the 64-bit bootstrap binaries have been around for years, but I am confused at what point things started being cross-built from 32 to 64 bit like you mentioned? <janneke>littlebo1eep: the cross build has always been part of the guix bootstrap <janneke>it's just sanity removing all traces of your bootstrap binaries, although none of the major gnu/linux distros bother to do so <janneke>littlebo1eep: yeah, its 'only mes and tcc' (and possibly gcc-2.95 and an early binutils) right now <littlebo1eep>janneke: Why do you need to cross build if you literally have 64-bit binaries of gcc, binutils, guile, coreutils, etc.? At what point does that switch to 32, then back to 64-bit again? <janneke>note that "only mes and tcc" cost me 6 years to get working for x86 <janneke>exactly, that's why i find your questions so weird <janneke>"why not do much more unnecessary work, why hasn't that been done?" <janneke>"<janneke> it's just sanity removing all traces of your bootstrap binaries, <janneke> although none of the major gnu/linux distros bother to do so [10:39]" <littlebo1eep>I'm not asking why it hasn't been done as much as trying to figure out what is missing <janneke>the cross build is from 64->64 in guix 1.0 <janneke>in guix 1.0, the cross build was 32->32, and 64->64 <littlebo1eep>I think I started with Guix 0.8.2 and used bootstrap binaries x86_64.... <achaninja>janneke: are you able to build many other programs with mescc? <janneke>in M2-Planet 1.10.0, #include is still a noop, right? <pabs3>muurkha: that wasn't intended :( <muurkha>pabs3: yeah, I didn't think you intended it <muurkha>it's not your fault you aren't cynical enough about Character Assassination News <janneke>stikonas[m]: okay, just sent a patch to oriansj for #include_next <stikonas[m]>It forks and calls M2-planet, blood_elf, M1 and hex2 <janneke>stikonas[m]: i'm first trying to catch-up with 1.10.0 <janneke>the post-1.8 features are pretty cool, only #ifdef'ing around "unsigned long" is some work <janneke>but most lib/m2/ duplications and hacks can be easily removed <stikonas[m]>We still are fairly limited in m2-planet but a bit less so than before <achaninja>fossy: ah I missed the --chroot option to live-bootstrap <stikonas[m]>janneke: so if I remember correctly, M2-Planet 1.10 supports simple constant #defines but M2-Mesoplanet can do function like macros , e.g. #define ADD(a,b) a+b <stikonas[m]>janneke: yes, it's kind of fork of M2-Planet with compiler stripped out <stikonas[m]>So we first build M2-planet, then kaem then M2-Mesoplanet <stikonas[m]>But M2-Mesoplanet has no requirement to work on baremetal, so we are a bit more flexible <stikonas[m]>And we don't have to keep to cc_* subset, we can use full M2 <stikonas[m]>But because M2-Mesoplanet supports includes, you might need to add some ifdefs to mes if we want to use it to build mes <stikonas[m]>Although I think oriansj added an option to disable #include functionality <oriansj>stikonas: yeah the option --no-includes will turn off following #includes for M2-Mesoplanet while still preserving all of the C macro expansion and spawning goodies <oriansj>muurkha: It is nice to see they realize when I said 24 hours, I did really mean non-stop for 24 hours. They are a bit wrong on the 100 characters per minute as cc_x86 involves a great deal of repetitive copy and paste bits (a very regular structure) <oriansj>yet it literally has a debug function at the buttom and they think that I did that "without testing anything"; now that is hilarious <muurkha>maybe they thought you didn't have time to test if you were typing 20 words per minute on average <oriansj>and the technical responses seem to be a FORTH C compiler that doesn't support structs, arrays, function pointers, inline assembly or variables more than 1 char long and a Lisp program that doesn't actually solve the technical challenge at all. <oriansj>muurkha: emacs C-x ( and C-x ) are muscle memory for me these days and boy does it make it quick to layout correct code fast <achaninja>random question, is mescc still being improved to handle more C code? <achaninja>or is it considered enough that it can build tcc <achaninja>that is to say, does mes aim to be a 'complete' C compiler, or complete enough. <oriansj>achaninja: well mescc is being improved (in fact it is currently the most advanced C compiler written in scheme right now) and has the very unique distinction of having the simplest/smallest requirements of any C compiler written in Lisp. <achaninja>I think building tcc makes it quite sophisticated <oriansj>there is an ancient Common Lisp C compiler but The subset of Common Lisp you would need to support would actually end up being bigger and more complicated than TCC itself <achaninja>perhaps I can try building chibicc using mescc <achaninja>I guess the only benefit of these other C compilers is just speed <muurkha>oriansj: if you want to indent or dedent code, C-x i now lets you move the block back and forth with the arrow keys <achaninja>I mean to say, if mescc could build just as much C code as tcc - theres no need for tcc in the bootstrap <achaninja>Another thing I thought a tiny bit about was if compiling an emulator is a valid way to bootstrap other arches <muurkha>a cross-compiler is good enough if your CPU isn't backdoored <achaninja>true, i guess you need a cross compiler regardless <achaninja>actually i guess not, you can do multiple seeds like is being done <stikonas[m]>Building tcc with mescc takes 10 min and fractions of a second to rebuild with tcc <stikonas[m]>And emulator is fine in principle even if overkill in some cases <janneke>yes, gitting rid of 0.9.26 would be nice <achaninja>yeah its trading time for complexity i guess <achaninja>the tcc code is sort of like an exercise in obfuscation - but it works <achaninja>and by complexity i just meant trading two compilers for one compiler <janneke>pflanze: ah no, there is no competition, at least not yet ;) <janneke>pflanze: stage0, of with cc_x86.c is now part, as a project started off from the bottom (hex0) and is working up <janneke>when stage0 just started, i created mes soon to be folowed by mescc, which aimed to build a version of tinycc and the remaining bootstrap for guix <janneke>the first versions of mescc had their own linker, which were later replaced by mescc-tools of stage0 <pflanze>"I created mes soon to be followed by mescc", do you refer with mes here to the Scheme interpreter? <pflanze>Or, is mescc also an independent project with the same name as the mescc inside the mes repository? <stikonas[m]>pflanze: basically we bootstrap from hex stuff to macro assembly, then cc_x86 then M2-Planet (simple c compiler in c) then mes and mescc <pflanze>cc_x86.s? Or is there a cc_x86.c, too (janneke mentions the latter)? <achaninja>janneke: I am not really a scheme expert, is the nyacc parser not in the mes repository? <stikonas[m]>pflanze: there is a prototype in C cc_x86.c which was later expanded into M2-Planet <stikonas[m]>Just different assembly syntax that is easier to parse <stikonas[m]>You need to download it separately and set up env variables to point to it <achaninja>I see, I didn't realise the C parser was in that project <stikonas[m]>pflanze: yes, that's for baremetal bootstrap, we have a version for POSIX bootstrap <stikonas[m]>There is cc_* for x86, amd64, riscv32, riscv64, aarch64 <stikonas[m]>M2-planet also has very similar algorithm but in C and extended with extra features <pflanze>Wouldn't it make sense to have the variants (both .M1 and .s, as well as architectures) in the same repo? <stikonas[m]>stage0 is baremetal bootstrap and stage0-posix is for running things on Linux kernel <stikonas[m]>stage0 is not easily automatable and might need adjustments to your hardware <stikonas[m]>There is a trivial shell kaem (757 bytes) to kickoff bootstrap <pflanze>I should try this out, does that work on Qemu? I guess there's a page that instructs me how to go about it, and I should just read through the https://bootstrappable.org/ first? <pflanze>I'll have to leave in a bit for a couple hours but will be back later. <achaninja>I fwiw i just finished a live-bootstrap in a chroot <stikonas[m]>There is an optional rootfs.py script to help you launch things <stikonas[m]>Should also work directly on hardware without qemu but I haven't tried yet <pflanze>I'll have a go trying out some bootstrap paths and get a proper understanding, maybe writing up what insights I have. (May take a few days, but I'll start tonight.) <stikonas[m]>Also a bit outdated and incomplete but we have a graph here <stikonas[m]>(Some stuff was reordered, etc). Some of the arrows later in the process are missing. And everything after bash 5.1 is not in graph <pflanze>Cool. If that info can be extracted automatically (scripts, dependencies), maybe I can contribute code to do that. But I'll first have acquaint myself with it. <pflanze>(On second thought, there are so many arrows, at least partially it already must be automatic.) *pflanze off for a few hours <stikonas[m]>No, it was manually written (there is .dot file somewhere) <stikonas[m]>Basically by following build scripts and looking at dependencies <stikonas[m]>Though that becomes harder/impossible once you start using autotools <janneke>hmm, m2-planet chockes on: #if __M2__ || !(__MESC__ && __arm__) <janneke>lib/mes/ntoab.c:32:newline expected at end of macro directive <janneke>and hmm, m2-planet does not define __linux__ *janneke will send a patch for __linux__ <janneke>hmm, m2-planet does not have "short" <oriansj>pflanze: not competing but working together to build a shared dream. <oriansj>Also to do mes-m2 in assembly would be a 3 month project for me but entirely possible to do (It took a bit to get mes.c into a state where gcc could directly compile it with having to do snarfing first) <oriansj>janneke: why would it define __linux__ ? it has no idea what environment it is running on <oriansj>hence why M2-Planet support -D __linux__ if you need it defined <oriansj>janneke: yeah, I haven't found a need for short yet but can be added with a bit of work. <janneke>oriansj: just like gcc defines __linux__ *janneke sent a patch, btw <janneke>m2-planet knows perfectly well which platform it's running on... <janneke>...but it does not know if it's cross-compiling, of course... <janneke>i'll add the -D <platform> to kaem.run <oriansj>janneke: how can it know what platform it is running on? <oriansj>it just defaults to knight assembly by default <janneke>right; it only knows when it's being built by gcc <oriansj>it is int main(int argc, char** argv); there is no envp to pull from to know about the world outside of what it is told <janneke>i have no pressing need for short, i think <janneke>stat gets possibly problematic without short, but using 'char foo; char foo_filler_;' now <janneke>btw, m2-planet has become very impressive <oriansj>or just char[2] label; will work as well <janneke>i'm able to remove the vast majority of the workarounds that made for 0.24 <oriansj>and you can do #if __M2__ .. #elif __MESC__ ... #else ... <stikonas[m]>janneke: yes, that was the goal of my m2-planet work for 1.10.0 <stikonas>I was deliberately trying to implement some of the stuff that would reduce the diff between between mes 0.23 and what is now 0.24 <oriansj>and ideally get M2libc to the state you can just use it when bootstrapping and just standard glibc libraries in standard usage <janneke>yeah, that's what the mes c library also does <oriansj>to reduce your total support work and help you with porting to new architectures <stikonas>one thing that I still don't particularly like about M2libc is that every low level function have to be rewritten in inline asm, so a lot of copy pasting rather than having syscall_NUMBER() helpers... <oriansj>(I might need to convert M2-Planet+M2libc to use MesCC's M1 DEFINEs first) <oriansj>stikonas: completely fair, I did that to make debugging syscall behavior easier <stikonas>well, yeah, you don't need backtrace in this case *janneke is not complaining! <janneke>okay, mes builds without --bootstrap-mode <janneke> +> ./bin/mes-m2 -c (display 'Hello,M2-mes!) (newline) <janneke>./bin/mes-m2 abnormal termination, signal number = 11 *janneke is afk for some dinner <oriansj>still need to figure out how to fuzz mes.c <oriansj>and clear out all of the segfaults that lurk inside <stikonas>(this and some riscv64 support are probably the only interesting things from mes-m2 that are not in mes) <stikonas>and then mes-m2 can probably be forgotten <janneke>i just rebased the riscv work, finally, but want finish this m2-planet supported cleanups first <stikonas>did a bit of testing: M2-Planet 0.10.0 -> mes 0.24 seems to work just fine, so no regressions <achaninja>how do you guy install nyacc for mes development? not sure exactly how to get he mes build to pick it up <stikonas>${sysa}/${MES_PKG}/src/${NYACC_PKG} is /path/to/nyacc, so you need /path/to/nyacc/module <stikonas>and also 2 directories inside mes itself <stikonas>oh, you are trying to compile using guile... in that case I don't have enough background to answer <achaninja>I mainly want to see what other C code I can run it on <achaninja>thanks that is helpful - I was missing some of these paths <achaninja>that was it, just with guile you do'nt need mes/module <stikonas>achaninja: hmm, looks like you are using wrong libc variant <achaninja>yeah, changing to CC=clang worked, its like my version of binutils was incompatible <stikonas>using mes and mesccon themselves is probably the least error prone but also the slowest <achaninja>I'm not sure what the build does exactly but it seemed to build meslibc with my host CC first <stikonas>yeah, I think if you run configure or something like that that's what happens <stikonas>the bootstrapping workflow is to use M2-Planet to build mes (for some reasons that binary is called mes-m2) <stikonas>and then use mes-m2 binary to build libc.a and optionally rebuild mes <stikonas>in live-bootstrap we don't bother building mes, there we use mes-m2 to run mescc to build tcc directly 