<yt>OriansJ: changes to the hex2/M1 versions of hex0 are up; they all translate to byte-identical versions of hex0 :-) <yt>I've added a little makefile to do that checking too, so it's easier to test that <OriansJ>QuickBootstrapQu: the answer to your question is nothing; as guile (running shepard) is the init and (running gash) is the shell and (running gash-utils) is the core utils needed and (running guix) as the package manager able to bootstrap GCC from source (running MesCC) as the compiler as well. <OriansJ>So only the bootstrap binary tarball is required (assuming guix fixed the gnutls-3.6.12 regression already); which we are in the process of significantly reducing further. <OriansJ>As once the guile compatible is bootstrapped from M2-Planet. One would only require to have a 737byte shell (kaem) and a 257byte hex assembler (hex0) and as kaem can run as an init and automatically do all of the build steps from source. Any minimal POSIX compatible kernel (with a handful of syscalls) will also work. <OriansJ>That is why the stage0 pieces are being done in a fashion which would allow bare hardware to be used for all of the bootstrap steps (until a proper kernel could be built from source of course) <OriansJ>yt: quick question you changed the base address form 0x400000 to 0x600000? I thought 0x400000 was the AArch64 standard address?? <OriansJ>Darius: don't worry I never take anything here personally. Although I apologize if I seemed agressive. I was entirely curious on what exactly your preference was and the reasoning behind it. As conflicting perspectives are essential to a greater understanding of possible solution spaces. <OriansJ>yt: reading through your AArch64/Development/makefile I guess you didn't realize that both M1 and hex2 support arbitrary number of input files. Top to bottom it is left to right. eg hex2 -f elf_header.hex2 -f hex2_program.hex2 -f anything_else_youwant <OriansJ>But your M1 code looks about correct and once you get that harmonized; I'll merge it <OriansJ>yt: your M2-Planet patch has been merged. <fossy>OriansJ: does M1 support 16 bit x86 mode (i.e. real mode) <fossy>i don't think it does but checking <OriansJ>it also supports the arbitrary mixing of modes too <OriansJ>if an instruction set or mode can be expressed in binary, octal or hex; M1 and hex2 support it. (Or are a minimal patch for new architectures) <OriansJ>The only thing that could trip M1 or hex2 is if your immediates or offsets required a weird encoding that needed to be added to M1 or hex2. (eg offset 0x4 needs be to b encoded as 0x696969^(0x4 << 3) or something odd like that) <OriansJ>The worst cases we have seen are Armv7l and AArch64 but they were 5 lines of support needed total <fossy>lets see if i can get this in M1 then <fossy>OriansJ: sorry, i think i am going to have some dumb questions, what does DEFINE XXX SOMEHEX do? <fossy>is the SOMEHEX a hex2 instruction? <OriansJ>fossy: DEFINE statements are line macros <OriansJ>so when you type DEFINE foo 12345; all instances of foo will be replaced by 12345 <OriansJ>So if add was 123; you write DEFINE add 123 <OriansJ>if register 15 is F; you write DEFINE R15 F <OriansJ>so that when you write add R14 R15; it'll convert into 123 E F <OriansJ>there is an example set of instructions for x86 in mescc-tools-seed in the file x86/x86_defs.M1 <OriansJ>(it couldn have been much nicer but I kinda rushed it) <OriansJ>now a more common thing is like: DEFINE LOAD_IMMEDIATE_ecx B9 <OriansJ>where you then want to do !42 or @42 or %42 depending on the size of immediate required by the instruction <OriansJ>(or $label or &label if what you really want is an address) <OriansJ>in which case you would write LOAD_IMMEDIATE_ecx %42 and it would convert into B9 2A000000 <OriansJ>which will exactly match the objdump of mov ecx, 42 <OriansJ>The names are anything you want them to be <OriansJ>but clarity and explicitness in naming usually will make life easier for you <pder>OriansJ: I wanted to try out your latest changes on blynn-compiler. Are you thinking that rts.c might not be necessary? <fossy>OriansJ: ok, what's the ! or @ or %? <fossy>um, any ideas for convincing <ericonr>fossy: I'm not sure you'll manage it... the argument I can think of is that binaries can vanish, but it isn't that great either, since the source can vanish just as well :/ <fossy>ericonr: i doubt it as well, and i kinda understand why they are saying no ***clever_ is now known as clever
***clever_ is now known as clever
<OriansJ>pder: I am thinking the compiling and rts.c might not even be required at all. Just unpacking it however is going to take a bit of work but should be managable. As it looks essentially larger and larger ROMs run on the virtual machine. with rts.c just being slightly different than vm.c <OriansJ>fossy: you can't convince them. They don't appear actually interested in bootstrappabilty of their binaries. As that would require them to worry about a potential future where someone might compromise their build machine. <OriansJ>even guix has a hard time unwinding a binary history because source tarballs can stright up disappear or change (resulting in different checksums) on third party suppliers. But I do hope guix and nix come together to solve that <fossy>OriansJ: yeah, i think you're right <fossy>OriansJ: do you think we can run everything in vm.c <deesix>OriansJ: Do you think that the RSHIFT_X0_X1_X0 change is the way to go? I'd think we want both kinds of shift defined and use arithmetic_recursion() to select between them at additive_expr_stub(), do we? <OriansJ>deesix: honestly you know more about aarch64 than I do and it appears that yt believes it would fix hex2's behavior (when built by M2-Planet). But yes the generally correct behavior in C is to have both. <OriansJ>So the question is hex2 built by M2-Planet needing that change? <OriansJ>as we do want yt's mescc-tools-seed work to get matching behavior in hex2, M1, blood-elf and M2-Planet on AArch64 as all of the other ports. <OriansJ>fossy: not everything but probably all of the stages that use rts.c (with possibly more extensions to vm.c still have to unpack all of that) <siraben>OriansJ: wrt. to the binary blobs in the beginning, I will be able to get rid of them later <OriansJ>but first I have to figure out how to unpack the generation of raw into discrete smaller steps; to make vm.c much more similiar to rts.c and thus allow a smoother transistion. <siraben>rts.c is code that is mean to be appended to the Haskell-generated C code to make a standalone file right? <OriansJ>siraben: prefixed to the Haskell generated output yes but most of it appears to be the same contents as raw <OriansJ>virtually all of it is in the prog blob <siraben>prog is the IOCCC entry I would not recommend using it <siraben>anything related to IOCCC should be removed no problem <OriansJ>siraben: I pulled all of the IOCCC out of vm.c <OriansJ>and there is no IOCCC in rts.c; only a prog blob which appears to match the contents of raw (in style but not content) <OriansJ>So I am guessing it just vm byte-code <OriansJ>which is why getting rid of rts.c is more reasonable <OriansJ>vm.c just becomes an interpreter of the byte-code that expands in complexity with each step. <siraben>I believe it's some weird base encoding too <siraben>Ah, so you want to keep it compiled all the time? That would be a good idea <siraben>Looks like all precisely.hs does is add bignums <deesix>OriansJ, I didn't look at the hex2 problem, and I think the question is more about M2-Planet than AArch64. <deesix>OriansJ, AFAIU the commit is trading a shift with the other, not supporting both. <OriansJ>deesix: I agree with that assessment; but the question does hex2 not behave correctly on AArch64 without that change, determines if we revert it or keep it. As the goal is consistent behavior on ALL architectures. So that a program made by M2-Planet doesn't behave differently on x86 or AArch64. <OriansJ>if one build mes-m2 on AArch64 and runs a scheme program, the output should be identical on x86 running mes-m2 built by M2-Planet <OriansJ>if the hardware instruction is doing the wrong thing there; we need to work around it. <OriansJ>if the hardware instruction is doing the right thing; the question is about why the behavior change or if it was just a mistake that needs to be reverted. <deesix>OriansJ, I don't think the hardware is doing anything wrong. Just M2-Planet not supporting both right shifts for AArch64. <deesix>I'll talk with yt about it, as I can't test my idea right now (no setup at hand). <deesix>OriansJ, about M1.scm of slow-utils... I'm pretty sure I have the lost file is you still want it. <siraben>OriansJ: M2-Planet not found, is the right command M2? <OriansJ>deesix: sounds like a good idea and of course I'd love to get a copy of your M1.scm work ^_^ <OriansJ>siraben: assuming you are trying go.sh; you need to build M2-Planet and install it into your path. <siraben>Ok, I'm making Nix expressions for them now <OriansJ>you can of course get them from mescc-tools-seed as they are all generated there <OriansJ>and if you want I will accept Nix and guix build scripts into mescc-tools and M2-Planet (and mes-m2 as well) <OriansJ>siraben: M2-Planet was just the named picked as the C compiler bootstrapped from M1 <OriansJ>as it is a PLAtform NEutral Transpiler <OriansJ>it was a logical step after I wrote the C compiler cc_x86.M1 <siraben>After building mes-m2 I only have mes-m2 in my bin folder <siraben>Ok, I was just confused where the binaries were coming from <OriansJ>don't forget mescc-tools for M1, hex2 and blood-elf <siraben>Heh, the blynnc-ompiler part is taking a while <siraben>OriansJ: what should I do for the install phase of blynn-compiler? At the moment there are none <siraben>OriansJ: I get 5f229bf7ec16c465e0c8267206348ff8cf69bb694572a036da78ba4d088d4fe8 as the sha256 for vm, is that right? <yt>deesix: happy to talk about the right shift/hex2 thing when you have some time <OriansJ>siraben: we are not yet at the level of install blynn-compiler yet and M2-Planet generated binaries can be up to 10 times slower than GCC binaries. <yt>OriansJ: I ended up changing to a 0x600000 offset because that's what I was doing in the bootstrap script, which I think came from amd64 originally <OriansJ>siraben: on blynn-compiler commit 7b52a3ec0f46333cb70bddd4fbbb7a7e3b8efbb9; I get 2c311678f1a2a516ee8a0177cbb4031091777c1f0bb3e776edcb0b1deb6d271c <yt>not wedded to any of them though, and I'm not sure if there is a "standard" as such on aarch64. surely it's just a linker implementation detail? <siraben>OriansJ: this was made by running `go.sh`. right? <yt>I'll have a look at your patched makefile and see whatever is going on with hex0 not being identical between versions... <siraben>What's the contents of vm look like? <OriansJ>but I'll double check, we can also compare bin/vm.M1 files to quickly find any differences <OriansJ>yt: yes it absolutely is a linker implementation detail but if you notice hex2 is our linker here. <OriansJ>and thank you yt for taking a look at that for me <OriansJ>siraben: does objdump -d bin/vm look more reasonable <OriansJ>nope still getting 2c311678f1a2a516ee8a0177cbb4031091777c1f0bb3e776edcb0b1deb6d271c <OriansJ>siraben: do you get f7abe737ad6415024575fe979bc353b858c16119bce8e07e9b2b36ee1ec36cf3 bin/vm.M1 ?? <siraben>I just cleared my bin folder, retrying <OriansJ>git checkout 7b52a3ec0f46333cb70bddd4fbbb7a7e3b8efbb9 && git clean -xdf && ./go.sh will always do the trick <deesix>yt, hello. If you look at the use of RSHIFT_X0_X1_X0 at cc_core.c, I think we can use arithmetic_recursion() like in signed/unsigned div, provided two different definitions for arith/logical right shift, sounds about right? <yt>deesix: if that's how M2-Planet is supposed to pick the correct type of right shift, then yeah, that sounds good. <yt>always using logical right shift is clearly wrong imo, as it just won't do the right thing on negative signed numbers <siraben>OriansJ: interesting, whatever I pass to vm it prints it to stdout <yt>which is what I hit in hex2, in a test calculating armv7l relative offsets (which need to be shifted right) <siraben>f7abe737ad6415024575fe979bc353b858c16119bce8e07e9b2b36ee1ec36cf3 bin/vm.M1 <yt>OriansJ: ho hum, I get identical checksums on all the versions of hex0, but a different checksum than you had <yt>are you using the most recent version of hex2? with the rel24bit offsets for aarch64? <deesix>yt, looking at other targets... they pick using that function (based on signed/unsigned of the type). <yt>deesix: that makes sense then. I can make that change instead <Hagfish>(should there be a script which is run before committing which rebuilds everything and stores the hashes in a config/log file, so that people who clone the repo can check the compilation on their machine?) <OriansJ>yt: oh I was using the version of hex2 from mescc-tools-seed; my bad. <siraben>Alright, I made the bootstrappable packages an overlay <siraben>blynn-compiler depends on mescc-tools, mes-m2, m2-planet <siraben>Only thing is, bin/vm seems incorrect <OriansJ>Hagfish: well we could steal the idea from M2-Planet of SHA256SUM files and make Generate-test-answers <OriansJ>yt: double checked and they are all identical. Your hex0 will be merged shortly <OriansJ>siraben: your blynn-compiler pull has been merged in <OriansJ>yt: not yet but I'll look at it promptly <OriansJ>oops; looks like I made another off-by-one error. Thank you yt; that will be merged promptly <siraben>OriansJ: do you want to check the hashes again? <yt>OriansJ: no worries! one of the most common errors in software, I'd bet. at least all it does is cause a null pointer dereference, not some horrid memory corruption (been there, done that!) <OriansJ>siraben: M2-Planet doesn't encode path information with the binaries it generates <OriansJ>so unless you changed the source code in vm.c or one of the input files; the checksums will never change. <siraben>2c311678f1a2a516ee8a0177cbb4031091777c1f0bb3e776edcb0b1deb6d271c ./bin/vm <OriansJ>siraben: algebraically and all of the *.hs for example <OriansJ>M2-Planet will always produce identical output given identical input with matching options regardless of host <siraben>Yay, so not only can we build blynn but also have the other parts all captured in Nix <siraben>I should enable the tests as well, ping me again when you're done reviewing <OriansJ>siraben: I was thinking of pulling the tests in vm.c out into a proper test folder structure to enable easier expansion and incremental testing without having to recompile vm.c <siraben>OriansJ: which of mescc-tools m2-planet mes-m2 have tests? <OriansJ>siraben: all of them have tests (in the test folder following the hello.sh standard (except for kaem because it is a shell)) <yt>OriansJ: thanks; I'll pull out and clean up hex1 next <OriansJ>yt: sounds great look forward to reviewing it <siraben>Hehe, m2-planet test failed because of no `which`, Nix makes dependencies quite explicit. <yt>OriansJ: ah, would you be able to add a hex0-seed for AArch64 to bootstrap-seeds? I think I forgot to include that repository in my contribution request at work (and someone somewhere might panic about committing binary code...) <yt>then I can add the first two steps of the bootstrapping script as well :) <OriansJ>yt: bootstrap-seeds needs two binaries for an architecture <OriansJ>as I doubt a x86 binary shell will work on AArch64 <OriansJ>but if you wish to do it as 2 seperate commits that is fine and I'll just merge the hex0 binary for you <siraben>OriansJ: not sure why, but when I enable tests in m2-planet the install fails <siraben>It tries to move M2-Planet to the out directory but M2-Planet is in bin/ <OriansJ>siraben: the tests don't need to be run <siraben>I guess I could make this a job for the CI, yeah. <siraben>I have experience with GitHub Actions. <siraben>It'll be very easy to add it once Nix build support is merged <siraben>That's one thing I noticed across these repos, no CI <Hagfish>i think it will really show potential contributors (and other people curious to learn more) how "alive" this project is <siraben>Before joining the IRC channel I thought it was stalled, actually <siraben>Didn't hear about m2-planet or mes at all, just stage0 <Hagfish>hmm, yeah, the publicity side of things has been hit and miss <siraben>Not only that we can use GitHub actions to produce binaries so people can download them and verify them <OriansJ>siraben: well I really didn't have much time to get CI included and honestly I was hoping someone who cared about it would be willing to help add it. <Hagfish>yeah, it's not been a priority so far, but if it's easy now it's maybe worth investing a little time in <OriansJ>Hagfish: I encourage you to lead that effort. <OriansJ>as bylnn-compiler is my current priority. <OriansJ>I guess writing 5+ compilers in assembly is less interesting than just writing 1 compiler in assembly. <siraben>So future commits and PRs will be run with the CI <siraben>I could add a separate workflow to check the hash of bin/vm and see if it's as expected <siraben>I think the README would be better updated to reflect this fork of blynn-compiler <siraben>It would be good to regularly bump versions of the dependencies as well <siraben>OriansJ: does mescc-tools development happen on savannah or github now? <OriansJ>but officially savannah is always the master <OriansJ>as savannah is an FSF approved source repository and this all is going to be part of Guix <OriansJ>siraben: so do you think yt should message dang (the HN moderator) when we merge AArch64's cc_aarch.M1 work? So that stage0 might remind people that it is very alive and active. <Hagfish>i don't like creating accounts on places, but HN is the one site i've made an exception for <Hagfish>anyway, if you want to get publicity from HN, the trick is to write a new blog post with a clickbaity/clear title, and post the link to that <siraben>OriansJ: I'm not sure. I haven't done it myself <siraben>I think beyond posting on HN and whatnot, "modernizing" the project would help <OriansJ>the funniest part is the cc_x86, cc_amd64 and cc_armv7l were all considered reposts of the knight cc_x86 <siraben>CI, badges, other flashy things, heh <siraben>Yeah, if it was considered a repost mentioning that would help <siraben>I could do the email perhaps, I'm pretty active on HN <yt>OriansJ: I still need to port kaem; I'm just using bash to run the script now, so yeah, two separate commits to bootstrap-seeds would be great <yt>maybe when the blynn-compiler is part of the bootstrap, we could do a blogpost for HN for a full source bootstrap on AArch64 ;-) <OriansJ>siraben: "modernizing" isn't exactly compatible with bootstrap everything from a 257byte hex0 binary but I get the desire to attract attention but I don't want to attract those programmers who do complexity for its own sake but rather those interested in the bootstrap problem itself. <siraben>That's a valid concern. I think people will self-select, however, once they see the goals of the project <siraben>OriansJ: it appears mes-m2 hasn't had a release tagged in a while, since Jul 16, 2019 <OriansJ>siraben: well life has been a bit crazy since my wife got pregnant and we had our son. <siraben>Of course, no worries. I'm just considering whether to track the latest master commit or tagged release <OriansJ>as no commit should ever break the build or the tests <OriansJ>and I am about to break mescc-tools-seeds by updating bootstrap-seeds; so everyone in 15 minutes update your copy of bootstrap-seeds and then mescc-tools-seeds as I have to chance both. <siraben>In which packages are the seeds used? <OriansJ>(and I am making boostrap-seeds a submodule in mescc-tools-seed to reduce future issues) <OriansJ>including janneke's guix bootstrap work <OriansJ>but this will be fixed in 5 minutes and there will be no external changes <OriansJ>now mescc-tools-seed will be entirely self-contained. a one and done repo of bootstrappablity <OriansJ>bootstrap-seeds is now a submodule of mescc-tools-seed <siraben>OriansJ: using the seeds is it possible to bootstrap blynn-compiler? <xentrac>CI would be pretty worthwhile, even without badges :) <xentrac>just in the sense of publishing continuous feedback on status <OriansJ>siraben: yes to the generated/lonely.c stage <OriansJ>siraben: once we fixup vm.c to remove the need for rts.c we will be all the way until generated/uniquely.c <OriansJ>siraben: mescc-tools-seed builds M2-planet and mescc-tools and mes-m2 <OriansJ>so we can simply make it a submodule and build it <OriansJ>(although we would probably write architecture specific versions for AMD644, armv7lo and AArch64 <siraben>I see. I'll try to build mescc-tools-seed then make mes-m2 use it as a dependency <siraben>OriansJ: using the seed then gcc won't be a dependency anymore for mescc-tools, m2-planet? <siraben>Heh, at this point it's probably putting too much into blynn-compiler, I'll maintain this as a separate overlay. <OriansJ>siraben: using the seed there is no dependency on anything <siraben>I'll keep the Nix-related changes to only be able to build blynn-compiler <OriansJ>siraben: using standard compilers is fine as the CI would only be checking for common development work flows <OriansJ>as it is much easier to debug gcc binaries than M2-Planet binaries <siraben>OriansJ: mescc-tools-seed tails during install: `fatal: not a git repository (or any of the parent directories): .git` <siraben>I could override it with a `cp` to $out/bin <OriansJ>siraben: and merged your latest pull request <siraben>I just tried to build m2-planet using the mes-m2 binary from mescc-tools-seed, and got a segfault <OriansJ>siraben: mescc-tools-seed builds M2-Planet too; so no need <OriansJ>wow yt you went overkill in hex1; with support for absolute addresses but I am guessing it was needed in AArch64 <yt>I originally only had 16bit absolute addresses, but after your suggestion earlier to use 32-bit addressing I had to add those as well! <yt>OriansJ: and the relative addressing help massively with b and bl addressing so that went in too <yt>I was really happy when I was doing hex2 to have those addressing modes; finally no more hand-editing address offsets :D <OriansJ>yt: well the relative addressing was the major pain point of basic hex0 programming and hex1 was just designed to be the minimal number of manually calculated branches required to add support for relative addressing. <yt>OriansJ: fair, I guess the absolute addresses only need to be calculated once and can be filled in everywhere <yt>but they're also easy to add to hex1 so here we are :) <OriansJ>yep, AArch64's hex1 is the most advanced of the bunch of hex1 <OriansJ>and biggest (x86 => 689; AMD64 => 861; knight => 472) <yt>OriansJ: I'm sure there is a lot that can be done to bring the size down for the AArch64 tools, at the expense of more different instructions and perhaps reduced readability <yt>The lack of a useable load immediate, or load address is really hurting on AArch64 <yt>At least the instructions themselves are 32 bits, or it would be *really* big <OriansJ>yt: I would not compromise readability for size reductions. <OriansJ>but I should be done auditing hex1 by tonight. <yt>OriansJ: great! I should have catm ready to go up when that goes in; no rest for the wicked ;-) <fossy>OriansJ: thank you for writing the generate_m0_definitions.org file it is v useful <OriansJ>yt: well one could say the first path to bootstrapping is "Math hard" but the second is just "Exercise hard" <fossy>i got a bootloader that boots and then halts the cpu! <fossy>(yes i know it is very basic and useless) <fossy>now time to try and get it to write hello world <yt>fossy: what system is that on?