IRC channel logs
2021-10-02.log
back to list of logs
<Hagfish>" The program is initially focused on rewarding the following work: Software supply chain security improvements including hardening CI/CD pipelines and distribution infrastructure." <Hagfish>i don't know if google fully appreciates the significance of bootstrapping work when it comes to SSCS, but they're spending $1m just to start, so there may be people/projects here that should look into this <Hagfish>actually they are offering grants of between $500 and $10,000, and i think this tier may be the most relevant: <Hagfish>"$5,000-$10,000 for moderately complex improvements that offer compelling security benefits." <xentrac>I think there are people at Google that do fully appreciate it <Hagfish>i think i've seen an acronym like that used before, but yeah, sorry, i sort of relied on it being reverse engineered from the context :) <xentrac>they developed their internal Titan CPU specifically to solve the bootstrapping problem we're working on <Hagfish>the question is, are those people at Google the ones that decide how the rewards are distributed? <xentrac>and they've also done most of the work on its OpenTitan successor <Hagfish>it's sort of hard to quantify the security benefit of bootstrapping <xentrac>I think they have spent significantly more than $1m developing OpenTitan <Hagfish>if it did uncover some 40 year old self-propagating backdoor, then the value in finding and fixing that is almost limitless <xentrac>yeah, but it's much more likely to prevent the introduction of a new one <Hagfish>right, that's a better way to frame it <stikonas>oriansj: so now with my local M2-Planet port I can build up to M1-0 and M1-0 seems to be working fine. hex2-1 is somehow not working properly (runs but outputs somewhat corrupted files) <oriansj>stikonas: that would be a broken chmod function <oriansj>oh, then that is different than the wrong permission bits issue I've seen with hex2 in testing <stikonas>blood-elf and M1 probably exercise quite a bit of M2's functionality <oriansj>well M2 just converts C to M1 without changing the things that would be valid in M1. (Like the inline assembly and immediates (unless the architecture requires it)) <stikonas>hmm, hex2-1 does seem to work on at least some files (e.g. catm_riscv64.hex1) <stikonas>it's probably something in ELF header that upsets it <stikonas>ok, it's the . notation that triggers some corruption <stikonas>might be something to do with unsigned overflows... <stikonas>since Development/hex2_riscv64.hex2 file works just fine (and it does not use . notation) but ".11111111 12345678" already fails <stikonas>hex2.word in '.' processing uses constant 0xFF000000 <stikonas>which is probably outside what we correctly handle with LUI/ADDI <stikonas>oriansj: yes, that's what was causing it <stikonas>would you be willing to accept patch to workaround this (we should probably fix large numbers eventually) <stikonas>i.e. replace 0xFF000000 with (0xFF << 24) <stikonas>with this change I can build up to at least full kaem <stikonas>but most likely M2-Planet will self-host too <stikonas>argh, somewhat annoyingly self-hosted M2-Planet is broken, I guess same issue of large immediates <stikonas>I think we need to implement large numbers manually... Binutils is doing something like that <stikonas>although, large numbers break a bit earlier in M2-Planet toolchain <stikonas>I think it's working up to 2^31 - 1 - 2^11 <oriansj>test1000 also tends to break real easily by design <stikonas>yeah, I haven't implemented test1000 yet <stikonas>although, it's mostly copy paste from mescc-tools-full.kaem file <oriansj>easy mistake to make. Happens all the time <stikonas>strange, I'm getting different hashes (compared to pastebin) in proof.answer <oriansj>well that means we are doing something different <stikonas>I get 92c8b0a0e142e2a825bd10cb7d7e2d581ca2ba608f4e21009c1d9837728205d2 test/test1000/proof <oriansj>less me check to see what M2libc I am on <stikonas>I've also added --little-endian to that blood-elf command in test1000 <oriansj>M2libc is at b58dbe11e6111489aca54c57a8bd1087a9d5db90 <oriansj>I am on M2-Planet commit f5f5f8942de9f89b2eed6290a2442d38cb2e1397 <oriansj>and I am doing an x86 and armv7l build test <oriansj>(which all architectures should produce identicial results) <oriansj>yet different build results when running RISC-V? <stikonas>well, that 92c8b0a0e142e2a825bd10cb7d7e2d581ca2ba608f4e21009c1d9837728205d2 hash is amd64 test <stikonas>oriansj: which one is the new and which one is the old? <oriansj>the old is the minus and the plus is the new <oriansj>you can also git apply the paste and git will understand <stikonas>well, I just wanted to check that it indeed matches, but it does <oriansj>and I'll have the updated blood-elf flags in place in a couple minutes <stikonas>and then I guess stage0-posix needs to pull-in new M2-Planet... <stikonas>(or maybe I should first try to fix large numbers) <stikonas>although, that should be a smaller change and only affect risc-v <oriansj>well we might as well get risc-v fully into M2-Planet <oriansj>and then I'll update stage0-posix x86, AMD64 and AArch64 ports to use the new blood-elf flag <oriansj>and the blood-elf flag update for M2-Planet is now up <oriansj>and I assume you can make test/test1000/hello-riscv64.sh to mirror what is in AArch64's test/test1000/hello-aarch64.sh <oriansj>which will make risc-v output part of the test suite <oriansj>actually that is fine as cross-hosting checks, even for broken output matters <oriansj>as all architectures given the same input (and flags) must always produce the exact same output <stikonas[m]>Well, I guess test will fail for now but rightfully so <oriansj>well how does this sound as a reasonable compromise: add the tests that currently pass on RISC-V and I'll verify that other architectures confirm matching results <oriansj>also if RISC-V is producing a correct proof result for test1000, it means it self-hosted just fine <stikonas>ok, let me pull in blood-elf changes and then I cna add tests <oriansj>because if you see: out=test/test1000/proof: OK when running test1000 for RISC-V you are done successfully porting M2-Planet to RISC-V <oriansj>and if it is the wrong checksum, we then compare the proof files to hunt down the problem. <oriansj>do you get 626ca275bbb320e6b5ebde0bff50d10bd1dc78ee8163d37f71fb77d10099ca7c test/test0020/tmp-riscv64/struct.M1 ? <oriansj>and 8e4c805f0aa40c8904f11811e827a5a4198165148b96f195d3f6acd027039202 test/results/test0020-riscv64-binary <stikonas>maybe my hex2 (riscv64 build) is indeed buggy <oriansj>is this your hex2? b6f289956fd3623f0545e8521ec2023b042109d83a3791c3aab4014b6116dd2d test/test0020/tmp-riscv64/struct.hex2 <oriansj>good. So now we have reduced it down to a local hex2 issue <oriansj>use GCC to build hex2 to eliminate self-host as a potential cause <stikonas>I can see 0xfffff000 in hex2_word.c where ~ is handled <stikonas>and AUIPC instruction is where crash happens <oriansj>in fact you should just do a GCC build of all of mescc-tools and use those binaries while hunting down issues in M2-Planet <oriansj>that way we are only testing one thing at a time <oriansj>after we get M2-Planet happish, we will go back to figure out why it isn't happy in the self-host with something known good to compare against <oriansj>eded68800781a8986f0a0703edee6ebf6f23251365b7fb40d1cc04462e322869 test/test0105/tmp-riscv64/lisp.M1 <oriansj>62d7d6b7bae3c5006bcce96c726d7fb918ff84971c5a2b6c3d397765cded3245 test/test0105/tmp-riscv64/lisp.hex2 <oriansj>0a1cb20f6420950ade08152e8e25b546158db8bc4f13fe9bc5747fa8240b48ce test/results/test0105-riscv64-binary <oriansj>trying to reproduce crash here (no crashes yet) <oriansj>how can two different people running the exact same static binary produce different behaviors <stikonas>well, I get occasional passes and occasional crashes <stikonas>and remaining tests pass with gcc'ed mescc-tools <oriansj>commit that. I'd like to verify we both have making checksums for all of the RISC-V binaries <oriansj>make Generate-test-answers will populate that after the tests run <oriansj>the proof only changes if the code that makes up M2-Planet changes <stikonas>so I think it's mostly hex2 issue that's left <stikonas>I can add another workaround like the one you merged... <oriansj>lets not jump to conclusions with hex2 <stikonas>although, I think M2-Planet also needs to learn to handle large numbers <oriansj>M2-Planet handles 32bit numbers just fine <oriansj>it uses 0xFFFFFFFF and 0xFF000000 without issue <oriansj>So we need to understand what special care RISC-V needs <stikonas>on risc-v it issues AUIPC + ADDI which fails for 32 bit numbers, works fine until routhly 2^31 <stikonas>I think just need to issue more assembly commands to shift everything to lower immediates <oriansj>we are not binutils, so we need to understand what needs to be done at the individual instruction encoding level <oriansj>for example do we need to do sign extension to full 64bits of the register? <stikonas>there might be some mess up with addi vs addiw <oriansj>So when one does 0xFF000000 what does the register look like and what should it look like <stikonas>well, I tested with basically the followign program, int i = 0xFF000000; convert to string; fputs(...); <oriansj>on amd64 and AArch64 it sign extends <stikonas>well, on riscv you can't just load 0xFF000000 <stikonas>but I think it's limitted to 32 bit signed integers <oriansj>ok and what do AUIPC+ADDI need to ensure the correct result? when needing to put 0xFFFFFFFFFF000000 into the register? <stikonas>if we use ADDI then it sign extends to 64 bits, if we use AUIPC + ADDIW then it limits to 32 bits and sign is in 32'nd bit <stikonas>sorry, LUI instead of AUIPC, but rest applies <oriansj>ok and what does LUI need to be for 0xFF00000000 ? <oriansj>armv7l, x86, amd64 and AArch64 confirming riscv64 test results <stikonas>well we do ~0xFF000000 which means LUI is getting FF000 as its immediate, hmm <oriansj>Ok, so is Hex2 outputting that value? <stikonas>checking, I did that ~0xFF000000 test and binary contains lui with 0 as immediate <stikonas>well, not it loads 0xffffffffff000000 into register <stikonas>seems to work but that might not be the ideal testcase because ADDI gets 0 as immediate <oriansj>and does hex2 produce 0 as the immediate given !0xFF000000 ? <oriansj>well it wouldn't be that hard to iterate through all 2^32 possible inputs and make sure hex2 and M1 do the correct thing for all of them. <stikonas>sorry, should actually point out to M1 code <oriansj>So if it works fine in M1+hex2; which means the binary is correct, what is the problem? <oriansj>where is 0xFF000000 causing an issue? <oriansj>M2-Planet is outputting the correct M1 instructions correct? <stikonas>well, the problem is that M2-Planet segfaults <stikonas>(and also remember that test0020 that failed with non-gcc'ed hex2) <stikonas>I think M2-Planet is outputting correct thing <stikonas>although, I'm not fully sure now, getting a bit confused with this bug <oriansj>So let us look closer at hex2 for a second. <oriansj>So build hex2 with M2-Planet and keep the gcc hex2 around <oriansj>and do the following have them both build test0020 binaries from the exact same hex2 file <oriansj>as hex2 should always behave the exact same regardless of host or target <oriansj>and where they are different let us take a closer look <oriansj>xeh will help with doing a compare with the resulting binaries <oriansj>addiw a0, a0, 1900 vs sd s1, 240(a3) mv s2, a4 <oriansj>well that would certainly cause a segfault <stikonas>oh, maybe it's not immediates but label jumps <stikonas>those differences are in the second line of AUIPC/ADDIW pairs <oriansj>can we distill the hex2 down into a smaller test by elimination <oriansj>it doesn't have to run but be different <oriansj>since they take the same hex2 as input. Cut it in half <oriansj>bisect the input hex2 file until only the parts that are different remain <stikonas>hex2 would complain once I cut out functions... <stikonas>well, yes, if I keep labels, that should work <oriansj>unless the issue is a specific offset size but then they will both match and you'll know you removed the diffference we are trying to distill <stikonas>well, I'm removing one function at a time while keeping label <stikonas>so at the very least we should get a smaller test file <oriansj>and if it is a difference you can replace lines with '00 00 00 00' to isolate the issue to the instructions in question <oriansj>must smaller problem space to search <stikonas>in the end it will probably be some stupid bug <stikonas>I can still try to minimize .hex2 file a bit <stikonas>once I have 1 difference, we can just calculate offset <stikonas>this is still producing 1 byte difference <stikonas>file size is 2060 - first and last instructions would be 2052 <oriansj>So the difference is the 00050000 .00000500 !FUNCTION_putchar 1B000000 <oriansj>is the value = value + 4; the problem? <oriansj>would moving it after shiftregister = shiftregister ^ tempword; produce the correct behavior in the M2-Planet tests? <stikonas>but that's why problem comes at 2060 and not at 2056 <stikonas>I specifically use addiw a0, a0, 4 and slliw there in hex2-0 <stikonas>but M2-Planet issues addi and slli there <stikonas>but using addiw in M2-Planet would break other things... <stikonas>all the pointer arithmetic would go wrong if result is restricted to 32-bit <oriansj>we don't use addiw when doing pointer work <oriansj>we use addiw when working with integers <stikonas>ok, I'll try to change some emit_out commands but after lunch <oriansj>and I'm going to try to distill your test further <oriansj>67800000 $_END_IF_fseek_5 6F000000 ?? <oriansj>(if so there is exactly two calls into UpdateShiftRegister which makes for easy simple stepping to see the problem) <oriansj>then we will know exactly where it goes wrong <xentrac>huh, I didn't know AUIPC was capable of causing a crash <stikonas>xentrac: it's not AUIPC that's causing a crash <oriansj>xentrac: it is if hex2 converts addiw a0, a0, 1900 into sd s1, 240(a3) mv s2, a4 <oriansj>So we are figuring out exactly why when built by M2-Planet hex2 does that <oriansj>no worries, shared understanding is essential to effective cooperation <oriansj>I guess I need to build hex2 from M2 myself to get a better look at this bug <stikonas>although, I should probably build hex2 with gcc-riscv64 <stikonas>amd64 version has different registers, so harder to compare <stikonas>well, we have 2 good hex2's one is hex2-0 and the other is gcc'ed hex2 <oriansj>one can also compare the M2-Planet build hex2 (amd64) against the riscv64 M2-Planet build as they should be doing functionally the exact same thing <stikonas>yes, there are a few things we can try that might all uncover where exactly things go wrong <stikonas>oriansj: so I can see some difference already <stikonas>but riscv64 register is 0xfffffffffffff000 <stikonas>because on amd64 version I'm printing "p tempword" directly <oriansj>well use gdb to hand set the register to see if it fixes the output <stikonas>hmm, something very strange is happening <stikonas>when I read shiftregister next time it's 0xffffffff <stikonas>probably need to go back and see how it got this value <stikonas>hmm, it got changed at some point between two invocations of UpdateShiftRegister <stikonas>oriansj: and yes, handsetting it to 0xfffff000 fixes the output <stikonas>should we just restrict it in hex2_word.c to 32-bits using bitmask? <stikonas>hmm, although, that's not really the right thing to do... <stikonas>should figure out why M2-Planet generates it incorrectly <stikonas>oriansj: hmm, arithmetic_recursion should issue the second string for unsigned? <stikonas>but it's right, the correct 2nd string was issued <stikonas>I can workaround this by restricting value to & 0xffffffff <stikonas>but I'm a bit confused how to fix this properly in M2-Planet... <stikonas>that UpdateShiftRegister function really assumes that it's variables are 32-bit <stikonas>with hex2_word.c workaround M2-Planet is self-hosting and seems to work fine <stikonas>although, let me check what happens if I replace add->addw in primary_expr_stubs <oriansj>can you confirm that the patch fixes hex2's behavior in regards to the M2-Planet produced hex2 <stikonas>that change add->addw etc breaks pointers, so I'll undo it <stikonas>well, as expected since it breaks all long int operations <stikonas>oriansj: oh, actually, that will only fix value if . is encountered <stikonas>swap is not inside UpdateShiftRegister_DOT <stikonas>no, I think I still get M2-Planet segfault <oriansj>(also trying to clear out things that might cause issues with M0) <stikonas>yes, those would cauase issues with M0... <oriansj>as now UpdateShiftRegister should not include any values exceeding 24bits <stikonas>I am seeing a couple of unrelated problems with my tests but maybe let's deal with hex2 first <oriansj>easiest to solve one problem at a time <stikonas>at least hex2-0 is not as slow as on aarch64 <stikonas>it takes about 2 minutes for hex2-0 to link everything <stikonas>GAS version of hex2 actually works much faster, maybe 10s <oriansj>well C-x e in bash really is so handy if one sets export EDITOR=emacs <oriansj>lets confirm and then isolate next possible problem <oriansj>I'll commit if it does fix the behavior in question <stikonas>yes, it looks like testcase is passign now <stikonas>when we had a lot of bugs, it's easy to confuse yourself... <oriansj>yep, now we are slowly knocking out the last <oriansj>So using gcc built M2-Planet, M1, blood-elf and hex2 compared against M2 built output, what is the first layer that is different? <oriansj>or is it identical all the way down to binary <oriansj>The first program that produces a difference is where we need to look for the next bug to crush <oriansj>so another hex2 bug or matching binaries? <oriansj>did you update the C source for hex2? <stikonas>(--- is the correctly behaving binary, +++ is bootstrapped) <stikonas>probably if ((value & 0xFFF) < 0x800) tempword = value & (0xFFFFF << 12); <stikonas>if value is still 64-bit in the register <oriansj>yet (0xFFFFF << 12) can only be a 32bit value <stikonas>I don't see anything else that might go wrong in UpdateShiftRegister <stikonas>oriansj: ok, I have a patch that fixes it... <stikonas>((0xFFFF << 16) + 0xFFFF) might be a bit over the top, I can try 0xFFFFFFFF <oriansj>(0xFFFF << 16) | 0xFFFF will always be exactly 32bits of 0xFFFFFFFF <oriansj>until I teach M2-Planet to properly deal with 16, 32 and 64bit integer types <stikonas>just doing final test with & ((0xFFFF << 16) | 0xFFFF); <oriansj>however you prefer as RISC-V is your success <stikonas>well, you did a lot of work too. Anyway, it works and I have change locally <oriansj>I'm just happy to see stage0 continue to make progress in ways previously thought impossible ^_^ <stikonas>well, with risc-v we had to add word-based M1 syntax <stikonas>which was slightly annoying to implement <oriansj>and once solid enough it'll enable MesCC to finally be able to support RISC-V <stikonas>the tools have some testing in the field <oriansj>and syntax details have been worked out <stikonas>not just written speculatively and hoping that it would work <stikonas>hmm, cd doesn't seem to work in kaem... should check M2libc function <stikonas>oh, I'm misreading things again :(. It's not cd but blood-elf needing new flags <oriansj>yep, I'm updating x86 blood-elf flags right now, then I'll be doing the AMD64 and then AArch64 <stikonas>need to add another ${ENDIANESS_FLAG} in mescc-tools-extra.kaem file <oriansj>well ENDIANESS will be the same for hex2, M1 and blood-elf and the flags are identical for all tools <stikonas>although, I have a very strange problem with kaem-minimal <stikonas>I added ../bin/kaem --verbose --strict -f mescc-tools-full-kaem.kaem <oriansj>well sanity check with the gcc built kaem to see if it would run without issue. Then we can see if we found a new issue <stikonas>and I think another bug is that kaem-minimal on riscv64 does not stop if child segfaults... <stikonas>ok, first issue is probably a bug in my kaem-minimal assembly <stikonas>looks like it does not pass through any shell variables <stikonas>that's the only difference I can see from strace <oriansj>well kaem should function just fine without any shell variables <stikonas>env -i ../bin/kaem --verbose --strict --file mescc-tools-full-kaem.kaem <oriansj>it is literally the standard nightmare-mode <oriansj>as one can't expect an environment when starting from kaem-optional <stikonas>although still, populate_env shouldn't crash on empty environment <oriansj>well there is an easy way to check to see if it is <oriansj>set --init-mode and see if it clears out the issue <stikonas>I'm just looking at what's causing thecrash <stikonas>while(n->next->var != NULL) n = n->next; <oriansj>guess we need a single require in populate_env <oriansj>just require(NULL != envp, "should not have received a NULL environment"); <oriansj>or return depending how you wish to treat a null envp <oriansj>yeah if(NULL == envp) return will do the correct thing <oriansj>yep, fossy took my minimal kaem and made it into a much more real shell <oriansj>impressive work honestly given the bugs of M2-Planet back then <stikonas>yeah, it might benefit from a few other improvements but it's quite impressive now for such small piece of software <stikonas>now live-bootstrap ends up with loads of kaem processes waiting <stikonas>kaem-micro actually does the right thing <oriansj>it is kinda amazing how well a buggy C compiler's output hangs together. <stikonas>it's far simpler than what I imagined C compiler would be <stikonas>needs some workarounds in C but definitely easier to write than assembly <oriansj>in contrast with scheme/lisp interpreters which are *VERY* touchy (especially in regards to garbage collection) <stikonas>oriansj: I think you need a * before envp there <oriansj>and honestly *envp is probably the wrong thing everytime its written <stikonas>I guess I'll leave riscv32 for somebody else to do :) <oriansj>possibly depending on the amount of differences <oriansj>and how skilled the developer is in assembly <oriansj>but a good learning experience none the less <oriansj>x86 and AMD64 done. Just need to do AArch64 <stikonas>will you then update M2-Planet submodule? <stikonas>I can then submit remaining kaem scripts to build remaining binaries <stikonas>riscv32 probably has very few differences... It's mostly using lw, sw instead of ld, sd and in some places 32-bit specific issues (pointers are 4-bit long, rather than 8-bit) <oriansj>will do that once I verify AArch64 is good <oriansj>stikonas: very slow, so I avoid changing it when possible <oriansj>but I probably need to byte the bullet and fix the hex2 performance <oriansj>xentrac: pointer size is register size <stikonas>xentrac: it's due to qemu issuing a lot of other syscalls <oriansj>hence a huge performance difference between hand written hex2 and M2-Planet hex2 (with M2-Planet's output being an order of magnitude faster) <stikonas>so by moving io buffer to stack, I was able to achieve 8x speedup <stikonas>well, it's mostly because that hex2 is buitlt with blood-elf <oriansj>stikonas: or doing far less calls to write with bigger blocks <oriansj>So doing a buffer and flush for hex2 would probably make a major difference but would be much harder to write in hex1 <xentrac>stikonas: is qemu issuing syscalls because hex2 is issuing a lot of syscalls? <xentrac>that's what I meant by "just a matter of buffering": less calls to write() with bigger blocks <stikonas>xentrac: qemu is issuing some extra syscalls because I guess it finds binary suspicious <stikonas>same binary compiled with GAS is much faster <stikonas>it's also calls to scratch space on heap <xentrac>I added output buffering to qfitzah a week and a half ago, which turned out to make it slightly *smaller* (on i386) rather than larger; not sure how much that correlates with "harder to write in hex1" <xentrac>especially "harder to write in hex1 for RISC-V" <stikonas>ok, so some data for final M2 linking (which is normally done with C version of hex2): <xentrac>I switched from `mov $foo, %al; call putchar` to `mov $foo, %al; stosb`. so it's sort of a quirk of the i386 that there's a 1-byte instruction for "mov %al, (%edi); inc %edi" <stikonas>C version: time ./hex2-1 --architecture riscv64 --little-endian --base-address 0x600000 -f temp -o test <xentrac>if I'd been hand-assembling the code it would have saved me a lot of jump offset calculations tho <oriansj>well jump offset calculations are quite cheap. <stikonas>and hex1 version of hex2: time ./hex2-0 temp test <stikonas>note that GAS and hex1 version are running exactly the same assembly instructions <stikonas>and this hex2 is already optimized, runs about 8 times faster than before <stikonas>well, jump offset calculations are quite cheap for program to do, but quite annoying to do by hand <stikonas>once you have hex1, rest of the stage-posix stuff becomes less tedious <stikonas>and more time is spend on programming rather than enconding binaries <oriansj>hex1 removes the biggest pain. hex2 pulls out the next big pain and M0 pulls out the last major frustration. <oriansj>then cc_* makes everything SOOOOOOOOOOOOOOOOOOO much easier <stikonas>yeah, after M0 I just wrote directly in M1 code <stikonas>yeah, with cc_* porting is basically done <stikonas>most of the other software is automatically unlocked <oriansj>and sorts out porting issues for those tools as well <stikonas>yeah, but you get a lot of C software for free <oriansj>I just hope we can find someone willing to work on the porting work for the pieces after TCC, which will become more important over time <stikonas>or somebody to improve mescc and mes libc to build newer TCC <oriansj>or enhancements to MesCC to build TCC without changes <oriansj>or depending on level of ambition, a version of TCC that M2-Planet can build directly <oriansj>or perhaps I'll pick up the M3 work in the future. which would provide a binutils compatible assembly syntax and standard linker sort of details <stikonas>I'm just forgetting that it has fewer features than coreutils version <oriansj>but perhaps after we get a proper RISC-V stage0-posix port out as a major release <stikonas>ok, mes will be a bit harder to build... <stikonas>it uses some of its own stuff instead of M2libc <stikonas>but yes, at we first need a new stage0-posix release with riscv support <stikonas>at least untar and ungz seem to work fine <oriansj>well I don't think I ever managed to convince janneke that using M2libc would be worth his effort and there are things in meslibc that are not yet in M2libc. <oriansj>I guess I probably need to find a week or two to incorporate all of those functions into M2libc and port to all of the architectures before he *might* reconsider <stikonas>when mes first appeared, I guess there was no other options <stikonas>well, stage0-posix still has somewhat smaller immediate range on riscv... <stikonas>and branches are now all indirect, so should support 20 bits... <stikonas>M2-Planet is about 250 KiB without debug sections... <oriansj>but uses a great deal more memory when running <oriansj>M2-Planet is exactly 272,452bytes right now (for AMD64) <oriansj>and I just fixed the mes-m2 kaem.run build file <oriansj>stikonas: in kaem array_length would return a zero if NULL == envp[0] <oriansj>forgot to also push to github for a second there <stikonas>it's fine, git makes it easy to pull from multiple sources <oriansj>true and savannah is always my preferred master source. ( just can't get bootstrap-seeds in ) <stikonas>well, yeah, I am not really a fan of github either <stikonas>have some personal projects on gitlab though... <oriansj>I never got gitlab to work without JavaScript <oriansj>but then again, I am of the extreme N0-js perspective even though a more moderate LibreJS-only perspective would probably enable a good deal of functionality needed. <stikonas>oh yes, I've whitelisted various gitlab servers in noscript... <stikonas>yeah, gitlab's JS is libre as far as I'm aware.. <oriansj>well the FSF's LibreJS plugin does check for you <oriansj>so does the kaem update fix the remaining kaem issue for RISC-V64 for stage0-posix? <stikonas>yes, it fixes that empty environment issue <stikonas>well, there is a smaller issue in kaem-minimal <stikonas>but that's less of a problem if you don't segfault <oriansj>perhaps the old kaem-minimal for RISC-V which read from kaem.run? <oriansj>as your RISC-V work will be the next major release for stage0-posix <stikonas>well, first you need to pull in M2-Planet... <oriansj>it is on: 057724344431736d72e55cb51a1f8f1c3df256df <oriansj>with commit fb5fe9c487c39a3e5349e022d199f1676da16bee for stage0-posix <oriansj>So unless I missed something all that remains is for me to do the AArch64 verification for the new fixed kaem and do a commit <oriansj>then all that remains would be the RISC-V kaem steps you are working on <oriansj>possibly improvements to HACKING or docs/M2-Planet.1 if either can think of something that needs to fix updated or fixed in them to provide improved understanding. <stikonas>yeah, we need to mention that RISC-V is also supported now in readmes... <oriansj>and hopefully capture anything that tripped you up when porting RISC-V to M2-Planet into HACKING <xentrac>your experience will be super valuable to people like me <xentrac>oriansj: yes, jump offset calculations are quite cheap — unless you're doing them in your head <stikonas>well, that's why this is the first thing we write in hex0 <xentrac>amusingly my subroutine table size optimization in Qfitzah eliminates the need to calculate jump offsets for procedure calls, because the offset in the call instruction is just the offset into the procedure table <oriansj>seems touchy to where it is loaded into RAM <xentrac>20-bit branch offsets don't care how much memory mes-m2 uses when running; you don't compile jumps into the runtime heap into the binary :) <stikonas>well, it's possible to avoid 20-bit limitation by doing jalr call... <stikonas>but we are unlikely to hit that limit with M2-Planet <xentrac>oriansj: what do you mean about the touchiness? <stikonas>so we can let more advanced compilers worry about that <xentrac>right now I'm writing Qfitzah with gas and GNU ld, so the procedure table gets populated by the linker from relocations when it builds the binary <oriansj>xentrac: by touchy I mean the task of getting mes-m2 into any form for GCC or M2-Planet is quite painful <xentrac>hmm? what were you saying was "touchy to where it is loaded into RAM"? <oriansj>we spent 2 years trying to get mes-m2 into a form that could be built by M2-Planet and still run MesCC <oriansj>you would need to do absolute loads or still do relative offsets <xentrac>what do you mean? the procedure pointers in the procedure table are absolute addresses <oriansj>xentrac: ok now if the location of the procedure table was changed what happens <xentrac>well, the startup code loads its address into %ebp. the startup code gets the procedure table's address from the linker, but if that weren't feasible, you could put the procedure table in the same segment with the startup code and do `call *(.+5); pop %ebp` (wait, is it 5? I forget, but whatever it is) <xentrac>I don't know if the trick is actually useful for hex0, hex1, hex2 <oriansj>so your table is at some offset from your binary that runs <xentrac>well, not in the way it's currently built; it's in the .data segment, and the linker pokes the proper immediate constant into the instruction when it links <stikonas>oriansj: it's trying to run /M2-Planet --architecture -f /sys/types.h -f //Linux/sys/stat.h -f /stddef.h -f //Linux/unistd.h -f /stdlib.c -f //Linux/fcntl.h -f /stdio.c -f /string.c -f /bootstrappable.c -f sha256sum.c --debug -o sha256sum.M1 <xentrac>in Qfitzah I came up with the procedure-table trick to save bytes, because although I'm easily under my 2 KiB goal, my 1 KiB stretch goal is looking dodgy without bytecode <stikonas>even though this invocation of kaem should have some variables set <xentrac>does kaem live long enough to look in /proc/12734/env? <oriansj>xentrac: yes but we are not going to make it Linux specific <xentrac>stikonas can certainly debug the bug in a Linux-specific way <xentrac>I'm not suggesting putting opening /proc/$pid/environ into the code :) <stikonas>well, syscall numbers are probably linux specific but yeah, we shouldn't put anything more <xentrac>yeah, lots of places support Linux syscall numbers but not Linux /proc <oriansj>stikonas: well netbsd and freebsd seem to support them enough (now that we fixed up the ELF-headers to be more strict) <stikonas>hmm, I guess if they want to support same C libraries it's simpler <stikonas>to share syscall numbers between OSes... <stikonas>even though Linux itself does not share syscall numbers between different arches <stikonas>ok, switching back to envp[0] check seem to fix kaem <stikonas>although, I'm a bit confused why checking for empty arrays break things <oriansj>notice that M2libc does ${ARCH}/${OS}/things <fossy>oriansj: stikonas, yes I think it is reasonable to just not read in the environment if there is a null envp <stikonas>yeah, that's true, we can support different oses there <stikonas>fossy: yeah, we already tried to do that but a couple of non-working commits <fossy>but I cant recall if there was special logic for the first item in the linked list <stikonas>let me try once more and you can both review <xentrac>what were the aspects of the ELF headers NetBSD and FreeBSD were choking on? <stikonas>fossy: maybe that's why current version fails to work <stikonas>I reverted the last commit that does not work <oriansj>array_length(envp) shouldn't change right? <oriansj>well that would be a null dereference if envp actually was a null pointer <oriansj>So a one line change would be to add the if(NULL == envp[0]) return; after the if(NULL == envp) return; <xentrac>just EI_OSABI and EI_ABIVERSION, eh? <oriansj>xentrac: yeah the Linux emulation layer checks to make sure it isn't running for native binaries