IRC channel logs
2022-11-30.log
back to list of logs
<rickmasters>I have another update on stage0-posix and live-bootstrap on the fiwix kernel (x86). <rickmasters>So far, these are the packages that fiwix can build beyond what builder-hex0 can build: <rickmasters>gzip, tar, sed, patch, bzip2, tcc-0.9.27, coreutils, heirloom-devtools, bash, <rickmasters>flex-2.5.11, musl-1.1.24, tcc-0.9.27 (musl pass 1), musl-1.1.24 (using tcc-musl), <rickmasters>tcc-0.9.27 (musl pass 2), sed-4.0.9 (with musl), bzip2-1.0.8 (with musl), m4-1.4.7, and flex-2.6.4. <rickmasters>I had to implement these syscalls so far: mmap2, readv, writev, stat64, lstat64, and utimes and I had to fix mmap. <rickmasters>I had to modify musl to bypass kernel supported thread local storage. <rickmasters>I have documented all of the system calls invoked by stage0-posix/live-bootstrap, <rickmasters>It typically takes many hours to track down and resolve each build failure, which is exhausting work. <rickmasters>And I still need to build and boot fiwix inside live-bootstrap which will also be a lot of work. <rickmasters>But I'm still chipping away and will continue to provide regular updates here. <stikonas[m]>Indeed, looks like a lit of work but slowly moving... <stikonas[m]>I was hoping that the number of new syscalls used will start to decrease <stikonas[m]>Well, it possibly will... After bison there is perl that might still be using something, but then a lot of packages are autotools which I hope are lighter on new syscalls <stikonas[m]>I'm slowly working on adding (u)int*_t types to M2-Planet. A bit more work than I initially hoped but should be doable <stikonas[m]>Well, for now I'm focussing on just amd64, will do other arches later <rickmasters>stikonas: I haven't been tracking. What was the reason for enhancing M2-Planet? Will it help with UEFI? <stikonas>rickmasters: yes, it will help with UEFI structs <stikonas>some of the UEFI structs have narrower elements, i.e. uint16_t or uint32_t which is a bit hard to access from M2-Planet <rickmasters>stikonas: sounds good. M2-Planet enhancements might also make is easier to implement other compilers down the road <stikonas>right now in M2-Planet I would have to do something like char length_1, char length_2 <stikonas>yes, I've already added support for global structs and structs on the stack <stikonas>before that only struct pointers were supported <stikonas>but that was not sufficient to implement smarter malloc <stikonas>struct pointers are mostly fine in general but they depend on having malloc <muurkha>struct n { .val = 3 } a; struct n { .val = 4, .next = &a } b; <stikonas>muurkha: that definitely wouldn't compile in M2-Planet <stikonas>before those changes, you could only use struct n*; n = calloc(1, sizeof(struct n)); <muurkha>I misunderstood what you were saying about struct pointers <muurkha>I guess you mean that before global structs and structs on the stack, struct pointers depended on having malloc? <stikonas>there was no other way to use structs in M2 <stikonas>well, M2-Planet itself of course still can only use struct pointers only that are calloc'ed <stikonas>but non-bootstrap M2libc can use lots of extra features <theruran>is there a citeable paper or book for the Knight machine? <stikonas>I haven't seen anything online at all (except for stage0 repository) <stikonas>ok, I think I've got uint*_t overflow to work correctly, which is probably sufficient for these fixed types <stikonas>haven't tested int*_t overflow but that's undefined behaviour in C standard anyway <oriansj>theruran: I don't think old magazine ads count as a citeable paper or book. <theruran>oriansj: I see. It's Thomas Knight of Symbolics and MIT CSAIL? <oriansj>theruran: nope; that is a different Knight machine <stikonas>I'm afraid my (u)int*_t change will be rather big one, and I don't think I'll be able to split it into individual commits <oriansj>I believe that one's proper name was CONS and its followup was CADR <stikonas>but it should at least refactor 5 or 6 calls blocks LOAD_INTEGER into a single function <stikonas>(that can deal with various cases of signed/unsigned, zero extend or not...) <stikonas>anyway, I think I've got x86 and amd64 working now. Except for compound operators case, i.e. += as it uses LOAD_INTEGER_ebx/mov_rbx,[rbx] instead of LOAD_INTEGER/mov_rax,[rax] <stikonas>maybe I'll just duplicate load_value function into load_value_2 <stikonas>and need to deal with other arches (riscv/arms/knights) <stikonas>but should be easier now that I have some working arch and working test <stikonas>oriansj: would you be ok, if new defines that I need for x86 will be "new-style" <stikonas>probably need to convert everything to those at some point <stikonas>so I don't want to add extra work for later <oriansj>stikonas: could you update the x86 defines and then do your change? <stikonas>hmm, possibly, though I'll have to rebase mu changes <stikonas>although, it would be easier to do it in the other order... <stikonas>well, I'll try to rebase, it would be cleaner than having mixed defines in some commit... <oriansj>utlimately do what you think is best and we can work around it worst case <rickmasters>Hooray! I was able to get through bison by changing musl's posix_spawn to use fork instead of clone. <rickmasters>stikonas: I'm working on coreutils-6.10. The rm command uses getdents64 which fiwix does not have yet. <stikonas[m]>rickmasters: I thought we don't build rm command there... <stikonas[m]>though I now noticed that makefile is unnecesserily complicated <rickmasters>sitkonas: yeah, I noticed that as well, but it also installs tmp/usr/bin, so I assumed it was in there, but let me check. <rickmasters>actually, it may have been the rm from coreutils-5.0 with musl <stikonas[m]>some of the stuff you might be able to fix by tweaking define flags <stikonas[m]>I'm fine with reducing functionality of those binaries if it's still enough for bootstrapping <rickmasters>that was right before coreutils-6.10. note its not the building of rm, its the running in coreutils-6.10 as part of cleanup I think <rickmasters>i set that aside for a bit, i'm making sure my changes musl work under linux. <rickmasters>But i've been tracking down something strange - it fails on a checksum error of the musl package <rickmasters>which makes sense because I changed it, but i thought i disabled the checksum check and it bypassed it the first time it built musl but not the second time <stikonas>and musl comes after bash, so both builds should use the same functions from helpers.sh <rickmasters>yeah, it looks like src_get is working differently for some reason. i'm trying to track it down <rickmasters>actually, I wonder if my musl change is the cause... <rickmasters>stikonas: ok, i think I figured out that our early version of sha256sum does not work with standard input <rickmasters>once the one from coreutils is built, it starts working as expected <stikonas>M2libc stuff deliberately avoids stuff like pipes <rickmasters>so the interpret_source_line function echos the checksums to sha256sum but should echo to a file and then use sha256sum -c $file <stikonas>yeah, sounds like a recent regression then <stikonas>you are already making quite fast progress <rickmasters>thanks, but it doesn't feel that fast when I spend 5 hours figuring out 3 lines of code. <rickmasters>Most of the syscalls I've implemented are just extended versions, like fstat64, which is just a copy and paste of fstat and changing a few types <rickmasters>I have learned some interesting stuff in the dark corners of musl. <rickmasters>I could probably implement set_thread_area after learning how it works but I'm hoping it won't be needed. I've worked around it for now. <rickmasters>i should clarify the 5 hours was figuring out that bison crashed because it tried to spawn and interact with m4 but musl was using clone to launch m4 which was not implemented. <rickmasters>And the 3 lines of code was commenting out clone and replacing with fork(); if (pid == 0) _exit(child(&args)); <stikonas>well, I'm also doing various other cleanups... I.e. should do that x86 define rename <rickmasters>stikonas: those new efines are really needed so thank you. I really struggled with those. I eventually found ways to disassemble the hex, otherwise I was lost. <stikonas>I don't think I've done converting earlier prototypes... <stikonas>but yes, now just started doing it for x86 <rickmasters>When I struggled, I was too new to the project to take that on and wasn't sure it would be well received so I'm really happy to see that change. <oriansj>rickmasters: we are just glad you joined us and we love seeing your progress; it is quite educational <oriansj>and honestly is looking to save me 20 of work figuring out how to make a working kernel and getting to anywere close to what you have achieved. <rickmasters>muurkha: thank you, I'm glad I found a good project I could help with. There has been so much done already that inspires me. <oriansj>call it mutual inspiration and encouragement <muurkha>well, in my case, I think I'm receiving an awful lot more inspiration than I'm providing <muurkha>haven't done much in recent years :( <rickmasters>muurkha: just having folks provide encouragement and comments helps more than you think. <oriansj>muurkha: you did a bunch of cheering from the sidelines and sometimes that is what people really need sometimes <rickmasters>A lot of this work is pretty tedious/difficult and so it helps to come here and report some progress and hear some positive feedback <muurkha>I know the tedium of spending all night tracking down a bug! <oriansj>plus people here who enjoy talking about this stuff is very much a good thinkg <oriansj>I consider everyone here one of my friends <oriansj>good people, good conversationa and good fun tackling good hard problems. <oriansj>so to the people quietly following, don't be afraid to ask questions; share ideas or just geek out a bit. You are among friends here. <rickmasters>Sadly, I'm going to have to check out for the night. Talk to you all later. Hopefully I can get past some more packages tomorrow on Fiwix. 'night <stikonas>oriansj: I've prepared the change to switch to new defines, though I had to undo that broken UEFI define stuff... <stikonas>argh, there seems to be an issue with test0105 once I rebased... <stikonas>hmm, no, it's actually introduced in new defines rework... Oh well, at least should be easier to fix <stikonas>ok, fixed, missed one DEFINE and it was evaluating into nothing <oriansj>stikonas: thank you for improving the defines ^_^ <stikonas>I think there was some issue with signed unsigned multiplication/division <oriansj>stikonas: how did that get flipped again?? <oriansj>x86 has the worst multiplication and division instructions <stikonas>anyway, once you merge M2libc changes, I'll do M2-Planet PR, you can't really test M2libc standalone <stikonas>and then I need to adjust stage-posix-x86 which I haven't done yet <stikonas>oh wait, the other probably needs rebase now <stikonas>now the hard rebase of my wip (u)int*_t changes... <stikonas>hmm, maybe I should do cc_x86 changes first... <stikonas>oh I think I forgot one thing, now after I converted to defines, the order is still the old one, so not very alphabetical... <fossy>oh, that's an easy fix for interpret_source_line - i must admit i didn't really consider that early sha256sum doesn't work with stdin <fossy>yes, not really something i thought of as i made that change <stikonas>now cc_x86 output spits out new style defines <stikonas>though I now noticed that on x86 kaem-optional-seed does not stop after child segfault... <stikonas>I wonder if that's a recent regression... <oriansj>well if it does segfault, nothing else in the chain builds <oriansj>as every step up until full kaem is very dependent on the previous step hitting success <stikonas>nothing builds, just tries to set some variables in kaem scripts and exits... <stikonas>ok, finally rebased (at least mostly) my wip changes <stikonas>got amd64 tests passing, now need to fix other arches (but even before all that x86 define work other arches were not done yet) <oriansj>I'm hoping in the future someone Refines Aarch64 using the new word based operations in mescc-tools <oriansj>as it is a much better map to the native instructions. <oriansj>(but then again one could say the same thing about x86 and octal and I wouldn't hold my breath on that one) <stikonas>though in general there was a lot of progress recently... <oriansj>recruiting is a hard problem, which we need more people working on <stikonas>anyway, I'm making some progress with fixed size integers <stikonas>I think now I have x86, amd64, riscv32 and riscv64 working <stikonas>I can probably get aarch64 working too, but not sure about arm and knight <oriansj>and never feel like you have to work on anything because of me, ok. Do only what you think is fun and worth doing. <stikonas>well, I needed that change for amd64, but then for M2-Planet we do need to get all features working on all arches <stikonas>anyway, once I get aarch64 working, I might push my changes somewhere <oriansj>well if you just get amd64 working and check it into a separate branch, I can pull and finish it off for the other architectures and then merge it into master <stikonas>well, let me push it now to a branch, and you can take a look at knight and arm <oriansj>sounds like a plan (now that my development setup is now sort of working) <stikonas>I should probably push M2libc changes with extra defines <oriansj>nice thing about M2libc is not every commit has to be perfect (only those we leverage in other packages) <stikonas>(in the test0030 I checked that uint8_t and uint16_t overflows correctly, we can add uint32_t if you want but it's more of the same. <stikonas>I didn't bother testing int overflow as that is undefined in C <stikonas>though for my purposes, I don't even need overflow, just need to be able to assign variable <stikonas>hmm, although I should probably check if what I have right now actually works for me... <oriansj>opt for what makes things better for you <stikonas>I might hit zero_extend branch if I do assignment like ptr->member = 5; where member is uint16_t <stikonas>well, right now I just converted current code, so that outputted assembly is unchanged <stikonas>anyway, if we need any changes, I think it won't be in defines <stikonas>so we might as well finish porting current arches <oriansj>I am sure we can find something that works well <stikonas>for sure. This is not something particularly complicated <stikonas>it's just that these load and store instructions were all over the place <stikonas>hence I had to add all those arguments like zero extend, signed/unsigned, etc... <oriansj>well if you are willing to make M2-Planet a little slower we could centralize ALL assembly instructions output <stikonas>well, I was thinking that we at least need to centralize simple operations like + and += <stikonas>not sure if we need to centralize even more... <oriansj>depends on if we wish to make it easier to add functionality or are approaching the limit of the functionality we feel needs to be in M2-Planet <stikonas>recently we have added quite a bit of functionality that we didn't know that we'll need <oriansj>well new roots come with new requirements <oriansj>up until now we never needed to deal with int16_t; prior to RISC-V we never *HAD* to support word based encoding <stikonas>ok, I think I've got aarch64 defines working... <stikonas>hmm, or maybe not yet, something fails after test 30, test0104 segfaults... <stikonas>(oh, except for last return, that should be t.c - 9 instead of t.c <stikonas>I think as it stands now, it wouldn't work <oriansj>sounds reasonable to improve the testing <oriansj>although I would set t.b and t.d after setting t.c <stikonas>and we can do some more complicated check to make sure all of them have the right values <stikonas>oriansj: oh, I might have fixed that extended test0030 <stikonas>I had a suspicioun that removing all zero_extend stuff might help <stikonas>but didn't expect all amd64 tests to pass <oriansj>well the test are the most basic ones needed <stikonas>ok, so far tests pass on aarch64, x86, amd64, riscv32 and riscv64, so that's good <stikonas>and I suspect knight works too, though I'm not sure how to test it <stikonas>oriansj: so I think the only things that are left are 16 bit load and store instructions for armv7l <stikonas>(luckily knight already had STORE16 and LOAD16 defines) <oriansj>well knight also had a generic load/store instructions where you could define the number of bits which were loaded and stored (load13s and store5 sort of thing) but because I didn't want to have to figure out how to support those instructions as well I pruned them <oriansj>and if you look at the old stage0 commits you can find the old pack and unpack instructions which although awesome would have been a good deal of extra complexity to add and were not actually ever needed. <stikonas>so for arm we have ldrb r0, [r0] which in defines is !0 R0 LOAD8 R0 MEMORY or 00 0 0D 0 E5 <stikonas>I guess I need ldrh r0, [r0] to load 16-bit value <stikonas>but that seems to be B000D0E1 which looks quite different from 00 0 0D 0 E5 <stikonas>soI'm a bit lost how to get something like !0 R0 LOAD16 R0 MEMORY <oriansj>DEFINE HALF_MEMORY E1 DEFINE B00D LOAD16 => LOAD16 R0 HALF_MEMORY <stikonas>I was a bit confused what the meaning of that MEMORY define was <stikonas>LOAD8, R1, and R1 are more or less clear <stikonas>but I have not looked at armv7 ISA at all... <oriansj>well I was trying to find some consistent mapping for arm instructions, the problem is the mapping in the docs is entirely wrong <stikonas>oh, that's not nice if docs are wrong... But I'm probably not surprised... <oriansj>like docs say [cond 4bit] [opcode 8bit] [reg 4bit] [immediate 8bit][reg 4bit][reg 4bit] but the reality is [immediate 4bit][reg 4bit][reg 4bit][immediate 4bit][opcode 4bit][reg 4bit] [cond 4bit][opcode 4bit] <oriansj>if the encoding was like the docs said, then it would have had beautiful hex encoding <stikonas>hmm, maybe I could do !B0 R0 LOAD16 R0 HALF_MEMORY <oriansj>The only way to make Armv7l encoding clean is to leverage word encoding <oriansj>which would require a change in mescc-tools, M2libc and M2-Planet and that would be it <stikonas>well, if somebody does arm, maybe that's where they should start with <oriansj>as I haven't actually done the armv7l stage0-posix steps yet <stikonas>anyway, I'll do something like !B0 R0 LOAD16 R0 HALF_MEMORY for now <stikonas>and it can be converted to word type later... <stikonas>because I think that LOAD16 is in fact B0_0D (where _ are skipped 2 bits for register) <oriansj>you can also define B0 to be NO_OFFSET <stikonas>so then we have NO_OFFSET R0 LOAD16 R0 HALF_MEMORY\n <stikonas>and NO_OFFSET R0 STORE16 R1 HALF_MEMORY\n <oriansj>until I take the time to redo armv7l right <stikonas>ok, actually LOAD16 and LOAD8 are the same... <stikonas>well, anyway, it's not really suitable without word encoding as you have said <stikonas>new defines, and once these are merged, I'll make M2-Planet PR <stikonas>oh I missed some of the other places where load and store integers are done... <stikonas>ok, I think I'll make PR with my current changes, and we can convert remaining load/store use in a followon PR <stikonas>I'll see if I can port remaining code to store/load_value tomorrow <stikonas>I must have edited something and forgotten to update checksum <oriansj>easy to miss but also easy to catch and fix <Christoph[m]><stikonas> "ok, so far tests pass on aarch64..." <- Are these tests done on virtual machines? Or do you have one of each at home? Or at your university? Or is bootstrappable blessed with funds I was unaware of and oriansj runs a testing farm somewhere? (I like the last version best. :-) <stikonas[m]>Christoph: these tests are done on qemu user mode emulation <stikonas[m]>I've just configured (using binfmt_misc) my amd64 system to be able to run aarch64, riscv32 and riscv64 binaries <stikonas[m]>later somebody tested it on real hardware and only found one issue <stikonas[m]>(needed to zero some register on startup rather than rely on it being zero) <oriansj>Christoph[m]: I have AArch64 systems, armv7l systems, x86, AMD64 and a prototype Knight system (Still a work in progress) <oriansj>(raspberryPIs work good for AArch64 and armv7l testing) <oriansj>I have a single RISC-V processor but I haven't yet figured out how to properly test on it as it doesn't support Linux <Christoph[m]>Oh! Bur you're bootstraping linux for RISK-V, so one day...? <oriansj>I probably will have RISC-V hardware to test on but right now I am limited to what I have