***midnight_ is now known as midnight
<oriansj>pabs3: I guess it was a good thing that I cloned it when I saw the thread <oriansj>although the readme with clone instructions is actually straight up wrong <oriansj>if you do git clone on the github url, it would allow proper cloning <oriansj>and 1GB of git history is a shitload if it is source and not generated crap. ***civodul` is now known as civodul
<oriansj>even if only half of that volume was additions (and the other half deletes); that is still 500 Million letters or 50 Million written words (assuming 10chars per word). Assuming a person typing 500 words per minute 24/7 that would be 100,000 minutes of typing or 69.44years of typing at that crazy high rate. <oriansj>even split across 10 people, it would still be 7 years of 24/7 work at 500 words per minute <stikonas>oriansj: in the end I think I'll reserve memory for that label table in hex1... <stikonas>a bit more code, but pointer will be easier to work with anyway for relocatable code <stikonas>yeah, 1 GiB of git history sounds like something wrong <oriansj>stikonas: sounds like a conservative plan and is quite reasonable <oriansj>also depending on relocation sections is a recipe for depending on functionality not currently in hex2/blood-elf <stikonas>on related note, apparently OVMF is able to run non-relocatable binaries <oriansj>well PC relative addressing is certainly something easy to do in AMD64 and we have hex2 support already so solves both of those issues <oriansj>although doing the PC relative version of LOAD EAX &label for function pointers is going to be eseential for cc_* and M2-Planet (not to mention mes.c) <stikonas>well, it most cases we just care about contents of varaible there <stikonas>although I don't remember now how exactly it is used in cc_amd64... <oriansj>in cc_* and M2-Planet (and mes.c) it is used for function pointers passed to call eax [call rax] <stikonas>oh, they are called using global addresses, ok... <oriansj>hence the need to figure out the way to do PC relative version of LOADI32 EAX &label <oriansj>but I am probably over thinking it and it is probably something simple like: mov rax, rip label or knowing x86 stupidity lea rax, [rip+label] <pabs3>oriansj: did you find out what took up 1GB of git? any generated/binary files? <oriansj>pabs3: well the git clone of the repo I have is only 8.3M in size <pabs3>I guess you got the new one? <oriansj>yeah, literally all of thre code in a single commit <pabs3>hmm, cloning the fork still gets me 8.3M too <pabs3>7448 commits in the fork though <pabs3>software heritage might have the old one. they plan to republish the old repo too according to that issue ***lukedashjr is now known as luke-jr
***hl_ is now known as hl
<oriansj>well a dozen or so failed git repacks could produce that much blob in a repo without it having anything to do with the git history itself. ***rkeene_ is now known as rkeene
***Noisytoot_ is now known as Noisytoot
<rickmasters>Hello, I'm back with another update on my bootstrap kernel work. <rickmasters>As a reminder, I reported on July 6th that I had built mes with mes-m2 <rickmasters>on the booostrap kernel (builder-hex0) and was proceeding to try to build tcc with mes. <rickmasters>We also talked a lot about EFI and I hoped to help with that. <rickmasters>Unfortunately, my hopes that I could get mes to build tcc quickly were dashed because <rickmasters>it crashed trying to compile tcc.c (after 16 hours or so). <rickmasters>This was perplexing because mes-m2 built over a hundred C files and <rickmasters>After two weeks of arduous debugging I managed to compile tcc.c into tcc.s. <rickmasters>Turned out that mes used more memory (over 570MB) than the kernel anticipated, <rickmasters>The next command I tried was to assemble and link mes-tcc but that also crashed. <rickmasters>After another two weeks of debugging, mostly stabs in the dark with a full day <rickmasters>of turn-around, I finally got confirmation this morning that the stack was not <rickmasters>restored properly when mes resumed after spawning M1. This was the clue I was looking for. <rickmasters>I am testing a possible fix for that now and hopefully I can produce mes-tcc and leave mes behind. <rickmasters>Mes is extremely slow. I am not using kvm which explains some slowness, but still, mes is painful to work with. <rickmasters>After mes-tcc, I will try to build tcc with mes-tcc. The compile chain is: <rickmasters>M2 -> mes-m2 -> mes -> mes-tcc -> boot0-tcc -> boot1-tcc -> boot2-tcc -> boot3-tcc -> boot4-tcc -> boot5-tcc <rickmasters>Running tcc is a new game with unknown syscalls so I can only hope it goes well. <rickmasters>I may disappear for a while again but I have been following the discussions about EFI and reducing the seeds which is great work. <vagrantc>mes has been languishing in debian since the default compiler switched to gcc-11 <Hagfish_>570 MB of memory sounds... generous, let's say. it's orders of magnitude bigger than most of the other numbers that get mentioned here :) ***Hagfish_ is now known as Hagfish
<rickmasters>Yeah, that much memory meant the kernel could not used a fixed size for saving a process when a child was spawned... <rickmasters>I had to look at the brk pointer and dynamically allocate the memory, so it took some rework <rickmasters>mes also used a different ELF header structure so I had to discover and support that before it would even run <rickmasters>I also made some other fun discoveries like our untar does not support long file names, silently corrupting those files <rickmasters>The basic tar format only supports 100 bytes for file names <vagrantc>if you can switch to ustar, that effectively gives you 151 (in some cases larger, but not all) <vagrantc>rickmasters: or are you talking about upstream tarballs of arbitrary formats you need to unpack during the bootstrap? <rickmasters>vagrantc: I am saying the untar command in stage0-posix/mescc-tools-extra does not support > 100 byte file names <rickmasters>so, I created a tar file using modern tar with long filenames and untar didn't extract it properly so I had to workaround that by using shorter relative paths <samplet>The GNU long name extension is pretty simple to implement. <rickmasters>samplet: TBH, it was easier to work around but yes that would have been a good improvement to add but I wasn't up for it <stikonas>rickmasters: well UEFI work is kind of parallel (and somewhat higher level) to what you are doing <stikonas>I'll probably have hex1 soonish but I was sick last week (still not healthy), so things were pretty slow <rickmasters>the pain was discovering what happened - the file name was truncated and you can't even see that without --verbose so figuring out that the compiler crashed <stikonas>but yes, you are doing incredible progress <samplet>rickmasters: Definitely. The only thing better than solving a problem is avoiding it! :) <rickmasters>stikonas: covid? several people I know got it recently, I got it in April so it seems to bounce off me now <stikonas>rickmasters: yes, covid, one co-worker got it, in a few days most of my other coworkers got it... <rickmasters>stikonas: this last variant is finding everyone it seems. its a wild fire <rickmasters>covid seems to never end but I'll say no more, don't want to side track on that <rickmasters>on a longer term topic - I've been thinking about how live-bootstrap could be restructured to support building bootstrap kernels. <rickmasters>sysa would need to be split up but that shouldn't be too hard <stikonas>yes, I agree, sysa would become a few systems <stikonas>a bit unfortunate that current numbering means we would have to rename e.g. sysc -> sysd <rickmasters>perhaps split after tcc-0.9.27. yeah naming is tricky <unmatched-paren>"note: Despite the name, sysd is not a bootstrap implementation of systemd." :) <rickmasters>or sysa1, sysa2 but not as clean in terms of naming conventions <nimaje>could use a unary numbering system <rickmasters>whoops. I should say a split somewhere after building tcc-0.9.26 and a new kernel because running make is the inflection point for a new kernel I think <rickmasters>the current bootstrap kernel can only spawn one child and presumes fork, exec, waitpid are run in an exact sequence <rickmasters>its possible that make -j1 might work but I wouldn't count on it <stikonas>live-bootstrap never introduced parallelization, partially because we didn't get to it, partially because there might be some new reproducibility bugs <rickmasters>ok, well I'll see how far I can get before this kernel runs out of steam, only testing will tell <rickmasters>stikonas: for the minimal boot, how is the Optional Data used? <stikonas>so if we run with hex0 amd64\kaem-minimal.hex0 EFI\BOOT\BOOTX64.efi then hex0 builds kaem-minimal <stikonas>You can set Optional Data in your boot entry <rickmasters>You pick hex0-seed.efi as the executable, but pass a different executable name "hex0" for the command line? <stikonas>in principle one does not have to add that first word at all <stikonas>but then we lose compatibility with UEFI Shell <rickmasters>so you would you use \bootstrap-seeds\UEFI\amd64\hex0-seed.efi for the command name if you were using the UEFI shell? <stikonas>e.g. on my laptop there is no UEFI shell <stikonas>rickmasters: also error checking is almost non-existant and OptionalData (command line) parsing also doesn't check for length limits <stikonas>but I might change that in later programs <stikonas>since it's much easier to do if we have hex1 to calculate jumps for us <rickmasters>the value of error checking isn't as clear in a bootstrap scenario so its not something i've put a lot of effort into to be honest, <rickmasters>although I have to admit it probably would have saved some debugging time for a couple of issues <stikonas>e.g. M0 does almost no error checking and prints no useful error messages <rickmasters>one distinction is whether the tool is used in a narrow way to build another tool or is re-used many times in ongoing new scenarios in which error reporting has more value <stikonas>that's why so far I've done minimal error checking <stikonas>although I've tried to keep to UEFI specs and released freed memory and close file handles