IRC channel logs
2023-11-16.log
back to list of logs
<pabs3>ACTION guesses this got discussed already? <Googulator>What's the significance of rebuilding meslibc with the "full" mes after it has already been built once with mes-m2? <Googulator>mes-m2 and mes are functionally requirement, if I'm not mistaken <Googulator>And that's especially true given that mes / mes-m2 is just the Scheme interpreter <Googulator>the actual C compiler is the same, mescc.scm, and it should certainly generate the same code regardless of the interpreter it's running in <Googulator>oh, I'm stupid... that's not what's actually done <Mikaku>Googulator: rickmasters patched the Fiwix kernel in the current live-bootstrap version to be able to have large initrd files, because in that time there was no possible to have such large initrd images <Mikaku>Googulator: Fiwix doesn't support ext4 <Mikaku>if you need larger initrd images, the latest Fiwix version (1.5.0 since yesterday), supports the virtual memory split 2GB/2GB, so theoretically this should permit you to have an initrd image of near 2GB <lanodan>I think for boostrappability-gamification status based tiers would make more sense than a number of dependencies. For example an maintained and officially documented bootstrapping path for A tier (A+ for multiple paths?), F tier could be source-less (rather than just recipe-less) blobs like with Mono shipping Microsoft DLLs. <lanodan>And I guess the rank would be capped to the worst ranked dependency. <lanodan>Rank cap because dependencies create "one bad apple spoils the bunch" kind of situations, for example because it often creates events like "Welp, now I can't use the software at all, there's not even binaries for my system". <Googulator>Need to clean up the code for this, but it's coming probably later this week. <Googulator>One small bug remains: nvidiafb loads just fine, but the console remains VGA 80x25. <Googulator>The cause? # CONFIG_FRAMEBUFFER_CONSOLE is not set <rickmasters>Googulator: The 1152MB initrd for fiwix stems from the memory constraints of the builder-hex0 kernel <rickmasters>Googulator: using larger partitions is possible but may take substantial work. <rickmasters>Mikaku and I spent an enormous amount of effort making live-bootstrap run from hard drive with Fiwix <rickmasters>So if we need to go down that path it is possible. The flush (f) command for builder-hex0 was implemented for that purpose. <rickmasters>Googulator: I'm grateful you put the work into running off bare metal! Great job! <rickmasters>Googulator: I meant to get around to that some day but I put it off. <rickmasters>Lately I've been working with Mikaku to upstream patches into Fiwix to support live-bootstrap so I can retire my branch. <rickmasters>Its a long and tedious process so I'm afraid I may not have a lot of available time until that is done. <rickmasters>I admit I haven't been following your efforts in detail so I'm not up to speed with your changes/requirements. <matrix_bridge><Andrius Štikonas> rickmasters: I guess to add more drivers to linux <Googulator>A bigger Fiwix initrd is needed because it constrains the size of the Linux kernel build tree, which increases quite a bit due to extra drivers needing to be enabled on baremetal <Googulator>With 1152MB, we run out of space when linking the kernel <Googulator>1280MB is sufficient, and it works with no apparent ill effects on my baremetal system - but due to different memory map in qemu, 1280MB already overflows into MMIO space there <Googulator>We could branch depending on baremetal vs qemu, with different kernel configs & initrd sizes, but I'd rather not do that <Googulator>plus, 1280MB works *on my hardware* but it may not work on other boards or BIOSes <Googulator>An easy way out right now is to kill the 5.10 source tarball used for linux-headers, and instead generate headers from the 4.9.10 tarball too <Googulator>that frees up 170MB of space in srcfs, which should shift the initrd enough that it no longer overflows in qemu <Googulator>Of course, removing 5.10 is a one-time fix only - next time we hit the limit, a more permanent fix is needed <Googulator>IMO the best way to go is to allow Fiwix to receive a smaller initrd than what its ramdisk is configured to hold, and then expand it to the full size of the ramdisk upon boot <Googulator>This way, we can fit everything we need to pass to Fiwix in a 500MB initrd image, but pass ramdisksize=1280MB to fiwix itself <Googulator>Since the image, as stored within builder-hex0's RAM filesystem, is only 500MB, we don't run the risk of overflowing onto MMIO area <Googulator>Then kexec-fiwix's trampoline code relocates the image to a lower address, and passes that lower address to Fiwix <Googulator>Fiwix then sees that the FS is only 500MB even though it's on a 1280MB ramdisk, expands the FS to the full ramdisk size (starting at the lower address, so even after expansion, it should be nowhere near MMIO), and mounts it. Now we have all the free space we'll ever need inside the ramdisk to comfortably build our Linux kernel. <Googulator>That 500MB will be further reduced to 300-350MB if we drop the 2nd Linux tarball <Googulator>(currently testing a build in qemu with 4.9.10 used for headers too) <rickmasters>Googulator: Eliminating the second linux version and expanding the file systems would be great improvements. <rickmasters>I've spent a bit of time in the past trying to change the headers but it wasn't easy and I moved on. <rickmasters>Another way to save a ton of memory in builder-hex0 is to move away from mes in the early bootstrap. <rickmasters>mes uses over 600 megabytes of memory in some invocations if I recall correctly. <rickmasters>I think someone was working on another compiler but I'm not sure what the status is or how acceptable removing mes is with everyone. <Googulator>IIRC that's in the process space, not file space <Googulator>builder-hex0 uses discrete memory segments for different uses <rickmasters>yes, but the only reason so much memory was reserved for processes was to support mes. <Googulator>& of course one improvement we absolutely *must* have for any serious baremetal support is proper e820 passthrough <Googulator>I got lucky that my motherboard's reserved memory ranges are _almost_ a subset of qemu's <Googulator>there's one small range that's available in qemu, but reserved on my board, but it sits below where the fiwix kernel is loaded to <Googulator>A few days ago someone did post a direct M2-Planet -> tcc bootstrap path (WIP - it depends on some JavaScript code in that prototype code) <rickmasters>i agree that the memory map support is poor right now and needs improvement <Googulator>If e820 passthrough is too hard to do, at the very least we need to provide a place to insert e.g. the e820 log from a Linux boot session during image generation, and inject it into kexec-fiwix.c <Googulator>that way we can generate board/BIOS-specific images for any board that can boot mainstream Linux <Googulator>of course, all of this is not great from a Trusting Trust resistance standpoint, since it requires running unbootstrapped code on the target board before bootstrapping, creating a path for a self-replicating parasite to inject itself into BIOS <Googulator>maybe write a small e820 dumping utility that can run under builder-hex0? <jcowan>of course, when the Linux kernel grows to 1 GB (which is foreseeable) then more space will be needed for the bootstrap <rickmasters>Googulator: kexec-fiwix could probably be adapted to call the bios routines and create the multiboot memory map accordingly, <Googulator>The ones I built for baremetal already exceed that <Googulator>rickmasters: is it possible to make BIOS calls at that stage? <Googulator>without needing help from the builder-hex0 kernel <Googulator>If kexec-fiwix can drop back down to 16-bit and call BIOS directly, then it's fairly easy <Googulator>But I'm afraid it will have to be added into builder-hex0 <rickmasters>Googulator: yeah it makes more sense to make the bios call from builder-hex0 kernel but perhaps the map can be converted to multiboot in kexec-fiwix <Googulator>If it's possible to do the call in kexec-fiwix, it would be preferable <Googulator>now I see why I sometimes get random failures in the builder-hex0 stage <Googulator>BIOS reserved area starts at 9EC00 on this board <rickmasters>If I recall, only a small part of the scratch buffer area is used, I'd be surprised if it used above 9ec00 <rickmasters>Googulator: the bios map could just be placed in a well-known memory location that kexec-fiwix could read. there is no memory protection :) <matrix_bridge><Andrius Štikonas> Googulator: ideally we should do early kernels multi staged too, not just written in hex0 <matrix_bridge><Andrius Štikonas> E.g. early loader written in hex0 then it loads kernel that can do hex1... <matrix_bridge><Andrius Štikonas> And only the last one would need to support full syscall list <matrix_bridge><Andrius Štikonas> Otherwise builder-hex0 will grow too big and will be hard to manually edit/inspect <vagrantc>does mes need tinycc patches? I tried building mes 0.25 on riscv64 with tcc, although I had to enable build of riscv64 on debian's tcc package ... <vagrantc>which was based on tcc 0.9.27+git20200814.62c30a4a <vagrantc>so presumably from git commit 62c30a4a from tcc git <vagrantc>++ /usr/bin/tcc -nostdlib -g lib/linux/riscv64-mes-gcc/exit-42.S -o exit-42 <stikonas_>vagrantc: I don't think we tested building mes with tinycc <vagrantc>lib/linux/riscv64-mes-gcc/exit-42.S:28: error: RISCV64 asm not implemented. <vagrantc>stikonas_: what is mes tested building with? <vagrantc>slightly confused by: tcc ... lib/linux-riscv64-mes-gcc ... shouldn't it be lib/linux-riscv64-mes-tcc ? <vagrantc>well, not the same build failure, but ... gcc was not considered tested either? 