IRC channel logs
2023-11-18.log
back to list of logs
<mihi>stikonas, vagrantc, what problem did you notice with M2-Planet's install target? When testing it (M2-Planet/mescc-tools) around 2021 in Cygwin, I noticed that the VPATH trickery in the Makefile only worked when "install" was called as a sole target (i.e. bin/M2-Planet already existed when make run started). At that time, I thought it to be a quirk of Cygwin, and made a note to test it on other systems, which I <mihi>a Makefile without VPATH trickery would probably be preferred for compatibility reasons anyway :) <mihi>mescc-tools doing the same btw :D <stikonas>mihi: install target is "install: M2-Planet" <stikonas>but the binary is actually in bin/M2-Planet <stikonas>so i think we just need to prefix it with bin/ <mihi>same for mescc-tools: "install: M1 hex2 blood-elf kaem get_machine" <mihi>VPATH (set at top of makefile) is supposed to lookup inputs in bin/ as well, but I suppose it only works when that already exists <mihi>for me, "make all && make install" worked, but "make all install" did not. <mihi>(assuming bin/M2-Planet did not exist at start) <mihi>stikonas, does "make all install DESTDIR=destdir/" work for you if bin directory does not exist at start of run? <mihi>and the "install" picked up the correct file? Good, then what I was seeing is indeed a Cygwin quirk. <mihi>for me, it compiled to bin/M2-Planet and then tried to move ./M2-Planet to destdir :D <mihi>so the question remains what problem vagrantc encountered :D <mihi>(in fact, it compiled to bin/M2-Planet.exe since it is cygwin :D) <stikonas>oh, maybe because its not looking in local <stikonas>cp M2-Planet /<<PKGBUILDDIR>>/debian/m2-planet/usr/bin <stikonas>but I don't see anything wrong in our makefile <vagrantc>i tried setting PREFIX and it seemed to ignore it <vagrantc>i can poke at it some more at some point <vagrantc>but basically i need to be able to override PREFIX and DESTDIR appropriately somehow ... either through environment variable or explicitly passing to make <vagrantc>hrm. PREFIX and DESTDIR were empty ... weird. <vagrantc>ok, it is clearly something in my packaging. <vagrantc>huh. ok. I have successfully passed PREFIX and DESTDIR .... <vagrantc>cp M2-Planet /<<PKGBUILDDIR>>/debian/m2-planet/usr/bin <vagrantc>cp: cannot stat 'M2-Planet': No such file or directory <vagrantc>is the "cp" call assuming to be run from some directory other than the toplevel directory? <vagrantc>well ... patching the makefile made it work as expected: <stikonas>M2-Planet is not a file target, it's another make target <mihi>vagrantc, do you explicitly mess with VPATH variable? <vagrantc>not that i am aware of, but i am using standard debian machinery <stikonas>well, VPATH is set in makefile to VPATH = bin:test:test/results <stikonas>$^ is expanded to M2-Planet rather than bin/M2-Planet <vagrantc>clearly. i guess the question is how is it even actually supposed to work. <stikonas>vagrantc: make should search for dirs in VPATH <vagrantc>well ... huh. only when DESTDIR already exists <vagrantc>first pass, it fails, second pass, it succeeds <vagrantc>cp: cannot stat 'M2-Planet': No such file or directory <vagrantc>make: *** [makefile:122: install] Error 1 <stikonas>vagrantc: that's locally outside debian packaging? <vagrantc>same version in the build environment too <vagrantc>ok, if i run make PREFIX=x DESTDIR=xyz/ && make install PREFIX=x DESTDIR=xyz/ ... it works <stikonas>yeah, but to reproduce you need to run make install <vagrantc>right ... which ... the packaging should be doing <vagrantc>e.g. bin/M2-Planet exists when i just do not even bother to call "make install" <stikonas>though I think the bug is also in makefile <vagrantc>that sounds like a plausible explanation of what is going on <vagrantc>well, i have workarounds until a proper fix comes up <vagrantc>at least two workarounds to choose from... <vagrantc>ugh. not enough sunshine to finish my mes build on riscv64 :/ <vagrantc>first workaround is just to manually install it from the debian packaging ... second is to patch the makefile to look in the "right" place. <vagrantc>the M2-Planet target *does* create bin/M2-Planet ... so it appears to work. <vagrantc>although then i still have to manually install the manpage :) <stikonas_>vagrantc: hmm, I'm a bit concerned about just patching that one line <stikonas_>but we don't have bin/M2-Planet target anywhere in the file <vagrantc>ah, right ... so it would not actually build M2-Planet if it was not yet present <stikonas_>how does it know that bin/M2-Planet has to be built using rules from M2-Planet: <vagrantc>because it effectively first does "make all" <stikonas_>but that's more work than there is time today <oriansj>we can always make M2-Planet and mescc-tools more proper <Googulator>Today, I've officially entered the 6th bit of my life :) and what could be a better birthday gift than seeing bootstrap go all the way to GCC 13 (and counting) on bare metal *with no more linux-5.10 dependency* - while watching Starship Flight 2! <Googulator>And I also verified that with linux-5.10 gone from srcfs, we can now fit a 1280MB sysa.ext2 below the MMIO limit on qemu, which is also bootstrapping as we speak (currently rebuilding musl, but I have already seen it reach sysb, where it failed due to a dumb mistake I've now fixed) <Googulator>bare metal sysa.img is down to 284MiB - with gz -> bz2 recompression, we could easily get back to below 256MiB, which is my target for the SPI or parallel flash bootstrap plan <Googulator>hopefully the bare metal bootstrap won't blow up at the last second in GCC 13, unlike Starship <stikonas>Googulator: nice! At some point we should start upstreaming your patches <stikonas>though there is probably quite a lot, it would be easier if you can split it into a few smaller PRs... <Googulator>Yeah, I'll have to properly refactor it & split into more logical units <Googulator>also the way grub is built is very unbootstraplike <Googulator>By the way, does anyone know a way to code a for loop in the early bash (2.0.something)? <stikonas>all those config files should be also moved outside bash scripts witgh cat >>> <Googulator>Neither the C-like syntax nor the for a in {1..1000} one works <stikonas>Googulator: sorry, I wanted to say scripts updated, bit grub upgraded... <stikonas>but we build first iteration of coreutils before bash <Googulator>tbh, the reason I need the for loop in the first place is because we're missing df at that point <Googulator>I'm trying to see how much space is left over after the Linux build, before clean up <Googulator>I tried df first - no dice, it's not built until much later <Googulator>so I came up with    for i in {1..1000000};  do  echo "interation $i";    cat /boot/fiwix >> /boot/wifix || break; done <Googulator>will tell us how big it got before the disk got full <Googulator>hmm, I could try nested loops of for i in 1 2 3 4 5 6 7 8 9 10 <Googulator>(I've already started a test with nested loops, will try seq if that fails) <stikonas>ideally we should also leave some space for future upgrades <stikonas>i.e. at some point we might have to add some more packages (e.g. gash) or upgrade other stuff, especially GCC <Googulator>That means, we can go up to 1350MB on the fiwix initrd right now <stikonas>but stuff like GCC upgrade might need quite a bit of space <Googulator>of course, any files added to sysa will eat into that <Googulator>so we have 70MB of space right now outside the initrd <Googulator>Also, as an unexpected consequence of enabling framebuffer console support, I now get 4 penguins on bare metal <Googulator>of course, your mileage (or rather, coreage) may vary depending on the hardware <stikonas>yeah, I'll probably try baremetal bootstrap with new hardware in a month or so <Googulator>should be easier to do than the current BIOS version <Googulator>but then again, I'm not so much interested in bootstrapping on new hardware <Googulator>too much recent, closed source firmware before you get to run any code <Googulator>and yes, you could get something with edk2, which is open-source - but is it bootstrappable? <stikonas>well, yes, they are not bootstrappable, but still coreboot/edk2 is better than random machine <stikonas>even if you don't get bootstrappable trust, you'll at least run more open code than proprietary UEFI <Googulator>yeah, it's way better to have a blob you built on a potentially untrustworthy machine with a potentially untrustworthy toolchain than an outright proprietary one <stikonas>especially given that with UEFI you even have some UEFI code left in the memory after boot <Googulator>but it's still a path through which a retrovirus (i.e. a Trusting-Trust-style backdoor) may sneak through into the bootstrap environment <stikonas>yeah, but we do have some help from diverse compiling here <Googulator>retrovirus, because HIV is basically a Trusting Trust on our immune system <stikonas>and if hashes match on old machine, e.g. what you bootstrap and new machine <Googulator>That's one good way to do it - bootstrap once on old hardware, then use the bootstrapped system to build edk2 for your new hardware <stikonas>yeah, there is still some window that all HW is infected... <Googulator>Flash that (preferably direct SPI from the old bootstrapped system), and then you can safely bootstrap on the newer system, modulo actual hardware-based backdoors/retroviruses <stikonas>but at that point those trusting trust attacks get ridiculously unlikely <Googulator>that's why I'm shooting for 15 to 25 year old HW <stikonas>well, the last step is that oriansj's Nexus atack <Googulator>old enough that it's unlikely that any APT would've persevered that long <Googulator>new enough that you can't straight up emulate it on today's hardware <Googulator>ironically, "nexus" would also be a great name for the initial binary seed (the boot sector) that gets to run before it can get printed <Googulator>since it's the narrow point that everything sort of has to pass through <Googulator>& hence the only place to hide a backdoor, outside of hardware/firmware <Googulator>make it small enough and incompressible enough, and you can pretty much exclude retroviral propagation from the host to the bootstrap environment <Googulator>and yes, I'm aware antivirus folks use "retrovirus" in a different sense, to mean viruses that try to target security software and shut it down - and I think it's a mistake <Googulator>the idea is that like HIV, such viruses target the computer's immune system <Googulator>but targeting the immune system is not what makes HIV a retrovirus - it's the fact that it writes itself back into the host's DNA <stikonas>well, at least if you have kernel bootstrap <stikonas>otherwise there is really big binary too <stikonas>and unlike hardware which "knows" very little about what it is running, kernel has way more control <Googulator>same for a "full fat" firmware like uefi or uboot <Googulator>which is basically a kernel with a non-POSIX ABI <stikonas>well, M2libc is some incomplete POSIX layer around that <Googulator>there's even a full port of Python to the UEFI interface <stikonas>though we are using UEFI in stage0-uefi in a very limitted manner <stikonas>basically just file I/O and spawn another process <stikonas>some of the UEFI compat code in M2libc actually resembles builder-hex0 a bit <stikonas>but stage0-uefi bootstrap seeds are quite a bit larger... <Googulator>anyway, with stage0-posix, you also have to trust the BIOS <Googulator>since it just translates everything to BIOS calls <Googulator>all the data read from disk passes through the hands of BIOS <Googulator>but I'm much more inclined to trust a 15-year-old firmware than a 2-year-old one <Googulator>an APT can anticipate a bootstrap strategy that will be made public 2 years from now and still appear relevant then, but not if it's 15 years from now <Googulator>by "anticipate", I potentially mean even the possibility that an APT will intentionally pretend to be one of us, releasing some bootstrapping code on GitHub - code that they have actually written 2 years ago, but not revealed <Googulator>& in that time, they already put backdoors into various firmware to subvert that bootstrap code <Googulator>basically a premining attack, but for bootstrapping instead of block chains <Googulator>The challenge in implementing such an attack (apart from needing APT-level capabilities) is in making sure that the bootstrap code/plan released seems relevant today, while still letting your N-year-old planted backdoor recognize and successfully subvert it <Googulator>but very large N permits another attack: because the hardware from that far back is so much less capable than modern stuff, it becomes possible to emulate it with a backdoor inserted on modern hardware small enough to dress up to look like thee vintage hardware it emulates <Googulator>e.g. an Intel P8080-2 that's secretly a Cortex-M0 running a backdoored 8080 emulator, packaged into a DIP that looks like a real 8080 and takes actual 8080 signaling <stikonas>well, that you can workaround buy buying older hardware <Googulator>This is what creates that "sweet spot" effect in the 15 to 25 year range <Googulator>making a fake Pentium 4 CPU in a convincing looking Socket 478 package is (hopefully) impossible even with our current technology <Googulator>a fake that also functions, that is - until you hit the backdoor <Googulator>alright, we have 78MB of free space left over inside the 1280MB Fiwix initrd <Googulator>& that is with the various tricks I use to bring over more stuff from sysa to sysc - without that, we can fit even more <Googulator>Bootstrap done on the bare metal system - and I just tested, it does reboot cleanly! <stikonas>I'm trying to upgrade bootstrappable tcc snapshot now <Googulator>just made a small fix to it; accidentally it still included the code for probing the remaining free space after the Linux kernel build (which made the build break at that point) <oriansj>I think nested switch statements in M2-Planet are working correctly too <oriansj>on test4 shouldn't the default case for the switch statement be hit? <oriansj>as case 6 does not exist and b is 6 when the test is run <oriansj>or is this me forgetting that an 8bit number is the limit for Posix return values <stikonas>maybe add expected = ${value}; statements to if's... <stikonas>and then subtract them from i in each case? <oriansj>yeah, moving to smaller numbers fixed it <oriansj>now we have 8 switch test cases; all working correctly <oriansj>ironically go's fallthrough would be simpler to implement than C's break <oriansj>Googulator: another option to expand our storage limits would be to add an unxz into mescc-tools-extra <oriansj>and with the new switch support we might even be able to do it with minimal code changes <stikonas>though a lot of older software does not have xz tarballs <oriansj>I guess I really lucked out. I have yet to find a failing test where gcc and M2-Planet don't agree on the output in a switch statement <matrix_bridge><Andrius Štikonas> It's not just luck. I'm sure skill was involved too. 