IRC channel logs
2023-12-19.log
back to list of logs
<Googulator>also, the swap creation script only matters if you use the -s option <Googulator>checksums are wrong in these, though, so use --update-checksum <stikonas>somehow python distfiles downloader seems quite unreliable now for me... <Googulator>this is one of the reasons why I implemented manifest-driven downloads (skipping files that are needed only after network is up) <stikonas>yeah, but python requests shouldn't be that bad... <Googulator>could even be AI defense - most AI is written in Python these days, so blockking Python UAs helps defend against potential AI bots (like Bing AI, but malicious) <stikonas>since I don't remember it being so bad even a year ago <stikonas>ok, checksum-transcriber on riscv should be 560df1e8527df9758252f6255c144cfd6b1555b9d2aa6162011204061af80ab5 <stikonas>maybe I should make a commit with all fixes first... <Googulator>checksum-transcriber also changed on amd64, even though it's pre-mes <Googulator>I'm guessing amd64 checksums weren't updated before the final merge of simplify <Googulator>also, simple-patch doesn't even have any amd64 checksums rn <Googulator>And arm64 gets no love at all, from what I've seen <Googulator>even though it's a bit more mature ecosystem than riscv64 <matrix_bridge><Andrius Štikonas> so probably got more attention because of that <stikonas>Googulator: for arm you could try switching to 32-bits in mes... <stikonas>though maybe not all boards support 32-bit mode <Googulator>that may not even work on boards that do have 32-bit mode <Googulator>v5 or newer code will run on v8/9 cores supporting 32-bit mode, but IIRC not v4 <Googulator>is that meant to run on top of an existing Linux kernel? <stikonas>for other arches we probably want uefi bootstrap <stikonas>well, ideally something that uboot can drive too <stikonas>otherwise each SoC just has different boot sequence... <Googulator>IMO it's fair to require enabling uefi support in uboot <stikonas>it's basically default option these days <Googulator>although some major vendors don't do that by default *cough* Rockchip & Radxa *cough* <stikonas>but I'm still struggling with UEFI even on amd64... <stikonas>though might take some time to support newer boards <Googulator>in the end, what we will need is to achieve bootstrap from the BROM up on _one_ decent board <Googulator>and also to build u-boot and/or edk2 in a bootstrapped environment <Googulator>then we can build trusted bootloaders for other boards using that <Googulator>Rockchip happens to be relatively decent in this regard, especially the 35xx chips <Googulator>32KiB embedded boot ROM, loads an insigned, unencrypted binary with minimal header from SPI to 1MiB SRAM <Googulator>(that would be an arm version of builder-hex0 customized for the particular SoC) <Googulator>of course, bootstrapping DRAM controller initialization & training is gonna be a challenge <Googulator>especially since RK35xx currently relies on a blob for that <stikonas>u-boot and edk2 is probably out of scope for early bootstrapped environment <stikonas>you might need a board with decent amount of SRAM <stikonas>e.g. rk3399 can trai DRAM as part of u-boot <stikonas>but SRAM is so small that relevant u-boot stage barely fits there <Googulator>IIRC 3399's bootloader is also nowhere near as open as later RK SoCs <Googulator>requiring an encrypted/scrambled image for the initial loader <stikonas>so BROM -> U-boot TPL (that does DRAM training) -> BROM -> U-boot SPL -> ATF-> U-boot proper <stikonas>maybe there are some boards where you need to do that <Googulator>AFAIK you do, it's just done by default using a well-known default key <stikonas>I thought only SPI needs some minor modification <stikonas>possibly writing every second byte or something similar <Googulator>or maybe it's the header that needs to be crypted <Googulator>but I remember you can't have a pre-35xx image be entirely in cleartext <Googulator>again, this is not a lockout, since the keys used (by default) are published <stikonas>yeah, cryptography / signing is not good for bootstrapping <stikonas>i.e. even if key is known, you don't want to sign hex0.efi <fossy>what is your bare metal system Googulator <stikonas>Googulator: I wonder if you are thinking of another boot sequence that rockchip supports and uses proprietary blob <stikonas>so far (but I haven't looked thouroughly) I can't see uboot doing any signing/encryption <Googulator>fossy: Asus P5K Premium with Q6600 and 4GB of DDR2-800 (dual-channel so 1600 effective) <Googulator>but the utility that generates the final image you can flash on SPI or SD/eMMC <Googulator>there's a header that needs to be prepended, and that's encrypted IIRC <stikonas>u-boot produces idbloader-spi.img and u-boot.itb files <Googulator>on 35xx, there's the RKNS header, which is now fully in the clear <stikonas>unless it does some inline header encryption <Googulator>on older SoCs, the header had to be encrypted (and it had a different magic vs "RKNS") <stikonas>well, begining of the file is a bit unreadable <stikonas>though it's not like I can read aarch64 machine code... <stikonas>yeah, not a problem for using stuff, but problem for bootstrappign <Googulator>the header is always RC4-encrypted, while the binaries referenced in it can be in the clear or RC4 <Googulator>at least RC4 is a stream cipher, so you can just ship a premade keystream & XOR with it <stikonas>but what's the point of all this mes if key is open... <Googulator>IIRC you can reprogram it by burning some efuses <Googulator>but 35xx just uses no crypto at all when it's not locked down <stikonas>yeah, I think I've read about those efuses <stikonas>you can kind of implement your own secure boot there <stikonas>but in home environment I'm more likely to just brick my board :D <Googulator>btw, any idea why building automake in chroot fails with "rm: cannot lstat `*/Makefile.in': No such file or directory" in WSL2? <stikonas>WSL2 just runs linux kernel in emulation, doesn't it... <Googulator>and if I run the same command outside the chroot (even using the binaries created within), it just works <stikonas>possibly by injecting static bash from later <Googulator>hmm, just realized it's using bash expansion in that command <stikonas>but you can't easily use our old bash for interactive debugging... <Googulator>../chroot/usr/bin/rm -- configure Makefile.in */Makefile.in */*/Makefile.in aclocal.m4 automake.info* -> works <Googulator>../chroot/usr/bin/bash -c '../chroot/usr/bin/rm -- configure Makefile.in */Makefile.in */*/Makefile.in aclocal.m4 automake.info*' -> error <Googulator>$ ../chroot/usr/bin/bash -c 'echo -- configure Makefile.in */Makefile.in */*/Makefile.in aclocal.m4 automake.info*' <Googulator>-- configure Makefile.in */Makefile.in */*/Makefile.in aclocal.m4 automake.info automake.info-1 automake.info-2 automake.info-3 automake.info-4 <Googulator>.in */Makefile.in */*/Makefile.in aclocal.m4 automake.info*' <Googulator>-- configure Makefile.in lib/Makefile.in m4/Makefile.in tests/Makefile.in lib/Automake/Makefile.in lib/am/Makefile.in aclocal.m4 automake.info automake.info-1 automake.info-2 automake.info-3 automake.info-4 <Googulator>and since this is WSL2, I'm worried this will come back to haunt us on regular Linux with a similar kernel <mihi>Googulator, is that inside drvfs (i.e. /mnt/c/...) or is this normal root filesystem (ext4)? <mihi>oh ok. With drvfs i had some strange encounters before... <Googulator>Binary files chroot/bin/bash and chroot/usr/bin/bash differ <Googulator>chroot/bin/bash is an absolute symlink to /usr/bin/bash AKA my normal, modern bash shell <stikonas>anyway, it sounds like globbing is broken <stikonas>though I'm surprised you even reached automake <Googulator>spun up a proper VM with Linux and copied this bash into it - it misbehaves the same <Googulator>it's somehow miscompiled in WSL, but compiles fine otherwise <stikonas>shouldn't this have been caught by checksuming? <Googulator>I'm using --update-checksums because I'm still generating the final checksums in qemu as we speak <stikonas>and this bash is even before ./configure scripts <stikonas>so it's not like somehow WSL would change build config <Googulator>Ran another chroot bootstrap, this time in a proper VM <Googulator>and echo "*/test" fails just the same when called outside chroot <Googulator>AArch64/LICENSE AMD64/LICENSE M2-Mesoplanet/LICENSE M2-Planet/LICENSE M2libc/LICENSE armv7l/LICENSE bootstrap-seeds/LICENSE riscv32/LICENSE riscv64/LICENSE x86/LICENSE <Googulator>even sudo chroot chroot/ env - PATH=/usr/bin bash -c 'echo */LICENSE' shows the same difference <Googulator>in strace, the only difference I can see is the order in which directory entries are returned <deesix>Globbing has a few configs... what 'set -o | grep noglob' says, for example? <Googulator>sudo chroot chroot/ env - PATH=/usr/bin bash -c 'set -o' is identical between the 2 systems <deesix>Is it also broken during interactive execution? <Googulator>this is the early bash built in Fiwix, it doesn't support any interaction <matrix_bridge><Christoph> I recall someone on this channel said that they circumvented mes and directly built tcc, and that this saved a lot of time. Even though there are many complaints about the time mes consumes, it is still in use. Why is it still necessary? <fossy>Christoph, primarily because no-one has implemented it yet :) <fossy>additionally, we need to ensure that mes libc can still be built using tcc <fossy>theres a few question marks still lingering regarding a mesless bootstrap <matrix_bridge><Andrius Štikonas> Christoph: also alternative path is x86 only <matrix_bridge><Andrius Štikonas> And far more work needed to port to any other arch due to number of steps <matrix_bridge><Andrius Štikonas> So maybe we can have an alternative path as optional choise <matrix_bridge><Andrius Štikonas> But we still need mes for those other usecases <mihi>Googulator, some things you could try: compare "tune2fs -l" of both ext4 filesystems, compare the base path (or its length) to the directory where you glob, compare kernel config (large file support), look at "ls -li" output for the directories if anything looks weird (e.g. inode numbers with weird bit patterns). <mihi>If you have coreutils ls built from same libc as your bash, you can also try comparing "ls -li" output, especially look for any zero inodes <mihi>(bash's own glob implementation ignores dirents with d_ino == 0) <matrix_bridge><cosinusoidally> Christoph: I've raised https://github.com/fosslinux/live-bootstrap/issues/355 regarding adding my alt tcc bootstrap path optionally. I'll raise a PR once fossy has stabilised their recent refactoring of live-bootstrap. My bootstrap path path is i386 only (and is difficult to port to other arches since I have about 13 versions of tcc in my tree). tbh it's probably "easier" for me to write an i386... <matrix_bridge>... emulator to run the bootstrap up to a version that supports other arches rather than trying to actually port it. <matrix_bridge><cosinusoidally> eg tcc_bootstrap_alt runs in about 10 mins when running under qemu binfmt_misc user mode emulation on a Raspberry PI 4. <matrix_bridge><cosinusoidally> I could do that, then get tcc 0.9.27 to generate an arm version of itself. <matrix_bridge><cosinusoidally> but tbh that's probably way down my list as I have other things I want to do first. <GoogulatorMobile>I was trying to fix the download timeout issue, and as a first attempt, I set the user agent to look like Firefox. <GoogulatorMobile>If it is, we can probably just use a curl user-agent in the Requests downloader. <stikonas>GoogulatorMobile: I think download-distfiles is reliable... <stikonas>hmm, is mes-0.26 much slower (or I guess memory bandwidth intensive...) <stikonas>though this time it might be CPU instead (as I'm running riscv64 bootstrap with qemu user emulation)