IRC channel logs
2024-12-18.log
back to list of logs
<oriansj>Lance R. Vick: generated programs tend to become deadends because no one truly understands how they work usually. And if you understood how it worked (by say just reading the Ada code enough to understand it); then you could just write the equivalent in about 2 months of work in straight C or C++ (as the understanding is usually the delay not the throwing code into files) <oriansj>deesix: well it exists for anyone who needs the files; they are also all available in a git repo for checkout/pull requests/etc <matrix_bridge><Lance R. Vick> oriansj: the neat thing about this generator, if you look at the example, is the generated c code adapts the structure and variable names of the original ada code, and can even include the original ada code as comments for each function if desired <matrix_bridge><Lance R. Vick> which is much better than any of the generation approaches I have seen at least <fossy>stikonas, deesix: the vibes i am getting are not to rely on the .tar.gz snapshots, as they can be enabled/disabled on a whim, and are intended as a web browsing convenince rather than something to rely on <fossy>and that it would be better for us to generate the .tar.gzs ourselves <fossy>i think this kind of opens up another can of worms regarding source distribution as to what we find acceptable. <fossy>currently we suppose HTTP (non-SSL) distribution as a hard requirement <jackdk>HTTP distribution being safe because we already know the hashes of all distfiles? <fossy>jackdk: yes, that is not limited to HTTP though. it is the de facto (and imho, correct) choice because it is ubiquitous <jackdk>Yes, and you can get away with echo and an nc-ish program if you really must <fossy>however, we have intentionally ensured that (for both ease and trust) we have "upstream" HTTP sources available -- yet we bend the rules on this a fair bit -- using mirrors, i think we even had archive.org at one point <fossy>so i think the question we need to consider more generally is: For those distfiles that must be pre-loaded into the system (i.e. before we have networking+curl available), do we require that they are available from a non-#bootstrappable, HTTP, source? <jackdk>I think that's unavoidable. There are too many unreliable upstreams that do things like re-issue distfiles, force-push over git tags, etc. I need to write to the guix people about some of their distfiles not matching their hashes <fossy>or is it sufficient that they are available from a non-#bootstrappable source of any kind, from which the tarball we use is either a) able to be created from or b) on HTTPS? <fossy>i should say, or b) the same identical tarball, just accessed through HTTPS <fossy>same thing goes for everything between networking and SSL support in the live-bootstrap chain -- can we just distribute those tarballs ourselves, and let users check later if those tarballs are OK once SSL support is available? <jackdk>from a trusting-trust perspective they're all untrustworthy until you know you've got the right files on disk because you've hashed them with a hashing function that you've bootstrapped, no? <fossy>so that's why i think our limitations at the moment might be a bit artificial <jackdk>I think however you get bootstrap dists, they have to come from a stable, ubiquitous protocol. The most reliable thing I can think of is the mirrors traditionally used in the Free Software world - they're more durable than torrents and don't have a SPOF. But they should be mirrors of upstream sdists, ideally, not mirrors of live-bootstrap-hosted sdists - you don't want to be so trusted that you can swap the sdists out on people <lrvick>Our single biggest problem in stagex is unreliability of the many mirrors for all the bootstrap sources. We hardcode hashes, but it is important that we maintain a provenance for the original/official location as the primary mirror, even if we sometimes have to fall back to a secondary mirror. <lrvick>Our hope is that we get enough people to mirror the dists that we can source from the mirrors at random with our fetch.py tool <lrvick>as long as all hashes match what we have hardcoded in our tree, we call it good <fossy>^ i think this approach makes a lot of sense <lrvick>I think the goal would be for the stagex project to host a full mirror of everything, but if say livebootstrap does the same, great <lrvick>we can source from each others independently managed mirrors in addition to upstream <fossy>lrvick: is "primary" = "upstream", or primary = some "official" stagex mirror? <lrvick>for now, primary will still be upstream, and just fall back to our own mirror <lrvick>jackdk: full source-bootstrapped, reproducible, multi-signed, and container-native toolchain to build all the things. <jackdk>Having the primary mirror be the real upstream, and ideally errors getting noticed and reported sounds good. There is a theoretical risk if a project like live-bootstrap becomes the trusted primary mirror, because then you can switch the dists and it'll be a while before people notice. <lrvick>but yeah, we added a toml file with sources to every project, and will incrementally be adding mirrors as we find them <fossy>that is my concern, i would want some kind of auto-diversification to avoid that jackdk <lrvick>once we have at least 3 mirrors, I would want to source randomly <jackdk>This also means there's another metric for the "quality" of the bootstrap: how long do you have to go building stuff before you have a checksumming program to check your sources? <fossy>for us that is "one of the first things once you get to C" :D <jackdk>makes sense. Though some brave soul might write one in one of the hex languages one day <lrvick>in our case, we download all sources up front with our fetch.py using the python standard library. Various team members have different host distributions <unmush>not related, but some day I'd like to see a special flash drive that lets you multiplex a first boot sector and boot partition using some physical mechanism. So you could, for example, move a slider to a position labeled, e.g. "x86-64", "armv7", "RISC-V", etc, and when you plug it in it will present the corresponding boot sector and boot partition. You could then populate the non-multiplexed part with almost exclusively source code and <fossy>Googulator had some "trusted flash drive" project/idea <jackdk>I don't even think you need crazy mutliplexing stuff for that, just a slider to choose which boot partition to present <unmush>is that not what multiplexing is? <jackdk>I thought you meant some kind of crazy hybrid format like the cosmoverse's APE executable format <unmush>oh no, I meant basically what you said <jackdk>lrvick: stagex looks like a cool project. Unfortunately guix is not bootstrappable at the moment - some sdists have different hashes to what guix expects and I can't proceed beyond openssl-1.1.1l because the test suite has certificate time bombs in it <jackdk>(I know you don't use guix in stagex, but your readme currently calls it bootstrappable) <lrvick>One of our goals at distrust/stagex is to ensure our source hashes are the same as those doing baremetal pulls via some of the actual "live" builds. That said, we have used stagex to bootstrap all the way up to secure enclaves, some of which actually do http TLS termination and DNS validation in the stagex-built enclaves... so nothing stopping them from being oracles to reproducibily <lrvick>download archives from various positions on the internet and confirm there is no dns poisioning or bgp attack nastiness. <lrvick>and these enclaves are remotely attestable and managed by quorums of people. <lrvick>By no means should that be the -only- solution, but if it matches everyone elses, helps stack confidence <unmush>jackdk: well, technically with sufficient permissions all time bombs can be worked around, you just have to figure out the right time to set the clock to <jackdk>unmush: yeah, and it would be nice if the builds did a faketime or something. But alas, that's not what's in their package set right now <lrvick>We have to libfaketime in a couple places I admit. Not ideal, but works <lrvick>sad guix is not currently bootstrappable, but I am happy to call that a temporary bug, given it is by design and has been in the past <lrvick>I want to be as charitable as I can be to others, given we have build breaking bugs from time to time too. <jackdk>Yeah I like the attitude in these spaces <unmush>hopefully guix is planning on adding faketime usage the next time the packages in question get updated <lrvick>I am mainly critical of the distros that have officially stated something as a non-goal. <lrvick>people should be aware of that when selecting a toolchain <lrvick>Guix has done heroic work on bootstrapping so even though they are not 100% reproducible, credit where due. We copied their rust bootstrap path for instance. <lrvick>in our case we actually can't release something that is not reproducible. Our release process won't allow it. That means we won't have ADA or haskell any time soon... And I need ADA to bootstrap coreboot. <lrvick>heads has network access now, so would be nice to be able to bootstrap from firmware up <jackdk>Guix is undoubtedly heroic and the most exciting GNU project I've seen in a long time. It's great to see free software at the forefront of something instead of trailing and trying to free users on existing proprietary software <fossy>feel free to say if you think any part is a bad idea, but say why :P <aggi>"And I need ADA to bootstrap coreboot." <aggi>it's mentioned "native graphics initialization" requires ada/gnat, otherwise coreboot does not need ADA <aggi>it should be possible to keep coreboot without ada, and let kernel initialize framebuffer which does not need ada then for neither kernel nor coreboot <lanodan>Or ditch framebuffer and connect via serial port? At least on machines where that's possible. <aggi>to my understanding coreboot can initialize VGA still, that's good, and kernel can initialize framebuffer if desired <aggi>anyway, the real concern is UEFI and CSM being phased out <aggi>this is an extremely aggressive move from Intel <aggi>the dominance of UEFI is bad enough, but at least CSM permitted 16bit real-mode boot <aggi>there's no excuse that option got removed <aggi>and it's breaking a major promise of Intel and Microsoft, for "backwards compatibility" <aggi>as a consequence most of the significant progress made by bootstrappable project with x86 is planned for obsolescence and beyond it is an unpreceeded attack against freedom <aggi>it too affects my approach to revive linux-2.4 with a complete system supported with an alternative unix toolchain on x86 <aggi>because such system would not boot on some recent laptops and whichever intel considers their servers <homo>uefi is not specific to x86, it also infects arm and riscv <aggi>yes, and even worse, arm and riscv need most recent kernels which are difficult to bring back support with pcc compiler or tinycc <homo>then all hope in hurd, plan9, inferno <aggi>at least the 12billion subsidy cashpot for the Intel fab in Magdeburg was terminated <Googulator>I respectfully disagree; choosing 16-bit BIOS as the target for kernel bootstrapping was, in retrospect, a bad choice on our side <Googulator>It is unfortunate that x86 doesn't really provide a "native" boot interface that effectively enables replacing the firmware - but even if it did, I would've considered bootstrapping a UEFI-compatible environment, and going further from there, a priority <Googulator>UEFI and multiboot are really the only choices for an open interface from the firmware to the kernel <Googulator>well, there was also OpenFirmware, but it's essentially extinct <Googulator>ARM and riscv chips usually provide a well-documented (but platform-specific) interface for loading firmware, x86 unfortunately doesn't <Googulator>But depending on such an interface without then bootstrapping a UEFI- or multiboot-compatible stage is problematic precisely because of the nature of such low-level firmware loading interfaces: anything that relies on them becomes, well, firmware. <aggi>presence of 16bit real-mode isn't a choice, it's mandatory <Googulator>Intel CPUs have been able to boot natively since Sandy Bridge at least; AMD probably a generation later. <aggi>x86 real-mode isn't IBM specific, and it's not intel specific either, zilog implemented it too <Googulator>And there's, as far as I'm aware, no modern architecture building on its legacy <aggi>"technical difficulties" <Googulator>Anyway, the goal of bootstrapping is not to get compatibility with 40-year-old systems <Googulator>What does that have to do with the need for retaining real mode? <stikonas>and even if that was bug in Ada, it doesn't mean that other languages (including C) don't have bugs <stikonas>and I'm sure 32-bit mode will also go away in a few generations <stikonas>once they see that world is not broken, they'll remove it <stikonas>it's a precious space on the silicon wafer <stikonas>if you waste it for compatibility, eventually, your competitors who don't keep backwards compatibility will outperform you <aggi>without x86 real-mode intel hasn't got a bootstrapping path <stikonas>yes, but how many sales does bootstrapping generate: nothing <aggi>and almost all other architectures including riscv or arm rely upon it indirectly at least <Googulator>In fact, it's pretty much gone on current CPUs as well, it's just emulated in microcode <aggi>Googulator: to my understanding the entire ISA is routed through some instruction decoder <aggi>but that's a whole other concern <Googulator>Modern x86 CPUs don't even really start up in 16-bit mode anymore since at least Sandy Bridge - Intel ME actually needs to configure the CPU for 16-bit emulation <aggi>Googulator: what's hiding inside the IntelME with it's minix? <Googulator>Intel ME is indeed a whole can of worms; AMD at least made a commitment to replace AGESA with an open-source solution starting with Zen 6 <stikonas>you can at least cripple Intel ME to some extent <stikonas>but anyway, all this microcode stuff is happening because you vendors can fix some bugs afterwards <stikonas>if you make a hardware with bugs, it would be billions of USD to fix it <Googulator>Bootstrapping AMD's openSIL will be an interesting challenge :) <stikonas>Googulator: but it's not like you can easily interact with your CPU before openSIL would load <Googulator>Since it's meant to be open source from the very first instruction that the management cores execute <stikonas>I would guess you best bet would be to cross-compile it from some other system that is easier to interact than PC <stikonas>yes, but how do you start bootstrap there? <Googulator>So to get a bootstrap rooted before openSIL, you would need to compile code on management cores <Googulator>AMD already supports (in hardware) loading AGESA & UEFI from USB <stikonas>so more similar to those riscv64 or aaarch64 boards that can load first stage bootloaders from e.g. sdcard or emmc <Googulator>so presumably you would put a stage1 on a USB stick, named as if it were an openSIL+UEFI package, and then boot with the USB boot override pin pulled low (which is a button on the IO shield of most AMD boards these days) <aggi>aarch got their separate ATF, at least that's opensource, but hasn't got a bootstrapping path <Googulator>Boot selection pin high -> AGESA loaded from SPI <stikonas>still, bootstrapping anything in these pre-firmware environments will be very time-consuming and target specific <Googulator>aggi: ATF is different, as it runs on the same Arm cores as the main OS, just in a fully privileged mode <aggi>Googulator: not sure, i recall some separate cortex-m0 or something exists on various boards, but don't quote me on that <Googulator>AGESA & ME are more like the VPU firmware on Raspberry Pi, which starts up the ARM cores from outside <Googulator>Some ARM systems use a Cortex-M for the same purpose, but it's not ATF that runs on it <stikonas>generally there are lots of micro CPUs in any hardware <stikonas>there are probably even riscv cpus these days there <Googulator>Recent RockChip ARM CPUs have several Cortex-M0s in them, but interestingly the first core to boot (from an internal boot ROM) is one of the "little" cores (Cortex-A5x) <Googulator>RK3588 IIRC starts the boot ROM on all 4 A55 cores, but all but one just go into a spin loop, waiting for the main OS to take control of them <stikonas>well, tha'ts the main system, but then each periferal would again have auxillary CPUs, e.g. network card <Googulator>then it loads a DDR training stage into SRAM, which starts the "real" RAM, and then loads ATF <Googulator>ATF then starts all the "management" cores like the M0s, and finally it loads the main firmware (uboot or EDK2) <stikonas>well, that's what U-boot's early stages usually do <stikonas>u-boot tpl or spl are loaded into sram and have ram training code <stikonas>(possibly hardware assisted but I'm not sure) <Googulator>Yeah, but RockChip AFAIK uses its own training stage, rather than uboot's <stikonas>u-boot can train some of them, e.g. rk3399 <Googulator>yeah, anything at that point is highly platform-dependent <aggi>after all, UEFI supposedly improved upon x86 real-mode BIOS (however that was implemented, probably ASM during the CP/M era which had some dark history attached of it's creator beaten to death) <stikonas>well, UEFI standartizes firmware interfaces <stikonas>whether it is a bit or small standard, that's another thing, but it's better than no standard at all <aggi>back to IBM, who "standardized" their PC with BIOS, and acquired MS-DOS <aggi>to comprehend some of the related history, arriving at those z80 systems and vt100 terminals <aggi>it is rather challenging, to re-create some vt100 from schematics fully opensource, both hardware and software <aggi>anyway, at least x86 real-mode boot did work, for decades <aggi>the software was written in ASM without excess dependencies, and the hardware could be soldered onto breadboard <aggi>any UEFI system is nowhere near that type of openness, never will <Googulator>You do realize EDK2, the UEFI reference implementation, is FOSS? <aggi>Googulator: is it written in ASM? <aggi>this is a principle issue, BIOS has to cope with low-level parts, and nothing else <stikonas>in general people don't want to deal with asm <stikonas>even at M2libc level, UEFIs support is almost fully in C, there is very little assembly there <Googulator>Also, it needs to be able to present a unified interface to the operating system, even in the face of the system's hardware changing - otherwise the operating system itself becomes firmware <stikonas>instead of syscalls, in UEFI you just call a function <Googulator>This is why Arm's SystemReady initiative is such a big deal - it enables Arm designs to clearly split off the firmware from the OS, which can remain software <Googulator>enabling users to _install_ an OS, rather than _flash_ one <aggi>from the standpoint of bootstrapping, this introduces a gigantic circular dependency which cannot be broken <fossy>what's the circular dependency? <fossy>i read backlog, but still a bit confused, lol <aggi>fossy: this means you need a c++ compiler at a stage of bootstrapping this could not be available <aggi>although uboot isn't written in ASM either <stikonas>nothing forces you to use EDK2 or U-Boot if you bootstrap firmware from the lowest stages (e.g. bootstrap OpenSIL) <stikonas>at that point you are bootstrapping for a very specific target, so unified interface doesn't matter too much <stikonas>few people would bootstrap their userspace, then fewer people would run kernel bootstrap and then again, almost nobody would be bootstrapping their firmware on a machine <Googulator>the idea is, you use a few custom, platform-specific initial stages written in machine code and hexX, until you get your assembler, then get to a C compiler & shell, and bootstrap just like you started from BIOS (or rather, we are treating BIOS as if we were truly bootstrapping from bare metal; which we really aren't) <stikonas>but if you are really one of those people who want to build up from pure hardware, you'll just figure out how to bring up enough hardware to initialize RAM and I/O <stikonas>and then just run your bootstrap form there <Googulator>Then, at some later point, you can build the final firmware stages (openSIL, uboot, EDK2, coreboot, etc.), and set them up so that upon reboot, they manage the startup process instead of the full bootstrap chain <stikonas>so the fact that edk2 or other firmware needs more dependencies, would not block you <Googulator>in fact, GRUB IIRC loads the kernel using its EFI stud, effectively implementing a mini-UEFI in the process <Googulator>in fact, pretty much a complete and proper UEFI in recent versions <Googulator>albeit not UEFI-PI, just the final interface to the OS <stikonas>yeah, Linux kernel on my arm machine starts booting with the following in dmesg: [ 0.000000] efi: EFI v2.100 by Das U-Boot <Googulator>UEFI-PI is optional, and not required by the main UEFI standard <aggi>stikonas: on ARM linux kernel does not need EFI to boot at all, fun one imo <aggi>with my rockchip system i disabled EFI with both uboot and linux-5.9 kernel, works perfect <Googulator>In that case, you're using the competing firmware interface standard, Multiboot <aggi>i fiddled together a custom firmware layout, and uboot directly loads kernel, without initrd nor anything <aggi>with uboot there wasn't any need for a further boot-loader nor multi-boot <stikonas>some drivers don't work well when compiled into the kernel <Googulator>aggi: uboot has 2 standard modes of loading the kernel, UEFI (bootefi) or Multiboot (bootm) <Googulator>I wonder if Fiwix still supports a direct 16-bit boot mode, or only Multiboot <Googulator>but the point is, UEFI and Multiboot are both just interfaces, not tied to any particular implementation from either the firmware or OS side <stikonas>possibly both but we use multiboot for kexec if I recall correctly <stikonas>indeed, bootm is not any more direct than bootefi <stikonas>just slightly different handover procedure <Googulator>also, bootm is MultibootV2, while Fiwix is loaded using MultibootV1, which is not compatible <Googulator>which is why Fiwix can't load Linux using its implementation of Multiboot kexec <aggi>this is how i load kernel with uboot: bootcmd=mmc dev 1 0;mmc read 0x1f00000 0x13800 0x800;mmc read 0x2080000 0x8000 0xb800;booti 0x2080000 - 0x1f00000 <Googulator>you could also argue that Linux's kexec interface is another firmware interface standard, although no real firmware ever implemented it AFAIK (only the Linux kernel itself) <aggi>then kernel loads a squashfs identified by a PARTUUID on the mmc MBR <aggi>the kerne image isn't compressed, but the rootfs is, with a few patches made to kernel <Googulator>"booti" is a legacy boot mode AFAIK before either UEFI or Multiboot were standardized on Arm <Googulator>which means, your kernel is probably tightly bound to your particular Arm board <Googulator>you're unlikely to ever successfully "booti" a generic Arm kernel <aggi>yes, and i think i removed usb support from uboot too <aggi>but i had to call it quit when digging even lower into ATF <Googulator>That means, your operating system kernel is effectively firmware - hence, by definition, you have an embedded system, not a general purpose one. <aggi>yes, i tried to decrease code-duplicate between kernel and uboot as low as possible <aggi>there shouldn't be anything remaining inside uboot which could do uninentional things with usb or ethernet <aggi>the "real mode" equivalent seems to be ATF however <aggi>ARM isn't my peace of cake <aggi>yes, and i tried to document an adress mappings table for the rockchip board <Googulator>ATF runs in high privileged native mode, basically the same mode builder-hex0 runs in during the x86 bootstrap <Googulator>AFAIK this is also the default startup mode of recent Intel and AMD cores, before ME/AGESA sets up real mode emulation <Googulator>which is when, by default, the main firmware gets control <aggi>besides real-mode is tied to a different address-mode, segment+offset addressing scheme, which is rather low-level in comparison to whatever an MMU hides inside <stikonas>segment addressing is probably emulated in microcode these days <Googulator>yes, likely no CPU produced in the last 10 years has any physical segment registers <aggi>nonetheless, just because an MMU is hiding the details of bank-switching doesn't mean the technique would not exist <stikonas>it's just a map from one address to another <aggi>probably, besides IOMMU, io/APIC <Googulator>also, current CPUs (since Haswell and Bulldozer) don't even fully emulate real mode <aggi>i dislike x86_64 for several reasons <aggi>gate a20 was coped with for decades, syslinux for example just made sure it's turned on/off, and no further trouble with it <Googulator>so FFFF:0010 and 0000:0000 are different physical addresses, unlike on a real 8086