IRC channel logs
2024-12-03.log
back to list of logs
<fossy_>stikonas: i'm re-investigating some of the kaem environment variable weirdness on bare metal in stage0-posix that I was running into <fossy_>i'm not certain that the use of get_next_variable_name in _get_environmental_variables (uefi.c in M2libc) is compliant <fossy_>(well, more i cannot figure out if it is) <fossy_>i'm trying to interpret the spec, what does it mean by " If a VariableName buffer <fossy_>on input is not a Null-terminated string, EFI_INVALID_PARAMETER is returned"? <fossy_>particularly, what does "on input" mean? <fossy_>it sounds like VariableName should be a null-terminated string always, and if it is an empty null-terminated string passed in, then it restarts the search?? <fossy_>also, where does the magic numbers in EFI_SHELL_VARIABLE_GUID come from <fossy_>oh, this is very nice, got it to work :D <fossy_>(on one of my machines it wasn't working on) <fossy_>spec states that if you call get_variable with NULL data you must have 0 in data_size, which we were not doing <fossy>another thing getting a "could not allocate memory for brk area" when trying to use posix-runner.. any thoughts? <aggi>i'm limited to 2Mbps uplink, without static-ip and without dns record (this requires expensive business contracting with the ISP which i cannot afford) <aggi>oriansj: seems a decent place if wherever your server is hosted got a little more bandwidth <aggi>i too notfied on tinycc mailing list, collecting regressions etc.; and some of this could easier be reproduced, tested and fixed if some prepared devdrop.iso was ready to boot <aggi>currently, the complete set of ~500ebuilds supported with tinycc including debugging symbols is 1.6GiB in size <aggi>i'll shrink that down to contain development utilities, and i have to rewrite the portage/ebuild scripting for re-integration into bootstrappable scripting <fossy>in other news, i am trying to decide a general rule of thumb for when to update toolchains in live-bootstrap.... <fossy>considering following debian stable as a general baseline as a "this is old but modern" <fossy>but not "too old that you'd need intermediate bootstrapping steps" <fossy>because i want to avoid e.g. someone going to try to bootstrap debian from live-bootstrap as a base and running into reverse bootstrapping problems (trying to compile older GCC with new GCC can go poorly) <jackdk>Can you define some sort of "layer" or "era" abstraction, where you build up to a certain point before you hand off to the "real" distro to go the rest of the way? <aggi>some no-c++ system profile containing all development utilities, without any direct or transitive dependency towards, well, c++? <aggi>that includes almost all compilers/toolchains nowadays (because those are written in c++ themselves, llvm, gcc etc) <fossy>jackdk: that is.. a lot more difficult than i think one might consider <fossy>the hardest part of that is defining "the rest of the way", how do you decide where to stop, and where to let the "real" distro continue from? <fossy>for me, out of the question is requiring distros to deal with anything historical, i.e. you need to go back in time for a rolling release distro, or use an old version of a versioned distro, in order to bootstrap from the output of live-bootstrap <fossy>to that end, aggi i have considerable issue with that, because that leaves us at ancient GCC, which is a non-starter <fossy>as you cannot realistically build any modern distros with that <jackdk>I'm sure it is =(. There's probably some uncontroversial part (up to at least a linux kernel and GCC 2.95 but probably 4 or even 10?) and then things get really fuzzy. If there's some definite bootstrapping path into the OS, then maybe you just need to be able to run that? For example nixos, which is what I currently know best, they've steadfastly stuck to nix-2.4 AIUI, so if you can build that, you can build a nixos and then lustrate into it <fossy>right. so GCC 10 is a bare minimum, as anything 13+ requires a minimum of 10 i think we found? or something like that, give or take a few versions <fossy>"definite bootstrapping path into the OS, then maybe you just need to be able to run that?" -- yeah, i'd tend to agree, i'm largely thinking about C/C++ toolchain, as pretty much every distro (exclude guix/nix(?)) do not have any kind of C bootstrap <fossy>maybe this is a much more difficult question to realise until people have attempted more distro bootstraps haha <sam_>15 needs 5.4 or so, 14 needs 4.8 <sam_>it's definitely not just a few versions <jackdk>nix will need C++. guix uses the nix daemon so probably will too. Debian will want to run `debootstrap` I suppose but if you're full-source bootstrapping `debootstrap` just to download binaries from the internet I notice that I am confused. (I suppose you can trust your gpg, at least) <jackdk>the equivalent in gentoo would be baking a stage1-equivalent tarball, I guess? Then you could unpack that into a chroot and build your gentoo? <fossy>sam_: hmmm, will need to check back on my notes/commit history, but there is some reason that in live-bootstrap we go 4.9->10->13 currently <fossy>ahhhh, okay, that was easy, because it's not 4.9, but 4.7 that we build currently <fossy>because that's the final C-only version, and then that can build up to GCC 10 <fossy>sorry, had that around the wrong way <fossy>i think there might have been a newer version of that somewhere <aggi>fossy: it won't leave you at ancient GCC, if the dependency chain does not block you from compiling any recent toolchain, and tinycc can do just that. <aggi>so this would be another criteria, a complete system integration for a distribution driven by tinycc (or any other alternative complaint toolchain) would exist <fossy>aggi: okay, i would like live-bootstrap to actually be usable though to bootstrap a modern distribution, and the reality is that everyone uses GCC or LLVM for their C compiler, and any modern GCC or LLVM needs C++ <aggi>the integration path i would propose, for what jackd considers "a certain point before you hand off to the real distro": <aggi>initial-seed/loaders -> mesc -> tcc-kernel/tcc-distro fully converged with live-bootstrap -> any other toolchain/distro <fossy>in relation to this step, "tcc-kernel/tcc-distro fully converged with live-bootstrap -> any other toolchain/distro" <fossy>any other toolchain/distro requires C++. tcc does not compile C++. how do you rectify that? <aggi>fossy: bootstrappable and too tcc already addressed the issue, because tcc can compile binutils and gcc-4.x, and from there you got various options to crossdev towards any architecture/toolchain desired <fossy>ohkay, this violates my requirement of live-bootstrap being directly able to build a modern distribution though <fossy>that's pretty much exactly what we already do in live-bootstrap <aggi>fossy: no, you haven't got a complete tcc distribution, which avoids gcc/binutils as mandatory dependency early in the dependency chain <aggi>and the question was, exactly when at which point system-integration was handed over to other distributions <fossy>fair. in this hypothetical situation, each distribution then needs to have a tinycc->modern toolchain bootstrap then, correct? <aggi>they always do, i would merely recommend a tinycc distribution existed for this, which was fully integrated with live-bootstrap <aggi>*i mean a complete tcc/linux distribution <aggi>live-bootstrap -> i486-tcc-linux-musl.iso (without gcc) -> any further system-integration <fossy>understood. i think what you are proposing is reasonable, but not what I want live-bootstrap to be. live-bootstrap's primary goal has always been a bootstrap of a "modern toolchain", very roughly defined as capable of building LFS (or similar), and i would not consider tcc a "modern toolchain" <aggi>bootsrappable heavily relies upon tinycc, it could just simplify the whole transition to gcc/binutils/llvm if you got a bootable and complete development ISO with tinycc <fossy>i entirely agree - that's why i say i think your idea is reasonable - even that's a good idea - but i don't want live-bootstrap to stop there <aggi>with another benefit, gcc (and too binutils) were _optional_, not _mandatory_ <aggi>i too agree, it must be possible to arrive at gcc and binutils, i would just prefer if that was optional for any distribution to decide <aggi>i too would prefer if Perl/autotools remained _optional_, but that's too much effort <fossy>sure. each distribution has to make a choice where they start their bootstrap <fossy>i have been thinking on and off that it might be nice to have different exit points from live-bootstrap, but that is not an easy thing to do properly <aggi>well, i think it is feasible to emit a complete i486-tcc-linux-musl.iso fully converged and re-integrated with live-bootstrap <aggi>although this required a "little" fork, and i am overwhelmed already with forking against linux-2.4, and forking against gentoo already <aggi>keeping tinycc synced to latest HEAD, was, well another little headache with regressions introduced <aggi>and, worst of it, i had proposed this project work to various employers in germany and a faculty, which was ignored <aggi>besides i got banned from #gentoo-toolchain and #gentoo-releng <aggi>last wednesday i asked at financial department if some ISP business contracting could be supported, so i could host my stuff, ... denied <fossy>can you elaborate on what "fully converged and re-integrated" means? <fossy>i think this would be a very difficult project to get funded, lol <fossy>in the long term, i would personally prefer tinycc to be removed from the bootstrap chain, it is one of the greatest pain points currently <fossy>... how did you manage to get banned?? <aggi>i got banned when discussing tinycc-toolchain integration into gentoo <aggi>fossy: this means some packaging/scripting is necessary, and live-bootstrap style seems a decent approach to replace portage/ebuilds <aggi>hence, i can follow the live-guistrap path up until some tinycc and maybe as86 assembler exist, then emit tccboot loader to compiler a linux-2.x kernel, and a sufficiently complete devutils-root.squashfs loaded <aggi>and i consider tinycc support some of the biggest benefits, since the bigger "pain points" are the mandatory gcc/binutils dependency far too early during bootstrapping <fossy>ok final question for you for now, what's the concern with gcc/binutils? <aggi>fossy: first and foremost, it isn't their GPL licensing, but technical issues, which is why i consider those developement tools inappropriate when those become a _mandatory_ dependency far too early during bootstrapping <aggi>both gcc and binutils are gigantic in size, lines of code, it's too much depending on circuumstances <aggi>and in practice, tinycc is much faster compile-time, which greatly improved my workflow ever since i got full support with my system profile for it <aggi>and i've seen gcc hitting oom with 4GiB RAM available far too often <fossy>"circumstances" must be pretty minimal requirements... no chance you can use most useful modern software with only tcc <fossy>faster compile-time i find to be a largely non-issue, especially as it generates slower (less optimized) code than gcc/llvm, but i see the benefit for development in some situations <aggi>that's the trick, i don't need to; and it is possible to optionally spawn any recent toolchain with tinycc <fossy>what are you compiling with tcc that gcc uses 4GB of RAM for, lol <aggi>fossy: for example i bootstrapped a gcc-6.x with gcc-4.x, "lol" <aggi>because i had covered aarch64 with my crossdev stuff, that needs a latest toolchain <aggi>which, admittedly, is a disadvantage with tcc, it cannot and maybe should not cover all ISAs <aggi>with compile-time/run-time performance tradeoff tinycc is far closer to the sweet spot than gcc or llvm could ever be <fossy>maybe a good performance tradeoff for the <1% of users who compile their own software on low-mid spec machines :) <aggi>tinycc compile-time is much(!) faster an all machines <aggi>and important, i would be far less excited about tinycc, if it wasn't possible to spawn a gcc/binutils with it, but thankfully it is <fossy>yes, but for anyone who's not building for just themselves, and not using incremental compilation, i.e. distributions/businesses, they care a lot more about runtime speed, ISA support, and quantity of software buildable when they choose a compiler, than runtime speed <aggi>tinycc does NOT lock you out from transitioning towards a full GNU <aggi>with regards to runtime performance, another hint, i think a linux-2.x is more efficient, if it had full support for NPTL and O1 scheduler <fossy>linux-2.x is more efficient than modern kernels?? <aggi>yes, i think so, regardless of BKL, i think it is missing O1 scheduler and NPTL which weren't rebased onto linux-2.4.37.11 which severly limited scalability of it <aggi>in parallel linux rushed towards their 2.6 series kernel, which quadrupled lines of code.. something i dislike in principle <aggi>otherwise i would invest more time and effort into supporting a later kernel version, but i think it is not worth it due to this <aggi>a problem major distros have, they cannot roll-back kernel or toolchain, optionally at least, they can't <aggi>bootstrappable.org chose a 4.x kernel version, which is missing time64 i think, hence i see no relevant disadvantage to roll-back further <aggi>*for POSIX 32bit, which is the basis of bootstrapping, indefinitely <fossy>answer to that is not difficult: hardware support. <aggi>see, linux-2.4 seems to hit another sweet spot, because usb-2.x, SMP, PAE are feasible <aggi>usb-2.x and PAE already booted, SMP is a little flaky when compiling with tinycc still <aggi>i already could boot a kernel patched for o1/ntpl too, that's flaky too <aggi>anyway, linux-2.x was the better kernel imo <aggi>and tinycc seems the better toolchain <aggi>i've not checked GNU hurd yet though, but i suspect it may not have USB support anyway <aggi>as far as "kernel bootstrapping" was concerned, and which options exist <fossy>NVME, AHCI, EFI, any still usable non-i[3456]86 ISA, USB-C, are all missing from 2.4 at least <aggi>ahci was unstable, yes, but it already got introduced <fossy>anyways, i need to go do other things, so i will leave our discussion here, but overall i don't think your philosophy, even if correct from a technical basis (which i find dubious), is really usable by most people for any real work <aggi>sata support is present, but not for all controllers, and i see no *practical* benefit, an NVME can be exposed as IDE device too <aggi>yes, linux-2.4 maintenance would need siginifcant efforts for backporting some stuff <aggi>for example, it hasn't got a 32bit POSIX filesystem with 64bit timestamps, and nilfs2 was introduced with linux-2.5/6 series <aggi>i _must_ backport nilfs2, and a few other things (usb-audio, usb-ethernet) <Googulator>I've not heard of any NVMe that can emulate IDE - but even if it can be done, IDE emulation on modern CPUs is problematic <Googulator>Non-AHCI SATA is just IDE emulation, and it has the same PMIO issue <aggi>yes, but the choice to begin with, is USB2.x or none thereof at all with any kernel-version chose <aggi>fiwix, gnu hurd, linux-2.x being candidates <aggi>some minix guy had joined discussion on tinycc mailing list <Googulator>Also, being tcc-compilable is not enough for kernel bootstrap <Googulator>Any kernel used needs to be tcc-compilable *on builder-hex0* <aggi>you need to distinguish compatibility for the use-case, that was either merely bootstrapping, or to emite a _complete_ distribution <aggi>i think it is possible to keep a clean dependency chain from build-hex0 towards tccboot/kernel and some devutil.squashfs loaded with it then <aggi>at the stage of hex0 and later mesc/M2-planet, there isn't a bare-metal bootable distro yet, but it contains the needed utilities to emit some <aggi>a major caveat seems to be 16bit real-mode asm processing, as86 and legacy linux-2.2 bootcode.S stuff might help <aggi>so all this doesn't need to be re-written from scratch <matrix_bridge><cosinusoidally> builder-hex0 -> tcc -> tcc-linux might be possible (similar to how live-bootstrap currently does builder-hex0 -> tcc -> fiwix). It may be easier to keep the fiwix step though, just for ease of use/easier debugging/testing. <matrix_bridge><cosinusoidally> fiwix could even be used to set up the hard drive before building/installing tcc-linux. <matrix_bridge><cosinusoidally> or is there some flakeyness around hard drive access from fiwix in the live-bootstrap version? I know you do the fiwix -> linux step in ram via kexec. <Googulator>Fiwix only supports legacy IDE-style hard drive access AFAIK <Mikaku>Fiwix currently supports PIO and MWDMA (only QEMU Intel PIIX3 PCI ISA IDE controller) ATA modes for hard disk access <Googulator>I mean legacy IDE as in non AHCI or anything that's exclusively MMIO <Googulator>because the concept of "IO ports" never existed outside of x86, and is problematic even on modern x86 <Googulator>Intel VT, AMD-V and ACPI all have hacks to deal with PMIO that unfortunately require OS support <Mikaku>AHCI is indeed not supported yet