IRC channel logs
2024-12-29.log
back to list of logs
<tmg1|michelson>but apparently i compiled without support for -Xbootclasspath which is kinda silly <tmg1|michelson>so, again, whatever configure options would be handy on the bootstrapping page <tmg1|michelson>and i guess i compiled without classpath being in its default location hrm <oriansj>stikonas: actually we looked into asic costs and we could do a custom asic (without pin restrictions) for as little as $9.75K or if we are willing to have pin restrictions: $500 https://tinytapeout.com/teaching/ <oriansj>So us doing custom asic work is definitely something we could afford if someone is willing to do the work on designing custom hardware to help with bootstrapping. <oriansj>homo: thank you for the link, didn't know anyone took the time to get that working in guix <oriansj>(skywater fab being 130nm which is sufficient for 64bit multicore CPUs with 1MB of L2) <oriansj>fossy: nothing; (I just didn't get to a link of 64bit RISC-V chips yet) <oriansj>the point being, we are not limited to what is readily available and we can do custom hardware to make bootstrapping easier (if we have hardware people interested in exploring that solution space) <Googulator>just for comparison, most of my bare metal tests have been on a 65nm quad-core, so 130nm is about 1/4 density <Googulator>& of course, with custom hardware, it can be designed in such a way to facilitate truly bare metal bootstrapping - e.g. with enough memory available right from power-on to get TCC bootstrapped, so that any memory training (if needed at all) can be done using C code <Googulator>plus readily available UARTs / SPI / etc. available via MMIO without any tricky initialization <oriansj>(and writing the firmware/bios is much easier of a problem for hardware we design) <Googulator>ideally, the firmwre is part of what we bootstrap :) <Googulator>I wonder if there is a simple, free IP core out there implementing basic x86-64-v1 <oriansj>not that I have seen found yet; plus we don't need *FULL* AMD64 support only the subset used by us <Googulator>should be possible given the age of the architecture - more than 20 years passed since the first implementation came out from AMD <Googulator>so any patents should be expired - again, I'm only talking about the basics (what GCC calls "v1") <Googulator>I'm interested in x86 mainly because of Haskell - I simply don't see any other architecture ever successfully bootstrap GHC without cooperation from GHC's developers, which seems to be sorely missing <homo>what are those unregistered HC files of ghc 4.08? <Googulator>Guix contains a comment saying that they cause segfaults <Googulator>But even if they do build (yielding an extremely slow GHC), it's gonna cause problems in later versions <Googulator>AFAIK unregisterised builds are dead in newer GHC <homo>and also arm support is pretty recent, so no way for arm boostrap <homo>riscv is in worse situation <Googulator>I'm not even sure if riscv is supported in the most recent GHC <Googulator>I wonder how we're even able to desktops on riscv then... <Googulator>given that modern GUI frameworks all seem to depend on Haskell in one way or another <sam_>that's what i keep saying over and over <sam_>haskell is important for pandoc but it's very, very easy to have a haskell-free system <sam_>stikonas said the same the last time this came up <sam_>most gentoo systems don't have it <homo>and gnome depends on rust <sam_>sure, rust isn't the same as haskell, rust is a different situation <Googulator>and a guarantee that rustc will never depend on its own quirks <homo>but that doesn't help on 32-bit <unmush`>last I checked rust doesn't have a standard, so isn't *all* rust behavior a rustc quirk? <Googulator>I'm referring to the rule that rustc version N, compiled in release mode (which disables all undocumented, experimental and in-development features) must be able to compile rustc version N+1 <sam_>(I'm not sure that's even a rule, fwiw, at least in the past I've not been able to find docs for that) <sam_>as far as I'm aware it's an unofficial rule of thumb for what may work <Googulator>AFAIK that's how they generate the official binaries <homo>I don't know of any modern gui frameworks that use haskell <sam_>it also certainly isn't true for all targets anyway, because i've filed bugs when it didn't work before, and the response was super unhelpful <homo>and I don't see them packaged in guix <sam_>and i had to figure out which patch fixed it by myself <sam_>so yeah, no such guarantee AFAIK in both senses <sam_>(N/N-1, and also release vs nightly) <homo>on the other hand, rust is infecting everything <unmush`>I would love to go around countermeming with "rewrite it in ADA!", but ada somehow is even worse for bootstrapping <oriansj>homo: xmonad uses haskell (but perhaps you might not consider that a modern gui framework?) <sam_>ada at least has a specification even if it is terrible for bootstrapping too <sam_>and actually has multiple production compilers (even if only one is FOSS) <homo>xmonad is a window manager, not gui framework <sam_>also cute that ada's spec is freely available (no hacks needed like for C, C++, and so on where you share drafts) <unmush`>one of the few times it was an unambiguously good thing US taxpayers funded something <homo>and yes, xmonad missed on wayland, so is not modern <unmush`>if only all research and development grants resulted in such open results <Googulator>I believe it's mostly indirect dependencies like the aforementioned pandoc <sam_>even then, entirely very much avoidable <Googulator>which is unfortunately required for everything, at least in Guix <sam_>it's nowhere near as critical as rust <oriansj>well the biggest problem with Ada bootstrapping is just how much of the language one would need to support to build any of the Ada compilers written in Ada (which is more effort than the year of work in the Ada Compiler we can bootstrap) <unmush`>hoping that that weight-watchers team goes somewhere <sam_>.. disable building man pages? <sam_>well yeah that's my point <sam_>it's a guix thing, not a fundamental desktop ecosystem thing <oriansj>not easily in guix but possible if you do your own tree <unmush`>you can, you just have to accept the call on the long night of local builds <sam_>which means it doesn't make sense to say "how can we have desktops on riscv w/o haskell" <sam_>but honestly even then, not that much even uses pandoc given this <sam_>(even if you always build the man pages) <sam_>haskell is really so insignificant in bootstrapping importance compared to rust <sam_>rust is borderline unavoidable for many usecases now <Googulator>The problem is, if X depends on Y which depends on Z which depends on... which depends on Pandoc, then Haskell is a transitive dependency of X <sam_>i fully understand, i'm saying: a) not much depends on it to begin with, and b) the issue you started with is guix-specific anyway <Googulator>Unfortunately the only other option that is buildable in a reproducible way is Nix, which has the exact same issue <sam_>oriansj: i had assumed guix had that technology now <sam_>also, doesn't nix have build options? I was sure it did <Googulator>AFAIK it lacks the ability to make dependencies conditional <sam_>so maybe can toggle build configuration but not the deps themselves or something <Googulator>Guix and Nix both lack a notion of a package being e.g. "a dependency for running unit tests" <oriansj>but if someone just made that metadata available it could <oriansj>and could probably simplify the build descriptions too <oriansj>as one could build a package's documentation and install it but not build the package itself <unmush`>I think it would require some ergonomic way of specifying attributes for edges in the dependency graph <homo>I myself actually get shocked how much gets built, like to build pipewire, you have to build gstreamer, qt5, qt6 and gtk3 <homo>and a lot of other elephant dependencies <sam_>i find this kind of baffling because this unnecessarily makes the depgraph huge for small installations <sam_>very surprised to learn this <sam_>i bet you have to have webkit-gtk for emacs too <Googulator>The only distinction that's made is build-time vs runtime dependencies <oriansj>homo: you can avoid pipewire for a bit but some packages hardcode it in <lanodan>And gstreamer and stuff are optional for pipewire <unmush`>guix graph: error: no path from 'emacs@29.4' to 'webkitgtk@2.44.1' <homo>a lot of optional stuff is enabled because why not <oriansj>also you can't install both ffmpeg and mpv in your packages (because mpv pulls in ffmpeg too and they are conflicting versions) <sam_>because that's obviously an issue we hit in gentoo <sam_>but like, isn't the whole thing with nix and guix that you can avoid that <sam_>because package installs are isolaed? <homo>webkitgtk and rust are the reason I will move away from gnome, I cannot easily rebuild those <Googulator>Pandoc is usually a build-time dependency, so you can install packages depending on it from substitutes without also pulling in a binary of Pandoc - but if you're building, you do need it <sam_>but it's a guix-specific issue, not some sort of deep ecosystem problem <unmush`>sam_: allow me to introduce you to a despised guix hack: propagated-inputs <sam_>unmush`: wrt webkit-gtk: it was/is for 'xwidgets' in emacs, but it looks like we ripped it out in gentoo in September because it broke w/ newer webkit-gtk (https://debbugs.gnu.org/66068#47) <sam_>(just explaining that it wasn't a random example) <homo>test depends are also a huge problem during build <sam_>test dependencies can often be bad, yeah <oriansj>yeah, guix and nix have the idea which is the dream but the details are where things go sideways, it is almost like work saving was opted over better build control. (which honestly makes sense for projects that started out as personal projects) <sam_>and of course we are not perfect either, i don't want to moan every time someone mentions guix or anything, i just don't like to see the problem presented as a general one <sam_>haskell is super avoidable in general and you can have desktops without it trivially <sam_>do i want a bootstrappable ghc? absolutely <oriansj>just, it'll take more time than anyone has yet been able to invest. <lanodan>webkit-gtk is a funny one because meanwhile I gave up on building firefox & chromium due to annoying dependencies and having a new version like every other week <oriansj>If I could throw $500 into a pot for anyone willing to solve the Haskell bootstrap problem; I would do so. (then maybe someone might have motivation to do that work which WILL suck) <oriansj>lanodan: have you tried telescope (gemini browser)? <lanodan>Nah, I got stuck to netsurf for like a week once though <Googulator>As for Gentoo, what's missing is reproducibility <sam_>yep, i don't dispute that <oriansj>I love kiwix files, I just wish the viewers didn't do a full firefox build to let me read a local wikipedia mirror <lanodan>Yeah gentoo having at least reproducible stage3 tarballs would be neat, that said part of why I got interested in bootstrapping is stage3 tarballs being like a huge blob that you can't verify <Googulator>mid-kid and myself did achieve building a stage3 tarball on top of live-bootstrap without pulling in a prebuilt stage3, but without reproducibility, it's of limited utility <oriansj>Googulator: well that is a pretty great utility of trust for a stage3 <lanodan>well gentoo is fairly manual and if it can be pretty much repeatable it's fine <Googulator>my hope there was generating a stage3 with the same hash as the most recent official stage3, thus validating that lineage of stage3 tarballs as not containing a Karger-Thompson attack - but unfortunately even 2 stage3 builds from the same source code on live-bootstrap result in completely different binaries <sam_>that sounds a bit surprising, the only real difference I'd expect there is some timestamps embedded in the odd application, it's not like we do anything special there <sam_>not binaries themselves or anything <sam_>would really need to see specific examples though <sam_>but it's doable, just needs someone to do it <Googulator>Guix does solve reproducibility, but you need to pull in bootstrap binaries at the start (because the hashes of those binaries are hardcoded in Guix; a shame especially given the unclear provenance of those bootstrap binaries), plus, you get the whole Haskell issue <mid-kid>Reproducing gentoo's stage3 builds is impossible because they're never rebuilt from scratch - they're always updated based on an existing binary cache <mid-kid>Portage has no way of knowing what specific package versions were used to build something. <sam_>that's not what he was saying <sam_>they're saying that *when starting from live-bootstrap*, building it again gave a different result to when they did earlier <sam_>we're not talking about "reproducing arbitrary gentoo stage3s" (although that is a nice thing to work towards) <sam_>(and I suspect we could get there for that, too) <sam_>i.e. they said they would like to do that bit, but "but unfortunately even 2 stage3 builds from the same source code on live-bootstrap result in completely different binaries" (so "even the easier part" couldn't be done, for some reason) <mid-kid>They specifically mentioned "generating a stage3 with the same hash as the most recent official stage3" <sam_>but they went on to say "but we couldn't even do that" <sam_>it's not really a meaningful point if you can't reproduce your own ones ;) <sam_>and if we get to the point where your own ones work, we can talk about making our own releng ones reproducible too <sam_>I have some initial ideas for making that easier <sam_>we can talk about that, I just have no idea why the ones built in live-bootstrap themselves were non-reproducible <mid-kid>I agree but even if we fix our own ones, the way releng is currently setup there won't be a way to reproduce them <sam_>just like portage right now needs patches, and so on <mid-kid>I don't think googulator realized that <sam_>also, it's not constantly evolving, that's kind of the thing with the ideas I have on improving the situation <sam_>we can/should add a metadata file to the stages (or maybe a side file, but I would like it inside?) <sam_>tree commit used, maybe hashes of binpkgs used too <sam_>but they start weekly on a schedule <sam_>so it's not total chaos, is my point <sam_>of course, reproducing them in general is going to be harder if you want to reproduce them from scratch, and that will be a fun problem <sam_>that might involve bringing back another stage to catalyst (which is fine) <sam_>but i think a first good step is recording that metadata and including it either in the archive or alongside (like the sigs are now) <mid-kid>I'm not sure recording metadata would help a whole lot in reproducing it... It's slightly too chaotic of a process to be practical in my opinion. <sam_>1. provenance is useful nonetheless <sam_>2. i don't agree, given the schedule i mentioned, and the possibility of tweaking that <sam_>but we can shrug and not bother doing anything if you prefer, of course <sam_>i'm, in general, well-placed to talk about: a) what we do, and b) what we can change (including me supporting such changes, and maybe proposing them) <sam_>i'm not saying the status quo is perfect, but i dispute that any of it is unfixable <sam_>it's very easy to be nihilistic, it just gets us nowhere <mid-kid>I have nothing to add, I agree that it's not unfixable, just commenting that the status quo is more complicated than was mentioned... (And I'm not sure if even practical to fully solve) <mid-kid>It'd require changing the way releng works <ztrawhcse>Googulator: I'd also like to just add that the pandoc bootstrap issue is easily avoidable by simply doing the reasonable, sensible, sane thing and avoiding pandoc in all software ever. Don't use it, don't hook it up in package builds, scold any upstream project that adds pandoc to their `./configure && make` <ztrawhcse>it's not even like pandoc produces good manpages, since it doesn't in fact produce good manpages <ludocode>Hello bootstrappers! A while back I mentioned my bootstrapping C compiler Onramp. I just hit a major milestone: it can finally compile Doom. <aggi>a problem i see with gentoo bootstrapping with the tinycc-route i proposed: python itself is blocked indefinitely <aggi>this is not gentoo-specific, but it's gentoo depending on python for packaging like no other system i knew <aggi>an alternative portage-utility exists, from exherbo iirc, but that's written in c++ hence not applicable with the no-c++/tinycc approach either <aggi>the original *BSD (netbsd pkgsrc and openbsd variant) tools that inspired gentoo were written in C instead <aggi>currently, i continued dependency-tracking and cross-compilation with gentoo-tooling on buildhost, but cannot emit a self-hosting stage-bundle anylonger, because i cannot retain python and gentoo tools for buildtarget <aggi>and, although i got a complete system-profile for tinycc/no-c++ with gentoo, it is almost impossible to merge that back <aggi>of those ~500ebuilds that i kept in maintaince, almost all of them needed changes inside a hotfix overlay <aggi>many important ebuilds/versions vanished from the main portage tree and had to be pulled back in from gentoo archive <aggi>finally, i can fully fork the tree, since there's almost nothing remaining that got pulled from the main portage repository for no-c++/tinycc support <aggi>i will however continue maintaining this giant fork, separated from and in parallel to a complete re-write of packaging to converge this with bootstrappable <aggi>another difference, i emitted stagle-bundles/ISOs with crossdev, instead of gentoo-catalyst tooling <aggi>hence i had to sanitize several dozen ebuilds for crossdev <aggi>cdrdtools were fun ones, perl, and of cause python... it's a miracle this thing didn't fall apart yet <aggi>since it's various variants covered for cross-compilation, gcc, tinycc, linux-5.9, linux-2.4, musl-1.1, musl-1.2, libressl etc etc <aggi>full static-linking support <aggi>although i can emit a minified stage3 package set, it too contains bigger ones such as mplayer, gdb, mutt <aggi>currently i can crossdev target towards aarch64, aarch32, amd64, x86, while tinycc/linux24 is covers x86 "only" <aggi>without qemu/binfmt btw. since cross-compilation got sanitized <aggi>to be precise, i can crossdev to any different ISA and keep gentoo-tooling, except for the the static/tinycc/linux24 variant, which is the most important one however <aggi>and i think complete alternative toolchain support had belonged into gentoo, for several reasons, including bootstrapping-related ones <oriansj>ludocode: congrats on your work, hopefully it was lots of fun ^_^ <tmg1|michelson>and now figured out how to get jamvm to run(wasn't obvious), next step: ant-bootstrap i think <stikonas>Googulator: in principle you can bootstrap haskell and other languages using qemu-x86/qemu-x86_64 <stikonas>Googulator: and yes, I don't have haskell on my gentoo system <stikonas>one could even avoid ruby if you want, but it turns out bootstrappping ruby is not too bad... <stikonas>tmg1|michelson: can't you just reuse existing bootstrap scripts for jamvm/ant? <lanodan>Oooh onramp looks like what I'd want to build to replace using tcc as a blob, nice! <stikonas>tmg1|michelson: but I have all ebuilds there from C++ to java 21 <stikonas>but I guess it's easier to extract distro independent bootstrap scripting out of gentoo ebuilds than guix packages <aggi>can't refrain, SATA and SMP both work now... when compiling/linking kernel with tinycc the order of linking objects is critical <aggi>i'm not happy with the timer subsystem yet, clock_monotonic, time64, among other things that linux-2.4 didn't have <aggi>with regards to kernel-bootstrapping, if tccboot loader (jit) were considered, smp remains a hindrance, since trampoline.S needs 16bit asm processing which tcc fails with <aggi>yet absence of smp would not block kernel-bootstrap, and with an AoT compiled kernel smp does work now it seems <aggi>md-raid and nfs too seem affected by linking order issues that panic smpboot, it's not urgent and can be debugged on demand <aggi>iptables is working which failed to initialize before, ethernet bugs still with autonegotiating link-speed, i'll get to this next <ludocode>tmg1|michelson: Onramp is only on GitHub at the moment. I don't have any plans to host it elsewhere. <ludocode>lanodan: Nice, what project is that for? It doesn't quite build tcc yet but that's what I'm working on next. <ludocode>lanodan: Neat. The other thing about Onramp is that it doesn't produce native code. It produces bytecode for its VM. So if you wanted to produce native binaries you'd still need tcc (or another compiler), but you could include only the source for that compiler and bootstrap it with Onramp. <stikonas>I wonder if onramp would work quicker in bridging gap between M2-Planet and tcc than mes/mescc <lanodan>ludocode: Yeah that's what I gathered, also might be mistaken but seems like core/build.sh requires utilities like echo, mkdir, … ? Which well would make sense to not need prior to onramp. <lanodan>I guess a stub version of them could be build in onramp VM bytecode, or in the case of echo as a sh.oe builtin. <ludocode>It's only like 20% done so far but that's the goal. <ludocode>The shell won't depend on any external tools to run those commands. When it runs an external tool it's prefixed with "onrampvm" <ludocode>So when running in Onramp shell, "onrampvm" is a command that loads the given program into the same VM. And when running in a POSIX shell, "onrampvm" actually runs a new VM. 