IRC channel logs
2024-02-11.log
back to list of logs
<stikonas>deesix: but why does it matter what was committed before <stikonas>if you need some older commit, git will just check out earlier local commits <deesix>Which might point to an older submodule that needs to download. <stikonas>and it's not like we need to go to earlier commits that much... <deesix>I don't think so, if is not a submodule "now". <stikonas>but git clone will have the whole repo of M2libc <stikonas>and if you do git checkout of older commit of stage0-posix <stikonas>I don't think it will redownloade anything at all <stikonas>it will be just taken from stage0-posix/.git/modules/M2lic <stikonas>you can even have multiple remotes there <stikonas>only if you checkout older commit, manually cd into submodule and do git fetch <stikonas>then you might download stuff from the old forge <deesix>I was thinking somethig like this: commit1 has submodule forge1/foo. That submodule is removed later (let's say commit100). At that moment the submodule is forge2/bar. A while later, I clone recursive. Does it get forge1/foo just in case I checkout a commit < 100? <stikonas>no, I don't think it will get forge1/foo <deesix>Yeah, I'm checking it out (say... for bisect or whatever). <stikonas>which is why you need to git init submodules again if new submodule if added <oriansj>deesix: there is no need to replace already commited URLs in history, any more than we would need to replace references to freenode in our histories. <stikonas>but I think it is even less of a problem than freenode <deesix>Well, oriansj; then one can get stuff from the "old forge" kind of unexpectedly (which makes the move not really complete). <oriansj>deesix: well the old forge will just be an updated mirror of the master forge <deesix>I'm not worried about the content, but the server serving it (the move out). <oriansj>so, it is trivial for anyone to detect bad behavior on old forges <oriansj>(assuming they are not breaking git tags too) <oriansj>and if they are breaking git tags, we have a much bigger problem than old forges acting badly. <oriansj>(hmmm looks like I will need to do a sys/socket.h if I want a minimal wget/curl in mescc-tools-extra) <stikonas>but is that even useful in mescc-tools-extrea <stikonas>we are unlikely to have any kernel support for networking so early... <stikonas>one could of course bypass the kernel and use userspace tcp stack too <stikonas>wrap.c is also only useful for posix bootstrap <stikonas>anyway, I'm not opposing in any case to minimal wget/curl <stikonas>just saying that it's hard to get meaning ful network in kernel bootstrapping case until we are quite far in the bootstrap chain <oriansj>and at which point there isn't any benefit to having M2-Planet compile it <stikonas>well, perhaps there might be some usecase <stikonas>UEFI could be any implementation (EDK2/u-boot or anything else) <oriansj>we can shelve the idea until someone comes up with an actual use case. As gcc can definitely handle wget/curl and those should be good enough to download any files we wish to remotely pull <stikonas>well, downloading stuff early in UEFI might be insteresting <stikonas>but I haven't looked at all into networking there <stikonas>but unless I get some help with UEFI bootstrapping, I won't have time to look into this any time soon <stikonas>and also how safe is it to run networking in ring 0... <stikonas>hmm, on the other hand Linux kernel also handles some parts of networking in ring0 <oriansj>stikonas: well that is a good question. <stikonas>anyway, I think small downloader might have some uses after all <oriansj>and atleast in bare metal making true memory ring buffers is trivial and mean buffer overflow attacks become impossible. <stikonas>but integrating it all nicely is probably a lot of work <oriansj>well I will take a look at sys/socket.h; see what we need to support for a trivial download program <oriansj>and after we get it working in stage0-posix we can then figure out what is needed in UEFI. <stikonas>first for UEFI I need to figure out why cp bin/mes-m2 bin/mes works on QEMU but gets stuck on my real hw machine... <quassel>I am interested in bootstrapping RPM-based distros (e.g. Fedora and RHEL-Clones). <quassel>I hope to use GUIX or live-bootstrap as a starting point and use parts of the procedure that Fedora used to bootstrap new arches. <quassel> Is someone of you aware of prior work in this area? <stikonas>quassel: no, I don't think there were any attempts to bootstrap RPM based distros yet <stikonas>some people are just starting to look at bootstrapping source based distros, e.g. Gentoo <stikonas>and for non-distro stuff, freedesktop-sdk is now bootstrapped off live-bootstrap <stikonas>well, fossy already merged some inprovements that should make bootstrapping gentoo a bit easier <stikonas>all that shadow / user support stuff is not merged <janneke>wondering if that could help to remove the dependency on tcc <stikonas>oh, cp in UEFI fails because it relies on getcwd call failing on non-directory input <stikonas>need to add a bit more logic to M2libc then... <stikonas>tried running mes's kaem.run and "cp bin/mes-m2 bin/mes" resolves to copying bin/mes-m2 to bin/mes/mes-m2 ... <janneke>afaics wile can be compiled without boem-gc, and it will prolly just eat all your memory <AlwinB>Thank you stikonas, for your answere. <janneke>it would be pretty interesting to see a full source bootstrap coming to fedora <janneke>AlwinB: any idea if the concept of reproducibility has landed in fedora yet? <AlwinB>janneke: My cursory search only turned up pages and repos that seemed to be abandoned for years. <janneke>fwiw, suse (courtesy Bernhard M. Wiedemann) is the only rpm-based distro i've seen contributing to reproducible builds <AlwinB>I am very new to this, but i would like to take a chance on enterprise linux. <janneke>linux isn't bootstrappable yet, we focus mainly on gnu/linux <AlwinB>I am surprised to hear that. When looking at GUIX and live-bootstrap, I was under the impression that you already made it somewhat possible. <janneke>no, most of us are not looking at the kernel for now <janneke>but yeah, linux will probably have to be addressed at some time <euleritian>There was recently a discussion about whether to use github or not. Here's one more aspect: github is blocked in mainland China (https://en.m.wikipedia.org/wiki/List_of_websites_blocked_in_mainland_China). I've been told that guix has a very active community in China, and I can imagine that there is a great interest in bootstrapping, too. Maybe, switching away from github enables support from the other half of the globe. <janneke>euleritian: yeah, github has been advised against by gnu even before the microsoft takeover <janneke>i've never understood why someone would choose to use it, but then again, i still don't understand how most people use their computers :) <janneke>note that nothing in guix's gnu/packages/commencement.scm references github <mihi>janneke, in my opinion GitHub shines when I'm *not* using my computer (i.e. I'm either using my phone or someone else's computer). The search, outline, xref features etc. in the code browser are just not comparable to other sites (most of them usable without logging in), and when logged in, you can fork, edit, run Actions, and do pull requests without being at your computer. And if the functionaltiy is not enough <mihi>Also, the same factor why so many people are on Facebook (where I am not) - everybody else is already there :D <janneke>mihi: that could be, i try not to use non-free software, however shiny it may be <mihi>janneke, I accept that decision. I my ecosystem there is lots of non-free software (and non-free websites), while I try to avoid them, I only do if I either don't lose much functionality or if I feel it in my purse (i.e. the proprietary software is expensive) <oriansj>mihi: the only reason the master repos are on github is because savannah rejected hosting them <oriansj>5 minutes after savannah would approve, believe me when I say the master repos would all be moved. <oriansj>and all of the git submodules updated accordingly <oriansj>I am using an x200, running gnu guix. (my configuration is readily available) given the option of doing the right thing, I will do it. <oriansj>but I don't have any method of getting them to change their mind and unless someone else does and is willing to help me; I am stuck between a set of less than optimal choices. <janneke>i've been in a tedious process to try and change the guidelines for packages, that should make it clear that such things are OK, but there's been no tangible progress for over a year <deesix>github or savannah is a false dichotomy. <stikonas>janneke (cc AlwinB): why do you say linux is not bootstrappable? <stikonas>at least in some cases it is already done <stikonas>on x86 with BIOS systems we could use rickmaster's buidler-hex and Mikaku's Fiwix <janneke>what i possibly should have said is that guix and live-bootstrap don't include linux yet <stikonas>but it's hard to see how one can integrate dthat <stikonas>for kernel bootstrapping you can't really start with package manager <stikonas>janneke: that said builder-hex0 does not work on some modern systems <stikonas>as those are starting to lose BIOS emulation layer <stikonas>e.g. my laptop (with coreboot) only supports UEFI api but not BIOS api <stikonas>but at some point stage0-uefi should be able work there and replace builder-hex0 <stikonas>but at least for now it needs x86_64 bootstrap path <janneke>oh my, so many things need to come together... <Googulator>stikonas: does't coreboot let you use SeaBIOS to get a BIOS API? <stikonas>but it's a good opportunity to advance UEFI bootstrap too <stikonas>it does not include SeaBIOS out of box anyway <Googulator>The problem with Savannah (besides refusing to host our projects) is no PR support <Googulator>moving to Savannah means moving to an email-based workflow, which would be a huge step back <Googulator>I wish there was some project hosting site offering Kallithea <Googulator>can do almost everything GH can (forks, pull requests, you name it - not sure about code search), sometimes even better (e.g. GH can't properly visualize a merged history as a DAG), and can even do Mercurial besides Git - but unfortunately you have to host it yourself, or at least I'm not aware of a "KallitheaHub" <stikonas>out of self-hosted forges I think gitlab and gitea seem to be the most common <stikonas>(gitlab is quite resource hungry and harder to maintain, although probably has more features) <Googulator>IIRC free-as-in-beer for open-source projects, but paid otherwise <stikonas>Googulator: no, I think they follow free sofware core model (possibly they have some proprietary extensions on enterprise version and the one that they run) <stikonas>and I think parabola is FSF approved distro <stikonas>at some point Debian was also shipping it but removed it due to vastly different release cycles of gitlab and debia <stikonas>but it's a big program written with ruby on rails I think <stikonas>gitea/forgjo on the other hand is a smaller program in Go <Googulator>I seem to remember there are CCP/Chinese government concerns around Gitea <stikonas>I think originally it was forked from gogs which has originated from China... <stikonas>oriansj: I think I have a fairly simple fix for cp problem <Googulator>doesn't Sourcehut have the same problem (only supporting e-mail workflows)? <oriansj>Googulator: I am good with email and irc; <oriansj>irc is good if you need to talk and have something addressed *now* and email is better for refection and deep discussion. <Googulator>I think many of us wouldn't be happy to move from a PR-based workflow to one similar to the Linux kernel <oriansj>Googulator: well that is the thing about here. Who does decides. What ever works best for the developers doing the work is what will be done. <Googulator>Email alone isn't the problem, but rather the requirement to use low-level SMTP <Googulator>Linux kernel-style workflows are really unfriendly to GUI users <oriansj>I don't expect git patch files; just simply saying my repo is here: ${URL} pull tag ${tag_name} <oriansj>and I do git add remote ${username} ${url} <oriansj>git fetch to get your updates; git lola to figure out what it is to review and merge <oriansj>and git merge when I like what you have done. <Googulator>and unlike the Linux convention of patches in the body of emails, it preserves Git commit identity <Googulator>which is my other big gripe with the Linux workflow <Googulator>is Sourcehut read-accessible for non-contributors? <Googulator>oh, so it's come a long way since the Audacity/Tenacity split <stikonas>anyway, looks like the main alternatives are sourcehut / gitea / gitlab (and each of these can be self-hosted or one can use one of the public instances) <stikonas>but self-hosting is probably not easy... <stikonas>if you keep registration open, you'll immediately get lots of spam <oriansj>well, we can use the lobste.rs user account model if we self-host. <oriansj>but I don't yet see a need for us to do so. <oriansj>everyone gets to do what they prefer; git remotes works around a great deal of those coordination problems and we need only agree on where the master repos should be located and if anything goes wrong the people actively working can collectively decide on the solution. <Googulator>very similar to Kallithea (in fact, Kallithea is a fork of RhodeCode), but unlike Kallithea, it has its own hosting option <Googulator>tl;dr: it's only free to use for now; will become paid later, even for open-source projects <oriansj>Oh, I have been paying to support development of AGPLv3 software <oriansj>just like I would have been willing to pay for savannah usage if it was an option. <oriansj>just like how I am willing to pay domain registration fees and VPS monthly costs to enable effective cooperation. (I just lack time to play administrator and am more than happy to give others the option to do so) <oriansj>I am certain a $50/Month VPS is plenty powerful to cover all of our collectively CI workflows and I am willing to pay for it. I just lack the time to setup and administrate it. <oriansj>I have the bootstrapping.world domain registered for the next decade for any thing we might want to setup infrastructure for. (only wiki.bootstrapping.world is kind of setup) <oriansj>(as a hedge against our main wiki getting lost) <stikonas>ok, with cp fixed, I think all of stage0-posix (except for kaem-optional-seed) and mes is working on UEFI (at least on my system, others might encounter some other issues as is common with new software) <stikonas>the next step is probably fix posix bootstrap on amd64... <oriansj>I was most excited about mes working on UEFI <stikonas>last year when I started stage0-uefi, I thought we'll have to port mes libc to uefi <Googulator>stikonas: "fix posix bootstrap on amd64" - as in the known issue of mescc apparently miscompiling tcc? <stikonas>but now with a fairly simple program (posix-runner) we seem to be able to reuse M2libc's support for UEFI without needing to do new work <stikonas>Googulator: yes, it is miscompiling tcc on amd64 right now <stikonas>also reproducible on normal posix bootstrap <stikonas>well, we fixed it sufficiently for tcc-mes to build <stikonas>Googulator: basically same issue right now if you try to build live-bootstrap with --arch amd64 <Googulator>last time I checked, tcc-mes couldn't even properly parse its command line <Googulator>it just printed what it should print with --version, then went into an infinite malloc loop <stikonas>and mes-wip branch works quite a bit better <stikonas>Googulator: you would still be missing some fixes even with 0.26 <stikonas>one could also go to individual c files rather than unified-libc <stikonas>then some files build, some cause crash with a few different errors <stikonas>the debug info that blood-elf emits is not super helpful for complex programs <stikonas>so adding debug prints is probably the way to go <oriansj>otherwise you get stripped binaries with no labels at all <oriansj>M1 and hex2 don't generate anything for gdb or objdump to leverage <Googulator>M2-Planet yes, but I wasn't aware mescc also uses M1 as its backend <oriansj>why do you think mescc outputs M1 assembly? <oriansj>mescc-tools was orginally written for mescc <oriansj>and eliminated whole classes of bugs for mescc (as range checking and the like produced easy to debug messages) <oriansj>compile, assemble, link -> binary is far easier to reason about and debug than compile -> binary <oriansj>nope, it spawns M1, blood-elf and hex2 to create that illusion. <stikonas>it even hides things a bit better than mesoplanet <stikonas>and supports things like -llibc+tcc to add libc+tcc.s and libc+tcc.a in M1 and hex2 commnad lines <stikonas>Googulator: you probably mixed it up with tcc <stikonas>it's tcc that produces straight binary code <stikonas>tcc debut output is a bit better, but at least for bootstrappable tcc is also not fully working <janneke>fwiw, until v0.7 mescc also wrote binary code directly <janneke>around that time, oriansj convinced me to write hex2 and/or hex3 and later mescc switched to M1 <Googulator>looks like we have a bit of a memory regression with Fiwix 1.5.0 <Googulator>so they impinge upon userspace-accessible memory <Googulator>memory: total=2097152KB, user=431908KB, kernel=1664724KB, reserved=520KB <Googulator>which explains why unxz was going OOM even after I fixed the memory leaks <Googulator>requiring additional memory usage reduction beyond just leak fixing <Googulator>by contrast, with 1.4.0-lb3, we had memory: total=1048576KB, user=705420KB, kernel=342756KB, reserved=408KB <Googulator>Mikaku: also, it appears both versions allocate the kexec space within the 1 or 2 gigabytes of "normal" memory, but at least 1.4.0 clearly tolerates initrd being outside that range <Googulator>so maybe it would be worth allocating the kexec ramdisk preferably outside "normal" memory