IRC channel logs
2023-12-14.log
back to list of logs
<oriansj>Guest25: well I am guessing your mescc-tools hex2 did something wrong; as it should result in file permissions of 0750 <oriansj>if it is unable to do so, it should atleast warn you with: Unable to change permissions\n <oriansj>cosinusoidally: definitely something worth exploring <stikonas>oriansj: well, with fossy's work on autogenerating live-bootstrap build scripts it might be easier to support multiple branches in the bootstrap <stikonas>it's much cleaner than gazillion of handwritten kaem and bash scripts with those build steps <oriansj>looks like a incremental apporach to declarative functional package management <oriansj>makes me wonder if we should resurrect the blynn-compiler work and do a proper package manager in the haskell subset it supports <oriansj>bringing a little gentoo USE flags and guix into the bootstrap <matrix_bridge><Andrius Štikonas> Might be a bit overkill so early in the bootstrap <Googulator>If it can, we get xz support as soon as we get mes running <Googulator>(at the cost of having to run Scheme code early in the bootstrap, which I'm not too fond of, given the extreme memory intensity almost rivalling AI...) <oriansj>Googulator: not yet if I remember correctly. But there is nothing stopping unxz from being added to mescc-tools-extra <oriansj>which already has unbz2, ungz and untar <oriansj>it'll probably be a week or two of work <Googulator>depends on the difficulty of writing an LZMA decompressor in m2 / mescc / tcc-compatible C, vs writing it in Mes-compatible Scheme <oriansj>well you can use an existing LZMA decompressor written in C and then just incrementally remove usages of C features that are not yet supported. <oriansj>(or depending on the usefulness of the C feature; I could add support for that feature in M2-Planet) <Googulator>although we do bring up xz before g++, so I guess it must have a pure-C implementation of it <Googulator>but IIRC Igor Pavlov's original implementation is in C++ <oriansj>and we can probably find many more in pure C <muurkha>but maybe that doesn't matter if you're running a nailed-down set of xz files through a nailed-down version of xz <oriansj>muurkha: completely fair and there is nothing stopping us from a minimal xz spec to support a handful of files and later expanding to meet growing bootstrap needs for xz <oriansj>stikonas: and I'll just be fixing those M2libc formatting details <oriansj>and the use of unsigned long isn't going to work in M2-Planet but I can just replace that with SCM and it'll work <Googulator>I've been looking at closing the "HTTP gap" - turns out, we're downloading 27 files totalling 166MiB over plain HTTP between the 1st and 2nd builds of curl, of which 68MiB is nss which is only downloaded for certdata.txt <Googulator>still too much to just shove into srcfs and expect to stay below 256MiB <Googulator>Recompressing it all as xz pushes it to 58MiB - combined with bz2 recompression of everything that comes before network, we might just fit the 256MiB limit <oriansj>stikonas: we might want to think about another M2libc release soon <Googulator>ok, recompression alone won't help, we end up with a 285MiB srcfs <Googulator>with unxz.c or other early XZ decompression, we can get down to 215MiB <Googulator>of course, I cheated a bit here and used solid compression - in practice, we could only recompress individual tarballs <Googulator>but I would still be much happier if we could postpone steps to after HTTPS is available <stikonas>Googulator: slightly different question: remember we discussed it makes sense to do first posix kernel in UEFI to be builder-hex0.efi or something like that. But can we easily do that, probably posix binaries that we use, will want to use specific memory addresses <stikonas>and we might not be able to get those blocks from UEFI... <stikonas>at least stage0-posix has all but kaem-optional.hex0 relocatable <stikonas>not sure what happens in mes and tcc though <stikonas>also it might be hard to implement brk :( <Googulator>UEFI itself provides no means for paging, right? <stikonas>well, in UEFI you can request memory pages, but I don't think you can configure MMU there <stikonas>we could e.g. request 100 MiB or so of continuous memory <stikonas>but even that might not be enough for mes <stikonas>and mes doesn't request all in one go... <Googulator>what do we use brk for? It's deprecated since 1997 at least. <stikonas>Googulator: M2libc and meslibc generally requests memory via brk <stikonas>well, for stage0-posix it really is the simplest way <Googulator>notably POSIX.1-2001 doesn't even consider brk and sbrk to be standard system calls anymore <Googulator>POSIX.1-1998 is the last one to mention them, specifically to declare them deprecated and warn that a compliant system may not even provide a real implementation - the only requirement is that it must error out if not actually implemented <Googulator>luckily that includes builder-hex0, Fiwix and Linux <stikonas>well, perhaps we need to add mmap and munmap <stikonas>well, for stage0-uefi we had a solution there <Googulator>Does mes require one large contiguous allocation? <stikonas>oh, actually we might not be using brk directly there <stikonas>so I just checked M2libc/uefi defines brk to return -1 <Googulator>If mes can tolerate e.g. 10 blocks of 100MiB each, that should be doable using UEFI memory allocations <stikonas>I imagine mes will call calloc gazillion of times with some small allocations <stikonas>but posix ELF binaries right now have calloc -> brk <Googulator>I'm hoping nothing actually relies on brk's contiguous extension behavior <Googulator>calloc certainly comes with no guarantee that what you get will be contiguous with a previous allocation <stikonas>no, but calloc itself will have to be improved <stikonas>is that if brk limit is reached, call brk to get more memory and continue from the earlier block <stikonas>so perhaps we just need to improve mes/lib/linux/malloc.c <stikonas>though I first need to fix uefi bootstrap binaries to work :( <stikonas>somehow pushing rbp to stack at the beginning of the program breaks my test in QEMU, but without that it doesn't work on real HW... <stikonas>probably something more complicated is going on... <Googulator>Also note: per POSIX, " These interfaces need not be reentrant. " <Googulator>As in, you're supposed to call brk or sbrk at most once in a process's lifetime, even according to the older standard that still defines them at all, <stikonas>well, both M2libc and mes is abusing posix then... <stikonas>but how were you supposed to get more memory then? <Googulator>You were supposed to know your memory needs in advance. <stikonas>man brk definitely doesn't talk about reentrancy <Googulator>also, POSIX considers the actual interface used by libc's implementation of malloc to be an implementation detail <muurkha>ACTION sips coffee, watching the flames <Googulator>Some parts are clearly outdated (e.g. we don't have make 3.80 anymore, and no mention of Fiwix) <matrix_bridge><Andrius Štikonas> Googulator: yeah, its an old snapshot of parts.rst <matrix_bridge><Andrius Štikonas> Ideally link to parts.rst rather than old snapshot <Googulator>Why do we even build coreutils 6.10 immediately after 5.0? <Googulator>coreutils 6.10 would be great to get rid of, as it's almost 9MiB <Googulator>interestingly, both 5.0 and 8.32 are waaaay smaller <Googulator>Do we really need another sha256sum at this stage? <Googulator>Won't the bootstrap one suffice until we build coreutils 8.32? <deesix>Googulator, you mean "call brk or sbrk at most once in a process's lifetime" because "need not be reentrant"? <deesix>IFAIU, reentrant is about calls from multiple threads, not about the complete lifetime. <Googulator>that sounds more like thread safety than reentrancy <Googulator>IIRC in a threaded context, reentrancy is specifically about calling the same code twice _from the same thread_ <muurkha>at least that's not how I've understood it <muurkha>the way I understand the term, code that isn't re-entrant at all can't be made thread-safe (except by wrapping it in a mutex to ensure that only one thread can ever be in it) <muurkha>but multiple threads isn't the only way you can hit re-entrancy problems <muurkha>recursion and interrupts are two others <muurkha>but code that isn't re-entrant at all can still be called safely many times from the same thread, or in a single-threaded process <muurkha>as long as it returns before each new call! <muurkha>the issue with brk or sbrk not being re-entrant in traditional (non-threaded!) Unix is that malloc commonly calls sbrk, and it's easy to accidentally write an interrupt handler that calls malloc <muurkha> https://en.wikipedia.org/wiki/Reentrancy_(computing) says, "A computer program or subroutine is called reentrant if multiple invocations can safely run concurrently on multiple processors, or if on a single-processor system its execution can be interrupted and a new execution of it can be safely started (it can be "re-entered"). The interruption could be caused by an internal action such as a jump <muurkha>or call, or by an external action such as an interrupt or signal, unlike recursion, where new invocations can only be caused by internal call." <muurkha>the article has handy examples of things that are neither thread-safe nor reentrant, thread-safe but not reentrant (which I said was impossible), and both reentrant and thread-safe <muurkha>so I guess it agrees with Googulator on the definition <deesix>I handwave-y "basically" because, one way or the other... nothing seems related to the whole "lifetime" and not be able to call brk more than once, which sounds very weird to me. <Googulator>I wonder what's the term for a procedure that can only be called once in a context, because it doesn't clean up after itself <Googulator>I always thought that's what "reentrancy" meant, not merely interrupt safety <Googulator>"Bison 3.4.1 is buggy and segfaults when perl 5.32.1 is built. This is probably because it was built with a hand-written makefile. We do not build the latest bison because perl 5.32.1 requires bison <= 3.4.2." <Googulator>That sounds like the difference is in how it's built, not the actual version. Why use both then? <Googulator>Rather than just rebuilding 3.4.1 with proper build system the 2nd time, or using 3.4.2 with the special makefile. <Googulator>Such a situation can commonly emerge when converting a standalone command line program into a library, or when adding a GUI or other interactive mode - and I've seen bugs related to this called reentrancy issues. <matrix_bridge><Andrius Štikonas> Googulator: bootstrap sha256sum has bugs with large files <Googulator>I guess we need to keep it before binutils/linux then <Googulator>Any idea why we download coreutils 8.32 from savannah-git (much smaller tarball, presumably without pregen files), but 5.0 and 6.10 from ftp.gnu.org? <Googulator>also note: git.savannah.gnu.org/cgit/coreutils.git/snapshot/coreutils-8.32.tar.bz2 works (we currently use gz here) <matrix_bridge><Andrius Štikonas> We should be able to switch to xz snapshot. <matrix_bridge><Andrius Štikonas> And I think we do snapshots here because release tarballs did not have enough info to regenerate pregenerated files <matrix_bridge><Andrius Štikonas> Googulator: perhaps we could extend coreutils 6.10 and remove 5.0 <fossy>stikonas, i'm just performing final tests, should be good to go with the PRs in the next 24hrs <fossy>i'll be merging in this order <fossy>#337 #340 #341 #343 #344, rebase #334 on master, merge #334 <fossy>if you're ok with that i'll proceed within a few days <fossy>oh and i need to push the arch specific checksums ofr simple-patch <fossy>mes 0.26 is looking very neat! <fossy>does gash run natively on mes 0.26 now <matrix_bridge><Andrius Štikonas> fossy: but with gash we will have to be careful <matrix_bridge><Andrius Štikonas> Even if we need to adjust meslibc build script a lot to are those new syscalls <janneke>fossy: for gash you need the wip-modular-mes branch <janneke>and from gash-utils you need the wip-mes branch <stikonas>we probably don't need to much from gash-utils anyway <stikonas>but having more powerful shell (than kaem) early on could be a big improvement <stikonas>on the other hand, it might make bootstrap simpler... <stikonas>so I guess there is a tradeoff between speed and simplicity... <janneke>stikonas: thinking (again) about how slow the bootstrap on riscv is... <janneke>i remember emilytrau mentioning they built mes C files all in parallel <stikonas>if the bottleneck is indeed RAM bandwidth <janneke>not if you have (only) one machine; right <stikonas>I was building gcc 4.6 from ekaitz branch on riscv <stikonas>and even 3 stage bootstrap is just 2h or so <janneke>so it's really mes that is exceptionally slow <stikonas>good is that other parts are reasonably fast <stikonas>but to reach them, you need a week or so on visionfive2 <stikonas>well, we don't yet have anything in the chain after tcc, we went directly to building gcc for now <stikonas>first to make sure we have a working GCC <stikonas>janneke: I wonder how much can be gained by switching those long if/else sequences to switch in mes.c <stikonas>wouldn't be much faster, but maybe can speed it up by a bit <oriansj>well, there are lots of potential improvements to explore and even more bugs to fix. (probably need to figure out why mescc-tools-extra sha256sum is giving wrong results and at what size) <oriansj>just checked a 5,296,912byte file and it matches the core utils sha256sum result <oriansj>I know it will break on files greater than 2GB due to the signed 32bit value <oriansj>and 1.5GB can result in a segment fault <oriansj>and checking a 472,700,928byte binary file I do see matching checksums from the x86 stage0-posix bootstrapped sha256sum <oriansj>which should be large enough for everything for a good while <stikonas>hmm, maybe whatever bug we had is fixed... <janneke>stikonas: iwbn if a fairly simple change as if/else => switch would improve performance <stikonas>well, if/else probably has far worse assembly <janneke>...and eval_apply is the most intensively executed code <janneke>so yeah, it might even be a happy suprise...hard to say! <janneke>just when everything seems so nice and stable ;-P <stikonas>well, oriansj already implemented switch there <janneke>oh, so "we"'re just a bunch of lazy laggards :) <janneke>(for the record, this has been one of my most intensive years, mes-wise, and i've been amazed by all your work too!) <GoogulatorMobile>I've just realized there's another neat perk to reducing srcfs to 256MiB, which is unrelated to the trusted Flash drive <GoogulatorMobile>I've been running with kexec_size=256MiB for a while with no ill effects <GoogulatorMobile>Which means, we can avoid the "nuke distfiles" step during the Linux build <GoogulatorMobile>& basically bring over an almost arbitrary amount of source code from the Fiwix world into the Linux one <stikonas>GoogulatorMobile: well, we could save on some stuff, coreutils is one thing... <stikonas>though we probably want to bring in gash <stikonas>and also give some space for future growth of some projects (stage0/mes) <GoogulatorMobile>Tbh, I'm not a fan of gash... or anything that would use the mes interpreter, for that matter <stikonas>yes, I know, but we need to solve that non-redistributable binary issue <stikonas>so at the very least I would like to run musl's ./configure script before bash <stikonas>but if it's not compatible with bison bootstrap, we might need both <stikonas>GoogulatorMobile: are you against because it is slow? <GoogulatorMobile>Yeah, I don't want to see m(olass)es in the bootstrap path for that reason <stikonas>well, configure script wouldn't take long.... <GoogulatorMobile>I have a feeling a performant Scheme implementation is not possible on our current CPU architectures <GoogulatorMobile>I mean, Scheme is a Lisp dialect... there's a reason why a custom architecture was originally designed for running Lisp <stikonas>as for sizes, perhaps we'll have to switch to more configurable bootstraps <stikonas>e.g. there would be different manifest files for different usecase <stikonas>that's becoming easier with that fossy's rework <stikonas>e.g. if we want to target gcc 4.6 rather than gcc 4.0 for riscv, we could do it separately <stikonas>half of that binutils size is gas directory <stikonas>perhaps you could write a postprocessing script <stikonas>that optionally repacks tarballs and injects new checksums <stikonas>though some people might prefer not to run repacking <GoogulatorMobile>slim binutils + early XZ support + coreutils improvements + pushing GMP & co to when it's actually needed = easy 256MiB, even if we include everything needed for HTTPS <GoogulatorMobile>We can safely use HTTP links if we so choose, but at no stage should we depend on the existence of HTTP links that don't redirect to HTTPS <GoogulatorMobile>Also, some ISPs or corporate environments might straight up block plain HTTP traffic <stikonas>GoogulatorMobile: you have bigger problems there in corporate environemnts <stikonas>you are unlikely to be able to run live-bootstrap there <GoogulatorMobile>I would expect plain HTTP blocking to become the norm in the near future <stikonas>doesn't apt download stuff from debian using http? <stikonas>though it's from the mirrors that are under Debian's control <fossy>janneke: thanks for the gash branches. i'll investigate them at some point (but i think in live-bootstrap we should probably wait for them to stabilise a bit) <muurkha>current Chromium does block unencrypted HTTP, as does HTTP/3 <janus>it seems to redirect to another page, even if i use incognito. surely it should fail before redirecting if your assertion were true <muurkha>janus: maybe you've already told chromium it's okay? <muurkha>or maybe you're not using the version of chromium where i've seen this behavior? <muurkha>also both chromium and firefox have dropped ftp support <janus>muurkha: that's why i try incognito, so that it doesn't know about my settings <janus>i am on chrome 120.0.6099.71-1 <janus>maybe google patched my chromium before packaging chrome <stikonas>yeah, http works on my ungoogled-chromium too