<oriansj>for example if the types out of box were: uint32_t, int32_t, uint16_t, int16_t, uint8_t, int8_t, float64_t, float32_t, float16_t with REGISTER_INTEGER/REGISTER_FLOAT available but hidden so that it is only used in the very special case performance libraries and kernels which need it. <Hagfish>oriansj: yeah, the constraints of making the instructions printable basically force all programs to be obfuscated <Hagfish>in fact, in a sense, it forces programs to be underhanded too, because it deliberately blurs the boundary be code/data/comments/strings <oriansj>also int 0x80 is: CD80 and syscall is: 0F05 so not possible to do anything on POSIX without non-ascii printable chars <Hagfish>unless you remove the "hard mode" constraint right? self-modifying code means you can (eventually) run whatever you want, i'm guessing <oriansj>Hagfish: well, yeah once you go self-modifying code throw everything you can assume about the code straight into the garbage. <oriansj>never trust self-modifying even if you wrote it yourself. <Hagfish>in fact, i'd say that anyone who can write self-modifying code is extremely sus (but that may be because i've never tried) <oriansj>well it was a skill required for fitting major programs in 1Kw of RAM <doras>stikonas: how do you suggest I approach updating mes? I want to test your MR that puts symlinks inside the tar archives to confirm it doesn't introduce any issues with the bwrap bootstrap, but the SUID issue keeps me from updating to the tip of master. <doras>mes doesn't have an official tarball for this commit, and we can't apply patches this early in the bootstrap (if I'm not mistaken). <stikonas>doras: so I think you need to 1. switch to new download url in sysa.py (git snapshot of required commit), 2. Update value of MES_PKG in sysa/after.kaem <stikonas>and then run a few times until you get passed initial kaem stages (each time updating checksums) <doras>stikonas: why does it require multiple bootstraps? Can't I disable the early checksum checks and update them manually later? <stikonas>doras: no, you can't disable early checksums right now <stikonas>kaem does have conditionals now (if, elif, fi) <stikonas>(at least not until you build coreutils) <stikonas>we do have match command in stage0-posix to compare to strings <stikonas>but then you need to get them into kaem script <stikonas>and right now kaem does not support sourcing <stikonas>doras: you can try to "emulate sourcing with catm" <stikonas>catm new_script.kaem bootstrap.cfg old_script.kaem <doras>stikonas: can the early sha256sum create checksum files, or only compare? <doras>Or maybe it's only relevant to stage0-posix internally. <doras>I think I'll just comment out the checksum command for now so the bootstrap doesn't fail, and then create the new hashes when the bootstrap is done. <stikonas>doras: you won't be able to create new hashes later <stikonas>almost all of the early binaries are overwritten <stikonas>with new versions or compiled against musl <doras>So I'll try to hardcode the checksum creation instead of the checksum compare, and maybe make it a conditional afterwards. <doras>stikonas: do you think we need to list all files individually like the commit above did? <doras>Sounds like using * may not be deterministic. <stikonas>that * from stage0-posix is from outside bash <stikonas>also redirects and command capture requires more syscalls <stikonas>doras: but it's still not completely clear what to do <stikonas>you don't have access to redirects in kaem <stikonas>so can't do sha256sum file > file.checksum <stikonas>you can extend sha256sum to have an argument to write to file <doras>What good is there in calculating a hash without being able to write it to a file? :) <doras>Unless I'm not looking in the right place. ***stikonas_ is now known as stikonas
<stikonas>yes, looks like it although ti was not documented in sha256sum --help <doras>stikonas: in the qemu-based bootstrap, are all the files created during the sysa kaem scripts stage kept only in RAM? Are they accessible in any way from the host? <fossy>doras: looking good, thx for PR <fossy>stikonas: good find with the symlinks <stikonas>yes, everything is much simpler now without .links packages <stikonas>there was a lot of hacky scripting around it <fossy>rickmasters, oriansj: IMO, we can keep builder-hex0 with a pretty small disk size (504MB more than sufficient), and worry about large disk size in 2nd-gen kernel in M2/tcc <fossy>unfortunately tcc cannot build linux kernel, due to assembly <stikonas>fossy: actually, what about tcc + binutils? <stikonas>mes is a bit too heavy to run on builder-hex0 <fossy>yeah, builder-hex0 is not sustainable past M2, really <fossy>(this is not a bad thing, we want to keep the hard-to-read hex0 code simple) <fossy>im actually uncertain about tcc + binutils, that may work <rickmasters>fossy, oriansj: at 504MB I think I can rearrange the kernel memory to support 1GB of memory files, .5G for source and executables, plus .5G for a memory disk image <fossy>but if we can get binutils we can probably run gcc.. gcc isn't very "syscall-intensive" <rickmasters>a memory disk image has the advantage of already supporting lseek <fossy>thats an absolute ton of space, rickmasters, i can't see that much being used under builder-hex0 <stikonas>fossy: why do we have two different linux tarballs? Do we need newer linux-headers? <fossy>not strictly required -- we can lock the versions together <rickmasters>fossy: i agree is very likely overkill, and the tradeoff being more hardware requirements, but just saying it would be easy <stikonas>that could reduce sysa download requirements by a lot <doras>stikonas: then I think I'll move the new checksums to sysc. Otherwise they can't be extracted. <stikonas>doras: if you want, but they can be extracted in chroot mode <stikonas>though other checksums are not available in chroot mode <doras>Running only two bootstraps is still better. I'll stop thebootstrap after the kaem stage completes, update the hashes, then continue with a qemu bootstrap. <doras>But yeah, I can also just move the checksum files to sysc. <doras>Though they all use the same name at the moment, so they can't be put in the same directory. <doras>A test run shows that the checksums are calculated and can be written to files correctly. <doras>fossy, stikonas, do you mind if I rename the checksum files to ${pkg}.checksum or anything along these lines? <doras>I'm not sure if I'll have any PR of any of this in the end, but I'm wondering about the general concept. Basically use the ${pkg}.checksum name for the checksum files, and when --update-checksums is used, we create and write the checksum files to /usr/src. This would allow them to end up in sysc's /usr/src and be extracted easily. <stikonas>doras: I'm fine with ${pkg} checksum, especially if they are auto-updated <stikonas>hopefully we don't need revisions so early in the bootstrap <stikonas>bash thingy uses ${pkg}_${rev} but we can't do this with kaem <doras>Having a unified file containing all checksums would definitely be better though, but we'll probably need additional functionality from stage0-posix to accomplish this. <doras>Maybe an --ignore-missing option for sha256sum. <doras>stikonas: so is catm our only viable option? It conflicts with the *.kaem script hashbang. <doras>Also sometimes called shebang :) <stikonas>you run catm in the previous kaem script... <doras>I mean, each ${pkg}.kaem file starts with #!/bin/sh. If we cat it with the bootstrap.cfg, the shebang would not be at the top of the file. <stikonas>you can do that with some higher level kaem file <doras>Maybe I don't understand what catm does. Is it not simply "cat"? <stikonas>it's catm output file input file1 [input file2...] <stikonas>shebang only matters if we invoke it via program loader <doras>So you're saying I can safely remove it? <stikonas>well, we have it at the top of the files for other reasons <doras>Or ignore its existence in the middle of the file? <stikonas>you'll only have its existance in the middle of the file at run-time <doras>But in general, catm is our only viable option, right? <stikonas>well, without much coding I think that's the only option <doras>But if it's the only way, I'll go with it. <stikonas>otherwise you'll have to implement source'ing <stikonas>e.g. move most of the stuff in after.kaem to run.kaem <doras>Ah, that's indeed a better idea. <doras>stikonas: do all "normal" variables considered environment variables by kaem? <doras>It looks like kaem is fine with executing scripts that don't have the executable bit set. <doras>I'll set it anyway in case this ever changes. <doras>I'm ~80% done, but I'll need to continue later. <doras>It ended up being a nice improvement <stikonas>executable bit matters only if you run something via ./script <stikonas>when you run it via ./script then it goes to kernel and kernel checks executable bit <stikonas>in all other cases you run executable interpreter and it simply opens the file <janneke>oh my, there is no bootstrap-seed for arm yet! <stikonas>janneke: no, we only have higher level prototypes for arm in stage0-posix <stikonas>there is aarch64 in stage0-posix but we don't have aarch64 in mes... *janneke goes to try an aarch64/arm melange for now ***lukedashjr is now known as luke-jr
<doras>stikonas: when is the first time we have "install" available? Only after coreutils? <doras>Do we want to get a newer GCC and other useful packages in live-bootstrap? <doras>We'll need to close the gap between live-bootstrap and freedesktop-sdk somewhere. The delta shouldn't be very large if I recall correctly. <doras>I think it would be nice to do it in the scope of live-bootstrap, so its final product ends up being generally useful for bootstrapping a full OS. <doras>There are a few full OS images built from freedesktop-sdk already. <stikonas[m]>Yes, we should add the packages that are useful for most distros <stikonas[m]>well, newer GCC/binutils and python will bring us much closer <doras>I'm not even sure yet which packages we actually need. <stikonas>we have most of the stuff that linux from scratch begins with <doras>I think it's basically these packages: <doras>Though it's hard to tell the exact minimum version we need of each package. Some of freedesktop-sdk's current seed packages may be newer than they have to be. <doras>One can find the package version that freedesktop-sdk's master branch currently uses for its bootstrap here: <doras>Basically packages from the current stable release (release/21.08) are used as the seed for bootstrapping the master branch. <stikonas>well, pre-bootstrap seems to be fully covered <doras>stikonas: "pre-bootstrap" is basically the last version's "bootstrap" + additional packages. So are you saying the additional packages (listed under `components`) are fully covered, or that also "bootstrap" packages (listed under bootstrap) are covered? <stikonas>the packages from bootstrap.bst are only partially covered <doras>stikonas: I see. The second link I mentioned (bootstrap.bst) is what stands behind the "bootstrap-import.bst" row. <stikonas>well, we don't have everything there but some of the stuff should be easy to build when you have gcc/binutils/bison/flex and autotools <doras>stikonas: have you checked versions as well, or only packages in general? <stikonas>but in general newer versions should be fairly easy to build <stikonas>even live-bootstrap have some new stuff, e.g. bzip, musl, coreutils, autotools, perl are all fairly new <doras>I think we need newer a version of GCC and probably other packages, but it's not clear exactly which version is the minimum we need. <stikonas>well, gcc and binutils would be good to update <doras>I think it would be fine for live-bootstrap to remain musl-based if this is the project's preference, as long as it can provide all the dependencies for building glibc in freedesktop-sdk. <stikonas>doras: it was not a preference but just happened because of simpler build system <stikonas>and then building old glibc sounded a bit of downgrade... <stikonas>but yes, we should provide all dependencies for building glibc <stikonas>hence I mentioned that we should build python <stikonas>but if possible I would leave python to fossy as he already looked a bit at it <doras>Ideally live-bootstrap would become the new `pre-bootstrap.bst`, and then freedesktop-sdk can use it as the seed for building the latest versions of everything listed in `bootstrap.bst` <Hagfish>that would be such a vote of confidence, and it makes perfect sense