IRC channel logs
2022-05-08.log
back to list of logs
<oriansj>achaninja: well samplet was working on getting mes to add proper scheme module support to mes.c (which I believe they finished) and after that finish getting gash working in mes.c I however don't have more details than that. <stikonas>achaninja: you can find some updates in the logs (maybe a week ago) <stikonas>probably slow too but that's less important <stikonas>fossy: oh and I think I've sorted that autom4te issue <stikonas>I was comparing old and new filesystems and the difference is /usr/bin/sh -> /usr/bin/bash symlink which is missing. I can create it manually... <stikonas>so broken pipes were somewhat of a red herring. I guess perl executes simple system("...") calls directly but more complicated are outsourced to system shell <markjenkinssksp>I believe my attempt to go through Nystrom's Crafting Interpreters with a compiler in primitive scheme and bytecode interpreter in M2-planet is now Turing complete now that I have if-else, while, and logical and/or <markjenkinssksp>My updated README.md reflects where I'm going to take it to have a bootstrappble Lox sub-variant, which will then be enough to be a language I can write further compilers and tools in <markjenkinssksp>before I go on to chapter 24, I should do more testing, including porting relevant parts of Nystrom's test suite <muurkha>if you have if-else, true, and false, you don't really need and/or <muurkha>and in Scheme, where you have recursion with tail-call elimination, you don't need while either <markjenkinssksp>indeed muukha, though I am trying to strike a balance on having enough practical features before I start using my own tool <muurkha>yeah, in Ur-Scheme there are an inconvenient number of places where I say things like (define (after-wsp s) (after-wsp-2 s (s))) <markjenkinssksp>No tail recursion in Nystrom's Lox or in my plans for it, once I have enough Lox I'll be writing C/JS style inperitive code... (sort of, because I'm going to have a pair() type that I'll for data structures) <markjenkinssksp>so my bslox (bootstrap-sub-variant) will be kind of a strange C/JS/lisp hybrid <markjenkinssksp>My logical AND and OR work was pretty easy today, so it was low cost to add with good payoff in terms of what I'll have <muurkha>Clojure makes do with something called "recur" <muurkha>which restarts the current function with new arguments IIRC <muurkha>I wonder if you can factor out the string-append that generates the label into a label-generating function <markjenkinssksp>the experience of forcing myself to use a relatively primative sub-set of scheme? <markjenkinssksp>am enjoying, though getting sick of the restricted scheme, but the good news is that I've never gone deep into into scheme to use things like module systems, macros, co-routines, and continuations before, so I'm not missing out of that stuff <markjenkinssksp>I did take a bit of a break for awhile when I found myself crashing the C/M2-Planet side of this project, but valgrind helped put me back on the right path <markjenkinssksp>plus, I haven't had much opportunity to get myself into memory errors in the C since as most of the work I've done has been in the compiler writen in scheme and the nice thing about a pure bytecode interpreter (with no compiler present) is that it's relatively minimal C, a good thing to say when working in M2-Planet C in particular <muurkha>not implementing define-record-type or defstruct was one of the decisions in Ur-Scheme I regretted <markjenkinssksp>My bytecode interpeter patches are pretty small at this point, plus Nystrom goes out of his way to minimize the number of opcodes, OR was a good example where he notes in the book that he could have added one (OP_JUMP_IF_TRUE perhaps), but he showed off by showing the OP_JUMP and OP_JUMP_IF_FALSE from the if/else part of the chapter were sufficient <muurkha>something similar happened in the early history of C, where at some point they added structs and the compiler got smaller <muurkha>yeah, that's a useful example to see <markjenkinssksp>I can't miss SRFI-9 records as I don't have experience with them, I'm pretty in experienced with scheme, it's pretty much this project and a very much work in progress transpiler project that I was working on (not published) ans shelved <muurkha>SRFI-9 records are basically C structs <markjenkinssksp>I wouldn't say my first compiler, I did take an intro to compilers univeristy course circa 2004-2005, which was mostly on tokenizing and parsing <muurkha>yeah, that unfortunately happens a lot with intro to compilers courses <markjenkinssksp>I also did some hobby (unpublished stuff where I was playing around transpiling a basic scheme into python using the structed python language gen stuff in python (not just doing a str output) <markjenkinssksp>and in one sense, this isn't truely me doing a compiler from scratch because I'm just working through Nystrom's book and taking his algorithms and data structures, making some feature/design simplifications, and doing the work in different language variants that his C <markjenkinssksp>so it's going to be very much a immitation is a sincere form of flattery kind of thing in the end, especially if I make a tool here that's useful for bootstrapping other things <muurkha>I think that's maybe the best way to do things <markjenkinssksp>I'm glad I dont' even know about various SRFI's like 9 because I'm keeping myself in some pretty simple scheme here, and so my attempts to minimally bootstrap that later will go nicely <markjenkinssksp>If the scheme I'm using isn't already compatible with the slow-lisp variant of mes in M2-planet or the M2-planet converted Mes, there's probably not much to do to make ti so <markjenkinssksp>but those scheme implementations asside, I'm also interested in implementing the scheme I'm using by way of some portable alternatives to M2-planet, but that's a ways off <markjenkinssksp>Nystrom is definitely giving me a good learning experience which will hopefully make a better toolchain bootstrapper out of me in the end <muurkha>sleep well! nice to see your progress! <markjenkinssksp>came back to add one last thing, also on my TODO for soon before I implement more features is to check if my bytecode interpreter is compatible with the new M2-Planet release and to then update my git submodules :) Much congrats on that release. Glad it happened, as I pretty much always pin my sub-modules to release tags <stikonas>not having /usr/sbin -> /usr/bin symlink breaks xbps (it can't find pkg-config) <doras>Hi. Regarding live-bootstrap, do we happen to already have a way to bootstrap only sysa and only sysc in two different steps? <stikonas>doras: well, sysc is automatically launched <stikonas>but we are doing some reworking now which might make launching just sysc easier <stikonas>doras: so first of all it's becoming easier to run live-bootstrap without rootfs.py <doras>What would be the main benefit? Installing only what's actually needed by sysc? <stikonas>so we don't keep broken stuff like tcc or stage0-posix <stikonas>and also it reduces number of stuff that we move <doras>I like the benefit, but doesn't installing packages as part of sysc's bootstrap complicate things a little? <doras>I feel like it would be better to actually a new sysroot for sysc that only includes what it actually as part of sysa's/sysb's would keep the output of those bootstrap steps simple: a single directory structure. <stikonas>well, it's fairly simple to install them <doras>to actually install a new sysroot* <stikonas>basically just untaring .tar.bz2 archives <stikonas>so we need to copy bash, tar, bzip2 binaries (also patch and gzip but that's because we don't have packages for those in sysa, they are built before bash) <doras>I mean, I'd like to think of sysa/sysb/sysc as separate input->output steps. Each starts with a seed and sources as input (can be a large seed with sysb/sysc), and each ends with a sysroot that the next step can consume as its input. <doras>This is a very nice quality of the existing design, I think. <stikonas>well, packages are basically input seed then <stikonas>though we also copy in older unused packages (for inspection purposes and checksums) <doras>Right, you change both the output and input. <doras>So if I try to define the flaw regarding the existing design, it's that sysa's (and maybe also sysb's) output contains many unnecessary things. <stikonas>well, depends on what you mean by output <stikonas>in /usr, yes they have some unnecessary things <stikonas>well, it creates unnecessary packages (older versions) <doras>Usually the output of a step (say, sysa) is the input for the next step (say, sysc). <stikonas>but right now they are propagated to the end, so that we can run sha256sum * on all packages and produces sysa/SHA256SUM.pkg file with checksums of all packages <stikonas>well, sysa moves all binary tarballs (output of sysa) to /sysc/usr/src/repo which we then install in sysc (so I would say that this is input to sysc) <stikonas>or at least part of the input (plus a few crucial binaries for unpacking) <doras>I'll try to explain my thoughts better by describing an alternative solution: instead of unpacking the packages as part of sysc's bootstrap, we could unpack them into a new /usr directory at the end of sysa's and sysb's bootstrap. <stikonas>the downside is that it would end up in far bigger initramfs image in sysb or more stuff is moved to sysc <stikonas>since sysa must fit in 3 GiB of RAM for running in qemu/baremetal <doras>The benefit is that the final step in sysa actually creates a useful sysroot that one can chroot into without needing any bootstrap operation to do useful stuff in. <stikonas>you can just run sudo chroot /path/to/sysc /init <stikonas>well, yes, you need to do some bootstrap operation, but it's fairly quick <stikonas>I do see some benefit for not having bootstrap operation <doras>I'm referring to ${SYSC} for a moment at the end of sysa's bootstrap. What binaries does it contain other than bash, tar, bzip2, gzip and patch? Only those, right? <doras>I can't "ls", "cp" or anything useful if I chroot into that. <doras>I must install packages first. <doras>Right. This is the downside I see here. That the output of sysa (the ${SYSC} sysroot) is not generally useful without an additional package install step. <stikonas>well, I guess it's possible to move some of this code to sysa and sysb <doras>But with that said, it's also fine to say that we don't want sysa's output to be generally useful to anything other than bootstrapping the next sysx. <doras>I think this is an entirely valid decision to make. <doras>I agree, both approaches have upsides and downsides. I'm not even sure which I prefer anymore :) <doras>But I like the fact that sysa's output will be much smaller and without unnecessary packages. Very nice :) <doras>And this will happen either way, so no complaints from me :) <doras>I'm currently trying to rebase my bubblewrap branch on top of the latest changes to see if it still builds. <doras>Previously I was left with only one workaround that I needed a solution for before I could open PR to have it merged. <doras>I still need a solution for it. <stikonas>well, bubblewrap should work with either approach <doras>Of course. The output of sysa and how sysc consumes it is entirely an implementation detail of the two. <stikonas>and it should be possible to kick it off <stikonas>we had a couple of issues, without tmpfs (tar order was messed up and something with automake but I think those are fixed now) <doras>I actually worked on two projects in parallel at the time which had a shared portion: one was to get live-bootstrap building in a sandbox environment without root permissions (bubblewrap), and another was to build live-bootstrap entirely within BuildStream with 0 dependencies other than a kernel that can chroot+exec. <doras>With the first project, python and ./root.py are still used, but instead of doing "sudo", creating device nodes and other privileged commands, I set up a bubblewrap sandbox that already comes prepared with those. <stikonas>well, both should be a bit easier now than a few months ago when you last asked <doras>Right. I noticed you've done a lot of progress on being able to build with 0 dependencies. Very nice. <doras>I'll probably have a lot of work to adjust what I had to the new approach :) <doras>But it would mostly be to simplify things, which is nice. <doras>But I wanted to start with getting the bwrap build working so I can propose it for merge to live-bootstrap. Basically unprivileged/safe bootstrap inside a sandbox. It's should be a nice feature for the project, I think. <doras>But the commit I mentioned was the last "hack" I had remaining in my side-branch. <doras>So I wanted to ask you what you thought would be a good solution. My issue is that I can't create a tmpfs for "sysa" because it requires escalated permissions. <doras>So up until now I simply commented out this part and instead relied on a normal directory for the bootstrap of sysa. <stikonas>yeah, I guess running witout tmpfs is possibility <doras>But I wasn't sure if this was an acceptable approach. <doras>I may be able to go a long way to actually create a tmpfs for sysa's bootstrap, but it would probably require creating a "sandbox" that includes the entire host OS, so I could then create another sandbox inside it that is where we'd actually bootstrap sysa. <doras>I'm not even sure if it would work, but it sounds like a complicated solution for something that may be a non-issue. <doras>Actually, I just checked and I think it can work and is not very complicated. Though I'm not sure if it will always work because it technically wraps the entire host OS in a "sandbox" (basically chrooting into / with user namespaces enabled). <stikonas>it does seem to affect some things (the order in which tar packed files) but we workarounded that now <doras>Not for performance reasons? <doras>I'm leaning towards removing that "shared code" for creating a tmpfs from SysA's constructor (is that the right Python term?), and then do it only for the modes where it makes sense to do it (all the current modes), and then do something else for the bwrap mode: either work without tmpfs or come up with a way to create a tempfs using namespaces or something. <doras>Because the creation of the tmpfs for sysa is unconditional and too early for me at the moment. <stikonas>we also need to add download only mode for rootfs.py... <doras>Hmmm... I think I managed to create an unprivileged tmpfs using user and mount namespaces. <doras>Let's see if it's actually temporary... <doras>It's indeed temporary, but also sandboxed, so the host can't see into it ;D <doras>I'm not sure if this is actually useful, then. The host can't actually see or access the result of the build process. <doras>I do wish to have a persistent on-disk output of the bootstrap process. Basically the sysc sysroot. So if the entire process is done inside a sandbox in memory, it's not really useful. <doras>Hmmm... I think I'll analyze the file structure after the recent changes in live-bootstrap to determine if I can work around this somehow. Maybe bootstrap sysa in a tmpfs and then sysc outside it. <doras>I remember the chroot (and therefore also the bwrap) mode was bootstrapping sysc in sysa/tmp/sysc, which was a bit odd. <doras>In my case if sysa/tmp is a sandboxed, memory-only, host-inaccessible directory, no output can actually be extracted by the host from the bootstrap process. <doras>But I guess this issue partially applies at least to the chroot mode as well. The tmpfs in sysa/tmp disappears after the bootstrap completes and the user exits the shell. <doras>So it's mostly an issue of live-boostrap in general not providing any actual persistent output from the bootstrap process, at least for the chroot mode. <doras>If we'll solve it for the chroot mode, we'll also solve it for the bwrap/rootless mode automatically. <doras>I guess there's the "don't unmount the tmpfs" when I exit mode, which kind of works around that. <doras>"don't unmount the tmpfs when I exit" mode* <doras>But I this is not possible to do when creating a new mount namespace. Once I exit from it, the tmpfs is automatically unmounted. No way around it. <doras>Oddly enough it doesn't get unmounted, but rather becomes inaccessible and the allocated RAM enters limbo. Very odd. <stikonas>hmm odd. But at this stage I think you might be more familiar with what's going on there <stikonas>than others (and me in particular) here... <doras>stikonas: is it possible that the chroot-based bootstrap doesn't currently work in live-bootstrap? <doras>It seems that "kaem-optional-seed" fails almost immediately for me. <doras> +> ./bootstrap-seeds/POSIX/x86/kaem-optional-seed ./x86/mescc-tools-seed-kaem.kaem <stikonas>I usually run both chroot and qemu builds before pushing <doras>I mean: ./rootfs.py --chroot <doras>It's as if the seed itself fails. Hmmm... <doras>I don't seem to have an "./x86/mescc-tools-seed-kaem.kaem" anywhere... <doras>Where is that supposed to come from? <doras>I figured it out, it was a submodule issue.