<stikonas>fossy: when you have time #120 and #122 can have some reviews. Also I'm looking at texinfo and gnulib stuff there is weird <stikonas>it's not a submodule like in other git repos... <stikonas>well, once I can identify revision, I guess it will be the same <vagrantc>oriansj: it looks good on the surface, but i find one of the main people pushing it has a history of burning bridges and burying projects after they get funded <vagrantc>i feel a bit awkward saying it so bluntly, but frankly, it has happened so many times i feel like anything less would almost be dishonest <oriansj>vagrantc: thank you for that information that I was not aware of. <vagrantc>it's hard, because they promote good ideas on important issues, and projects i really *want* to see succeed ... but i eventually told myself "never again will i get my hopes up" <vagrantc>maybe there are enough people on this one that it might work out... <oriansj>vagrantc: all success depends on the people and some people shouldn't be trusted based on their previous behavior alone. <Hagfish>the only thing that looks odd to me is there various different gnulib snapshots being downloaded <Hagfish>i'm not sure if the reasons for those snapshots is explained anywhere <oriansj>there, now it is no longer possible for M2-Planet to show wrong line numbers <oriansj>now on every byte read, it will update line numbers if '\n' is found. <oriansj>I should probably add a proper sha256sum to mescc-tools-extras <oriansj>38KB isn't half bad for a static binary of sha256sum <oriansj>build with M2-Planet --architecture ${ARCH} \ <oriansj>So a little ugly and needs a pinch of cleanup but it works and behaves just like standard sha256sum (minus a slightly more helpful message when files don't have matching checksums) <mihi>oriansj, why is the output filename of your sha256sum utility "sha3sum"? SHA-3 is actually a different algorithm than sha256 (which is the 256-bit variant of SHA-2) <fossy>holy crap stikonas[m] that was fast <stikonas>but yes, our runs are getting larger and longer <stikonas>even if we switch to chroot/no tmpfs, this will only give us a bit of headroom <fossy>on my computer we are approaching 50 mins i think <fossy>i'm working on kexec at a later stage (just after gcc), so then we can do a sysb transfer and hopefully cache sysa <siraben>Do we have a Docker image that would allow people on macOS to run it as well? <fossy>that should speed it up a bit <fossy>docker image for live-bootstrap? no <siraben>Or I think the QEMU should work, I haven't tested the live-bootstrap yet <stikonas>siraben: no, but Docker image would be nice to have <stikonas>basically just chroot with some extra isolation <fossy>I mean we don't currently have one. But, not sure in the long term a Docker image is something that should be entertained as a "production" type idea <fossy>like i wouldn't recomment using chroot to someone who isn't a developer <stikonas>well, yes, chroot is just nice to get files in/out <siraben>Oh I wasn't thinking of it like that, more like, getting it able to work in more environments <stikonas>also, if it works, maybe rootless podman is better than docker <fossy>"getting it able to work in more environments" is an anti-design feature <fossy>live-bootstrap is made to run idealistically for one single set of software <stikonas>well, I was thinking it might be good for CI... <fossy>the only current changing factor we really have is the kernel <fossy>hm, yeah that's proabbly where docker could come in handy <siraben>like I'm running on macOS, I would have to use QEMU to get live-bootstrap to work right? <fossy>how does one run docker on macos anyways <stikonas>not sure how docker works or doesn't on macos <siraben>for blynn-compiler what I did was forward the Nix build to a Docker container running NixOS <qyliss>they have a proprietary program that runs a VM for you <siraben>Oh the Docker app is proprietary? aw <qyliss>so only features that can cross VM boundaries work -- you can't e.g. share a socket with a container <fossy>not that that matters for live-bootstrap though <siraben>fossy: I guess, with Hyper V framework instead of emulation <fossy>anyway the idealistic target platform for live-bootstrap is bare metal <fossy>so apart from CI usecases which is a good enough point for implementation I don't see all that much usecase in docker/etc <fossy>but i would still like it cause CI <fossy>but CI is just getting longer and longer, stikonas is right <stikonas>and I think it might get much longer soon... <stikonas>we probably can do something like this next <stikonas>libunistring->guile->autogen->gcc (maybe newer binutils before that, possibly glibc) <stikonas>I think guile is probably slow to compile, gcc too... <oriansj>mihi: yes the output program name was wrong (it was a copy and paste from a sha-3 implementation for M2-Planet (that never quite worked right)) <oriansj>isn't docker mostly just a wrapper around qemu? <oriansj>in which case docker support is just a few generate and spawn functions and that is it. <amirouche>not specific to docker per se, but they were created to enable lightweight "vm" such as docker. <amirouche>on macos, IIUC, it rely on qemu to emulate linux, then docker is run inside that full vm <oriansj>amirouche: So the only meaningful difference is on Linux where the kernel is that of the host system. <stikonas>yeah, on linux it's basically chroot + some namespaces (e.g. process namespace) <amirouche>I am not sure I understand. And I forgot about the name of the syscalls used by docker, and lxc/lxd. lightweight vm such as docker, will not emulate anything, it is somekind of sandbox everything it has access to is given to it by the kernel. <stikonas>yes, so docker uses pid namespace (for process isolation), net namespace(managing network interfaces), ipc namepsace, mount namespace and some uts namespace which I'm not fully sure about <stikonas>I think podman on top that uses user namespaces that allows it to run without root unlike docker <amirouche>podman runs without a daemon started by pid 1, unlike docker. <amirouche>docker can be used by a user inside the correct group. <stikonas>but having no daemon also allows running without root <amirouche>I believe docker runs with a daemon is an historical artifact, because before now, they were no way to access the syscalls as a regular user. <oriansj>so a docker image for live-bootstrap wouldn't be a freedom issue (even guix produces docker images) but rather one requiring someone to choose to put in the effort required. <stikonas>I would think docker image for live-bootstrap is just development aid, just like chroot <stikonas>as end-user if you want to do bootstrap, you run it on real hardware instead <stikonas>as developer, it might be easier to develop on chroot or inside container <stikonas>and distros also usually build either in chroot or inside container <stikonas>anyway, it's probably not too much of an effort to produce docker image <stikonas>basically we already have contents of what should be inside docker image (same thing as elsewhere, e.g. inside initramfs) <stikonas>just need to add a tiny bit of metadata to specify what process we want to start <amirouche>guix produce docker image and can use the linux syscall itself. but the problem (imo) of docker, is that it is fake sandbox, it is not meant to enable new features for the endusers, it was meant to do "like chroot a little less broken".. <amirouche>being able to run docker images is somewhat a freedom issue imo. <amirouche>guix can not run docker images without docker. <oriansj>and from what I understand the Windows binaries are also proprietary. <amirouche>i mean, you can but you need to re-implement docker. yes. that is my minimalist pov (sorry for the noise) <oriansj>So I don't disagree that there are freedome issues for Windows and MacOSX users of docker. <oriansj>Hence my suggestion that isn't something we should feel that we should do but it isn't also something we should reject if someone shows up and is willing to contribute that work. <siraben>I brought this up because I wanted to make it more accessible to more developers, even if we here choose not to use docker <siraben>Currently what are the dependencies to run live-bootstrap? <oriansj>siraben: literally nothing except a posix kernel <oriansj>siraben: only needed if you wish to virtualize the execution <oriansj>but the chroot isn't actually required either. <oriansj>as all the steps can be done outside of total isolation but they are done in isolation to prevent your development environment to contaminate the build results. <Hagfish>stikonas: thanks for refreshing my memory about gnulib. am i right in thinking that patch and gettext are using 3 different versions of gnulib? <siraben>Where can I get the kernel that's requried? vmlinuz <Hagfish>stikonas: err, actually, i think sysa.py is missing a comment on line 498? <oriansj>siraben: vmlinuz will work but so should any reasonable posix kernel. <siraben>oriansj: right, but I was wondering where to get it <siraben>hm, I'll check for NixOS, /boot doesn't exist <siraben>pinging Melg8 as well, I think they would know how to get it to work on NixOS <Melg8>that's example how i used qemu with 4.4 kernel " nix-shell -I nixpkgs=channel:nixpkgs-unstable -p python3 python38Packages.requests qemu linuxPackages_4_4.kernel" -just using nix shell - <Melg8>"sudo nix-shell -I nixpkgs=channel:nixpkgs-unstable -p python3 python38Packages.requests qemu linuxPackages_4_4.kernel --run 'python3 ./rootfs.py --kernel /nix/store/kg5wa8hww2g6zc03yas7llbzrbip4b5b-linux-4.4.267/bzImage --force_timestamps --tmpdir ./temp'" <Melg8>that with execution of command <Melg8>siraben is it what you asked about? <oriansj>hmm. looks like the sha256sum doesn't work quite right on 64bit targets. guess I need to fix that <stikonas>Hagfish: yes, everything uses its own version of gnulib <stikonas>the version that maintainers used when they made releases <stikonas>Hagfish: thanks for spotting missing comment <stikonas>siraben: well, for preparing initial initramfs we use python as well <stikonas>but for running live-bootstrap, kernel is the only dependency <nimaje>why does rootfs.py use sudo to find the chroot binary? and why does it shell out to python for that instead of just run('which' 'chroot') or using that python code directly or just ignoring the absoulte path of chroot and simply calling it? <nimaje>(ok, it needs the absolute path because PATH gets set to just /bin) <siraben>`sudo: unable to stat /etc/sudoers: No such file or directory` hmm <siraben>I'm trying to run Melg8's commands in a docker image running NixOS <Melg8>if you run there as root already maybe sudo is not needed? <siraben>Ok got past that by adding a sudoers and adding sudo to my shell <siraben>new error: `sudo: unable to initialize PAM: Critical error - immediate abort` <siraben>nimaje: yeah, it using sudo is a problem when sudo is not installed <amirouche>I mean those you speak too are not interested in your project. <nimaje>well, I complain about it using sudo where it don't need it <Melg8[m]>sorry xD i was trying to find original post with "forget" in it through element.io) and instead posted this word)) <stikonas>nimaje: because some distros put chroot in sbin <stikonas>so if you don't use sudo to find chroot, it might not be in PATH <stikonas>if you are already root, just remove that sudo prefix form rootfs.py <stikonas>(probably also from the place where tmpfs is mounted) <stikonas>nimaje: or you can write a patch to only search with sudo if search without sudo does not find anything <Hagfish>stikonas: i'm glad my review was helpful. i wish i could suggest more important fixes, but you keep writing such good code, it doesn't leave much room for improvement :) <Hagfish>how did you discover what the correct versions of gnulib were for each step? <Hagfish>they're not even sequentially ordered (at least in terms of filenames) <nimaje>ok, live-bootsrap chroot mode only works if the kernel can execute linux binaries <stikonas>Hagfish: usually its just going to git repo and checking what version gnulib git submodule is checked out <stikonas>in case of texinfo, there was no gnulib submodule, so I just found commit by date <stikonas>nimaje: not just chroot, also for qemu, kernel has to understand elf binaries <stikonas>for other kernels, you would have to replace elf header with approprate other header <Hagfish>stikonas: i guess it wouldn't make sense to try to generate those version "numbers" programmatically? it would make the numbers less magic, but be more indirect/fragile, right? <nimaje>I trought qemu runs some other kernel, so it doesn't matter what the host is as long as the kernel executed by qemu can execute linux binaries <stikonas>those version numbers are just commit hashes <stikonas>so I don't think we want to generate them <stikonas>nimaje: yes, qemu runs some other kernel indeed, but I mean if you supply it with non-linux kernel <stikonas>it would be the same as chroot on non-linux kernel <Hagfish>stikonas: i've never really been a fan of non-sequential version numbering systems, because it feels like it lacks critical information, but i guess all version numbers are arbitrary in some sense <Hagfish>if i see a dependency on, say, gcc v7.x, where i'm expecting gcc v3.x, then i know something is wrong, but if i see gnulib a8b93e instead of 9b3d1a then that's not human readable <Hagfish>that's not your fault, of course, it's just something that gnulib makes harder <nimaje>why didn't they use some release of gnulib? or does gnulib not do releases? <stikonas>well, nothing you can do, that's how git works. You can't assign stable version numbers to git <stikonas>you are meant to include some modules from gnulib in your tarball <nimaje>and how do you debug mes in live-bootsrap (chroot mode on freebsd with linuxemu stuff loaded)? only getting "Subprocess error 139" isn't really helpful (seems like it tries to compile tcc) <stikonas>hmm, is that when process exits with non-zero exit code? <stikonas>you can always try to use gdb for debugging <stikonas>although, by default compiled binaries have no debug symbols <stikonas>if it's segfault, then I guess try to add debug flags and use gdb <stikonas>I don't think anybody tried live-bootstrap on freebsd, maybe m2planet only <nimaje>(ah, yes lldb on freebsd can't really debug i386 linux binaries :(; trying to run the program step for step segfaults lldb), but I think the normal run is okish, and that says signal SIGSEGV: invalid address (fault address: 0x4749464e) <oriansj>Hagfish: The reason why the ELF headers are all hand written and not generated is because different architectures interpret the ELF spec differently. For example PowerPC64LE instead of putting the e_entry address in the e_entry field, it puts the address where the e_entry address can be found in the binary (In complete and total violation of the ELF standard) because that is how AIX does it. <oriansj>The asme reason why binutils spends 800+ lines of code just dealing with ELF headers. <oriansj>and HP-UX and Digital Unix and Xenix. <oriansj>fuckit let us just blame every Unix that was sold. <oriansj>The BSDs and Linux for allowing that stupidity infect them. <Hagfish>oriansj: interesting, thank you, but i wasn't asking about ELF headers <Hagfish>if you have thoughts about how to make version numbers less opaque (especially when they are just commit hashes), i'd be interested to hear about that too <Hagfish>to some extent, the code is saying "these are the versions we found, and they work", which is self-documenting <stikonas>well, you can always fork gnulib and add tags... <Hagfish>hmm, that's an interesting alternative <Hagfish>or have a script-generated file which fills out a lookup table of tags and commits? <Hagfish>that's really over-engineered for something that isn't really a problem, but it's helpful for me to think about <Hagfish>any process that tries to add back the missing semantic information is necessarily going to add some indirection, and i don't think there is a way to reduce the complexity of that misdirection sufficiently to justify the information gain <Hagfish>actually, the gettext-0.21.tar.xz tarball contains a gnulib-local directory, whose most recently touched file is from 2020-07-26, which is the same date as commit 7daa86f of gnulib <Hagfish>that's a bit of a fuzzy process, and probably wasn't how that version was found, but it's interesting that there is this trail in the artefacts themselves <Hagfish>stikonas: i probably trust your human judgement more than a script, but if you felt that the process you used was time consuming or error-prone, you could consider writing a script next time <Hagfish>after all, why repeat a 3 minute task 5 times, when you could spend an hour writing a script to do it for you? :D <mihi>I can also use repeated use of cat if you prefer that (test vector is 1 million letter "a") <mihi>(that branch is WIP, will have to verify that I did not break M2-Planet build first, on a machine/VM where M2-Planet compiled binaries can run) <stikonas>well, there is nothing bad with perl in general <mihi>perl is infamous for being a write-only language. But for one-liners I think it is ok. e.g. "perl -e 'print chr for 32..127'" to get all printable ASCII chars. <xentrac>yeah, perl is great for that kind of thing <mihi>I also like that you can redefine string delimiters. Useful for pasting some text and having no need to escape. But I understand that if you don't know that feature, it will make the code really unreadable... <stikonas>well, a few months ago without knowing any perl I still managed to rewrite one perl program in awk (for live-bootstrap). And I think it was shorter than perl version <mihi>xentrac, true. Both for computers and for humans. <xentrac>dunno, I think s/<(\w+).*?>/[\1]/g is a lot more readable than the Python equivalent ***Noisytoot is now known as [[
***[[ is now known as Noisytoot
<mihi>xentrac, perl -pe 's{<(\w+).*?>}([\1])g' still? <xentrac>mihi: I didn't know that was legal but the intent is clear <mihi>perl -MO=Deparse will give it away, though <xentrac>well, it's clear enough how it works <xentrac>but I did run it to figure out what it actually output <mihi>then your perl knowledge is better than most people. alias y/ for tr/ seems to be quite obscure in my experience. <mihi>also the q.a...q.z. confuses many (redefining quote character to . is evil) <xentrac>y/// is undesirable when you're writing code for readability, but for an interactive shell oneliner it's just a minor shortcut <mihi>once I submitted a perl oneliner (perl -e '$_=$ARGV[0];1while s/\(\)//;print+(/./?"Not ":"")."OK".$/;') for a computer science exercise. I'm curious if you can spot what problem it solved. <drakonis>this looks like it checks for accidental colon placement? <mihi>xentrac wins :) more particular, check whether the input string consists only of correctly nested parentheses. <xentrac>though I think the string-replacement mechanism it's using is actually Turing-complete, so using it to recognize a context-free language is maybe overkill (though I don't know of a better option in Perl 5) <xentrac>I mean of course it is Turing-complete if you have multiple pattern strings and corresponding replacements <mihi>in particular, it is not the most efficient way, but it was definitely the shortest submission <xentrac>correctly nested parentheses and newlines outside any parentheses, to be more exact <xentrac>if you rule out extraneous characters such as "\n" and "0" then you could save yourself a stroke with $_? instead of /./? <xentrac>I've been toying with a compact PEG syntax that would represent that language as <s ;'('<s>')'> <mihi>does that include expressions like '(()(())())'? <xentrac><foo> is the nonterminal foo, <foo bar> is the nonterminal foo which is defined as bar, and foo;bar is one or more foos separated by bars, so foo+ becomes foo; and foo* becomes ;foo <xentrac>and concatenation and quotes work as you'd expect <xentrac>so s is defined as zero or more parenthesis pairs each containing s <mihi>even more cryptic than Perl :) <mihi>but yes I've been guilty of having significant whitespace in some of my toy scripting languages too <mihi>probably you could replace <foo bar> by <foo=bar> or <foo:=bar> to make it more clear <xentrac>yeah, it doesn't have significant whitespace except in quotes; < s; '(' < s > ')' > means the same thing <xentrac>I did think about the = thing but decide that raised the line noise level even more <xentrac>actually no I didn't, or at least I don't have notes of having done so