<stikonas>fossy: why are we not installing bzip2 to DESTDIR? <stikonas>because bzip2 -dc "/usr/src/repo/${pkg}_${revision}.tar.bz2" | tar fails? <stikonas>maybe in that case it's better to go back to two-step process? <rkeene>Why would that fail ? bzip2 supports streaming decompression (rzip is the only thing I can think of that doesn't want to do that) <stikonas>rkeene: no idea... I just saw fossy's comment <rkeene>I assume you're not doing bzip2 -dc x.tar.bz2 | ( cd "${DESTDIR}" && tar -xf - ) ? <rkeene>You're just missing ${DESTDIR} in the linked line <rkeene>Nothing to do with the contents of any archive <stikonas>well, DESTDIR is missing there because of the comment\ <rkeene>You can replace the binary that's running as long as you don't open the file for writing but write a new file and rename(2) it into place <stikonas>but I'm aware that we can replace binaries. We do that all the time... bash, tar, etc... <rkeene>Yeah, copy to bzip2.new then rename to bzip2 <stikonas>I think install should just do the right thing <stikonas>install as in binary from coreutils and not makefile target <rkeene>It just says it copies it, which implies open()+splice/read+write, which will cause existing mapped segments to become mismatched <stikonas>yes, install indeed does open and read + writes... <doras>What? "install" is supposedly a safe command for replacing binary files. It's not at all like "mv", for example. <doras>As in, shouldn't affect the running binary, which should remain mapped to the original binary before the replacement. <doras>At least this is the observed effect of this command at least a few years back. <muurkha>yeah, it unlinks the target file before opening it O_WRONLY|O_CREAT|O_EXCL <muurkha>that's not "safe" in the sense that if your machine crashes or power fails at that time you can be left without the binary upon reboot <muurkha>but it's safe in the sense that it shouldn't segfault any running processes by overwriting their memory-mapped executables <doras>Yeah, it doesn't do atomic file replacement as far as I know. <doras>I'm not even that Linux provides a simple method through which atomic file replacement can be done. Especially not for multiple files at the same time. <doras>Maybe it can be implemented using a FUSE filesystem. <muurkha>rename does work for atomically replacing a single file <achaninja>haha its not common, maybe I got the name wrong, but im pretty sure its an api that lets you swap two files atomically <achaninja>Trying to find it... Hopefully im not misremembering :P <achaninja>as a flag, then you can atomically swap them <achaninja>Let me know if you try it, it's not finished, but gets as far as master tcc <achaninja>it trusts the host system a bit more in order to get some more flexibility <achaninja>and it has a bit of a smaller scope than live bootstrap <Hagfish>"Importantly you do NOT require an existing compiler." <Hagfish>yeah, highlighting that is really important <Hagfish>it's like a magician pulling a rabbit out of a hat :) <oriansj>achaninja: I look forward to seeing your progress. and you can leverage the stage0-posix-$ARCH repos if you wish to more deeply embed stage0-posix into your workflow (and remove the need for changing of directories entirely) <ekaitz>achaninja: interesting that you decided to go for cproc and qbe, they look promising <doras>Regarding live-bootstrap, do we usually ignore the /usr/src/repo/x86-repodata file for checksumming purposes? <doras>After running the bootstrap with the "--update-checksums" flag (and some modifications), I see changes in /usr/src/SHA256SUMS.pkgs (in the final System C), but are there any other checksum files that get modified that need to be updated in git? <doras>I haven't changed any sources, only commands during the bootstrap. <doras>I see we also have checksums in tmp/sysa/*/checksums, but they seem unchanged. <rickmasters>I stumbled upon stage0-posix a few months ago. very cool. <rickmasters>I wanted to join the fun, perhaps contribute in some way... <rickmasters>So I wrote a small 32-bit x86 posix kernel in hex0 that can be used to build x86 stage0-posix up to M2-Mesoplanet. <rickmasters>The 32 bit kernel can be built using an included 16-bit "mini" bootable (MBR) kernel which is only 384 bytes of code. <rickmasters>The mini kernel compiles the "full" kernel binary which is a 3K bootable PC disk image. <rickmasters>There are details in the README files on how it all works but happy to answer questions. ***lukedashjr is now known as luke-jr
<muurkha>rickmasters: holy crap, that sounds amazing! <stikonas>doras: SHA256SUMS.pkgs is the main one for now. There are also some /sysa/*-*/checksums files for individual checksums before bash helpers are available <Hagfish>rickmasters: you absolute legend! you'll fit in well here with these other incredibly talented people <stikonas>kernel is indeed something that we really need <doras>Thanks stikonas. It seems that our tar trick did not work. It seems "--mode=go=rX,u+rw,a-s" didn't have the expected effect. I think the permissions remained the same. <doras>I'm hitting a failure in perl-5.003_0.tar.bz2, and then the bootstrap stops. <stikonas>but with --update-checksums it shouldn't stop <doras>I haven't tried ignoring those failures to see if I get more. <stikonas>I'm just asking to understand if it's just 1st tar that is broken <stikonas>rickmasters: by the way, have you seen live-bootstrap? <unmatched-paren>well, at least, most of us are talented. unmatched-paren ∉ "incredibly talented people". <doras>stikonas: I ran with --update-checksums in only with an unchanged QEMU-based bootstrap to get a reference for the expected hashes. <stikonas>rickmasters: it would be nice to eliminate kernel seed in it <stikonas>doras: you can try to run your mode with with --update-checksums <stikonas>and see if checksum mismatch ends at some point <rickmasters>stikonas: yes, but just getting through stage0-posix was pretty difficult <stikonas>if you read logs here, everybody is aware that kernel is tough <stikonas>getting bootstrap kernels might be harder than going from hex0 to gcc <rickmasters>tbh, probably the most difficult program i've written (40 yrs programming) <unmatched-paren>i suppose this kernel will be linux-compatible (with only the things needed for bootstrapping, ofc) <stikonas>unmatched-paren: not necesserily, if we can build linux without gcc <Hagfish>this feels like the moment when Linus announced his kernel as an addition to the GNU project :) <stikonas>but you can always go via some other intermediate linux compatible kernel <stikonas>anyway, at least some keeping Linux compatibility would make it easier *unmatched-paren looks at BSDs (not linux-compatible, but gcc can be built for it...) <rickmasters>i should say, i'd hesitate to take this kernel much farther - hex0 is exhausting and tedious <stikonas>just like cc_* in assembly and M2-Planet are almost the same thing <rickmasters>but it can bootstrap M2 or whatever if we lockdown the syscall footprint <rickmasters>i didn't implement chmod, permissions, the file system is hacky <stikonas>everything can be executable, etc early in bootstrpa <stikonas>well, in some sense BIOS is a shortcut, just like POSIX is even bigger shortcut <stikonas>and unexpected! we didn't know that somebody is working on it <rickmasters>i wasn't sure i could pull it off, didn't want to promise anything <rickmasters>i avoided paging and preemptive multitasking with some righteous hacks <stikonas>yes, oriansj was reading up a bit on kernel writing <rickmasters>not sure exactly how to fit it in but hoping to collaborate <unmatched-paren>(meanwhile, i'm still struggling with such a ridiculously simple task as rounding an arbitrary number up to the next power of 2 in C...) <muurkha>unmatched-paren: there's probably a ridiculously convoluted way to do that in five lines of C in Hacker's Delight that GCC will recognize and replace with a single SSE2 instruction <muurkha>if you don't know about Hacker's Delight you'll probably enjoy it <muurkha>the amd64 instructions for this are apparently BSR and LZCNT, on ARM it's CLZ <muurkha>unmatched-paren: the nvidia news sounds great <stikonas>that's not something I would have expected anytime soon <stikonas>I guess it can't go beyond stage0-posix right now? <Franciman>hi everyone, I'm new here, and i already love all your projects! <stikonas>but in any case, writing kernel in M2 would be much easier <rickmasters>plug in your own after.kaem and give it a go - remember to add your source files... <stikonas>yeah, I'll try, though maybe a bit later... <rickmasters>yeah, i'll try more too but i needed a breather after stage0-posix <stikonas>I don't think oriansj or fossy expected that we'll be here any time soon <muurkha>unmatched-paren: yeah, §3–2 of Hacker's Delight is on this problem; it calls it clp2() <stikonas>Franciman: you came here at the time of a very big reveal :) <rickmasters>yeah bootstrappings cool but i struggle to explain it to my wife :D <stikonas>if mes-m2 somehow runs inside boot2now, I guess it will take hours :D <dzho>I heard you like to make software, so we're going to make software that makes software that makes software ... <stikonas>mes is much slower than M2 stuff written in C <stikonas>and in boot2now, M2 stuff is as slow as mes is on Linux kernel <unmatched-paren>dzho: a perfect description of the Autotools build system in the gcc repo :P <unmatched-paren>autotools generates makefiles which generate code which generates more code <rickmasters>stikonas: if you look closely, you'll notice it runs helloworld inside qemu and then on your local system <muurkha>stikonas: that's the most momentous Hello World I've ever heard of <Hagfish>the first line of bootstrappable.org is helpful for explaining to "normal" people the basic idea behind software bootstrapping: <Hagfish>"Do you know how to make yoghurt? The first step is to add yoghurt to milk!" <stikonas>rickmasters: oh, I didn't look at it that closely yet <rickmasters>HagFish: i tried the yoghurt analogy with my wife already but i'm not sure she got the connection <Hagfish>i'm glad you tried, but yeah, i imagine that i would have a strange reaction if someone told me they invented yoghurt :) <Hagfish>there are lots of things i'm not an expert in, and someone explaining their work to me in those fields would probably sound like gibberish <muurkha>Hagfish: I've found that this is rarely the case <Hagfish>well, the real experts are the ones who can explain their ideas clearly <muurkha>but it often takes a lot of effort to really understand what they're saying rather than just having a vague idea <muurkha>I don't think real experts in cryptography are going to be able to explain why elliptic curve Diffie-Hellman is considered safer than standard Diffie-Hellman to someone who doesn't know what a Galois field is <muurkha>they can explain at a WIRED-article level, which is enough to give you the illusion of understanding <Hagfish>well, that's probably an analogy that captures some glimpse of it, but some analogies are more helpful than others <Hagfish>yeah, "the illusion of understanding", i like that <rickmasters>with bootstrapping, explaining what, is easier than explaining why in my experience <muurkha>a real explanation of that issue is one that gives you a good enough understanding that if the conclusion were wrong you would notice <Hagfish>sorry, i meant "there's probably an analogy" <Franciman>so we could have an optimizing scheme compiler <muurkha>oh, yeah, there's probably an analogy, agreed <Hagfish>"if the conclusion were wrong you would notice" => yeah, i was watching a video recently about an AI trying to explain a joke that a human told it <Hagfish>they told the AI multiple jokes, and it explained them perfectly, and they even included an anti-joke <Hagfish>the AI realised that the joke was funny because it wasn't funny / didn't make sense <Hagfish>it's like the problem of a test suite that only tests positive cases and no negative cases <lagash>"Why'd the chicken cross the road? To get to the other side!" <Hagfish>"Because there was no native toolchain for the road on the architecture that the chicken used" <Hagfish>gcc didn't support it on principle, because it was a non-free toll road <stikonas>something very strange is going on with live-bootstrap hashes after I updated package further in bootstrap chain... <stikonas>first and second builds of tcc exchanged hashes... <muurkha>maybe rickmasters secretly backdoored your system with the boot2now build process ;) <muurkha>nice thing about working on a time machine: when you get it working you can go back and give yourself the result <muurkha>rickmasters: can your wife deal with children's books? or does she hate them? <muurkha>because I think the best book for explaining bootstrapping is Rémy Charlip's "Arm in Arm" <stikonas>muurkha: that's just with linux kernel and live-bootstrap, nothing to do with boot2now <muurkha>stikonas: the joke is that anything you ran on your linux kernel could compromise it and change the results of anything you did on it afterwards <rickmasters>muurkha: i'll be honest - she has other interests - but she supports what i'm passionate about 100% even if she doesn't need to know the details