IRC channel logs
2022-05-09.log
back to list of logs
<stikonas>oh yes, you need to either init or update submodules <stikonas>but easy to forget since it's one off step <doras>I did, but I think it somehow kept the previous checkout I had or something. <doras>So apparently mes is missing my fix: <doras>I was sure janneke took it when we talked about it at the time. <doras>I can't find any documentation about how to contribute patches to the mes project. <stikonas>yes, I thought you sent it to upstream mes too <stikonas>doras: what was the problem without that patch? <doras>Basically file permissions are messed up. <doras>I don't remember the exact effect, but permissions are confused in some way. <doras>I'm not sure if it's related (yet), but I now see that "cp" from mes seems to remove the owner read permission. <doras>I submitted the patch to bug-mes@gnu.org. Hopefully it's the right place. <doras>Yeah, the bug basically it made S_ISUID == S_IRUSR. <doras>So if SUID is removed through some mask, it actually also removed the owner read permission. <doras>Similarly, if using "ls", all files with the owner read bit enabled also appear to have the SUID bit enabled. <doras>The root user doesn't seem to care about read permissions, so it isn't affected. Once you a non-root user, you start getting failures in basic operations. <doras>I think I'll wait for mes upstream to pick up the patch and tag a release, and for live-bootstrap to update to it. <stikonas>well, at least mes upstream is now the only mes <stikonas>and hopefully mes 0.25 will come much quicker than the time it took for 0.24 release <stikonas>and janneke seems to be very busy incorporating improvements from M2-Planet 1.10.0 <stikonas>I can see plenty of nice improvements since 0.24 <doras>Maybe I'll rebase my branch temporarily to the commit before the removal of mes-m2. Just to be able to proceed. <doras>I seem to be getting wrong hashes with both the chroot and bwrap modes, but in different ways. <doras>The only change I have locally for the chroot mode is that I'm not using tmpfs with it either. Basically the commit above, but with conflicts resolved. <doras>Well, the commit I mentioned previously* <doras>I don't have time to look into at the moment, but it does look like something regressed somewhere. <doras>This is without the last MR that switched away from mes-m2. <stikonas[m]>doras: oh, tmpfs fixes might be after mes-m2 to mes change, you need to pull it into your branch <doras>stikonas @stikonas:kde.org: I think I had this one applied. The move to mes is the very last commit in the master branch, and I only excluded this one. <stikonas>doras: anyway, you can try to use mes snapshot rather than 0.24 now that janneke accepted your patch <janneke>doras: yeah, sorry for taking so long! <doras>stikonas: I think I can't do that until I can trust the hashes I get. The change in mes will definitely change some of them, and I need to be able to distinguish changes that are due to an issue and changes that are due to mes fixing the SUID issue. <doras>How come we didn't have the tmpfs-related issues before? <stikonas>doras: well, one issue was introduced by adding tar packages by fossy <stikonas>since tar was packing them in undefined order (tmpfs vs non-tmpfs) <stikonas>also we are checksumming more stuff now (basically everything), so maybe some of the issues were present <doras>Like additional reproducibility issues. <doras>stikonas: is there any chance you could send me your tmp/usr/src/repo/util-linux-2.19.1_0.tar.bz2? <doras>I don't have a reference for the file behind the hash we expect. <doras>So I can't compare it to anything. <doras>stikonas: looks like I'm getting different file contents :\ <doras>In mount/disk related commands: blkid, fdisk, findfs, etc. <stikonas>remote resources disappearing wouldn't be a disaster since we have a lot of local caches of these tarballs <rkeene>stikonas, A local cache doesn't scale over time <rkeene>Because local systems die and are replaced over time <stikonas>but that's only a problem if remote site dies <rkeene>New people join and existing people leave <stikonas>and we have quite a few people running this <rkeene>In other words you have an ad-hoc mesh of files of varying ages and no way to know if anyone has the complete archive <stikonas>well, strictly speaking yes, but contents don't change often <rkeene>There's a reason every project that spans decades has a mechanism for holding a copy of the data they rely on in a remote repository <rkeene>stikonas, As far as you know -- although you've just been given an example of it apparently changing :-) <stikonas>the problem is some historical tarballs disappearing <stikonas>mes is specifically written with bootstrap in mind <stikonas>and new releases will not break bootstrappability <stikonas>the risk is that some other older tarballs would disappear <stikonas>but most of them are from gnu project with good mirroring <rkeene>Or that they will change (which is why it's content addressed) <doras>It's most likely a configure thing again. <doras>Unless someone can analyze that giant diff into something meaningful, I'll need the build log that ends up with the expected hash. <doras>Regardless of util-linux-2.19.1_0.tar.bz2 failing the hash check in the chroot mode, I have perl-5.003_0.tar.bz2 failing the hash diff in my bwrap mode. <doras>It's funny (or sad) how all of this worked a few months ago. <doras>Regarding the bwrap mode, it seems that the tar contains different file permissions. <doras>--rw-r--r-- for the chroot mode, and -rw-rw-r-- for the bwrap mode. <doras>Also --rwxr-xr-x vs. -rwxrwxr-x <doras>So it seems that the group write bit is set for the chroot mode for whatever reason. <doras>I'd be happy if someone could diff it with the working build log. <doras>So basically since we started adding everything to tar archives, file permissions are now also hashed. <doras>Which means the umask of the user that executes the bootstrap is part of the hash. <doras>This can differ in different systems and environments. <doras>Do we know what umask is set for the user in the bare metal bootstrap? Where would this value be coming from in that case? <doras>I don't have umask when I already fail. <doras>The first thing that fails is perl-5.003_0.tar.bz2. <doras>And in that point we don't have the umask command yet. <doras>The "umask" command comes from the bash package in my Fedora system. <rkeene>I use "tardy" to post-process my own tarballs (though I tend to use cpio archives for this) <doras>Oh, it's possible that umask is actually a built-in. This is good. <rkeene>doras, umask has to be built-in (like cd) because it affects the process attributes <vagrantc>are you talking about the mode of the tarball, or the mode of files inside the tarball? <vagrantc>that reminds me, i should add --mode in there... <stikonas[m]>but I think -numeric-owner --owner=0 --group=0 --mode=go=rX,u+rw,a-s would work, it seems to be available in tar 1.12 <doras>stikonas: indeed it doesn't :( <doras>Sounds like something that may break many hashes <doras>But since I still don't trust the hashes I get in my system (I haven't managed to reach a full bootstrap yet), if it does change the hashes, I'll need someone that can complete the bootstrap to update them. <doras>I never tried qemu. I can try. <stikonas>well, try, it would be easier for you to develop if you have trusted state (hashes, logs, etc) <doras>What's the actual procedure to use qemu for the bootstrap? <doras>Because I'm getting "mkfs.ext4: No such file or directory while trying to determine filesystem size" when running "./rootfs.py --qemu" <doras>Where does one obtain a 32-bit kernel these days? :D <vagrantc>debian still ships at least one 32-bit x86 kernel <doras>stikonas: thanks for the tip. <doras>Any special drivers needed, or does the i386 defconfig good enough? <stikonas>there are a few things that are needed on top of allnoconfig but pretty much everything would work <markjenkinssksp>Regarding the current bootstrappable mailing list thread, and the point re common objection "you’re building from source but you’re not gonna audit all that source code anyway, so why bother?" <markjenkinssksp>It's also worth always going back to the conclusions of https://dwheeler.com/trusting-trust/ . We can make inferences about the behavior of the source code of a large program like GCC through a combination of bootstrapping and diverse double-compiling. <rkeene>I do bootstrapping to be able to support really long life software projects, since I need to be able to create binary patches for 10+ years <markjenkinssksp>If for example we someday bootstrap clang/LLVM to self-hosting without GCC and we're getting the same result as bootstrapping clang/LLVM with gcc, then we can assume the GCC source is okay and doesn't contain a "do the malicious thing if I'm not compiling clang/LLVM" <markjenkinssksp>and so audit full source of a shorter bootstrap does allow inferences about the source of larger build tools used in alernative bootstraps <oriansj>markjenkinssksp: I honestly don't get how someone could say you're not gonna audit all that source code when everything needed fits on 171 pages (single sided size 12 font) <oriansj>after that you have sha256sum, ungz and untar. Which means collective audits become possible for every single step after that <markjenkinssksp>exactly oriansj, audit the short stuff, don't worry about auditing the gcc source. <markjenkinssksp>perhaps we'll even have video lectures walking through all the short stuff some day... <markjenkinssksp>People might also respect the auditability of the short stuff more some day if we have it buyable in print as a way to support these efforts <oriansj>well audit *all* of the things but do it in a fashion that allows multiple independent parties to attest their audits and allow reputation to be known <oriansj>well I guess I could do a book on bootstrapping that covers all of the source and everything one would need to audit it <markjenkinssksp>exactly, let the gcc devs audit GCC, let the Linux devs audit Linux, let the bootstrappable.org community audit the bootstraps, and draw additional conclusions about the behavior of big programs like Linux, gcc, clang/llvm by way of DDC <oriansj>but that would take a couple months of work <markjenkinssksp>I'm assuming there will be another kernel available for bootstrap purposes at some point as well that will help make DDC conclusions about the behavior of Linux when its running dev tools <markjenkinssksp>or similar conclusions if the same set of audited software is used to bootstrap GNU/Linux on one side, FreeBSD on the other side <oriansj>markjenkinssksp: well yes that is the plan <oriansj>getting there however is going to be a bit of a minor mountain to climb <oriansj>my plan was get a rough but usable kernel written in C and then just manually write it in M1/gas assembly. <oriansj>as I am giving up on the idea of doing a kernel written in the M2-Planet C subset as I have come to realize that hardware is less forgiving to missing C features than a POSIX is <markjenkinssksp>I'm kind of interested in seeing if Bellard's tccboot is reviable and then it plus a initrd buildable in a simpler non-kernel environment, though I recognize that even with a full environment to run mescc to build tccboot, there would be real c compiler work to get tccboot and mescc compatible <markjenkinssksp>I also have hardware that's compatible with the drivers available in the version of Linux that tccboot could build and boot, so at least I personally need to do any driver dev for this particular shower throught <oriansj>but testing needs to be done to find viable kernels to run live-bootstrap on <markjenkinssksp>interesting, if one could do the work to get tccboot to build tilck in addition to the old patched linux that tccboot could build, then two diverse kernels could be bootstapped to run live-bootstrap independently <oriansj>well the smaller and simpler the root POSIX is, the easier it will be to audit the hand written assembly version of it will be <oriansj>*sigh* kernel is going to be the next major bootstrapping fight <oriansj>and it is going to be much harder than the bootstrap from hex to GCC fight