<fossy>stikonas: pder how important do you think it is to checksum interim binaries <fossy>for example m2-mes in mes.kaem, flex-tmp in flex-2.5.11, boot*-tcc for tcc, etc <fossy>that's what i've been doing so far <fossy>so, it's currently fletcher16 until coreutils, then sha256sum from then on <stikonas>I tried calculating hashes for all binaries in /after/bin though <fossy>fletcher16 will be replaced by crc32 once we migrate to m2libc <stikonas>(not all temp binaries are installed there) <stikonas>if I remember correctly, we might have more problems after binutils <fossy>it checks everything installed into /after/* after each "Part" <fossy>i kinda want to add a reproducibility check to ci <fossy>but that would double the length <fossy>yeah, well that's what i'm doing, lol <fossy>also stikonas i don't think checking /usr/include files is required, particularly for mes, do you agree <stikonas>fossy: yeah, those are usually just installed or coppied <stikonas>OriansJ: congratulations, thta's PPC stuff? <OriansJ>stikonas: yes the key to PowerPC64le that was driving me insane <OriansJ>## e_entry isn't the Address of the entry point for PowerPC64 but where the entry address is stored!!!!!!! <OriansJ>and it explains the weird as shit behavior in gdb <stikonas>well, after you spend some time on it, you appreciate it more once you solve it <OriansJ>I spent 2 hours commenting a hex dump of an assembled program to finally figure it out <OriansJ>ouch and it can't be in the elf header???? <stikonas>hmm, it took like 3 weeks to implement bison bootstrap in live-bootstrap. Any guesses how long perl will take... <OriansJ>build with: hex2 --little-endian -f foo.hex2 --base-address 0x10000000 -o foo <OriansJ>(I'll add the --architecture PPC64le after I get the test pretty) <OriansJ>but this might cause some trouble with blood-elf <OriansJ>dftxbs3e: can you please bare metal test that it does infact work on actual PowerPC64le hardware <OriansJ>It should display hello world and return 42 <OriansJ>readelf -h will say: Entry point address: 0x100001e8 but the actual truth is: 0x10000080 <OriansJ>dftxbs3e: we finally have a working hex2 test for PowerPC64le <OriansJ>please build and run to verify it works on real hardware as well <dftxbs3e>OriansJ, yes doing so now! Thanks a lot! <dftxbs3e>OriansJ, what version of hex2 minimum required? <OriansJ>dftxbs3e: just grab latest but honestly every version after Release_0.1 supports little-endian output and base-address (if I remember correctly) <OriansJ>(older versions use old style flags but that wouldn't change the output anyway) <dftxbs3e>OriansJ, I say that because I have some older version of GNU Guix working on ppc64le <OriansJ>it should build just fine; if it complains about unknown arguments I'll put it in old form for you <OriansJ>and if you want the latest version in guix, you can just git clone and guix build -f guix.scm <dftxbs3e>Well I have some unmerged stuff required for latest version of GNU Guix (WIP) <OriansJ>(old version needed --exec_enable if you wanted to skip the chmod) <OriansJ>then we realized I did it backwords and added --non-executable <OriansJ>Ok, we officially have a working hex2 program for PowerPC64le that works on baremetal. <OriansJ>next to figure out jumps (it looks like it might be off by one) and branching <OriansJ>then figuring out blood-elf integration; then begins the long ugly trail of getting M2-Planet working. After which janneke will have what he needs to port MesCC to PowerPC64le (it the proper short form ppc64le ???) <dftxbs3e>I would prefer if it was called ppc64le everywhere <OriansJ>for the branching and jumping testing you'll be doing dftxbs3e you'll be pulling from master <dftxbs3e>Debian uses ppc64el, Fedora uses ppc64le, GCC and LLVM use powerpc64le, so go figure <dftxbs3e>Otherwise the ISA is called "PowerISA 3.1" for example <OriansJ>mostly because you like it and are willing to help and that is good enough for me <OriansJ>ok looks like branches have a +3 going for @label <dftxbs3e>OriansJ, note: in GNU Guix Ludo called it powerpc64le so we may want to stay consistent at least on the GNU Guix-related things side <OriansJ>I'll give them until the next mescc-tools release to complain otherwise eh <dftxbs3e>OriansJ, they wont complain, it's just me wanting to keep it consistent in GNU Guix :-) <OriansJ>dftxbs3e: well on guix if one does get_machine one gets: ppc64le <fossy>ppc64le is more common afaic <dftxbs3e>If get_machine is Linux's uname -m, then yes <OriansJ>dftxbs3e: think of it as the low-level version of uname <dftxbs3e>It's fine, but if it needs to be passed down somehow it means we will need even more conditionals <OriansJ>get_machine with no arguments returns what hex2 and M1 would expect for --architecture <dftxbs3e>OriansJ, I don't know the specifics of your tools, I just meant it from a packaging perspective <dftxbs3e>due to this inconsistency, I some times have to give aliases for the different names to particular programs in packages <OriansJ>dftxbs3e: these tools are only used to bootstrap GCC <OriansJ>So only people here have to deal with them (or people who love working in hex) <dftxbs3e>well even then, the simpler it is, the better, right? But since it auto-detects, that's a non-issue. <OriansJ>dftxbs3e: ok for this next test; you are going to need to get the latest mescc-tools commit <OriansJ>( 722a555d1813087a8f826ab8737d1d80d9723cdc to be precise) <OriansJ>hex2 --architecture ppc64le --little-endian -f elf64-PowerPC.hex2 -f foo.hex2 -f retard.hex2 --base-address 0x10000000 -o foo && ./foo || echo $? <OriansJ>This is making sure the relative and absolute addressing matches with the virtual machine <dftxbs3e>OriansJ, okay doing so now. By the way, if I give you access to an lxc container running on baremetal, is that baremetal enough for you? <OriansJ>dftxbs3e: This is the last test needed for a while <dftxbs3e>my email is: lle-bout@zaclys.net in case you want to send more tests and I'm not here <OriansJ>It'll only be fore the blood-elf spot check, M2-Planet spot check and final M2-Planet pass. as I can do alot with these basics <dftxbs3e>OriansJ, where do I get the 3 .hex2 files from? Do I just use the single file from the paste or..? <OriansJ>dftxbs3e: the single file from the paste is the cat of the 3 files <dftxbs3e>Running: ./pre-inst-env guix environment --with-commit=mescc-tools=722a555d1813087a8f826ab8737d1d80d9723cdc --ad-hoc mescc-tools -- hex2 --architecture ppc64le --little-endian -f /tmp/1185290 --base-address 0x10000000 -o foo && ./foo || echo $? <OriansJ>dftxbs3e: building mescc-tools is just make <OriansJ>or if you wish to keep it even simpler: gcc -D_GNU_SOURCE -std=c99 -ggdb -fno-common hex2_linker.c functions/file_print.c functions/match.c functions/numerate_number.c functions/require.c functions/in_set.c -o bin/hex2 <OriansJ>(feel free to replace GCC with any C compiler) <OriansJ>looks like I-Form instructions in ~label 48 (forward bl) and ~label 4b (backward bl) work without issue <OriansJ>I should have the M1 work for PPC64LE done tomorrow. <Hagfish>they're really suggesting that every time you download a piece of software, you check whether a public append-only log has seen that software (with that hash) before <Hagfish>you'd really want to do those checks over Tor or something, to not reveal to a log that your company is running old versions or adopting some interesting new technology <Hagfish>but yeah, that should make it harder for companies to insert backdoors in their updates <Hagfish>i guess there could be some clever timing attack where they release a malicious version just before you check for updates, and then immediately release a non-malicious version afterwards <Hagfish>really you'd want a cooling off period on all updates, to give auditors a chance to check for backdoors <Hagfish>if there is an emergency update rushed out for a security vulnerability, probably the right thing for the app to do is to exit after doing the update check <Hagfish>that means that an attacker who gains control of the signing key could at worst cause a DoS <bauen1>Hagfish: couldn't you just download the entire log ? or would that grow too big after some time ? <gforce_d11977>for F in $(busybox cut -d' ' -f3 1.txt); do ( cd /after/bin && busybox sha256sum $F ); done >2.txt <stikonas>well, I was using chroot and /after/bin/sha256sum <gforce_d11977>i think we should check from externally (CI-script), if the hashes are good, but NOT abort the build if something changes <stikonas>does it really matter whether it's internal or external... <gforce_d11977>stikonas: no, i does not matter, but for now we *do not have* sha256sum build <stikonas>I've built it some time ago in live-bootstrap <stikonas>and that doesn't require cryptographically safe hash <OriansJ>bauen1: there is no point in downloading a full history when you only need a subset, think git shallow clone with the option to download more if needed <Hagfish>good points. i think the technical side has mostly been solved now. what's needed is the social side, i.e. setting up useful trust relationships between entities <Hagfish>maybe Let's Encrypt or the Internet Archive could run a server that's dedicated to downloading open source projects and reproducibly building them, then serving their hashes (and inclusion proofs) as an API <bauen1>OriansJ: that's what i was thinking, makes sense <OriansJ>Hagfish: the social side doesn't need to be solved. Society function best when there is a small group of crazies (that anyone can join) shoulder the burden of keeping things working. <OriansJ>There are 7billion people and what 5-11 developers here? Bootstrapping progress is going just fine. <OriansJ>They need only 1-4ish people to work actively on maintaining that chain forever and they could easily hook into guix challenge to do most of the work. <OriansJ>get a checker that can get built by M2-Planet, a public mailing list and a few random anonymous crazies (So they can't be targeted) to catch if the maintainers are being targeted. <OriansJ>It also looks like hex mapping to ppc64le might not work the way I wanted. Now to try binary mapping to see if that fixes the problem. <Hagfish>it's true that a lot can be done without massive social structures (indeed, many of the best innovations and advances have come without that), but it's generally not good to concentrate such important work in the hands of so few, long term <Hagfish>i guess bootstrapping is different from other software projects in that the aim is not to keep adding features, but produce a permanent, well-studied set of artefacts <OriansJ>Hagfish: bootstrapping isn't different; there will be refinements *FOREVER* and new ports to new architectures and new bootstraps to new languages. <OriansJ>It is a never ending trend where we keep running forever and never getting closer to the end. <OriansJ>Nothing here is magically different than any other software development project. <OriansJ>The only real unique thing about this bootstrapping project in comparision to all the others is we got more than 1 active programmer at a time. All the other failed bootstrapping projects were solo works that died when the programmer gave up or got stuck before producing anything interesting. <OriansJ>Here we support each other and help each other get past problems that the other couldn't/wouldn't solve. <OriansJ>Those lines of documentation, the presentations, the spell checking, the distro specific testing and personal feature additions that look wasteful are actually a reflection of the huge amount of energy invested in this project by everyone here. Without the sheer numbers here, such impressive results simply are not possible. <OriansJ>That is why the first job of everyone here is to make the job of the next person easier whenever possible. (because that person might just be you) <OriansJ>we are approaching the 5 year anniversary of #bootstrappable <Hagfish>this project seems to have found an "attractor", in the mathematical sense, or a "critical mass" in the physical sense *janneke being curious if something exceptional happened <OriansJ>once two or more developers actively work together for greater than a year, the odds of attracting additional developers goes up. <OriansJ>Guix as a community for people finding a common goal and spawning work together basically gave us a big push. ***ChanServ sets mode: +o rekado
<stikonas>fossy, pder: I'm trying to build miniperl from perl 5.8.5. I have about 10 errors left (and for now I'm ignoring bison parser generation) but I think half of them are the same problem. Maybe a second pair of eyes would help... <stikonas>I'm getting build errors like gv.c:1946: error: 'copy' undeclared or regexec.c:1187: error: 'PL_utf8_' undeclared and a few others and they all seem to use CAT2 define <stikonas>it looks like CAT2 should be in config.h, grep found #define CAT2(a,b) a/**/b <fossy>I had to patch out fastcall for linux kernel, this is probably a similar problem <fossy>stikonas: gforce_d11977 it *does* matter, and quite a lot too, whether it is internal or external <fossy>if it is external we can count the hashing as null and void for the most part, as it has not been bootstrapped and is hence untrusted <fossy>and we certinaly need to fail if the checksums are incorrect, that means it is probably not reproducible which is very bad <stikonas>ok, I think for miniperl I've now have config.h that can build all .c files (with a couple of TODOs for later). Now I get maybe 70 linking errors... That's way more than with earlier programs <stikonas>oh, much better after clean rebuild, only 18 left <stikonas>fossy: oh a##b I think concatenates two strings <fossy>stikonas: pder how are we even making tinycc link against must <fossy>musl installs to /after/lib/musl <fossy>i see libdir is reset to /after/lib/musl <stikonas>fossy: argh, I have more serious problems with perl... <fossy>stikonas: yes for checksumming <fossy>stikonas: what kind of cyclic deps <stikonas>ignoring this file (and perly.y for now, which might be doable since we have bison) I get 3 linking errors when building miniperl... <stikonas>and embed.pl is not particularly simple script <fossy>stikonas: first maybe look at Perl 5.0? <stikonas>no, there is a scriped called embed_h.SH instead of embed.pl <stikonas>perl will definitely need very careful review once something is done...