<stikonas>well, usually if you force-push, it updates PR and there is a link to changes... <stikonas>hmm, one more issue left with my Makefile for coreutils <fossy>stikonas: ah; there are a number of causes of that that i ran into with bash <fossy>1. cd 2. some types of characters that actually should be escaped inc ( , ] [ ) } { ' " <fossy>but look for stransge characters <fossy>also, restructure nearly done <stikonas>maybe $(BINARIES) : % : %.o needs /bin/sh? <fossy>i bet its somewhere in 109-114 <stikonas>it immediately went to building basename <fossy>maybe that is unsupported by the make version <stikonas>fossy: ok, I think it's space in PACKAGE_NAME=GNU Coreutils <fossy>stikonas: ohhh yeah a space would do it <stikonas>I guess sh is needed if I do it in one go... <fossy>stikonas: i'm going to finish the restructure, then either me or you can fix your PRs for the restructure (its just moving files around) and then i'll merge them <stikonas>well, let me test what I have and then I can try to make PR so that you can take a look and maybe restructure <stikonas>the whole install thing is much simpler than I initially wrote, it's just $(SRC_DIR)/install $^ $(bindir) <stikonas>make is definitely nice for coreutils... <stikonas>it would have been a nightmare to do everything in kaem <stikonas>so the biggest missing things from coreutils are ls and mknod <stikonas>hopefully they'll be easier with better libc <stikonas>although, we don't need them until we want interactive shell <stikonas>build process should be slightly easier now with coreutils... <stikonas>might be easier to install into chroot than mix c libraries... <stikonas>in principle we can just have different folders <fossy>well sooner rather than later i'm going to put linux 2.4.26 into it and then we will have sysb <stikonas>fossy: anything after coreutils will now use better cp <fossy>linux 2.4.26 compiled as sysb ***xwvvvvwx- is now known as xwvvvvwx
<Hagfish>Guix and reproducibility get shoutouts: <Hagfish>"The most prominent example is Nix and Guix, that use a purely functional deployment model where software is installed into unique directories generated through cryptographic hashes. " <Hagfish>heh, they add a clarification in a footnote: <Hagfish>'EDIT: It has come to my understanding that I’ve conflated the meaning of “reproducible”, as in reproducing bit-for-bit identical software, and “reproducible”, as in recreate the functionality of an operating system in this article.' <gforce_de1977>stikonas: bauen1: new numbers: a kernel 5.10.8 with 'tinyconfig', without procs/sysfs/printk ist 680k compressed and 1124k uncompressed, so much larger than 'myunix' 8-))) <gforce_de1977>and the most interesting: it boots in ~70 milliseconds to kaem <bauen1>gforce_de1977: ah, that looks much more comparable, myunix is still lacking most features, so let's see how that turns out when i have the first userspace programm running with basic io, fork and mmap support <bauen1>> Another approach would be to find an ARM SOC that has a bootrom that has been dumped and reviewed where users can easily input code and that has a display controller that don't need complex software to be used. <bauen1>good idea, i do have a pine h64 where i've reviewed the bootrom (and for a pine a64), no idea how complex it would be to add a display and keyboard, you probably need a usb stack for that <bauen1>i also have a 65c816 "computer" build from chips, which could function as a simple spi burner perhaps, writing the initial code for it using dip switches, but it currently relies on an atmega for IO <gforce_de1977>bauen1: i was thinking about ARM and in general: there is IMHO no need for peripherals (display, usb, block, net), because we can just dump out the resulting binaries on serial console (e.g. base64 encoded). generally spoken, the machines must be as simply as possible <gforce_de1977>i'am a fan of doing the builds in qemu, so fiddling with real hardware is something for later 8-) but i understand the needed thrustchain / reproducibility concerns <gforce_de1977>bauen1: is coreboot an option? it seems to be available for some platforms <gforce_de1977>(and coreboot can be provided as an BIOS replacement for QEMU) <bauen1>gforce_de1977: the only binary blob the allwinner chips used on those boards is the bootrom, and it can't be changed <bauen1>gforce_de1977: i would disagree, you at least need a way to review the code on the "trusted computing base", you only really need 1 button to confirm that the source code looks like it should <bauen1>but yes, block and network access are optional (but very helpful) <stikonas[m]>Hmm, I have rockpro64 that also only needs bootrom, but bootrom is not open <bauen1>^ i've found a 2 bugs in the sbrom, one of which you can use to bypass secure boot :/ <bauen1>in anyway 32kb binary blob is a lot bettern that your average x86 board ... <stikonas[m]>On x86 you have BIOS which is what u-boot does on arm, and u-boot is free <gforce_de1977>stikonas: but in theory you can write an uboot or coreboot to a flash/rom <bauen1>stikonas[m]: i mean you probably only need a small "bootloader" that sets up dram, and (maybe calling into the bootrom) loads some more chunks from spi flash <bauen1>probably don't even need a C compiler or u-boot <bauen1>stikonas[m]: essentially reimplementing the spl in assembly <stikonas[m]>Anyway, early bootstrap is unfortunately always hardware specific <stikonas[m]>So probably will only be done on limited number if hardware ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<stikonas>hmm, although, later chroot will be harder once we have sysb <stikonas>I don't think we can emulate other platforms in ci... <stikonas>without kvm, it will take a day to complete the run (and we haven't even started bootstrapping gcc) <gforce_de1977>stikonas: talking now about speed does not help. all these computers out there have time 8-) lets compare real numbers, if we have arm64-bootstrap up to tcc. I doubt it needs a day in emulation <stikonas>well, I only meant you can't run automatic free CI on somebody's else runners... <gforce_de1977>stikonas: and running './rootfs ci' on my computer is free-of-charge anyway 8-) <bauen1>speed in our case is more limited by money (or free credits or free capacity) than anything else, not something to worry about (too much) right now <gforce_de1977>bauen1: yea, it's better to *have* tests, even when they are slow, than not to test "because its slow" 8-) ***card.freenode.net sets mode: +o rekado
<OriansJ>pabs3: interesting article. I guess the bigger question is how much bootstrapping work needs to be done on a trusted machine. Because if one needs GCC to have any trust for another machine, there are bigger problems which bootstrapping GCC will not address. <OriansJ>eg like why does firmware that just initializes the memory and loads 512bytes into memory need GCC to be built? <stikonas>I guess if you can bootstrap up to M2-planet on trusted machine and check that hashes match it should be fine... <stikonas>well, if hardware is compromised then indeed bootstrapping will not help.. <OriansJ>or why a CPU has more than 1MB of Microcode <OriansJ>bootstrapping of software alone will not solve these sorts of hardware problems. <OriansJ>as once you get past cc_*, it stops being possible to properly check that the software is doing what you expected on hardware you don't trust; if you don't have a machine you can trust to compare against. <OriansJ>Modern GCC needs 1+GB of Memory to be built if I remember correctly. <OriansJ>also the number of supported architectures is limited <OriansJ>mostly by the oldest version of GCC in the bootstrap chain. <gforce_de1977>OriansJ: thanks for the reference, i was in hope, that it 'comes from somewhere' (university?) <OriansJ>gforce_de1977: long dead computer company's advertisements actually. As I never managed to find a copy of their actual manuals <gforce_de1977>OriansJ: in the appendix, there is a pointer to Motorola 6809..., strange: there is not even a wikipedia entry for Knight-Architecture i know of... <gforce_de1977>OriansJ: you selected knight, because of the possible easy implementation path? <OriansJ>gforce_de1977: no patents that would still apply, Could be done in individual logic gates and because it was different enough that a trusting trust attack in software wouldn't include it. <OriansJ>aka, it is weird and essentially forgotten <OriansJ>One needs to remember the more arbitrary hardware we can throw at the lower levels of the bootstrap the better. <OriansJ>I want college students to think up arbitrary crap, get the stage0 steps up to cc_* running on it and make hiding hardware backdoors impossible. <OriansJ>then I plan on doing libresilicon to get a trusted CPU and use that as a basis of trust. (That anyone can build) <gforce_de1977>OriansJ: i very nice idea! hopefully some teacher gets impressed and the show can start <OriansJ>As trying to buy trust just doesn't work <OriansJ>well it does make for a good instruction to how software works. <OriansJ>Starts with no linker, builds a minimal linker, then assembler and then a C compiler <OriansJ>it allows one to teach an introduction to assembly; where there is no magic and the students get to define their own instructions. <OriansJ>as every piece must be trivial to understand both in how it works and why. If any of it fails that, I need to fix it. <OriansJ>As all of the early stages must be something that fits inside one's head <mid-kid>If I understand correctly, Guix builds each "package" in a deterministic manner, and this applies to commencement.scm as well. <mid-kid>However, binutils' `ar` tool likes to store the timestamps of files in .a archives, making them mismatch. <mid-kid>And I see no use of the ar -D option or --enable-deterministic-archives, or any patch to disable the storing of timestamps. <mid-kid>How does Guix deal with this, does it reset the timestamps during build? Does it use a filesystem without timestamps? <mid-kid>binutils-2.14 doesn't even have a -D option to disable the storing of timestamps. <xentrac>OriansJ: I really like this idea and I wish I had taken a class like this <xentrac>but I am not experienced enough with teaching people to know whether it would be more popular or effective for most people than the more common approaches to it <stikonas>mid-kid: probably resets timestampts but ask on #guix <civodul>mid-kid: for ar, Guix builds Binutils with --enable-deterministic-archives, which always resets timestamps <civodul>as for the files in /gnu/store, their mtime is always set to 1 (one second after the Epoch) <stikonas>mid-kid: but in any case guix does not build each package in deterministic manner. It tries to do as much as possible, but definitely far from every <mid-kid>oh? I thought that was the point? Might be misled... <mid-kid>civodul: the binutils used for commencement.scm packages doesn't have this option. <stikonas>mid-kid: also commencement.scm takes a few shortcuts in bootstrapping stuff (some of them due to guile driving the build process, if we don't have guile, we can't run e.g. gash)... <mid-kid>stikonas: Right, but it's the only working "full" bootstrap chain I've found, so I'm trying to understand it. <stikonas>mid-kid: well, we have some bootstrap chain (based on guix) that does not take shortcuts, but it's not finished yet <stikonas>we have built tcc, sed, coreutils, gzip, tar, bzip2, m4, make... <stikonas>but properly building bash and autotools is much harder... <stikonas>e.g. guix just runs configure scripts that are in packages <stikonas>and bash source has some autogenerated C files too <xentrac>OriansJ: the Knight was originally a 16-bit machine, right? I'm trying to find documentation for the original machine and failing <stikonas>well, original manufacture was gone before internet appeared <stikonas>so I guess it would be hard to find anything online <stikonas>mid-kid: not that bash is included there but it's not "properly" done like I mentioned above, we need to bootstrap flex/bison <stikonas>mid-kid: but you can run quite a bit of it <stikonas>(either in QEMU, chroot or on real hardware) <stikonas>so it starts with 357 byte hex0 and 737 byte non-interactive kaem shell) <stikonas>mid-kid: but getting autotools will be hard :( <stikonas>autotools depend on perl, and perl also needs perl for full build <stikonas>(Configure script used for building perl is generated with perl) <mid-kid>I'll check it out, I was basing my scripts on the guix bootstrap chain, with about 2MB of mes+busybox as seed, as I thought mes-m2 was still unusable. <mid-kid>I see it doesn't go beyond mes though? <xentrac>stikonas: is there an autoconf ./configure alternative to ./Configure for Perl, or am I misremembering? <xentrac>(or does autoconf depend on Perl too?) <mid-kid>As for autotools, in some cases you can make your own config.h and avoid it, especially with the older GNU packages. <stikonas>it's tcc + sed, coreutils, gzip, tar, bzip2, m4, make... <stikonas>mid-kid: that's what we did for those packages <stikonas>but we'll have to bootstrap it at some point <stikonas>mid-kid: initially it was just plain "kaem" shell scripts that we used, but now we have make, so we are using hand-written makefiles <stikonas>mid-kid: we probably need to build musl too... <mid-kid>Haven't found a way to build musl (even older versions) with tcc+mes-libc yet... <stikonas>mid-kid: I did try the other day, but still some of the things need fixing <stikonas>(but not latest git, because that one does not build on mes libc) <stikonas> /usr/bin/automake-1.13: Perl script text executable <civodul>mid-kid: ah ha! it could be that it didn't exist, but not that most makefiles do "ar D", which is equivalent <stikonas>well, you can also make .a archives using cat <xentrac>stikonas: yeah, I think autoconf would probably have been written in Perl too if it had been started a few years later <stikonas>yeah, now it's shell script that uses m4 and awk... <stikonas>well, we have m4, awk still needs bison... <mid-kid>civodul does binutils 2.14 have the option though? <mid-kid>it's not mentioned in the manpage at least... <xentrac>stikonas: hmm, are there no awk implementations without yacc? surely someone must have written a recursive-descent-parsing awk at some point <civodul>mid-kid: interesting: it would seem that gcc-core-mesboot0 is not reproducible <civodul>checking whether this is due to that <mid-kid>wouldn't be too surprised. I think I'll end up patching 2.14 and 2.20 to force removing the timestamps. <stikonas>xentrac: maybe there are, but I haven't looke to omuch since we need yacc anyway <xentrac>civodul: not reproducible in the sense that the results aren't bitwise the same, or not reproducible in the sense that it doesn't build at all? <xentrac>stikonas: ah, and awk isn't sufficient for autoconf because it also depends on Perl, and awk isn't otherwis necessary for bison? maybe some kind of graphviz diagram would be helpful for this kind of thing <stikonas>xentrac: yeah... diagrams might be helpful <civodul>xentrac: in the sense that they're not bit-reproducible, due to timestamps in .a files <stikonas>bison and flex depend on each other but there is some bison-bootstrap path via heirloom lex/yacc that we can try <xentrac>civodul: we might want clearer terminology to distinguish FTBFS from "nondeterministic" <civodul>xentrac: ah sure, i'm used to the Guix and R-B contexts where it's clear that it's bitwise reproducibility we're talking about :-) <xentrac>original-awk also depends on yacc (but presumably any old yacc, not specifically bison, which might be a distinction worth making in light of nyacc) <xentrac>awk from 9base (the Plan 9 awk) also depends on yacc. although the plan9 yacc in 9base doesn't seem to depend on yacc <xentrac>that is, 9base_6.orig/yacc doesn't contain any *.y files or anything that looks like an LALR transition table encoded in C <xentrac>I was concerned that maybe it would be written in Plan9 C instead of ANSI C but I don't see any obvious sense in which that is the case <xentrac>(also presumably Debian's 9base package doesn't require Plan9 C to build!) <fossy>gforce_de1977: I see no reason to add unnessecary complexity to rootfs.sh when those exact commands can just go in the ci script, since we would just be using it for CI <OriansJ>xentrac: building for and running on are two very different problems. <xentrac>yeah, I don't think GCC can even run on a sun2 <xentrac>(because the 68000 ignored the most significant byte of not just the physical address but also the virtual address!) <stikonas>fossy: I've added a couple of more utils to coreutils PR... <stikonas>but I guess I still need to update everything to your refactor <stikonas>fossy: I also had another quick look at musl (still manual testing)... Hit a few problems btu hopefully they are not intractable <stikonas>first of all we need newer tinycc... secondly, I might be hitting the same problem you had with mes libc (I guess that's why you had to create unified sources) <stikonas>and binaries were still segfaulting for me... (although they do compile) <pder>stikonas: earlier you said that perl depends on perl. Where are you seeing that? <stikonas>pder: Configure script is generated using perl <stikonas>so if you don't want to use pregenerated files (that are not source) then you get this dependency <stikonas>it's not a "strong" dependency like e.g. gcc depends on gcc... <stikonas>so we might be able to write some manual build script to build miniperl