IRC channel logs
2021-01-14.log
back to list of logs
<fossy>stikonas: are you using the makefile from the bzip2 project or no? <fossy>because there is no makefile checked into that commit... <fossy>I will not do the same thing for bash despite it being handwritten, as bashs makefile is much too complex as it uses things like install <stikonas>fossy: I only build one target from that makefile in bzip2 <stikonas>well, there is also bzip2recover but we definitely don't need it for bootstrapping <stikonas>fossy: did you mention make binary is also non-functional? <fossy>bbl need to go so something irl <stikonas>oh, bash doesn't have tar.bz2 file. Well, in that case bash and bzip2 can be in any order... <stikonas>maybe it makes more sense to build bash first <vagrantc>and of course, also worked for me when i uploaded, and several tests i've done now <vagrantc>main difference that seems like it might be relevent is the user's shell... but i can't reproduce it by changing the user's shell on my local builds <vagrantc>well, will have to follow up on mescc-tools another time ... <bauen1>xentrac: yup, it works, i never really changed the irssi default keybindings <fossy>jeez bash has a lot of files <bauen1>and requires so many different syscalls <stikonas>are you going to push make build first to live-bootstrap? <stikonas>that's why I was wondering when make will be available :) <siraben>fossy: would live-bootstrap work on macOS? <gforce_de1977>siraben: at the moment i'am preparing am qemu-image for this task <stikonas>more interesting and known to fail would be arm bootstrap... <stikonas>gforce_de1977: yeah, but I couldn't get any screen output on arm qemu... <stikonas>I've recompiled my arm kernel with some driver but that diddn't help <stikonas>for github action it might be simpler to use chroot build... <stikonas>then we don't have nested virtualization <stikonas>in CI we wouldn't have KVM so qemu would be really slow <bauen1>stikonas: do you know if gitlab has KVM / nested virtualization for the public runners ? <fossy>gforce_de1977: look at my pr... i have started ci <stikonas>bauen1: on gitlab you can have you own runners, but then you need to host your own builder <fossy>gforce_de1977: nested virt is needed for it to run at any reasonable speed <fossy>hence why if you look at my pr i am using cirrus ci <fossy>because they officially support nested kvm <stikonas>but maybe something more lightweight is enough for CI? <fossy>well now that we have chroot mode i'll adapt it to use that <fossy>anyway make is working welll so i will push it <fossy>also, FWIW, stikonas this is my current bash makefile, if you want to draw inspiration at all <fossy>Hm, does anyone remember who boootstrap bison <stikonas>fossy: maybe you can have simpler list for all those objects and append extention/folder using make functions? <fossy>stikonas: no curently unrelated to live-bootstrap, just wondering who did it <fossy>stikonas: plan to get to that at some point, but wasn't bothered at the time of making the list <fossy>if you come up with something for m4 i'll apply that <fossy>otherwise i'll make my own make rule for that <stikonas>fossy: yeah, I'll try to write something this evening before you wake up <fossy>grr. bison depends on flex which uses bison to generate something else :( <stikonas>well, I guess that's why gio write his code... <gio>stikonas: You can build bison without autotools, manually calling gcc. <gio>It's not beautiful, but you can do it. <stikonas>we we manually calling tcc for evrything up to now, but we have now bootstrapped make <fossy>gio: your statement that "it is possible <fossy>to compile Flex without using Bison <fossy>is not true unless you use a pregenerated file <fossy>as per included with a source distribution <gio>I do that in nbs, using stuff from the Heirloom project. <gio>It requires some more steps, but it can be done. <gio>In nbs only binary stuff available at the beginning is tinycc and musl. Everything else cames in source form, unless something slipped though. Let me know if you see anything autogenerated in nbs. <stikonas>gio: so we only have any posix kernel (for now linux), hex0 (357 bytes) and kaem shell (753 bytes) <gio>Hairloom might probably give you a lot of shortcuts. <stikonas>and this point we have tcc/tar/gzip/make/diff/patch/sed <gio>What is your first compiler? <gio>What is your first real compiler? <stikonas>well, tcc is first fairly powerful compiler <gio>In other words, what do you "compile" with hex0? <fossy>gio: well it just goes through mescc-tools-seed to start off with <stikonas>another hex assembler: hex1, then hex2, then cc_x86 <stikonas>cc_x86 then builds M-2 Planet which is the first compiler in C <fossy>then it uses janneke's wip-m2 of GNU Mes to compile Mes using M2-Planet, then uses MesCC to compile TinyCC and then we move on <fossy>abliet hard to read without appropriate background, but that's low-level assembly <gio>Ok, I think I still prefer my path through asmg. <gio>(not to dismiss your effort, of course, everything is good; it's just that I don't completely agree your count of 357+753, as it leaves out a lot of other not-really-easy-to-check stuff) <fossy>the linux kernel is much too large for an end project <fossy>i kinda want to implement asmc as an alternative starting point <fossy>and M1/M0/hex* is not all that easy to verify <gio>I believe that Linux kernel would not be far from the reach from asmg, it I had time to spend on that. And maybe eventually I will have some, but not now. <gio>Linux is not that complicated; there are some linker shenanigans and some assembler syntax that tcc doesn't handle, but that's all. <fossy>gio: theoretically, it would be possible to run asmc from a floppy right? <gio>It's possible, depends on whether you can fit the source code it needs on it. Eventually it boots iPXE, so at that point it can fetch further code from the network. <gio>Not necessarily; for the moment it is the most Linux-like thing it is able to compile, and it was basically an exercise for me to show that something Linux-like could be compiled at boot time, without directly attacking Linux. <gio>If you manage to compile Linux-with-network (hard) and a stupid HTTP client (easy), you don't really care for iPXE. <gio>So if you can pack asmg (very little), the C compiler in G (quite little), tinycc and enough of Linux source code on a floppy, there you are. <stikonas>well, if we can compile linux without network, and have a bit more space, we can easily compile linux with network <gio>So far I did my experiments with hard disks, but it doesn't change much. <stikonas>and I don't have floppy drive at home... <gio>stikonas: Yes, the hard part is Linux itself. Especially the linking and loading part. <fossy>gio, afaict, asmc does not have a libc involved, is this true? <gio>There is actually nothing intrinsically difficult, it's just a few linker magic that you have to unwind and implement your way. <gio>fossy: No, it doesn't. <stikonas>well, you only need libc one you have kernel <stikonas>and even then you can do syscalls directly... <stikonas>we can definitly look at this once we are done with curent live-bootstrap <gio>stikonas: Yes, although at some point you'll want to have an actual libc anyway, and that might be a tad more difficult (unless you've already solved this with live-bootstrap). <fossy>Mes LibC is what we currently use <gio>Once you have tinycc and musl, you can start back from nbs, in my plans. <gio>Or from live-bootstrap if you can join there. <gio>Yeah, libc's are hard... <gio>I find them hader than kernels. <fossy>kinda following guix's bootstrpaping route to a full toolchain in live bootstrap rn but filling in holes like autotools <stikonas>musl is probably possible but need some patching to build with tcc <stikonas>guix is now using 50 MB driver to drive 357 byte hex0 <gio>What is a driver here? <stikonas>so in guix guile prepares build environment and also runs some tools (gash, bootar, etc) <stikonas>although, tar is already done, and bash is almost done <civodul>stikonas: that's a temporary situation though <civodul>(plus nobody's done any better, right? :-)) <stikonas>civodul: well, our live-bootstrap will be doing better, but if you talk about distros, then yes <civodul>nice, i didn't know about live-bootstrap <civodul>we didn't pursue it, but the idea was to "unfold" the operations normally performed by the Guix build daemon & co. <civodul>so you would boot a system that starts building from scratch the package graph <bauen1>in theory every distribution should be interested in bootstrapping, if only as a way of breaking cyclic dependencies that are annoying to deal with <stikonas>civodul: yeah, for now we use linux kernel, hex0 and kaem <pabs3>(CI for cross-bootstrap stuff) <janneke>stikonas: the guile-2.0 driver is much smaller than 50MB (although still huge0 <janneke>du -sh $(guix build -e '(@@ (gnu packages bootstrap) %bootstrap-guile)') <janneke>14M /gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0 <stikonas>yeah, I guess I didn't remember it correctly... <stikonas>I thought guile is the biggest part of current reduced bootstrap (which is guile + mes) which is about 60MB... <janneke>stikonas: that's right, i assumed we were talking about the full source bootstrap: <janneke>i.e., where mes' wip-m2 branch was first shown to be feasible for the bootstrap <janneke>stikonas: didn't you catch that i managed to build mes (wip-m2 branch) with m2-planet? <janneke>people are now trying to reproduce outside of guix too <stikonas>janneke: so we have a fairly good progress now <stikonas>yeah, and we are not using pregenerated scripts like configure, <stikonas>we don't have any of the gash or bootar like guix does <stikonas>they don't seem to run on mes and require guile <stikonas>after bash we need to bootstrap autotools (m4/autoconf/automake) I guess... <janneke>that's right, running gash, gash-utils on mes is one possible future path <stikonas>well, we just wrote enough kaem scripts to build what we need <stikonas>well, no more kaem scripts from now on, since we now have make <stikonas>janneke: by the way, we managed to build sed 4.0.7 instead of very old 1.18 that is used in your branch <stikonas>janneke: 1.18 was of no use for us, had to build at least v4 <stikonas>(without gash we needed -i flag in sed for in place editing) <janneke>stikonas: oh, that's nice; i'll have a look <janneke>i was hoping we can gradually get rid of most ancient softwares <janneke>i don't remember why that didn't work at the time, could be need for a newer awk, or bash <janneke>right, going around the problem altogether <stikonas>(since bash is no available for us yet when we build sed) <janneke>i wanted to use guix's gnu-build-system as much as possible initially <stikonas>(we use sed for early patching of other software) <stikonas>janneke: yeah, but in live-bootstrap we dont' have gnu-build-system, we need to bootstrap our build systems... <stikonas>and building patch also needed some patching :) <janneke>yes, iwbn to get a series of "bootstrappable" releases that don't need patching <stikonas>well, most of the patching is very light <stikonas>and much smaller than rewriting build scripts :) <stikonas>civodul: so if we say that handwritten things are source, then to full extent <stikonas>since we can't do any advanced stuff in kaem... <stikonas>civodul: well, hex0/1/2 are even less "source", but they are also handwritten <stikonas>in any case, mes.kaem is basically a list of files to compile <stikonas>we run the same command over and over again <civodul>stikonas: i thought this shell scripts were automatically-unrolled makefiles <civodul>(to me, "source" is the "preferred form for making modifications to it", as the GPL puts it) <stikonas>not really automatically... although I did look make output :) <stikonas>well, I was making modifications to this file, not some makefile and run script on it <stikonas>so I guess it satisfies GPL definition of source <civodul>though the form we really "prefer" is the original makefile (or similar) <civodul>but i guess that's always the case, we always "prefer" the higher-level thing <stikonas>yeah, people also prefer "configure" files even though that's definitely not preferred form for modifications <civodul>i meant "prefer" as in "prefer to make modifications" <civodul>configure files are definitely not "preferred" in that sense :-) <stikonas>gforce_de1977: remember that asterisk that I got wrong inside quotation mark <stikonas>shell scripting is so easy to get wrong :( <gforce_de1977>stikonas: i send the proposal: for FILE in foo/${var}.*; do test -f "$FILE" && ...; done <stikonas>yeah, that's what I have now in my commit queue too <gforce_de1977>stikonas: fossy: finally my arm-qemu-setup-CI-runner is ready, i'am tired now and will send the pullrequest tomorrow. I also had to fight with arm to output text 8-))) <stikonas>gforce_de1977: oh, you managed to solve it, nice! <gforce_de1977>stikonas: fossy: because of the asterisk: will send the PR also tomorrow morning <gforce_de1977>stikonas: arm is really tricky..., but now the way to aarch64 is also clear... <fossy>how will live bootstrap work on arm <fossy>janneke: whats the total seed size inc guile for full source bootstrap? <janneke>fossy: %bootstrap-guile is 14Mb, and hex0 is 357 bytes, oh and we have kaem ;-) <stikonas>so I'm still confused why older reduced bootstrap is 60 MB. Didn't it have just guile and mes? <stikonas>fossy: I have a few patches to review while I'm working on m4 <stikonas>fossy: I've built bzip2 (with some patching to patch out dependencies on bash and coreutils) <janneke>stikonas i believe that was calculated when we aimed to switch to guile-2.2, so that number is over-estimated <stikonas>oh I see... my confusion is finally resolved :) <janneke>bootstrap-mes and bootstrap-mescc-tools are about 12MB <stikonas>ok, so you'll go down from 26 MB to 12MB <janneke>we aimed to halve the size of the bootstrap twice and that sort of worked <stikonas>that's an order of magnitude better than anybody else already <fossy>hm, stikonas, i think that i still prefer doing patch/make/install/test in a kaem subfile, even if we are only doing make <fossy>otherwise after.kaem.run will get very long very fast <stikonas>fossy: I thought we'll call after.sh file soon <fossy>stikonas: we will but consistency <fossy>i think i still prefer splitting it out <fossy>well, only issue is that if someone ctrl-c's in the middle of a download then it will use a corrupted source code tarball <fossy>but does significantly reduce time.. <stikonas>fossy: ok, pushed new change with split out file, will test it now <stikonas>nothing ever breaks on trivial changes /s <fossy>well we should keep all of the tar/gunzip/cd's outside or all inside <bauen1>shouldn't the downloaded files be compared to a checksum ? <fossy>as per make, the patch dosen't have any prefix on the file <stikonas>yeah, I agree that we should do checksum <stikonas>probably as part of that get_file function <fossy>and rm it if it dosen't match <stikonas>bauen1: I think you just volunteered to make a patch :D <stikonas>anyway, if bauen1 doesn't do that I can do it (maybe tomorrow) <stikonas>need to get the right defines for CFLAGS there... <bauen1>stikonas: i haven't tried live-bootstrap yet, so you should do it <bauen1>stikonas: i'll be sure to add :+1: on the merge request though :p <fossy>stikonas: if you look at your bzip2 PR you'll see what i want you to do to make the blocks consistent <stikonas>ok, let me check, I haven't looked on GH <fossy>stikonas: ahh, very nice manipulation with addprefix/suffix <fossy>is there a need to redefine %.o: %.c? it's fine if there is but just interested <stikonas>actually, I don't even define INCLUDE_DIRS <stikonas>and no clean target until we get coreutils :( <stikonas>but I think by then we'll have autotools <stikonas>anyway, I'll try to test in live-environment too now <fossy>stikonas: remember we need to do coreutils pre autotools <fossy>because configure sripts use coreutils <stikonas>well, probably autotools can be compiled but I guess can't run configure <stikonas>hmm, probaly try to build as many as possible... <stikonas>hmm, I wonder if we need to build perl before autotools <stikonas>I think for autoconf we can do without it