IRC channel logs

<oriansj>2 char labels don't save much complexity relative to arbitrary length labels unless you require all labels to be exactly 2 chars

<stikonas>oriansj: I was thinking exactly 2 characters

<stikonas>but maybe we won't need that

<stikonas>anyway, I'll let rickmasters to investigate this in the future

<stikonas>(i.e. whether first smaller kernel can run hex1/hex2 or maybe will have built-in hex10

<oriansj>well 2 characters would need 256KB of memory just for the label table

<stikonas>oriansj: I only though about getting 256 with printable chars

<stikonas>i.e. to hex to combine into 1 byte...

<stikonas>but anyway, that might not be necessary

<stikonas>let's wait a bit until smaller seed kernel materializes

<oriansj>well 4KB of hex0 is within the margin of auditable

<stikonas>yes, it is

<stikonas>but we might be able to get it down a bit

<oriansj>it is far from ideal from an editing persective but shrinking builder-hex0 is probably something that can wait unless someone thinks it'll be really fun.

<stikonas>well, we already have builder-hex0-mini, it's just that it needs a reboot

<oriansj>but you gave me an idea

<stikonas>and then it will still only run hex0 for now

<stikonas>which is why I was proposing that we somehow get hex1 or hex2 there

<stikonas>so that then we could directly build builder-hex0.hex2

<stikonas>oriansj: what's the idea?

<oriansj>sparse executables loaded into a shared address space

<stikonas>by the way, I've started converting cc_x86.M1 to new GAS style defines...

<oriansj>then absolute addresses would rarely need to change and only relative jumps inside of a single function would ever need to be calculated.

<stikonas>that's the last file that was still using old defines

<oriansj>stikonas: thank you for that ^_^

<stikonas>(at least in stage0-posix, maybe there are some prototypes in stage0 itself)

<stikonas>maybe once I'm done we could make new relases...

<oriansj>definitely would deserve that

<stikonas>well, it's not really because of defines

<stikonas>but we have quite a few other things...

<stikonas> (https://github.com/oriansj/stage0-posix/blob/master/CHANGELOG.org)

<stikonas>argh, just noticed that org mode converts double dash -- into em dahs

<stikonas>I guess I would need =--file= instead of --file

<oriansj>hmmm,my idea isn't probably worth the complexity as BootOS could already be used to write builder-hex0's 4KB kernel to disk in 8 512byte blocks and with a minor tweak boot directly to it

<stikonas>well, the biggest practical problem is of course preparation of source disk...

<stikonas>nobody would really be able to type e.g. linux source code into BootOS prompt...

<stikonas>or even read all source code...

<oriansj>well we could write a network client which just gets a block number and a 512byte payload and use an external system to populate. We could write a trivial filesystem unpacker which reads the contents of a DVD/CD/Tape Drive/etc

<oriansj>I could port SET

<stikonas>in practice it's unlikely to be used by many people...

<stikonas>I suspect even if somebody wants to bootstrap, they would most likely limit themselves to kernel bootstrap with pre-prepared disk image with source code and seeds

<oriansj>well builder-hex0 in its current form is pretty darn close to something one could bootstrap from zero in a month

<oriansj>and you are right in that even stage0-posix-x86 is 403,588bytes of source

<oriansj>and that isn't even counting M2libc, M2-Planet or anything written in C

<oriansj>live-boostrap wouldn't even fit on a CD

<oriansj>so for the sake of simplicity, let us just assume there was a DVD or other media with contains of the source code. We either assume a custom filesystem (like builder-hex0 does) or a program which just converts from a more standard filesystem to the filesystem that builder-hex0 expects.

<oriansj>as there is no way to escape the need for some prepared disk/disc/tape image with the source code; but we could skip the seeds as they could be built or a different image (say on the bootsector of a floppy disk)

<oriansj>so let us assume the builder-hex0 kernel, the stage0-posix source code and BootOS on the bootsector of a 1.44MB floppy disk; then loading the rest of live-bootstrap can be done in programs written in C/assembly

<oriansj>as say an after.kaem script

<muurkha>nice

<oriansj>I can write a 512byte less program which bootOS can build and run; then one can audit the source code on the disk if they want

<oriansj>fossy: download-distfiles.sh should have #!/usr/bin/env bash instead of #!/bin/bash as it is more portable

<fossy>this is true

<fossy>doras: agreed with stikonas on all of that :)

<fossy>oriansj: fixed, thank you :)

<muurkha>oriansj: why don't you patch bootOS to load and save entire floppy tracks instead of single sectors?

<fossy>FWIW, i'm working on GCC 12 now

<stikonas>fossy: nice!

<fossy>seems to be rather simple so far :)

<stikonas>yeah, should be easy now

<stikonas>GCC 10 is not ancient

<stikonas>we already have new toolchain and all dependencies

<muurkha>then you have 18 sectors (9 KiB) for the stage of the bootstrap following bootOS

<fossy>i've enabled PIE and SSP as well, not very useful for live-bootstrap, but i suspect some distros will bork if they aren't available

<fossy>(for GCC 12)

<oriansj>muurkha: well we only need 4KB for builder-hex0 and the rest of the space can be used for tools which deal with the ability to manipulate the full floppy

<oriansj>so 18 sectors is plenty for our needs

<oriansj>hmm sysa/distfiles/heirloom-devtools-070527.tar.bz2: FAILED

<oriansj>(good thing I have a copy with the correct checksum available)

<muurkha>yeah, that's the kind of thing I was thinking of

<muurkha>I think you only need a few more bytes of code in bootOS to loop up to 18 when it loads or saves a "file"

<fossy>hey, rickmasters, (whenever you are here), with your branch fiwix-1.4.0-lb-e1bf632, should disk now work with Fiwix?

<fossy>in live-bootstrap that is

<stikonas[m]>Fiwix need real HDD though (so not USB storage)

<fossy>yes, not very problematic IMO

<fossy>i assume ata

<fossy>"IDE/ATA hard disk device driver." <-- yes

<river>i wrote a blog post about the potential for LLM based worms. I added a section about bootstarpping https://gist.github.com/rain-1/cc67caa8873ee38098aefb0f35f46014

<gforce_d11977>@oriansj: i read to often "floppy", which i'am used to - but keep in mind, that the path ideally should be similar for different platforms. think about risc-v or motorola 68k (very different). so an simple large drive (or pumped in via serial-port into memory) with the sources on it should be enough

<gforce_d11977>(to => too)

<stikonas>fossy: maybe we should metion this channel on live-bootstrap's README?

<oriansj>river: well the longer one thinks on AI, the more possible uncontrolled runaway appears.

<oriansj>gforce_d11977: i don't think we should hard code any single path to bootstrapping up to a POSIX kernel. And once we have a POSIX kernel, then the details of the hardware matter far less

<river>yeah

<stikonas[m]>We are probably still somewhat far from AI runaway but something like that is definitely plausible...

<rickmasters>fossy: I have to do more testing, but the Fiwix kernel is probably stable enough to run live-bootstrap off hard drive.

<rickmasters>The main blocking issue is that builder-hex0 cannot write an ext2 image large enough to build sysa.

<rickmasters>The HD geometry that qemu/SeaBIOS provides limits the legacy CHS int 0x13 interface to 1008MB which is not enough.

<rickmasters>There are different ways to resolve this.

<rickmasters>We have discussed this before but I'll repeat the options again as a reminder and so others can follow.

<rickmasters>If qemu/SeaBIOS would present a different geometry it would allow writing larger images with current builder-hex0.

<rickmasters>This is known as ECHS but so far I haven't been able to invoke this behavior/feature.

<rickmasters>But I haven't yet given up on this option because it may be the easiest and has few downsides.

<rickmasters>Another option is to modify builder-hex0 to use the LBA BIOS interface.

<rickmasters>LBA is how everything works these days, I think, and it should be fairly easy to do.

<rickmasters>But it means that hard drive/BIOS combinations prior to the mid 1990s will be left out.

<rickmasters>To me, this is acceptable and so it could be the best option in terms of cost/benefit.

<rickmasters>A third option is to change live-bootstrap sysa to keep within 1008MB.

<rickmasters>The low hanging fruit here is that linux-headers uses way more space than is necessary.

<rickmasters>I recently tried this option because it is currently the easiest way to test Fiwix.

<rickmasters>Even using a very small linux-headers tar file I still ran out of space building linux (which was surprising).

<rickmasters>Assuming I could get that to work, the downside is that it would leave sysa size-constrained.

<rickmasters>Part of me thinks that constraint is a good thing but I've sensed that you and stikonas are not fans of that constraint.

<rickmasters>The fourth option is to format a larger second partition after Fiwix boots.

<rickmasters>I think this is the option you favor and I'm OK with that option if it makes the most sense.

<rickmasters>But to be honest it is not my favorite.

<rickmasters>There is new work needed to transition to a second partition which further complicates live-bootstrap.

<rickmasters>And we end up with a useless 1G partition.

<rickmasters>Just changing builder-hex0 to LBA avoids this, for example, so I'd like to make sure we've reconsidered all the options first.

<rickmasters>The End. :)

<stikonas[m]>rickmasters: well, we could try to unify Linux and Linux headers tarballs too

<stikonas[m]>We now ship both

<stikonas[m]>(Headers are newer but without any real reason...)

<stikonas>rickmasters: in general I'm not really against size constraint

<stikonas>but we need to figure out where all the space or RAM is running out

<rickmasters>stikonas: I spent some time trying to use linux-4.9.10 for headers but ran into a problem.

<stikonas>I was not sure why linux kernel uses so much

<stikonas>rickmasters: we could go for newer kernel and older headers

<stikonas>meet somewhere in the middle

<stikonas>I think with new binutils, we should be able to build somewhat newer kernel

<rickmasters>Compiling dhcpcd-9.5.0 I got: /usr/include/linux/if.h:79: error: syntax error before numeric constant

<rickmasters>And I couldn't figure out why

<rickmasters>Just to test fiwix I rebuilt the linux-5.10.41 tar ball with just headers, saving 173MB.

<rickmasters>This should have been enough to fit sysa but I still ran out.

<rickmasters>Because the ram drive build fits in 1152MB

<rickmasters>It's a mystery I should probably track down.

<rickmasters>stikonas: The linux 4.9.10 tar file is 136M (M=1024*1024) and the untarred source is 763M so thats 900M right there.

<rickmasters>stikonas: now that I think about it, the linux build script deletes all source tar files before building,

<rickmasters>stikonas: so reducing the linux-headers tar file wouldn't help.

<stikonas[m]>Yeah, it's indeed a mystery where the space goes...

<stikonas[m]>We must be missing something

<aggi>i think recent kernel versions aren't desireable anyway

<aggi>checked the statistics, tarball-sizes, lines of code, and linux blew up since 2.4-2.6 transition, due to various features introduced (ACPI, USB, drivers), features which aren't necessary for bootstrappable

<aggi>and features which aren't necessary for a stable and scalable kernel

<rickmasters>stikonas: There are two mysteries maybe but I just resolved the one I was talking about.

<rickmasters>stikonas: The mystery was: If we can build sysa in 1152MB then why can't we build it in 1008 if we save 173MB from linux-headers?

<aggi>besides, full tcc-integration with linux-2.4 is much easier to; Kbuild system related, with linux-2.4 it was possible to avoid Kbuild entirely and use some tiny posix schell script to build the kernel (which crashes still)

<aggi>*too

<rickmasters>stikonas: And the answer is that the linux headers tar ball is deleted before linux is built so it was not part of the problem.

<stikonas[m]>Oh, indeed, we do some cleanups

<rickmasters>stikonas: The other mystery is: generally, why does linux take so much space that we can't build it (and the rest of sysa) in 1008MB?

<stikonas[m]>Still, what is using so much space...

<aggi>linux-2.4: 191M total

<aggi>stikonas[m]: recent kernels too ship a notable amount of firmware.bin clutter

<stikonas[m]>aggi: no, we don't use any of that

<stikonas[m]>We use aggressive deblob script

<aggi>stikonas[m]: sure. didn't know you already removed the firmware.bin junk

<rickmasters>stikonas: But is that really a mystery if the linux source is 763MB alone and then we add in all sysa artifacts?

<stikonas[m]>Yeah, perhaps...

<rickmasters>stikonas: I'm not sure about the deblob. Linux build currently outputs: cp: cannot stat `/sysa/distfiles/deblob-4.9': No such file or directory

<aggi>is there any reason to choose linux-4.x instead of 2.x ?

<rickmasters>stikonas: ... because it deleted it before starting the build.

<stikonas[m]>Hmm, maybe it regressed

<stikonas[m]>aggi: it's just more modern

<stikonas[m]>More hw will be supported

<rickmasters>Stupid question: What is deblob used for?

<stikonas[m]>2.x can't be deblobbed either

<aggi>well, for x86 and too x86_64 linux-2.x should suffice

<stikonas[m]>rickmasters: to remove binary blobs

<aggi>stikonas[m]: 2.4 didn't require deblobbing, because there wasn't any firmware.bin bundled with it, to my knowledge

<stikonas[m]>Like firmware from drivers

<stikonas[m]>aggi: at all?

<stikonas[m]>It would need checking, there might be a some in inline arrays

<aggi>there seems some support to load firmware, but it wasn't common iirc

<aggi>the WiFi stuff, which required it, at that time, used the ndis-wrapper magic instead

<rickmasters>stikonas: the linux builds script moves deblob before deleting distfiles/* but doesn't move it back like the tar file...

<stikonas[m]>OK, we need to fix it...

<aggi>of cause, there is less available filesystems with 2.x (if those weren't available out-of-tree, such as squashfs, don't know, or overlayfs/aufs etc.); y2038

<aggi>and, UEFI, however, if you depend on legacy-BIOS int13 maybe UEFI is out-of-scope, hence 4.x series less relevant

<rickmasters>stikonas: It may be working without being moved back to distfiles. I'm not sure where the cp error is coming from.

<stikonas>yeah, I'll take a look...

<rickmasters>stikonas: its coming from extract_file in helpers.sh which expects the file to be present as listed in sources

<rickmasters>stikonas: so it's probably working but maybe the linux build script should move it back to distfiles so extract_files can do the copy without erroring

<stikonas[m]>Yeah, I think you are right

<gforce_d11977>I just tested to build an 3.18.140 kernel with target "tiny", and it still needs ~1010 megabytes diskspace)

<gforce_d11977>(x86)

<gforce_d11977>idea: patch the compiler to delete compiled sourcefiles while working. or is there maybe a flag/switch for this?

<stikonas[m]>gforce_d11977: there wouldn't be such flag

<stikonas[m]>How would compiler know that this file won't be used

<stikonas[m]>In future steps...

<rickmasters>So, interesting development. If I download, compile, and use the latest SeaBIOS, I get the disk geometry I'm expecting to support larger disks!

<rickmasters>Oh shoot, I'm confused. The geometry looks the same with older SeaBIOS.

<rickmasters>I'm not sure what's going on. builder-hex0 should be working with this geometry and a 1152MB disk. So maybe I just have a bug I can fix...

<rickmasters>Wow, I must have really screwed up my prior investigation or attempts to make this work.

<rickmasters>It appears that I now have builder-hex0 able to write an 1152MB disk image. But I'm still testing and a bit confused so I may be wrong...

<rickmasters>A significant issue is that builder-hex0 takes a very, very long time to write an image of that size to disk.

<rickmasters>The code is not optimized and it takes so long that I thought it was frozen/crashed. But I could have sworn this just never finished in the past...

<rickmasters>I could have sworn I was always getting 32 heads for an 1152MB image, which is not enough.

<rickmasters>But I'm actually getting 64. I must have tested this incorrectly in the past. Yikes, I'm losing my mind.

<rickmasters>Anyway, I just completed my first successful kernel bootstrap build with Fiwix finishing sysa on a hard drive!

<rickmasters>fossy, stikonas: Sorry for my confusion. We can forget about LBA mode or reducing sysa to 1008MB.

<stikonas[m]>OK, that simplifies things!

<rickmasters>But if sysa gets bigger we might have to create a second partition.

<stikonas>well, source of sysa wouldn't get much bigger

<stikonas>well, we have sysa source limit (whatver builder-hex0 supports)

<stikonas>but we also have fiwix RAM limit

<stikonas>to keep all build files

<rickmasters>There is one last option. We could make builder-hex0 write the ext2 partition without preparing the full image in memory first.

<rickmasters>I'm not sure we're seeing any memory pressure on Fiwix yet.

<rickmasters>Once Fiwix runs off hard drive its just using memory for compiling and it has 1G for that. I'd be surprised if anything uses any where near that much.

<rickmasters>What I mean about builder-hex0: When it creates a large ext2 image the partition only has source in it to begin with, so most of it is empty.

<rickmasters>I think the only thing in the empty part of the ext2 image are root superblock copies. Those could be written manually.

<rickmasters>In other words, builder-hex0 could write up to 1152MB of source into an ext2 partition and then do what resize2fs does to expand it.

<stikonas[m]>Yeah, that's an option if we start hitting memory pressure but I guess not strictly required for now

IRC channel logs

2023-04-16.log