IRC channel logs

<oriansj>Googulator: well stage0 will always opt to support any and all architectures one may wish to use; so we may need to have PowerPC support in live-bootstrap in the future. But removing it today in a way that doesn't make it unreasonable to add it back in the future is entirely reasonable.

<oriansj>well, we always can improve unxz and we probably should.

<rickmasters>oriansj: he submitted a PR: https://github.com/oriansj/mescc-tools-extra/pull/22

<GoogulatorMobile>oriansj: isn't PowerPC a moribund architecture?

<GoogulatorMobile> https://android.googlesource.com/kernel/msm/+/android-7.1.0_r0.2/arch/powerpc/sysdev/micropatch.c is the file in question btw

<GoogulatorMobile>(sorry for the android link, I'm on mobile atm)

<GoogulatorMobile>I don't know if this is relevant at all for modern PowerISA

<GoogulatorMobile>Also, IIRC the latest generation of PowerISA has no hope of ever being booted blob-free - the DRAM controller requires a proprietary blob signed by Synopsys to be loaded before RAM can be initialized

<stikonas>uu, ugly...

<stikonas>GoogulatorMobile: by the way, sam_ suggested using another bc implementation

<stikonas>(that also doesn't depend on ed)

<stikonas>there is bc-gh

<stikonas> https://git.gavinhoward.com/gavin/bc

<GoogulatorMobile>I'd rather not remove ed though, it's really useful in interactive mode

<GoogulatorMobile>Since we have no other editor or pager

<GoogulatorMobile>In fact, I'd love to build ed earlier than we do now

<oriansj>GoogulatorMobile: who cares, armv7l is dead too but if anyone wants it supported to bootstrap through all the stage0 steps, I am game.

<oriansj>rickmasters: thank you, I never really check github.

<oriansj>GoogulatorMobile: merged

<GoogulatorMobile>Is there any other editor maybe that we can build before the first Bash?

<GoogulatorMobile>Ideally, when bash-2.05b is built, and the tty2 shell is spawned, we should already have some kind of editor & pager ready, to enable pausing for audit

<rickmasters>there are a million editors but I'll toss pico into the running as its from my alma mater.

<stikonas>that's basically nano now

<stikonas>isn't it?

<GoogulatorMobile>Nano would be ideal, but it requires ncurses :(

<GoogulatorMobile>I doubt we can build that using just kaem, make & tcc

<GoogulatorMobile>and meslibc

<GoogulatorMobile>As that's all we have when we bring up bash

<stikonas>no ncurses is stricky

<stikonas>you probably want at least musl

<GoogulatorMobile>Is there a heirloom-curses?

<GoogulatorMobile>wow, that sounded accidentally British

<stikonas>argh, but more non-redistributable stuff...

<stikonas>we should really move heirloom after basgh

<GoogulatorMobile>plan9 to the rescue

<stikonas>sorry after musl

<stikonas>GoogulatorMobile: there are some other random things that might just work

<stikonas>e.g. https://github.com/adsr/mle

<stikonas>oh this uses lua..

<stikonas>probably not good then

<stikonas> https://viewsourcecode.org/snaptoken/kilo/ ?

<stikonas>source is at https://github.com/antirez/kilo

<stikonas>hmm, probably won't build with meslibc...

<stikonas>#include <termios.h>

<oriansj>GoogulatorMobile: well there is always SET

<oriansj>one can write a version in M1 if one wanted or in C

<GoogulatorMobile>Isn't that dependent on knight?

<oriansj> https://git.sr.ht/~oriansj/stage0/tree/master/item/stage1/High_level_prototypes/SET.c

<oriansj>nope

<oriansj>it only depends upon fgetc and fputc

<oriansj>I was thinking of adding it with more.c and sin.c to mescc-tools-extra

<oriansj>as a sort of be able to detect utf-8 sort of invisible character attacks and fix them.

<oriansj>and we probably could make it an order of magnitude more efficient in regards to memory usage.

<oriansj>as a general rule, every thing I do; I try to ensure it can run on as many architectures as possible. (real, theory, hobby, etc)

<matrix_bridge><Andrius Štikonas> My gas bug got fixed :) https://sourceware.org/bugzilla/show_bug.cgi?id=30308

<oriansj>nice

<GoogulatorMobile>I wonder where "AT&T-syntax" x86 assembly even came from

<GoogulatorMobile>As far as I understand, it's basically VAX assembly syntax adapted to x86

<GoogulatorMobile>But why create a new syntax long after x86 became a thing?

<GoogulatorMobile>Intel syntax must have long existed by the time anyone attempted to port Unix

<GoogulatorMobile>Especially since before the 80286, there was no hope of porting Unix to the architecture, and that was the 3rd iteration of x86 already

<oriansj>well; x86 was an after thought for unixes

<oriansj>it was this weird, slow and cramped target

<oriansj>so they did the least effort work to port their assemblers to x86

<oriansj>it didn't matter that Intel's syntax was better

<GoogulatorMobile>Yeah, x86 was a microcomputer architecture that came well before anyone expected microcomputers to be a success

<GoogulatorMobile>But what I don't get is, whoever invented AT&T syntax must have already been intimately familiar with Intel syntax

<GoogulatorMobile>So why port VAX's assembly syntax instead?

<oriansj>AT&T syntax wasn't VAX's assembly syntax either

<GoogulatorMobile>Wasn't it based on Digital's official docs for the VAX?

<GoogulatorMobile>Or maybe even earlier, PDP-7 or PDP-11

<oriansj>it was the PDP-7's assembly syntax

<GoogulatorMobile>But VAX was derived from that, wasn't it?

<oriansj>which are extremely primitive compared to the VAX's assembly which had proper macros and type checkers.

<oriansj>different family line within DEC

<GoogulatorMobile>AFAIK early VAX, or "VAX-11" as DEC liked to call it, was PDP-11 with a virtual address extension

<GoogulatorMobile>(& a 32-bit address space)

<oriansj>yes but PDP-11 and PDP-7 are different family lines

<oriansj>pdp-7 was 18bit; pdp-11 was 16bit

<GoogulatorMobile>Oh, so it wasn't a direct progression from PDP-8->9->10->11

<oriansj>pdp-7's assembler came from the pdp-4; pdp-11's assembler came form the pdp-8

<oriansj>DEC's product lines often had very little relation.

<oriansj>For example the PDP-6 was the parent to the PDP-10 which was a 36bit machine

<GoogulatorMobile>So PDP-8 wasn't an upgraded PDP-7

<oriansj>PDP-8 was a 12bit machine

<GoogulatorMobile>So using PDP-7 syntax was established in Unix before PDP-11 even became a thing

<oriansj>nope.

<oriansj>PDP-11s were available as of: 1970. Unix came out 1971

<oriansj>(unix development started in 1969)

<oriansj>They wanted a GE 645 mainframe and got stuck with a computer that no one wanted.

<oriansj>it was all in assembly language until 1973

<oriansj>so changing out assembly syntax would have been impossible

<oriansj>and by the time, it was ported to C (which had a boatload of assembly too); the base had grown to the point changing assembly syntax was also another non-starter.

<oriansj>It wasn't successfully ported to another architecture until 1977

<GoogulatorMobile>And I guess today it's perpetuated by FSF's desire to do things differently than Microsoft

<oriansj>and that port used the same syntax

<oriansj>well no, FSF used the standard for Unix and supports other assembly standards as options.

<oriansj>it is why one can write .intel_syntax and just use intel syntax when working in assembly

<GoogulatorMobile>Today, they do support Intel syntax, but as I see it, there's a strong preference for AT&T

<oriansj>basically people know it and opt to use it despite a better option being a single line away. It is like how people tend to use languages for problems for which they are bad fits.

<oriansj>people don't like learning new syntax and will opt to stick with what they know and thus that is what the next generation learns too.

<GoogulatorMobile>Anyway, this all came from trying to figure out what that weird instruction in https://github.com/fosslinux/live-bootstrap/blob/master/steps/linux-4.9.10/patches/bad-asm.patch means

<GoogulatorMobile>movw 68(%esp), %di

<GoogulatorMobile>Or in Intel syntax, mov di, dword ptr WTF(esp, 68)

<oriansj>or mov di, [esp+68]

<GoogulatorMobile>Is it just that?

<oriansj>trivial to check

<GoogulatorMobile>Every AT&T assembler I tried just considered it a syntax error

<GoogulatorMobile>I was guessing it's mov di, [esp] with a prefix of 0x68

<oriansj>well movw is move word, a word is 16bits %esp is a memory reference and 68 is the offset from the memory reference

<GoogulatorMobile>Similar to how 0x66 is register size override

<oriansj>well no, prefix encoding isn't involved here

<GoogulatorMobile>oriansj: what's your opinion on the deblob situation?

<GoogulatorMobile>(of course, in the end, it's up for fossy to decide)

<oriansj>personally; I am using an x200 with a libreboot kernel running vanilla guix with no substitutes. So I'm fine giving up drivers personally but I also respect that some people use vanilla linux and install blobs that they want/need. So my perspective is if possible let the users choose for themselves what they want.

<oriansj>^kernel^firmware^

<oriansj>make the default whichever you the developer is willing to be responsible for supporting.

<oriansj>but be kind to those who like the old path and create a method for them to opt in to that alternate path; ensure it works when you do that change and after it will be up to those that care about that alternate path to maintain and support it.

<GoogulatorMobile>That's not really the issue

<oriansj>the use the linux-libre deblob process or just delete the folder (and a couple of files with blobs in them) and be done.

<GoogulatorMobile>Rather, it's about deblobbing to prevent blob-mediated compromise of the trusted bootstrap environment, vs. deblobbing & removing the firmware loader mechanism (and all the other things the script does or tries to do) for FSF Free Software compliance

<oriansj>agreed

<GoogulatorMobile>4.9 unfortunately still has the /firmware directory, which is hard to remove (properly) without a debug script

<GoogulatorMobile>I'm hoping to upgrade to 4.14, where /firmware is gone, and removing the 3 drivers that actually embed blobs masquerading as code via a patch, and not touch the FSFLA deblob scripts at all

<GoogulatorMobile>4.14 is the last LTS release that still supports gcc 4.0.4

<oriansj>yep

<GoogulatorMobile>its support period ended a month ago

<GoogulatorMobile>Unfortunately deblob-4.9 does things that I really disagree with

<oriansj>well good thing live-bootstrap can be run in isolated environments.

<GoogulatorMobile>Starting with removing drivers just because they have the ability to optionally do device firmware updates is requested by the user

<GoogulatorMobile>And going as far as to inject text into documentation about why saying "GNU/Linux" instead of "Linux" is important

<oriansj>I understand

<GoogulatorMobile>Which is totally irrelevant for an automated bootstrap where no one is gonna read that documentation

<oriansj>(and there is no point in building that documentation in the first place)

<GoogulatorMobile>especially when the bootstrap process involves a liberal mixture of GNU and non-GNU code

<GoogulatorMobile>it's mostly documentation that doesn't need to be bult

<GoogulatorMobile>Just text files

<GoogulatorMobile>The script even tries to replace the Tux framebuffer logo because it's "giving too much credit to Linux where credit is actually due to GNU" - in 4.9, it fails

<oriansj>GoogulatorMobile: I understand your perference.

<oriansj>I just merely suggesting a figleaf, set your preference to the default but set things such that if an environment variable like FSF-COMPLIANT is set that it'll do the linux-libre deblob script instead. That way the user choice isn't eliminated and we get a good default because I know you'll do a good job on the deblob.

<oriansj>we can alway remove that variable and path if no one steps up to maintain it.

<oriansj>and because a single environment variable check is cheap and easy to do, it ends any potential argument from those who might not agree with your recommended default.

<GoogulatorMobile>It's a bit more complicated than a single environment variable

<GoogulatorMobile>2 separate kernel configs need to be maintained

<GoogulatorMobile>And potentially 2 separate sets of checksums for packages built after the Linux kernel

<GoogulatorMobile>Plus, a way to pass all of that to a bare metal environment, where the deblob script makes the biggest difference

<oriansj>but you only need to maintain 1; the checksums for user space packages should not change between the two different paths as they are both Linux and yes the linux-libre deblob will probably break very shortly after but that isn't your problem and worst case someone else steps up and live-bootstrap gains another maintainer.

<oriansj>hence a figleaf and not another burden for you.

<oriansj>or no one steps up, it gets dropped due to lack of support and we all move on with your new default path because ultimately: who does decides.

<GoogulatorMobile>Unfortunately user space package checksums do change as a result of deblobbing using the FSFLA scripts

<GoogulatorMobile>Not because of deblobbing per se, or even removal of the firmware loading mechanism, but because of all the other changes it makes

<oriansj>the name change I am guessing

<oriansj>which builds tend to capture (if the proper flags are not set to stop that)

<GoogulatorMobile>Yeah, uname output is changed

<oriansj>so a handful of reproducible build tweaks may be required.

<oriansj>which is probably a good thing.

<GoogulatorMobile>And all of this is just to take choice away from.the user

<GoogulatorMobile>Given that it's not actually blobs that are being removed anymore, just code for loading blobs from files if they are presen

<oriansj>well, much like how nonguix is not to be ever discussed on #guix

<GoogulatorMobile>(and they're not, for live-bootstrap doesn't include linux-firmware - intentionally and rightfully, since IMO it would genuinely compromise trust and provide a path for Karger-Thompson retroviruses to get in)

<oriansj>purity isn't a requirement, it is the default and if someone wants to do the stage0 steps on Windows 11; let them choose their own roots of trust.

<oriansj>The defaults are us collectively doing our best to make a secure, trust-worthy system and bootstrapping it entirely from human written source code.

<oriansj>if people trust Linux or BIOS or UEFI or need to make their own hardware from sand; we will give them the option.

<oriansj>but we need only be responsible for maintaining what we care about.

<oriansj>and we will be collectively more secure by having more diverse platforms and roots that all converge on the exact same results.

<Guest991>HI, how to create live-bootstrap ․img on bare metal

<samvel_harutyuny>HI, we successfully finished bootstrapping with chroot. Now It will be fine if we would be able to run it on bare metal. But instructions seem not enough. Can you help us?

<fossy>rickmasters, most interesting, thanks!

<fossy>stikonas, Googulator: firmware loading support is quite OK with me; it's just pregenerated non-firmware things I'm more concerned about

<fossy>firmware is currently unavoidable tbh, and sufficient hardware to avoid firmware blobs is fairlyyy rare

<fossy>Googulator: regarding update, it should be possible, now that binutils is updated

<fossy>there was a historical reason for 4.9 i think it was binutils, but i'm pretty sure all 4.14 criteria are supoprted now

<fossy>FSF compliance in that regard (deblobbed kernel) is not something I'm particualrly attached about for the bootstrap

<fossy>also - if you're using free hardware then none of the non-free stuff will be used anyway

<mid-kid>How do I extract the init.img built with ./rootfs.py -q?

<mid-kid>From what I read it's using a very tiny boot sector that reads the files in the sectors that come right after

<mid-kid>and builds them directly

<mid-kid>but I can't really map where the final filesystem is located

<mid-kid>nevermind it's in external.img

<fossy>:)

<stikonas>mid-kid: there is also --bare-metal that will create just that image

<fossy>I think this is probably Guest<random numbers>, but I got an email from them with finally some debugging stuff

<fossy> https://greenhosting.am:333/db/get/video5359835051846550769.mp4?id=54b8b2df-1a4c-4e9d-9d3b-6e7b2ca1faf8-b8551d3f-98a5-4c69-b2f0-e0b0271e5d45 this is the video I was sent of the behaviour

<fossy>not one i've seen before

<fossy>Googulator any thoughts?

<fossy>i have similar hardware available to me so i'll try and reproduce

<stikonas>hmm, fails when building kaem?

<fossy>appears so

<fossy>it's very odd, haven't seen anything like it

<Googulator>Could be a memory map issue

<Googulator>Was this done with the e820 patches already included?

<Googulator>though it probably won't help, because e820 isn't actually used by builder-hex0

<Googulator>it's just saved for later use by Fiwix

<Googulator>...it's indeed a memory map issue

<Googulator>this is the e820 map I got in my PMs:

<Googulator>[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable

<Googulator>[ 0.000000] BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x0000000000059000-0x000000000009efff] usable

<Googulator>[ 0.000000] BIOS-e820: [mem 0x000000000009f000-0x00000000000fffff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a685ffff] usable

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000a6860000-0x00000000a6860fff] ACPI NVS

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000a6861000-0x00000000a688afff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000a688b000-0x00000000ab403fff] usable

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000ab404000-0x00000000ab779fff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000ab77a000-0x00000000ab7b1fff] ACPI data

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000ab7b2000-0x00000000abe16fff] ACPI NVS

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000abe17000-0x00000000ac42afff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000ac42b000-0x00000000ac4fefff] type 20

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000ac4ff000-0x00000000ac4fffff] usable

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000ac500000-0x00000000afffffff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved

<Googulator>this is on a machine with 16GB physical RAM installed

<Googulator>and regular Linux kernels somehow see all of that, despite reporting that e820 map

<Googulator>either way, I don't think this is a system we can support for kernel bootstrap with the current approach

<Googulator>the memory map looks similar to this: https://bbs.archlinux.org/viewtopic.php?id=275750

<Googulator>that 0x58000:0x58fff memory hole seems to be a feature of some TianoCore-derived UEFIs

<fossy>ah, and builder-hex0 is writing to 58000-58fff?

<fossy>where is builder-hex0's expected memory map?

<Googulator>probably not

<Googulator>[ 0.000000] BIOS-e820: [mem 0x00000000a6861000-0x00000000a688afff] reserved

<Googulator>this is the problematic region

<Googulator>but the 58000 hole is a useful identifier for this kind of system, which apparently is a UEFI

<Googulator>(even though Guest991 insisted to me it's a BIOS)

<fossy>right, that makes more sense

<fossy>lol

<Googulator>so IMO the way forward for these systems is posix-runner

<Googulator>just treat them as Class 3

<fossy>yeah agreed

<Googulator>CSM is technically present, but too broken for bootstrapping

<Googulator>meanwhile, prototyping the kernel upgrade to 4.14.336 - had to add 2 patches so far, but we already build all the way up to drivers/gpu/drm, where I had my last failure

<Googulator>hmm, is there a reason why we build findutils with tcc, after gcc is already built?

<matrix_bridge><Andrius Štikonas> Probably fossy or me reordered some packages

<matrix_bridge><Andrius Štikonas> And didn't update compiler

<Googulator>so it's not like "it crashes the first GCC build"

<fossy>i have no recollection of any reason that would be, so i'd take it as a "no, there is no reason"

<Googulator>hopefully with gcc, we can upgrade to a version of findutils that actually supports musl (right now, we have "# Musl is not recognized, pretend to be uClibc")

<matrix_bridge><Andrius Štikonas> Yeah, sound good

<oriansj>Guest991: do you mean creating the initial live-bootstrap image? from no running system or from a running system?

<oriansj>samvel_harutyuny: So you are looking for more detailed steps for bare hardware?

<mid-kid>I was just able to build LFS based on a completed live-bootstrap, without modifications or messing around with installing python and whatnot

<mid-kid>Major kudos for that

<mid-kid>Last time I tried this was some years ago so maybe it's been like this for longer "^.^

<Googulator>mid-kid: great news!

<Googulator>IMO next step would be building a Gentoo boot image & stage3 tarball on the LFS system, achieving a fully bootstrapped Gentoo

<mid-kid>I'm gonna do this right now by using the LFS system as a hopping point, primarily to avoid dealing with cross-compilation

<mid-kid>chapter 5 and 6 of LFS is essentially making a cross-compiler and cross-compiling a bunch of tools to make a minimal system from which to build the rest

<mid-kid>*ideally* you'd just build a cross-compiler and then use portage to build a stage1 system with that, but it's a bit less trivial

<mid-kid>I should mention that I'm using the chroot method for everything here, I don't really have a machine to put aside for this...

<Googulator>Please document your journey somewhere, and we will try to replicate it on bare metal.

<Googulator>You can also use qemu mode to test with kernel bootstrap, without needing dedicated hardware.

<mid-kid> https://mid-kid.root.sx/git/mid-kid/bootstrap/src/branch/master/gcc/notes/gentoo/gentoo_notes.txt

<mid-kid>this was my adventure back in 2020

<mid-kid>though it wasn't live-bootstrap but rather a set of scripts I had based on the guix bootstrap, nowhere near as good as what live-bootstrap is today

<mid-kid>I was wordier back then than I care to be right now, but I'll document this expedition as well :+1:

<matrix_bridge><Andrius Štikonas> mid-kid: really nice! I would like to replicate this too but I only have UEFI machine to spare

<matrix_bridge><Andrius Štikonas> So need to fix mescc->tcc step on amd64 first

<matrix_bridge><Andrius Štikonas> And then boot into fiwix...

<mid-kid>UEFI can't boot 32-bit systems? that sucks

<matrix_bridge><Andrius Štikonas> mid-kid: it can

<matrix_bridge><Andrius Štikonas> The problem is not uefi itself

<matrix_bridge><Andrius Štikonas> The problem is that posix-runner does not support paging

<matrix_bridge><Andrius Štikonas> So it can only run code that is position I dependent

<matrix_bridge><Andrius Štikonas> Independent

<matrix_bridge><Andrius Štikonas> So that means amd64

<matrix_bridge><Andrius Štikonas> once we build 32 bit fiwix we can boot into it

<matrix_bridge><Andrius Štikonas> mid-kid: well, another option would be to shut down uefi services

<matrix_bridge><Andrius Štikonas> And conver POSIX runner onto full kernel

<matrix_bridge><Andrius Štikonas> But that also need a lot of work

<matrix_bridge><Andrius Štikonas> We need to set up framebuffer and fonts

<Googulator>yeah, because in the UEFI model, nothing guarantees that the display output hardware has a native text mode

<Googulator>ar: drivers/input/built-in.o: Too many open files in system

<Googulator>...damn

<Googulator>but at the same time, it got to the end of drivers build - just needs a higher open files limit in Fiwix

<Googulator>this is with kernel 4.14

<sam_>mid-kid: nice!

IRC channel logs

2024-02-15.log