IRC channel logs

2021-01-17.log

back to list of logs

<stikonas>strange...
<stikonas>something must be different...
<stikonas>well, I only tested in chroot
<stikonas>maybe because I had to inject busybox into /after/bin/sh
<stikonas>hmm
<stikonas>but anyway, we still need to somehow properly reach bash...
<stikonas>fossy: I was looking a bit at musl now, any idea if this error is fixable? src/signal/i386/sigsetjmp.s:8: error: invalid displacement https://gitlab.com/giomasce/musl/-/blob/master/src/signal/i386/sigsetjmp.s
<stikonas>looks like it chokes on jecxz 1f
<stikonas>but my assembly is not strong enough to fix this :(
<stikonas>I guess 1f is offset for jumping
<siraben>OriansJ: did my email properly send to the mailing list? I don't use mailing lists often so I may have just replied to you
<fossy>stikonas: iirc it might have not worked under busybox but did under kaem
<fossy>siraben: sorry my assembly is not very good either
<stikonas>fossy: for now I replaced that jecxz with I think jm and then it compiles
<fossy>stikonas: interesting
<stikonas>but I don't know if it breaks something
<stikonas>actually jecxz with jz
<OriansJ>siraben: yes it did, you can see it quite clearly in the archive https://www.freelists.org/post/bootstrappable/wipfullsourcebootstrap-from-a-357byte-hex0-to-hello,20
<siraben>OriansJ: Oh, thanks.
<stikonas>fossy: so I think it might be possible to get musl compiled if that helps us at all...
<stikonas>(I don't have any proper scripts yet, most of my testing was manual)
<fossy>stikonas: that would be useful....
<fossy>yeah it would
<fossy>more complete than mes c library
<stikonas>fossy: so for now I did the following
<stikonas>added https://gitlab.com/giomasce/musl/ to my chroot
<stikonas>configured it with CC=tcc AR="tcc -ar" sh ./configure --target=i386
<stikonas>but I guess without shell we can still pre-create configuration include file
<stikonas>it's fairly simple
<stikonas>then removed src/complex src/fenv, patched that sigsetjmp.s file
<stikonas>and ran make CFLAGS+="-DSYSCALL_NO_TLS"
<fossy>stikonas: is it autotools or no
<stikonas>this needed some manual help due to missing coreutils
<stikonas>fossy: no, it's not autotools
<stikonas>manually written configure file
<fossy>ooh
<stikonas>which creates about 20 line config.mak file with variables
<stikonas>CFLAGS, prefix, etc...
<fossy>what if we used heirloom bash
<stikonas>if we can build it...
<fossy>yeah
<stikonas>it might depend on those wchars...
<stikonas>but that configure script is not essential
<fossy>grr wchar
<stikonas>well, musl has it
<fossy>is it still segfaulting
<stikonas>what is segfaulting?
<stikonas>fossy: config.mak is like this https://paste.debian.net/1181439/
<fossy>stikonas: yacc was segfaulting right
<fossy>or just failing?
<stikonas>well, linking was failing
<stikonas>some missing symbols
<stikonas>I probably need to intall musl more properly...
<stikonas>although, I think I was able to recompile libtcc.a and tcc
<stikonas>and then use it to recompile musl again
<stikonas>but I failed to compile hello world...
<stikonas>so some more investigation is needed...
<stikonas>but I think it's possible
<stikonas>but this is unpatched yacc...
<stikonas>anyway, I'm going to bed now...
<deesix>OriansJ, yt_, I'm switching more archs to M2libc in M2-Planet tests (just saying to avoid duplicated work).
<bauen1>fossy: currently the only syscall supported is exit, i'm working on a proper process model and the open syscall, bash requires quite a few different syscalls (mostly related to process groups) so i'm not sure
<bauen1>fossy: if you can give me a list of syscalls (normally) used i can tell you if it is feasible to implement without bloating the kernel
<fossy>bauen1: okey
<fossy>yeah it uses process groups quite siginificantly from strace
<fossy>also the rt_* family (idk what that is)
<fossy>then mostlry read and write
<fossy>not very feasible
<fossy>stikonas[m]: i see two options for flex/bison
<fossy>1. musl (full) or just src/multibyte make it into a library then nbs
<fossy>2. use older versions of flex
<bauen1>fossy: it kind of depends on how much you can patch out (while retaining critical functionality) tbh, implementing stub syscalls that just return -EPERM (or similiar) is also very easy
<bauen1>fossy: but yes, it will be a bit before i start to think about process groups, job control or controlling terminals
<stikonas[m]>I think I'll try to get full musl first...
<stikonas[m]>We don't really need interactive terminal for bootstrap kernel
<stikonas[m]>fossy: since older version of heirloom flex might not be able to bootstrap flex. Although, if full musl fails, that's an option
<stikonas[m]>Hmm, for musl i need at least mkdir and rm...
<malina>stikonas[m], kinda funny, I am building/bootstrapping my musl lfs now, (4 years since I did the original glibc one) and I came across some of your comments or repo and so on. where you mentuoenc heirloom. I happened to compile it myself just some hours ago but for different reasons. I didn't have tr even so kernel headers wouln'd install :D
<malina>I am trying to keep the toolchain set as minimal as possible so I grabbed the tr from heirloom and compiled that :)
<malina>the flex however, failed iirc. but I didn't realyl poke around much in it [heirloom] right now.
<malina>being totally new to musl so. but by the time I return to the heirloom , I am sure you troopers are already far ahead again
<stikonas[m]>Can't be me, I've never heard about heirloom even 3 days ago
<malina>OH, it seemed from above you are giomarco or something?
<malina>or is that fossy perhaps?
<malina>suckless/heirloom/bb/toybox , all KISS.
<malina>[02:21:08] <stikonas> added https://gitlab.com/giomasce/musl/ to my chroot <- I guess this was it, and I misunderstood lol :)
<stikonas[m]>Well, by added I meant just cp -r
***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<malina>ya, I ddn't check, was just saw 'giomasce' then came across it later today myself bychance, since im doing finally a musl strap. nvm.
<fossy>stikonas[m]: i mean older versions of GNU flex
<stikonas[m]>Hmm, maybe...
<stikonas[m]>Anyway my testing yesterday showed that musl might be feasible
<stikonas[m]>Though, it is quite a bit of work
<stikonas[m]>And I need to understand implications of jecxz better...
<Hagfish>i know javascript tooling written in rust isn't an obvious topic for this channel, but i'm starting to be impressed with how Deno is replacing NodeJS
<Hagfish>they had an interesting failure mode in terms of bootstrapping, which they hint at in this blog post: https://deno.land/posts/registry2
<Hagfish>it's something that is often taken for granted, but somehow they had managed to get wrong in their v1.0 design
<Hagfish>basically, all dependencies would be defined as just a package name, without a specific version, and even when you specified a version, there's no guarantee that the code associated with that version wouldn't change
<Hagfish>complete madness
<Hagfish>but now they guarantee: "All files served from the registry are immutable. They can not be changed or removed by the package author."
<Hagfish>(at the cost of centralisation, although in principle it should be possible to host your own registry)
<Hagfish>and they plan to improve things further: "Give all modules a score based on ... are dependencies pinned to specific version"
<Hagfish>it'll be interesting to see how well packages deal with security vulnerabilities in their (transitive) dependencies
<Hagfish>i don't think they have an equivalent of "npm audit" yet, or a culture of packages updating their dependencies to non-vulnerable versions
<Hagfish>there's also the question of whether packages backport security fixes to old versions, or if the solution for a vulnerability in version 3.x is to upgrade to version 8.y
<Hagfish>maybe next year it will be clearer what the engineering culture is like in that ecosystem, and whether the packages available are enough to base any serious projects on
<stikonas>ok, finally got some more progress on live-bootstrap, although, on slightly orthogonal thing. grep seems to build fine
<stikonas>(and somewhat newer than guix in commencement.scm, got 2.4 working rather than 2.0)
<stikonas>bauen1: so maybe then we build bash later, not in bootsrap kernel. but do you know if it would be possible to easily reboot into newly compiled kernel?
<stikonas>bauen1: something like kexec?
<Profpatsch>Hagfish: are they just making all the golang mistakes all over again?
<Profpatsch>Combined with all npm mistakes I assume
<gforce_d11977>fossy: are you there? i have a proposal for CI-integration of live-bootstrap: question: i will introduce an ARG1 of 'CI', which does the things -> (./rootfs CI) - are you OK with this approach?
<gforce_d11977>(I'm preparing the pull request)
<Hagfish>Profpatsch: yup, it seems like it, except they are also trying to fix them. maybe that's easier with a newer ecosystem, although it would have been easier not to make the mistakes in the first place
<gforce_d11977>fossy: i have problems starting 'init' from your live-boot in a x86_64 qemu-instance. "file init" says it is "ELF 32-bit LSB executable, Intel 80386, version 1", maybe this is the underlying problem?
<gforce_d11977>fossy: Run /init as init process -> Failed to execute /init (error -8)
<stikonas>gforce_d11977: hmm, it works fine here
<stikonas>init is indeed 32-bit variable
<stikonas>s/variable/executable
<stikonas>(it's kaem seed)
<gforce_d11977>stikonas: i "compiled" the AMD64 hex0 kaem and will test (fingers crossed!)
<stikonas>well, I only tried AMD64 manually on mescc-tools-seed
<stikonas>not in live-environment
<stikonas>x86 binary is smaller anyway
<stikonas>and x86 should just run on x86_64
<stikonas>fossy: grep is ready https://github.com/fosslinux/live-bootstrap/pull/15
<stikonas>I inserted it before bash
<stikonas>since bash is so heavy on syscalls, it might be easier later to do more stuff earlier on bootstrap kernel if we insert it before bash...
<gforce_d11977>wow, now further progress on QEMU, there seems to be more wrong than just init. init now works, but i get:
<gforce_d11977>Run /init as init process +> ./bootstrap-seeds/POSIX/x86/kaem-optional-seed mescc-tools-srandom: fast init done
<gforce_d11977>eed.kaem.run
<gforce_d11977>Subprocess error
<gforce_d11977>ABORTING HARD
<gforce_d11977>Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
<stikonas>well, non-zero exit code
<stikonas>random: fast init done is unrelated
<stikonas>that printk from kernel that just happened to be printed at the same time
<stikonas>gforce_d11977: I think you need to look at why your qemu fails to run 32 bit binaries...
<stikonas>or maybe your kernel?
<stikonas>that's probably more likely
<gforce_d11977>stikonas: the error is not on my side 8-)
<gforce_d11977>stikonas: when i just run on my laptop: bootstrap-seeds/POSIX/x86/kaem-optional-seed mescc-tools-seed.kaem.run
<gforce_d11977>i get: +> ../bootstrap-seeds/POSIX/x86/kaem-optional-seed mescc-tools-seed-kaem.kaem
<gforce_d11977>Subprocess error
<gforce_d11977>ABORTING HARD
<gforce_d11977>(laptop is x86_64)
<gforce_d11977>in theory 32bit binaries should not be a problem, i know...
<gforce_d11977>the error is strange and i a chroot the live-bootstrap works for me, but the real test (a minimal x86_64 qemu) should also work
<gforce_d11977>fossy: do you test in qemu or chroot? for the qemu-case: what is your kernel .config?
<stikonas>I think both fossy and I use both qemu and chroot
<gforce_d11977>stikonas: your kernel .config?
<stikonas>one moment
<stikonas>but first check if you have enabled
<stikonas>CONFIG_IA32_EMULATION
<gforce_d11977>no, i have not enable CONFIG_IA32_EMULATION
<stikonas>ok, that's the reason
<gforce_d11977>ok...will do
<stikonas>you need it to run 32 bit binaries
<stikonas> https://cateee.net/lkddb/web-lkddb/IA32_EMULATION.html
<stikonas>You should likely turn this on, unless you're 100% sure that you don't have any 32-bit programs left.
<gforce_d11977>stikonas: thank you, i did not know this, i was always thinking, the CPU handles it automatically
<stikonas>well, running executables involves both kernel and CPU
<stikonas>so both have to support it
<gforce_d11977>(*IT*DOES*RUN*DANCING*ON*THE*TABLE*)
<gforce_d11977>Tasks state (memory values in pages):
<gforce_d11977>[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
<gforce_d11977>[ 72] 0 72 161 125 24576 0 0 kaem
<gforce_d11977>[ 73] 0 73 402 365 28672 0 0 kaem
<gforce_d11977>[ 92] 0 92 1108 1072 28672 0 0 kaem
<gforce_d11977>[ 108] 0 108 33072 14238 143360 0 0 vm
<gforce_d11977>oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=vm,pid=108,uid=0
<gforce_d11977>8-))) how much RAM should be enough?
<stikonas>well, VM requests 8 GB
<stikonas>I don't think we use all 8 for now
<stikonas>we are building minor stuff...
<stikonas>gforce_d11977: how much RAM do you have?
<gforce_d11977>stikonas: 128mb
<stikonas>well, RAM is needed to store all sources...
<stikonas>oh
<stikonas>that's not enough
<stikonas>where did you get a box with 128 MB ram?
<gforce_d11977>(for QEMU) - i have to raise this...
<stikonas>oh
<stikonas>yeah, I run it with 8
<stikonas>compressed initramfs is already 13 MB
<stikonas>then you need to unpack it + kernel
<stikonas>and most of your ram is already used up before you even start bootstrapping
<gforce_d11977>wow:
<gforce_d11977>+> cd tcc-0.9.26 +> catm config.h +> /after/bin/mes --no-auto-compile -e main /after/bin/mescc.scm -- -S -o tcc.s -I /after/mes/lib -I /after/mes/include -D BOOTSTRAP=1 -I .
<gforce_d11977>system-error: ("No such file or directory" . "tcc.c")
<gforce_d11977>assert fail: ERROR
<gforce_d11977>mes[559]: segfault at 0 ip 00000000010187d4 sp 00000000ffbc3454 error 6 in mes[1000000+1b000]
<gforce_d11977>Code: 83 c4 04 85 c0 8b 45 08 89 45 fc b8 00 00 00 00 89 45 fc 85 c0 b8 00 00 00 00 bb 00 00 00 00 50 89 d8 8b 5d fc 01 d8 89 c3 58 <88> 03 80
<gforce_d11977>Subprocess error 11
<gforce_d11977>during live-bootstrap. is this known in that early stage?
<bauen1>stikonas: yes, you could implement something like kexec in myunix, or you could implement module loading, so you could have the core kernel very simple and then as you get tinycc available build + load more complex modules (like an journaled ext4 rw driver)
<bauen1>stikonas: it would probably be easiest to make myunix (or a similiar bootstrap kernel) capable of (cross) compiling (and all dependencies necessary) the linux kernel, glibc, bash, gcc, and other toolchain tools you need
<bauen1>stikonas: so you somehow get to myunix, then build the intermediate toolchain (in stages, starting from hex0 or something similiar) up until you receive a minimal linux system from which you can continue bootstrapping other things with potentially more complex feature requirements against the kernel (maybe utilising containers and other features if you want to)
<bauen1>stikonas: this minimal linux image could then serve as the base for distributions to compile their own distribution-specific (patched) toolchain
<bauen1>i personally would really like allowing modules to be loaded into the myunix kernel, care must be taken to ensure that there is no cross-contamination, but makes bootstrapping the bootstrap-kernel easier
<bauen1>while you can write a rw ext4 driver in (pseudo) assembly, it isn't very fun and probably harder to verify that it does the job correctly
<bauen1>in the end a modular to micro kernel might be the best thing for the bootstrap-kernel
<stikonas>hmm, yeah, module system sounds nice
<stikonas>we can probably reach tcc and some basic utils with fairly few syscalls
<bauen1>do keep in mind that myunix is an experiment of mine, so has some weird C code, e.g. using enums as error values, so while it can be compiled using tcc it might not be a good fit for bootstrapping
<stikonas>well, we'll see...
<stikonas>at first we can try to use some slimmed down linux kernel...
<stikonas>with few modules
<gforce_d11977>stikonas: my qemu-instance runs with an 'allnokonfig' kernel + 6 options (e.g. ELF + 32bit support 8-))
<gforce_d11977>stikonas: so really slimmed down...
<stikonas>ok, it might be worth documenting which options are needed
<stikonas>how big is the kernel?
<gforce_d11977> https://github.com/bittorf/GNU-mes-documentation-attempt/blob/main/kernel.bin
<gforce_d11977>but it still has procfs,sysfs and printk support, so it can be around 300k smaller
<gforce_d11977>it's latest 5.10.8
<gforce_d11977>(but when it comes to small kernels, we can also boot an early 2.6kernel which boils down to ~300k
<bauen1>that's quite good, very close to ~280kb myunix currently uses if you disable debug informations
<stikonas>linux kernel is probably compressed
<fossy>gforce_d11977: please do document your config
<fossy>if you could pastebin it I would be very ahppy
<fossy>But it sounds like it is not working?
<fossy>gforce_d11977: can you elaborate on how the CI would work?
<fossy>I have a CI PR done.'m
<fossy>stikonas: any progress on musl? all good if not
<stikonas>fossy: kind of but indirect
<stikonas>fossy: first of all I have grep (but that is a bit orthogonal)
<stikonas>fossy: now I'm working on coreutils
<gforce_d11977>fossy: https://github.com/bittorf/GNU-mes-documentation-attempt/blob/main/build_qemu_image.sh
<stikonas>which will help with musl
<stikonas>fossy: since musl build needs rm/mkdir, etc...
<gforce_d11977>fossy: this is about the CI integration
<gforce_d11977>fossy: i will add documentation about kconfig/config tomorrow
<stikonas>fossy: it's a huuuge Makefile for coreutils...
<gforce_d11977>stikonas: you are right, these sizes are compressed, so 'myunix' is much smaller
<fossy>gforce_d11977: I have an almost complete CI for live bootstrap https://github.com/fosslinux/live-bootstrap/pull/3
<fossy>gforce_d11977: myunix is not complete enough yet for use
<gforce_d11977>fossy: yes, but is does not work locally
<fossy>stikonas: unsuprsing.
<stikonas>fossy: I've got most important utils working (last one is "ls")
<stikonas>that's in my test environment, not in live-bootstrap, but usually it's close enough
<fossy>gforce_d11977: its going to be because of the slimmed down kernel, some option will be missing
<fossy>stikonas: cool, what tools do you think we need?
<stikonas>well, I've got most of them working
<stikonas>basename cat chmod cksum csplit cut echo expand factor false fmt fold head id kill li\
<stikonas>nk ln logname mkfifo mkdir mv nl od paste pathchk printf ptx pwd readlink rmdir seq split sum tai\
<stikonas>l tr tsort unexpand unlink wc whoami
<stikonas>also md5sum rm mv sha1sum
<stikonas>fossy: some are a bit broken, but I think important ones are working
<fossy>nice!!
<stikonas>e.g. seq prints zeroes...
<stikonas>I probably need another 2hours or so to finish this
<fossy>take your time
<stikonas>anyway, review and merge grep
<stikonas>it was fairly easy
<stikonas>fossy: ls will probably have to wait
<stikonas>until we get better libc
<stikonas>or we add more patches...
<stikonas>but I think we can live without ls for a bit longer
<stikonas>fossy: fixed that PR and tested again
<fossy>stikonas: I think we can too
<stikonas>anyway, I'm now starting to test coreutils in live-bootstrap
<stikonas>fossy: hmm, something is wrong with GH PR request...
<stikonas>I've pushed those fixes into my branch, but somehow PR didn't spot new changes...
<stikonas>I've opened a new one https://github.com/fosslinux/live-bootstrap/pull/16/files