IRC channel logs

<achaninja>ekaitz: yeah I'm using cproc and qbe because I know the authors of both

<ekaitz>achaninja: they did a nice job on that. I really like the ideas behind those projects

<stikonas>well, I guess right now cproc and qbe are alternatives to newer tcc versions

<stikonas>for now one still has to go via tcc-0.9.26

<stikonas>it would become very interesting if at some point cproc/qbe could be build with mescc

<oriansj>rickmasters: looks very promising and I am very glad you selected https://www.gnu.org/licenses/license-list.html compatible licensing.

<rickmasters>thanks, i am flexible on licensing

<stikonas>I guess the same license applies to wrappe scripts in top level directory of boot2now?

<oriansj>The use of hex0 is impressive to say the least; especially when if I am reading this correctly, you could be bootstrapped by https://github.com/nanochess/bootOS

<rickmasters>stikonas: yes, whoops i neglected to add the notices, will do so

<stikonas>initially I even neglecged to look at notices...

<stikonas>oriansj: bootOS for typing in source?

<stikonas>i.e. something like monitor program

<rickmasters>orianjs: i provided a 384 byte bootstrap so i guess it depends on the goals

<stikonas>that's why I'm a bit confused...

<stikonas>it's not clear to me why we need bootOS

<oriansj>stikonas: we don't need it but it is another route to build the hex0

<stikonas>oh you mean bootOS->hex0 and do builder-hex0.hex0->builder-hex0.img

<oriansj>as in the hex0 in builder-hex0.hex0 could be built by bootOS

<stikonas>oriansj: anyway, with builder-hex0 you probably won't have to worry about writing kernel in assembly. Next kernel can be written in M2-Planet subset

<oriansj>stikonas: oh yeah

<stikonas>though I think you said M2-Planet subset is still a bit annoying to write kernels in

<stikonas>or maybe we get lucky and mes-m2 runs on builder-hex0

<stikonas>I haven't tried yet

<doras>stikonas: I see that the permissions are different all the way to guile-3.0.7_0.x86.xbps.

<doras>So... something is fishy here.

<oriansj>well mes.c does require a couple more syscalls than what was listed.

<stikonas>oriansj: do you remember which syscalls?

<stikonas>maybe they are not essential

<stikonas>at least some of that was just detecting interactive mode

<rickmasters>even hex2 would be a lot easier than hex0, those jumps...

<stikonas>oh yes, I'm well aware of that...

<stikonas>I added riscv support in stage0-posix

<stikonas>actaully even hex1 is alright

<stikonas>you are just limitted to smaller programs

<stikonas>I guess for kernel hex1 is too simple

<stikonas>and builder-hex0 is maybe three times as big as hex0+kaem-optional-seed

<oriansj>rickmasters: M0 is even a viable option for bare metal

<stikonas>yes, but since builder-hex0 can go to at least M2-Mesoplanet we don't need to use M0 later

<stikonas>well, at least for x86...

<rickmasters>it seems like a kernel would have a mix of assembly and higher level

<rickmasters>i don't know enough about the syscall requirements beyond stage0-posix

<oriansj>well M2-Mesolplanet does support inline assembly so we can definitely do a Kernel in it

<rickmasters>builder-hex0 does not really implement chmod or access so they would need to be reimplemented as well

<oriansj>rickmasters: chmod+access would only matter if we needed them for the filesystem we would be working with.

<rickmasters>the file system is ... primitive, probably needs a rewrite - you can't really expand a file after closing it

<oriansj>rickmasters: that isn't an issue for us

<oriansj>mescc just does a single write as well

<oriansj>TCC and above however would be an issue

<oriansj>and I was thinking of https://github.com/oriansj/GFK a filesystem one could write by hand

<rickmasters>i thought i read somewhere here that tar was capturing file modes and affecting checksums

<stikonas>rickmasters: yes, recently live-bootstrap switched to creating some hand-rolled packages

<stikonas>basically just .tar.bz2 with contents of make install DESTDIR=/tmp/destdir

<stikonas>rather than just checksumming binary files

<stikonas>but with old tar the problem is that order is not deterministic

<stikonas>seems to work fine in tmpfs but if you run it outside tmpfs, the order might be different

<stikonas>that's what doras is struggling with

<rickmasters>stikonas: are you open to changing tar?

<rickmasters>sitkonas: or do you try to stick to the originals?

<stikonas>in principle yes, we use patches in live-bootstrap

<stikonas>but that might not be the easiest solution

<stikonas>so right now we start with tar 1.12

<stikonas>that's the latest tar that can be built with mes libc

<stikonas>fossy tried to add this as workaround for sorting https://github.com/fosslinux/live-bootstrap/blob/master/sysa/helpers.sh#L40

<stikonas>but then we still have an issue with permissions if one tries to run everything rootless

<rickmasters>i should also mention that builder-hex0 only supports one outstanding child with fork, if anything forks a lot (like make), it won't work

<stikonas>well, make is only useful after tcc

<stikonas>and tcc is already too complicated for builder-hex0

<stikonas>I was actually thinking of moving make immediately after tcc in live-bootstrap

<stikonas>that was before I've read builder-hex0

<stikonas>*read about

<stikonas>but I guess that's still fine

<stikonas>I've also experimented a bit with newer tar 1.13 (haven't tried even newer ones)

<stikonas>tar 1.13 is missing two functions in meslibc

<rickmasters>a much more capable kernel beyond builder-hex0 would be needed for make

<stikonas>strtoumax and strncasecmp

<stikonas>yes, but from what oriansj said, even tcc wouldn't run on builder-hex0

<stikonas>so as long as we keep make after tcc, we should be fine

<stikonas>and wouldn't add any additional constraints

<stikonas>well, in bootstrap we deliberately used few syscalls early on

<stikonas>e.g. kaem does not use dup or any other similar stuff

<stikonas>we can't implement $(command) then but at least it can run on simple kernel

<rickmasters>yes, i was surprised how for it got on so few syscalls

<stikonas>we only start using pipes once we reach bash

<stikonas>before that scripting does not use "tar xzf" but uses two steps (unpack and untar)

<stikonas>but it was still a bit of guess work since we didn't have any kernel

<stikonas>just in general attempted not to use complex stuff

<stikonas>fossy: https://github.com/fosslinux/live-bootstrap/pull/159

<rickmasters>adding it up: file system, paging, multitasking, pipes, dup, etc

<rickmasters>the next kernel would be some work

<stikonas>bauen1: had some work done on that https://gitlab.com/myunix/myunix

<stikonas>but it's not M2-Planet buildable

<rickmasters>nice - looking for existing work is my instinct, maybe porting it M2-Mesoplanet

<stikonas>also it would be nice if newer kernels supported something like kexec...

<stikonas>then one can easily do automatic chaining

<stikonas>hopefully kexec shouldn't be hard to implement, just loading new kernel somewhere and jumping into it

<oriansj>we might also end up having to extend what M2-Planet/M2-Mesoplanet supports

<stikonas>well, last upgrade already extended it quite a bit

<stikonas>1.10 has quite a few extra things compared to 1.9

<oriansj>but rickmasters your work is epicly impressive and you should feel very proud

<stikonas>I agree

<rickmasters>oriansj: thank you so much, that means a lot

<rickmasters>oriansj: i feel the same way about you and stage0

<rickmasters>oriansj: and contributors, honestly don't know the history, but it seems you are the focal

<stikonas>depends on project

<oriansj>rickmasters: everyone here is excellent and if it wasn't for their help it would have taken another decade to get this far

<stikonas>oriansj maintains stage0-posix

<stikonas>fossy and I maintain live-bootstrap

<stikonas>janneke: maintains mes

<rickmasters>stikonas: thank you, i've read a lot about live-bootstrap and mes, its all great work

<rickmasters>an epic puzzle being put together here

<oriansj>and a great deal of fun too

<stikonas>well, it was inspired by guix and follows work done there

<rickmasters>not too familiar with guix, did it approach boostrapping more from a top down perspective?

<stikonas>yes, it started as normal distro

<stikonas>and was trimming down its bootstrap seed

<stikonas>live-bootstrap was done bottom-up and hence is more strict with pregenerated files

<rickmasters>stikonas: i see

<stikonas>but it was made possible by combining both approaches

<stikonas>mes/mescc came from top down approach (with the goal of building gcc)

<stikonas>and stage0/stage0-posix was building from hex0

<stikonas>and it was about 1.5 years ago when the gap got closed

<stikonas>though it took some time to get everything into released versions

<rickmasters>so, besides the kernel, is the bootstrap to gcc completely done?

<rickmasters>100%?

<stikonas>yes

<stikonas>well, live-bootstrap demonstrates it

<stikonas>even goes to g++ 4.7.4

<stikonas>although we couldn't bootstrap gnu autogen for regenerating top-level configure script in gcc

<rickmasters>stikonas: ok, i watched a presentation that said there was some missing work on mes or something but that may have been old

<stikonas>but that script is optional

<stikonas>(this is created using autogen https://raw.githubusercontent.com/gcc-mirror/gcc/master/Makefile.in)

<stikonas>but running ./configure in gcc subdirectories works around this

<rickmasters>if it works and is auditable then seems fine

<stikonas>yes, it's not too bad https://github.com/fosslinux/live-bootstrap/blob/master/sysa/gcc-4.0.4/pass1.sh

<rickmasters>actually, the less tools the better, in my opinion

<oriansj>so yeah, your kernel work, puts us very close to doing a bare metal x86 to full Linux distro

<stikonas>yes, I expected it would take a few years to get to this stage

<oriansj>assuming we get a little more advanced kernel written in C that M2-Mesoplanet/M2-Planet can build supporting maybe 31 more syscalls; we would be able to build Linux and the rest would be details

<oriansj>we might actually be done before bootstrappable hits the 10 year mark

<rickmasters>oriansj: that's pretty exciting

<rickmasters>oriansj: i've been thinking that there would need to be utilities build to format a file system and lay down bootloaders, grub/lilo

<stikonas>rickmasters: worth emailing your work to bootsrappable mailing list

<oriansj>absolutely

<stikonas>well, live-bootstrap builds util-linxu

<oriansj>it is mega news

<rickmasters>there's a mailing list? lol

<stikonas>but live-bootstrap does need a real disk for system c which is preformatted and loaded with source tarballs

<stikonas>it's a low volume mailing list

<oriansj>yes: bootstrappable@freelists.org

<stikonas>maybe an email once every few months

<rickmasters>ok, i'll take a look

<rickmasters>i guess the transition from 32 bit to 64 bit linux is well established?

<rickmasters>just cross compile

<oriansj>indeed and trivial after one has binutils, gcc and guile

<rickmasters>just wondering if its been automated before but shouldn't be a problem

<stikonas>guix has it automated

<stikonas>live-bootstrap for now stays in 32-bit mode

<rickmasters>which makes sense, a full 64 bit bootstrap is hard to imagine

<rickmasters>i mean starting in 64 bit, like with a 64 bit kernel, i'm not signing up

<oriansj>rickmasters: no worries

<oriansj>do only the things you find fun

<rickmasters>is cross compiling to other architectures a sufficient strategy for those architectures or will is it an open goal to try to boostsrap every architecture from bare metal?

<oriansj>rickmasters: personally, I want to do *every* architecture from bare metal but doing just 1 is enough to show that it is possible and end the debate

<oriansj>and provide a root capable of enabling DDC to find all trusting trust attacks in the entire ecosystem

<oriansj>doing all of the architectures would even allow us to even detect hardware level subversion

<rickmasters>oriansj: i see. thats ... ambitious and i don't disagree because 386 may not last forever

<oriansj>well yes.

<rickmasters>carting emulators around will get tiresome - the bootstrap will need to keep up with the times, someone will port to apple M9 in 2093 and never look back

<rickmasters>the more implementations the better i guess is what i'm trying to say, but cross compiling seems find to me for now

<rickmasters>fine

<oriansj>well you just showed that the root kernel can be *MUCH* simpler than I planned and going by time to completion for the cc_* rewrites: ^_^ that isn't very far off

<oriansj>if we do the C kernel in a cross-platform manner, we create another universal base and porting to new architectures becomes a 4 step process: M2-Planet+mescc-tools, stage0-posix, C kernel and builder-hex0 rewrite.

<rickmasters>yeah i guess i'm imagining how hard it would have been to write builder-hex0 in 64 bit from scratch - just unknown to me. maybe its not so bad

<oriansj>rickmasters: well, if it had been me: Write in M1 first and then use emacs macros to do the conversions to hex2 and then calculate the labels and then jumps/calls

<rickmasters>i decided not to do that on principle, but it would have been a lot easier

<oriansj>rickmasters: completely fair

<rickmasters>spending 6 hours isolating a typo'd opcode is no fun

<oriansj>and honestly all the more impressive

<rickmasters>well less fun

<rickmasters>oriansj: excited to see where this goes. great meeting you and everyone. near my bedtime but i'll be back in 9 hours or so

<oriansj>a pleasure rickmasters and stay awesome

<fossy>rickmasters: this is absolutely insane, amazing work

<fossy>rickmasters: this is a *massive* development, thank you very much for your contribution

<Hagfish>yeah, it's like Christmas has come early for the bootstrappers this year :)

<fossy>arches my personal opinion is cross compiling is good for everything BUT i would like it if we can someday make an open arch the "base" arch that everything can be cross compiled from

<rickmasters>fossy: thank you!

<fossy>stikonas[m]: FYI i've nearly finished sysc network source download, but disk-source behaviour will be retained, behind command-line flag for rootfs.py & used if found within the disk

<stikonas[m]>fossy: sounds good

<stikonas[m]>fossy: can you add an option to specify remote mirror for downloads?

<stikonas[m]>E.g. I can put my cache dirs somewhere on my server

<stikonas[m]>Or is that non trivial?

<fossy>stikonas[m]: no, that seems simple to me, i'll add that

<Franciman>has anybody tried mes on big projects?

<Franciman>Like how fast is it?

<Franciman>mescc is reasonably fast?

<stikonas[m]>mescc is quite slow

<Franciman>hmm i see

<stikonas[m]>Takes about 10 min to compiles tcc

<Franciman>i was thinking about how much would it take to add a bytecode interpreter

<stikonas[m]>Whereas tcc rebuilds itself in a fraction of second

<Franciman>thanks t184256

<Franciman>ops

<Franciman>sorry

<Franciman>thanks stikonas[m]

<Franciman>i'm working on my bootstrappable scheme (before i knew about mes, but now i keep going cuz it's fun lol)

<stikonas[m]>fossy: BTW, are you going to format disk in sysb in download mode? Or not planned right now?

<fossy>stikonas[m]: yes, disk formatted in sysb, if it has no existing partitions

<fossy>rickmasters: is there a known limit to the size of helloworld.src? i tried adding mes to it, but it fails in the src file src file stage, kernel crashes

<rickmasters>fossy: hmm, no, but i did run into bugs as it got bigger and bigger, so not too surprised

<rickmasters>i'm happy to take a look with some more detail

<rickmasters>perhaps a link to the specific file you are adding?

<rickmasters>wait, now that I check - there is a 1M limit total in build.sh near the top

<fossy>oh, ok, that makes sense, it was around 5M

<rickmasters>yeah, i should probably put in a check

<rickmasters>you should be able to just change 2056 to (size / 512) + 8

<rickmasters>you're in uncharted waters :D

<fossy>:P ok thanks

<stikonas[m]>fossy: trying to run mes-m2?

<janneke>so, what's up with: https://github.com/torvalds/linux/blob/v4.19/arch/s390/kernel/syscalls/syscall.tbl ?

<janneke>the numbers seem all off / don't seem to work?

<stikonas[m]>janneke: s390?

<stikonas[m]>I don't think mes support s390

<janneke>eh, isn't that riscv(64)?

<stikonas[m]>I sometimes look here https://marcin.juszkiewicz.com.pl/download/tables/syscalls.html

<janneke>ah, something had me terribly confused by equating riscv64 with s390?

<stikonas[m]>It worked for m2libc development

<stikonas[m]>s390 seems to be some IBM thing

<janneke>thanks

<achaninja>janneke: what does the -DBOOTSTRAP do in your tcc?

<achaninja>I also wonder about LONG_LONG_STUB, but i guess that just emulates LONG_LONG by computing only the low 32 bits?

<achaninja>I guess it looks like it removes some features mescc can't support

<achaninja>does that sound right?

<janneke>achaninja: yes, something like that

<janneke>mescc does not support floats, so they are compiled in in several steps

<janneke>same for long long

<janneke>achaninja: see boot.sh, which is called several times by build.sh

<achaninja>thanks, I made the mistake of not using them properly and ended up with a mostly working tcc

<achaninja>but not quite right

<achaninja>janneke: I have new appreciation for the work you put into patching tcc

<achaninja>good job!

<janneke>achaninja: hehe, thanks

<janneke>it's all pretty much a best-effort thing

<janneke>not really what we want the "final" full source bootstrap to look like ;)

<achaninja>Well I will be happy to see the progress as it occurs

<stikonas[m]>Well, progress is happening. We can now do bootstrap using released mes. Hopefully gash will soon run on mes. Then we'll be able to use mes/mescc build system

<stikonas[m]>And now we got bootstrap kernel

<stikonas[m]>janneke: have you seen the new kernel? Might be nice if we get mes running on it (if that is possible)

<janneke>stikonas[m]: oooh, that's awesome rickmasters

*janneke would like to see that!

<rickmasters>janneke: thank you

<unmatched-paren>rickmasters: thanks again for your work! i can't get builder-hex0 to run on my guix system though; i'm running `make` in a `guix shell qemu xxd make`, but all i get is "No bootable device" from the qemu firmware

<unmatched-paren>what am i missing?

<rickmasters>unmatched-paren: i guess i would start by seeing what did work, maybe ls -ltr

<unmatched-paren>so, looks like the `-mini` .bin and .img files have been built

<unmatched-paren>but not the regular ones

<unmatched-paren>that's the only difference from the git repo

<unmatched-paren> https://paste.sr.ht/~unmatched-paren/192afa2ca9a35d0a5ac776e1d0e5ed1a44a4439f

<rickmasters>didn't get far, check for cut and xxd

<unmatched-paren>xxd and cut both exist

<rickmasters>cut builder-hex0.hex0 -f1 -d'#' | cut -f1 -d';' | xxd -r -p > builder-hex0-seed.bin

<rickmasters>thats the command maybe the options differ?

<unmatched-paren>that command works fine, should i try make again?

<rickmasters>does it run without error but produce an empty file?

<unmatched-paren>oh, i see. that command produces the right file, but the mini-seed is 0 bytes large

<rickmasters>so sorry cut builder-hex0-mini.hex0 -f1 -d'#' | cut -f1 -d';' | xxd -r -p > builder-hex0-mini-seed.bin

<rickmasters>its this one that failed

<unmatched-paren>the seed isn't 0-sized anymore

<unmatched-paren>woo it works!

<unmatched-paren>i think?

<unmatched-paren> https://paste.sr.ht/~unmatched-paren/a2dca67164b5b2c69a0394ed93a3d231d7f978a6

<unmatched-paren>this is the output

<rickmasters>that looks good

<rickmasters>are you in the boot2now repo, that was just the builder

<unmatched-paren>ah, i'm not

<unmatched-paren>i cloned builder-hex0, which is probably the problem, right?

<rickmasters>yes, that's just the kernel/compiler

<rickmasters>boot2now combines that (it has its own snapshot of builder-hex0) with the stage-posix source

<rickmasters>you don't need to clone builder-hex0

<stikonas[m]>Would got submodules make sense for boot2now?

<stikonas[m]>Or is stage0-posix modified

<unmatched-paren>rickmasters: btw, i just noticed a number of improvements that could be made to the builder-hex0 makefile

<unmatched-paren>mind if i send a pr?

<unmatched-paren>unless there was a reason for not using $@ $^ et al

<stikonas[m]>Although boot2now only has x86 part of stage0-posix

<rickmasters>stikonas: i am not opposed to git submodules but it wasn't my immediate preference

<stikonas[m]>No, it's up to you, I was just asking if there is some particular reason

<rickmasters>stage0-posix can add stuff (like access recently) that i may not support yet, so i was trying to control what snapshot to use

<rickmasters>i'm sure there is a way with git submodules but after an hour or two of research it wasn't crystal clear so i just did the copy pattern

<stikonas[m]>I know some people don't like git submodules...

<rickmasters>unmatched-paren: yes, open to pr

<stikonas[m]>git submodules point to specific commit, so that should be fine

<rickmasters>funny, you'll find $@ and $^ in my history

<unmatched-paren>rickmasters: surely you can just do git submodule add https://github.com/oriansj/stage0-posix <COMMIT>?

<unmatched-paren>to freeze to a commit

<rickmasters>i removed the make variables to its easy to cut and paste at the command line (as demonstrated above)

<unmatched-paren>ah i see

<rickmasters>but i agree its kinda best practice for maintenance of the makefile

<rickmasters>unmatched-paren: no doubt submodules would work, i don't have a strong feeling about it

<rickmasters>i just read scary stories about merging and branches and forgetting to update and just said, ah screw it

<rickmasters>lame, i know :p

<unmatched-paren>i _think_ the branching/merging problems only happen if two people add submodules seperately and try to merge

<unmatched-paren>forgetting to update sounds like more of a problem with just copying it in :P

<rickmasters>then i read about git subtrees and experienced a little analysis paralysis, had to move on

<unmatched-paren>ok, i'm running boot2now now :) seems to be working

<unmatched-paren>"Hello World!" it works \o/

<rickmasters>woo hoo!

<Hagfish>fantastic

<rickmasters>unmatched-paren: i'll work on switching to submodules

<unmatched-paren>oriansj: does m2 support `restrict`?

<unmatched-paren>as in, ignoring it instead of throwing an error

<stikonas[m]>unmatched-paren: no

<stikonas[m]>unmatched-paren: see https://github.com/oriansj/M2-Planet/blob/master/cc_core.c#L2351

<unmatched-paren>thanks

<stikonas[m]>You might be able to ignore it here

<unmatched-paren>i'm implementing some things in m2libc that seem to be defined using restrict in typical libcs, so i was wondering whether i should use it

<unmatched-paren>*with arguments that are defined using restrict

<Franciman>is there any document explaining the subset of C supported by mescc and m2-planet?

<janneke>no

<stikonas>Franciman: you can get some idea of what is supported by M2-Planet if you look at function names in cc_core.c

<stikonas>and there are sometimes comments above them that display what is processed, e.g. https://github.com/oriansj/M2-Planet/blob/master/cc_core.c#L2254

<Franciman>thanks

<stikonas>there is also very basic macro pre-processor in https://github.com/oriansj/M2-Planet/blob/master/cc_macro.c (basically handles defines and their define expressions). There is a more sophisticated pre-processor M2-Mesoplanet which can also handle includes and more complicated function-style defines

<stikonas>e.g. #define SUM(A, B) A + B

<stikonas>I'm less familiar with mescc but mescc is far more capable

<Franciman>m2-planet seems already very capable!!

<stikonas>yes, it's not bad

<stikonas>but some stuff is missing, e.g. i++ wouldn't work or int i[2][3]

<stikonas>also no support for structs on stack, structs have to be used via pointer

<stikonas>and no support for . operator

<stikonas>M2-Planet itself is written in simpler dialect, so that cc_x86 and other cc_* compiler could build it

<Franciman>neat

<muurkha>stikonas: would (i++) work?

<stikonas[m]>muurkha: no, not implemented

<stikonas[m]>i += 1 does work now in 1.10

<stikonas[m]>Before it was only i = i+1

<muurkha>oh, you just mean in the compiler, I thought you meant in the preprocessor

<stikonas[m]>Yes, in the compiler

<muurkha>I want to point out that #define SUM(A, B) A + B is bad style unless it's working around a preprocessor limitation

<muurkha>because SUM(x ^ y, z) ends up as x ^ y + z which misparses as x ^ (y + z)

<muurkha>#define SUM(A, B) (A) + (B) solves that problem

<stikonas[m]>Well, that works too, I was just pointing out that function type macros work

<stikonas[m]>What's on the right hand side is not important

<stikonas[m]>It can be even another define "function"

<stikonas[m]>M2-Mesoplanet supports that

<muurkha>yeah, it's a tangent, but I thought it was worth mentioning because that's a "red alert" line of code

<muurkha>sorry to distract from your intended point

IRC channel logs

2022-05-12.log