IRC channel logs

2021-02-01.log

back to list of logs

<stikonas>so far I think I did the following https://stikonas.eu/files/musl.txt
<OriansJ>xentrac: what about IBM/360? its instruction set seems to map beautifully to hex and and runs Linux and is supported by GCC 4.7.4?
<OriansJ>It is also Big Endian
<OriansJ>hopefully working versions that are old enough to have the ability to toggle in a boot program and thus start a trusted boot chain.
<OriansJ> http://bitsavers.org/pdf/ibm/360/GC20-1667-1_intro360arch.pdf
<deesix>Which would be the preference for the next arch? Wirth's RISC, IBM/360, RISC-V, POWER9, PowerPC, SPARC, 6502, Z80...?
<OriansJ>deesix: well that is a hard question as all of those are likely to be supported to a degree (getting M2-Planet to fit/run in 64KB when self-hosting might be a problem or just require some complexity)
<OriansJ>although part of me is pushing towards picking up the armv7l work in mescc-tools-seed first and finish it before starting another M2-Planet port.
<OriansJ>but you should be done with your unification work soon (Which is a massive help btw and *THANK YOU*)
<OriansJ>Would getting M2-Planet's pointer arithmetic more C compliant seem like a good idea or is there some other behavior of M2-Planet you see as more lacking?
<OriansJ>as we probably want to start taking things out of known_issues.org to make M2-Planet porting easier in the long run.
<deesix>Yes, unification (Knight apart) should be done this week, so possibilities for next work^Wfun are popping in my head :)
<deesix>I'm not writting much M2 lately. While prototyping ANSI C parser I encounter more cases of me wanting logic short-circuit than pointer arith, but the later might be more feasible.
<deesix>In any case, you documented the workarounds, which is great.
<OriansJ>deesix: well the short circuit would require a very ugly change but not entirely harder
<deesix>Adding test cases to M2-Planet is another possibility. There're a few things without tests.
<OriansJ>deesix: and would definitely be needed for major code changes within (and possibly find more C behavior we don't match)
<deesix>Yes, not a minor change I recall.
<deesix>live-bootstrap and blynn-compiler are also in my radar, but that would need a bit more time (and reading for Haskell/lambda theory) to get up to speed.
<stikonas[m]>fossy: I've got initial work on musl in a branch https://github.com/stikonas/live-bootstrap/tree/musl
<stikonas[m]>It goes until linking now
<stikonas[m]>Then that "tcc -ar" issue...
<stikonas[m]>And make install will need adjusting paths...
<OriansJ>deesix: all one would need to do to make M2-Planet short circuit is make && and || branchy
<OriansJ> https://godbolt.org/z/KsoKfr
<OriansJ>tack on a cmp; branch over the rest of the statement and act like nothing happened, otherwise all the other evaluation in M2-Planet would be identicial but a bunch more labels prefixed with _ being output
<deesix>Yep, that was my intuition about it.
<OriansJ>so ugly but not hard to add
<OriansJ>impossible to do in assembly but not a problem in C
<fossy><stikonas> fossy: should we keep musl binaries in the same prefix /after?
<fossy>Yes
<Hagfish> https://github.com/stikonas/live-bootstrap that README is like reading the plot of an incredible action movie
<Hagfish>i can't wait to read more :)
<Hagfish>(i don't know if it's deliberate that the "Part 23" heading is bigger than the previous ones)
<stikonas[m]>Hagfish: typo but I'll remove flex 2.5.14 later and will build 2.6.4 using 2.5.11 after musl
***pgreco_ is now known as pgreco
<Hagfish>stikonas[m]: thanks for the insight
<Hagfish>do you mind if i submit the link to Hacker News?
<Hagfish>well, i guess https://github.com/fosslinux/live-bootstrap is the more appropriate link
<stikonas>well, you can submit it but it's still very much WIP. But I guess that's fine
<Hagfish>i'm thinking that EU afternoon and US morning might be a good time for it to receive some upvotes, and the repo has a good title/About
<Hagfish> https://news.ycombinator.com/item?id=25988977
<Hagfish>there we go
<deesix>upvote obsession, just don't play the game
<bauen1>gotta catch all of those free fake internet points
<bauen1>btw, just how far is live-bootstrap from gcc+linux+"whatever else is essential" ?
<stikonas>bauen1: so need to get musl, then try to finish bison, gawk, probably need to rebuild some of our old stuff (should be able to build more coreutils). That will probably let us run ./configure scripts. Then we still need to do perl for autotools. After that it should be fairly easy (binutils and gcc)
<stikonas>anyway, after autotools, things should get much simpler...
<bauen1>so in theory it's very simple lol
<bauen1>this is quite nice
<stikonas>yeah, we already have quite a bit of stuff finished
<stikonas>we have a shell (ancient bash but it will do)
<stikonas>I'm almost done with proper C library
<stikonas>and we have a reasonable C compiler (tcc)
<stikonas>so we are almost entirely out of mescc land (most recent difficulties were due to mes libc)
<stikonas>and musl compiles, I just have problems linking it
<rain1>cool!
<siraben>Hagfish: upvoted, hopefully it gets to the front
<siraben>though I think a title like "live-bootstrap: automating GCC bootstrap from a <1 KB binary with initramfs" might have been clearer
<stikonas>it's not <1KB yet...
<stikonas>I think it's 1094 bytes now
<stikonas>although, maybe kaem-minimal can be replaced with something smaller?
<bauen1>making (assembly) code smaller, doesn't necessarily make it easier to verify
<stikonas>something that hardcodes how to build hex0 and kaem-minimal ?
<bauen1>or write from scratch
<stikonas>well, that's true...
<stikonas>I was just saying kaem-minimal still has a lot of features...
<stikonas>but yeah, at these small sizes it's not always better to go smaller
<bauen1>stikonas: arguably you don't need kaem, you just need a binary that issues a few predefined commands, waits pid and finally execs kaem
<bauen1>no file read, just hard coded syscalls
<stikonas>well, that's what I was basically saying
<stikonas>you only need to issue 2 commands, build hex and build kaem
<bauen1>imo anything that can be hand assembled in a reasonable time span is fine (so something like <6h if you have some experience and a hard code of the i386 manual)
<bauen1>s/hard code/paper edition/
<xentrac>1KB will be awesome! not that 1094 bytes isn't
<stikonas>well, it shouldn't be too hard to modify kaem to instead issue 3 hardcoded commands (hex0-seed hex0_x86.hex0 hex0, /hex0 kaem-minimal.hex0 kaem-0, kaem-0 -f kaem.run) and rip out all parsing code...
<stikonas>I suspect that will take this modified "kaem" much closer to hex0 size
<bauen1>stikonas: actually, you should integrate keam and hex0 into a binary that builds hex0 and kaem minimal or something like that
<stikonas>well, I thought about that too, that would save some space on elf-headers
<stikonas>although, maybe less readable?
<bauen1>stikonas: it depends, maybe it could reduce the total amount of hex0 code, at the cost of a slightly larger seed
<siraben>Maybe we should get nanochess to help us, hehe
<bauen1>might still be worth it, since hex0 is basically a glorified binary format with comments
<siraben> https://nanochess.org/
<xentrac>like, fork, exec, and wait three times in a loop?
<bauen1>yes
<siraben>He can do insane stuff in 512 bytes
<bauen1>xentrac: yes
<xentrac>probably worth mentioning that Oscar Toledo and his father, mother, and sisters have built their own OS, computers, and web browser (though I think we're talking like Netscape 3 class browser, not Firefox 70)
<xentrac>I assume they used precompiled compilers to do get it bootstrapped
<bauen1>nanochess.org is him ?
<siraben>bauen1: yes
<bauen1>huh i've heard of him bevore then, mostly from random hacker news articles
<JeremiahOrians[m>well he would definitely kick my ass at making the seeds smaller.
<JeremiahOrians[m>except for the x86 native as that has to be 512bytes exactly but his would probably do far more.
<xentrac>bauen1: I wrote some incomplete notes in https://dercuano.github.io/notes/bootos-notes.html on his bootOS
<xentrac>in theory you could start with a floppy disk with those 512 bytes in its boot sector and an IBM-compatible with a keyboard and monitor, and bootstrap from there to a working system using just your brain and fingers, no further software
***nimaje1 is now known as nimaje
***jelly1 is now known as jelle
<stikonas>fossy: somehow I'm struggling to get unified .a files with tcc...
<stikonas>fossy: so I'm trying to run tcc -ar cf ../libc.a *.a
<stikonas>tcc: ar: Unsupported Elf Class: aio.a
<fossy>stikonas: oh, yeah, sorry I tested it with tcc ar yesterday it dosent work
<fossy>only with binutils ar
<stikonas>yeah...
<fossy>:/
<stikonas>I had some luck with cat... but I think the resulting .a is only partially working
<stikonas>so not good
<stikonas>hmm
<fossy>another option is make 2-3 .as and recompile tinycc to link against all of them....
<stikonas>we probably need more than 2-3...
<stikonas>I think at least 5
<fossy>hm, I see
<stikonas>but that's an option...
<stikonas>another option is to try to find the cause of that limit
<stikonas>and probably not an option is to build binutils ar...
<stikonas>it's more complicated than even bash...
<stikonas>fossy: oh, I also tried a bit with dynamic linking, but .so file also seems broken
<stikonas>i.e. compile 10 or so .a files into .so...
<fossy>stikonas: tinycc does /not/ work with dynamic linking in our setup like at all
<stikonas>fossy: that's because we are using mes libc
<fossy>even with musl iirc
<stikonas>oh?
<stikonas>why not?
<stikonas>I think we just need to specify elf loader
<fossy>yeah I know about that, but I think something is broken in it as you said
<stikonas>well, I just couldn't link .so file
<stikonas>I can try to do it with binutils...
<stikonas>and see if that .so is functional
<fossy>cant link even with -nostdlib?
<stikonas>no, linking worked fine for me
<stikonas>but 1) .so was about 1/3 of expected size. Then hello world was linking against libc.so but crashed on startup
<fossy>hmmmmm.....
<stikonas>well, crash might be caused by 1/3 of expected size
<fossy>yeah
<fossy>but the question is why it is crashing
<fossy>I mean
<fossy>why it is so small
<stikonas>yeah...
<fossy>what command were u using to convert the .as to the .so
<stikonas>something like -shared *.a -Wl,--whole-archive -o libc.so
<stikonas>but can't reproduce it now
<stikonas>maybe I used tcc-musl
<fossy>Right
<stikonas>hmm, so far the best I got was with cat
<stikonas> https://stikonas.eu/files/fix_linking.sh.txt
<stikonas>yeah, this doesn't work...
<stikonas>cat'ing them is useless...
<stikonas>fossy: well, I can try to make some unified c files
<stikonas>if we can get some smaller number of files
<stikonas>that might be enough to link .a...
<stikonas>although, musl has a lot of redefinitions...
<fossy>yeah I was gonna say
<fossy>might be harder with musl
<fossy>cat wont work its an archive
<stikonas>strange that guix didn't hit any of this in gcc...
<stikonas>I would think gcc should be just as large...
<pder>tcc -ar is failing when the command line is too long?
<stikonas>pder: yes
<stikonas>pder: I think when command line is 1024 chars or longer
<stikonas>pder: but it's probably not inside tcc
<stikonas>tcc only gets argc=236
<stikonas>so already truncated
<stikonas>and I think there are 1261 .o files in obj directory
<stikonas>pder: any ideas what to do?
<pder>I dont have any idea at the moment. Would be nice to track this issue down so we can avoid workarounds
<stikonas>it might be mes libc bug since argc is already smaller
<stikonas>and hard to workaround that
<stikonas>(since tcc will stay linked to mes libc)
<stikonas>(because tcc 0.9.27 is not self-hosting)
<stikonas>well, maybe we can rebuild 0.9.26 too...
<stikonas>janneke: do you know anything about command line limit in mes libc?
<janneke>stikonas: yeah, mes libc still has some hard coded limits
<stikonas>oh ok...
<janneke>it suffices for the guix bootstrap, though
<stikonas>well, because guix doesn't try to bootstrap bison or autotools...
<stikonas>just uses pregenerated bison .c files or ./configure scripts
<janneke>yeah
<janneke>probably easy to pick somer larger arbitrary limits
<stikonas>anyway, I can try to do some experiments to confirm that is is just mes libc...
<janneke>a proper fix is more work
<stikonas>well, we probaby need at least a few times more
<stikonas>fossy: interesting observation, so I used binutils for experimentation. Then I built tcc-musl and it is self-hosting unlike tcc-0.9.27
<pder>Trying some experiments inside live-bootstrap chroot. running /after/bin/echo 1 2 3 4 ...
<pder>to see what the max value is before failure
<stikonas>pder: and also tcc-musl -ar works fine
<stikonas>so it's def mes libc bug
<pder>Yeah, and the behavior is not predictable
<stikonas>oh, it's not always the same?
<stikonas>I do get the same size with tcc-ar
<pder>/after/bin/echo 1 2 3 up to 400 only prints up to 144
<pder>/after/bin/echo 1 2 3 up to 200 prints up to 200
<stikonas>yeah, I can see...
<stikonas>so maybe it's not 1024...
<pder>and /after/bin/echo 1 2 3 up to 255 prints... nothing
<stikonas>yeah, I can also see that
<stikonas>you are testing with /after/bin/echo $(seq 1 255) ?
<stikonas>(where seq is busybox builtin
<stikonas>254 is the last one that works
<pder>I was actually generating a one line bash script from another script
<stikonas>257 shows 1 again
<stikonas>so there is some overflow?
<stikonas>at 256
<stikonas>hmm, ok, so it's actually number of arguments
<stikonas>not length of command line
<pder>Maybe something in mes under the exec functions
<stikonas>and 1261 % 256 is 237
<stikonas>so one off from argc=256 that I'm getting in tcc -ar
<stikonas>so definitely this issue
<stikonas>va_list is limited?
<stikonas>hmm
<stikonas>pder: if we can't figure this out, I think workaround would be to build individual folders as individual libs instead of libc.a, build tcc against it with extra -l flags and then rebuild tcc and musl properly
<stikonas>doable, but messy :(
<pder>I'll take a look later this evening.