IRC channel logs

2021-04-22.log

back to list of logs

<OriansJ`>indeed fossy right now the problem is mes is not distributed in a .tar but a .tar.gz and if we want to remove git submodules; it is something that needs to be addressed.
<OriansJ`>so we either work it from the distribution side (janneke making .tar releases or mes-m2 doing a .tar release) or by finding something I can convert to M2-Planet C to solve the .gz problem with mescc
<OriansJ`>or something like this: https://github.com/franzinc/zlib/ that can run on mes-m2
<OriansJ`>as that would also exist at that stage0
<OriansJ`>but stikonas's link does look like something that can be converted in a week of work
<stikonas>OriansJ`: well, that was just simple online search
<OriansJ`>It would be easier if someone did some prep work to strip out all of the config options to only that which is needed by mes' .tar.gz file
<stikonas>I am not claiming that it's the best/easiest option
<stikonas>it just seems that it might be possible to find something
<OriansJ`>indeed, it is just a matter of putting some search time in to find a reasonable starting place.
<OriansJ`>Then putting time into simplifying it with GCC to something elegant that I can deal with converting to M2-Planet's subset
<OriansJ`>I mean we literally went from a hex-monitor to a C compiler and a scheme able to run a C compiler powerful enough to get us bootstrapped to GCC. So getting a gunzip for M2-Planet is just a matter of putting in the effort
<OriansJ`>now if gunzip requires floating point support or bit masking arrays, it might be beyound M2-Planet's ability to cope but at first glance, this looks like something entirely reasonable to obtain.
<xentrac>it does not
<xentrac>it does require bit field extraction though
<OriansJ`>xentrac: foo = 0x1200 & blah works just fine in M2-Planet
<OriansJ`>as does foo = (0x12300 & blah) >> baz
<OriansJ`>bitwise and, or and xor function great along with bitshifts both left and right so extraction will be fine as long as the bit field isn't larger than something that could be expressed in a struct
<xentrac>should be fine. though I haven't uh actually implemented flate
<xentrac>but definitely no floating-point :)
<OriansJ`>well the best way to know for certain is to have a working minimal program that does exactly what we need and nothing more.
<OriansJ`>let us either find a simple 1 file ungz for compile one from a simple library and strip it to the bone
***terpri__ is now known as terpri
<OriansJ`>god people seem obsessive about making a *fast* unpacker
<OriansJ`>or look at this assembly version that gets inflate into x bytes
<OriansJ`> https://github.com/madler/zlib/blob/master/contrib/puff/puff.c seems like something we can put in order with reasonable effort
<OriansJ`>might need to strip out the setjump and longjmp bits as M2-Planet doesn't yet support that.
<ericonr>800 lines with a bunch of comments :o
<ericonr>that's tiny
<OriansJ`>just need to sort out the gzip wrapper format and bundle it together into a tiny ungz.c file for me to later port to M2-Planet's C subset
<ericonr>hm, I might be interested in doing that
<ericonr>depending on how long you can wait for that :p
<ericonr>and if no one is available for it yet
<OriansJ`>ericonr: well take a shot at that if it looks like fun
<OriansJ`>it should be able to be built via gcc ungz.c -o ungz and aim to keep it simple and easy to understand.
<OriansJ`>also most people here are already over-allocated on tasks. So if you see something simple or fun that'll help the work being done, feel free to do it and share with us the amazing thing you did.
<OriansJ`>because everytime one of us make a little bit of progress on the problem; we all benefit and because we are all making little bits of progress here and there. We have made huge amounts of progress together.
<stikonas>fossy: I think we might soon hit a problem with cirrus CI
<stikonas>it seems to time out in 2h
<stikonas>I guess that's their limit
<stikonas>(.yml has timeout of 3h)
<Hagfish>that would be an interesting constraint to optimise for
<Hagfish>i'm not sure how justifiable it would be to split the process over two jobs
<OriansJ`>speedrun bootstrapping, see who can get the farthest in 2h of build time
<Hagfish>heh, yeah
<Hagfish>gcc% :)
<Hagfish>fixed seed, glitchless
<OriansJ`>lol
<OriansJ`>rewrote GCC in hex0; speedrun time 20seconds beat that bitches lolz
<Hagfish>would it make sense to do each step of "parts.rst" as a separate job, which uses specific outputs of previous steps?
<Hagfish>segmented, new game plus
<OriansJ`>ironically not even that far off from janneke's original fosdem 2017 plan for bootstrapping mes
<Hagfish>on one hand that would expose any unexpected dependencies between the steps, but on the other hand it means that something might break when actually trying to run all the steps for real
<OriansJ`> https://video.fosdem.org/2017/K.4.601/guixsdbootstrap.vp8.webm
<OriansJ`>I think I managed to finally convince janneke not to go that route
<Hagfish>i think that live-bootstrap is a very reasonable approach
<OriansJ`>I wonder if we could do a CI chain
<OriansJ`>live-bootstrap-0 builds the first set of binaries, then live-bootstrap-1 depends on live-bootstrap-0 and builds out and repeat for infinite build time?
<OriansJ`>say at every time the libc is updated for improved.
<Hagfish>yeah, changes of libc might be a good way to conceptually split the process
<ericonr>OriansJ`: got it, cool :) if it works out and if you have a reference for the C subset, I might be interested :p
<OriansJ`>ericonr: The C subset is touched on here: https://bootstrapping.miraheze.org/wiki/Stage0 but honestly getting a single file ungz.c would be progress, even if it uses more than the subset in its initial version.
<OriansJ`>in short we have structs, unions, arrays, FUNCTION pointers, chars, int, long, unsigned and pointers for datatypes
<OriansJ`>we support function calls, returns, gotos, if/else statements, for/while/do loops, break/continue statements and inline assembly.
<OriansJ`>we also have some macro support as well for #define statements and #if/#elif/#else statements if condition inclusion is absolutely required.
<OriansJ`>I have already implemented a subset of the C standard library in http://github.com/oriansj/M2libc and can add any additional primitives you might need.
<OriansJ`>In short if it can be built by GCC and be easily tested, I can always trim it down to something M2-Planet can build.
<OriansJ`>But I do get stuck on programs I can't properly test or break into easy to update chunks
<OriansJ`>The key is don't aim for perfection, just help get the ball rolling in the right direction.
<ericonr>OriansJ`: okdo! thanks
<fossy>OriansJ`: that is the plan through sysabcdefgh etc
<fossy>but first I plan to finish kernel work
***nckx is now known as raghavgururajan9
***raghavgururajan9 is now known as nckx
***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<gforce_de1977>iiii
<gforce_de1977>(sorry)
***pgreco_ is now known as pgreco
***bauen1_ is now known as bauen1
<gforce_de1977>stikonas: i have started the mkdir/sync tweak + masstest, lets see if this changes something: tomorrow morning, we have results: here the mkdir-minipatch: http://intercity-vpn.de/bootstrap/helpers.sh.txt
<stikonas>ok, hopefully it resolves some of the issues
<stikonas>at least it would make sense then why you see it more often (less RAM)
***Noisytoot_ is now known as Noisytoot
***deltab_ is now known as deltab
<gforce_d11977>if this is a RAM issue, i can make a mass-run with 4GIG instead of 3, but i'am in doubt
<gforce_d11977>run.sh: line 15: sync: command not found -> Subprocess error 32512 -> ABORTING HARD
<gforce_d11977>so sync is not a 'builtin', we we have to A) provide and compile a sync.c or B) tune our libc to always sync() when doing a writeXY() somehow...
<gforce_d11977>maybe also fflush()
<stikonas>gforce_d11977: sync is indeed not builtin
<stikonas>but we build id at the same time as mkdir I think
<xentrac>gforce_d11977: you don't need sync(1) (sync(8)?) for bootstrapping
<stikonas>xentrac: there are some intermittent issues
<xentrac>unless you're running on a CPU so long that your bootstrapping time is large compared to the mean time between power failures
<stikonas>xentrac: if you have other ideas it might be nice to hear
<xentrac>alias sync to /bin/true
<stikonas>xentrac: OriansJ deliberately wanted to run real sync
<xentrac>or am I totally misunderstanding the topic?
<stikonas>well, sometimes code like
<stikonas>mkdir $dir; cd $dir fails
<stikonas>so OriansJ suggested inserting
<stikonas>mkdir $dir; sync; cd $dir
<xentrac>that's not what sync does
<stikonas>e.g. see https://github.com/fosslinux/live-bootstrap/issues/103
<xentrac>if mkdir $dir; cd $dir fails then either your kernel has a bug, your shell has a bug, mkdir failed, or $dir has spaces or other special characters in it
<stikonas>dir definitely has no spaces
<stikonas>it's just e.g. libcpp
<xentrac>if it's a timing-dependent bug then adding more stuff in the middle may mask it
<xentrac>sleep 1 would probably work more reliably than sync for that
<stikonas>basically this code fails https://github.com/fosslinux/live-bootstrap/blob/9f83e1213ea65cf6f2687c0c789b969cdf4cb93a/sysa/gcc-4.0.4/pass1.sh#L72
<stikonas>(quite infrequently)
<stikonas>oh, and mkdir also shouldn't fail
<stikonas>because it's set -e
<stikonas>so if mkdir fails, then script should abort
<stikonas>maybe some kernel timing bugs
<xentrac>good plan
<xentrac>seems likely. or the shell
<stikonas>yeah, or the shell...
<stikonas>especially since it's build with mes libc
<xentrac>I mean you could imagine a shell bug where it does a wait() and some unrelated child process exits
<xentrac>so it continues before mkdir finishes
<stikonas>although, mkdir should be quite fast...
<stikonas>it's in the RAM
<xentrac>it should, yeah
<stikonas>well, there is of course context switch...
<xentrac>if I were maintaining this code I would definitely put doublequotes around $dir though. I only leave shell variables undoublequoted if word splitting is something I actively want
<xentrac>Bourne chose the wrong default
<xentrac>I don't see anything in this script that could be running in the background and happen to terminate at the wrong time
<stikonas>well, I usually put double quoutes, but in this case $dir is defined just above
<stikonas>and without spaces
<xentrac>yes, I didn't mean to imply that it was a bug
<xentrac>your practices are valid :)
<stikonas>there are a couple of other issues if you have any ideas: https://github.com/fosslinux/live-bootstrap/issues
<stikonas>(there were more, but those are fixed now, some where real issues due to pre-generated files)
<xentrac>congratulations!
<xentrac>I have no ideas
<stikonas>but they look a bit similar to the first one
<stikonas>e.g. 2nd is configure is missing for some reason
<stikonas>so a bit similar to this one where directory is missing