<fossy>stikonas: nice!! all it needs to do is build bison, so it dosen't matter too much <stikonas>fossy: actually it only needs to build flex1 <stikonas>so we really need to only run it on that single file <stikonas>fossy: by the way, is there any particular reason why you chose patching with -p0? <stikonas>I'm trying to find nicer way to produce patches like that but most tools give me -p1 <stikonas>fossy: ok, good news is that flex 1 builds in my chroot (with busybox) <stikonas>so I'll push lex now and later start working on flex 2.5.1 <stikonas>I want to start from tarball, not gio's submodule <stikonas>it's a simple shell script that produces another file, but we don't have shell yet... <fossy>stikonas: honestly because that's what i'm used to, lol <fossy>in void all patches are -p0 so i default to using that :P <stikonas>did M-x replace-string to get rid of those prefixes <fossy>stikonas: hm, mkskel is rather ugly <stikonas>I was thinking maybe write a C app that does the same thing <fossy>ah.. m4 only does it to stdout <fossy>lol, so, i'm trying to figure out how to link the linux kenrel, so i was using tcbcoot, so i extracted the iso, but then there was a romfs filesystem (wtf?) inside it, so i recompiled my kernel to add support for it, so now i can finally read it, and finish the kernel xD <stikonas>by the way, can we make kernel optional? <fossy>i am unconvinced that everything will always work with all linux kernels <stikonas>fossy: I think I have an idea for mkskel.sh <stikonas>provided that sed works for that file (I haven't tested it) we run sed and do catm skel.c header sed'ed file footer <stikonas>those header and footer are very simple anyway <stikonas>well, I can create header and footer with patches... <stikonas>probably don't have to be in patch format <fossy>ill merge once I get back home :) <stikonas>hmm, newer flex has much more annoying mkskel.sh <stikonas>I wonder if we can skip it until we have bash <stikonas>haven't been in office since last March... <stikonas>so I'll go to bed and maybe tomorrow evening I can start looking at flex 2.5.1 <stikonas>since I was already using it to test correctness of yacc and lex <stikonas>then I think I'll try to skip newer flex and go directly to vison <OriansJ>--tty-in and --tty-out have been added to stage0's vm to allow easier testing <fossy>stikonas: ah, sucks over there rn, hope youre doing ok <stikonas>yeah, I'm fine. I have builder's half mask for diy, those work very well against corona too <OriansJ>deesix: I have the revised knight-native tests nearly done and should be up shortly <OriansJ>the knight-posix will take a little longer but I should have it up before this weekend. <OriansJ>just make sure to have stage0's vm at commit 086c5f0ed2f234700f37b8be690825b12c855412 or newer <deesix>OriansJ, looking at the diff for 2043a574c982 it the scripts are now testing [ 0 = $? ] for programs that returns (usually) 42, is this correct? <OriansJ>deesix: well yes because HALTing a machine doesn't exactly have a defined way of returning a value. <OriansJ>normally I'd use Knight.py and set a breakpoint at the halt to see what is register R0 <OriansJ>knight-posix will actually return a value because the SYS_EXIT HALCODE does exit(vm->reg[0]); ***duncanm_ is now known as duncanm
<deesix>Still special cases for the unified scripts but now we can run the tests, which is great, thanks! <OriansJ>deesix: well knight-native might not ever fit in the unified scripts but knight POSIX should be able to without much complexity <OriansJ>just a single variable set (say KNIGHT_EMULATION) and an else if will be sufficient to put knight-posix into your unified script. I'll get an example ready for you so it'll be more obvious <deesix>Yes, for native is also a conditional branch. <OriansJ>deesix: native doesn't have to be in the unified because it is sufficiently unique <deesix>It's 90% the same, like the rest... <deesix>If not, just a bit different and I guess unify-able, but sure... a bit more ugly. I guess we can skip it. <OriansJ>deesix: well yes; just like x86-userspace is 90% of x86-native. it has syscalls, arguments and environment variables; whereas x86 bios don't. <deesix>Sorry, I'm lost now xD I was talking about about the content of the scripts... the calls to M2-Planet, M1, ... then the verify zone. <OriansJ>deesix: I am just saying unification with knight-native is not a priority nor likely to yield much benefits <OriansJ>knight-posix however is a different matter. <deesix>OK, so test/common_knight/* won't be replaced soon. <OriansJ>deesix: it'll be the last of the test/common_* to be removed. <OriansJ>419,265,798 instructions for knight-posix to self-host M2-Planet; even at 1Mhz that is only 7minutes <OriansJ>I could probably cut that significantly if if I didn't tokenize all of the input and cache all of the output <OriansJ>So the current design goal for bare metal knight: 1Mhz or faster and 5MB of RAM <OriansJ>I'll probably end up making knight-posix more posix compatible. ***qyliss- is now known as qyliss
<fossy>lol, 1185 LOC patches for linux ***mephista is now known as spicy_icecream
<gforce_de1977>fossy: can you upload/show/pastebin your .config for a minimal 2.4.26 kernel? <fossy>gforce_de1977: well, we are not really using a .config, as this is before makefile ***Server sets mode: +cnt
***mephista is now known as spicy_icecream
<gforce_de1977>fossy: till now the minikernel/kritis did not work with kernel 2.4, so i'am trying to get an idea why (strange compiler complaints) <gforce_de1977>fossy: but seems, that this old kernel need GCC-2.95.3 or at least GCC<4 ***nimaje1 is now known as nimaje
<pder>stikonas: in live-bootstrap are you treating bash as not currently available since it isn't properly bootstrapped? I am wondering why you say you need a workaround to mkskel.sh? <pder>I dont understand, bash is built and runs scripts <pder>Is it due to the way it is built? <stikonas>pder: we used pregenerated file to build bash <stikonas>once this work is done we'll have bash properly available <stikonas>took me maybe 2 weeks :D to get yacc and lex working <pder>Yeah, nice work by the way on yacc and flex <stikonas>flex 2.5.11 should be done later today if I'm lucky <stikonas>but I don't expect any difficulties with flex 2.5.11 <pder>Ive been busy with other stuff, so havent had a chance to investigate why doubles are so broken in tcc. Is this possibly a known issue? <stikonas>then hopefully I can skip latest flex version (postpone after bash) and go for bison <pder>I noticed in gio's asmc project he patched tcc to use softfloat <stikonas>seq still prints correct number of zeroes <bauen1>this discussion reminds me, didn't tccboot use some tools to pregenerate / preprocess files and make them usable with tcc ? <bauen1>fossy: true, maybe you can answer that <stikonas>I initially thought makefile will be used but apparently it's kaem... <stikonas>anyway, for now I'll to go get bash bootstrapped <stikonas>hmm, I barely managed to get hold of flex 2.5.11 tarball... Well, it wouldn't have been a disaster since there is git history but there were only two places left <stikonas>first flex-tmp which uses simplified lexer written for heirloom lex. Then flex-tmp builds very lightly patched flex. Then very lightly patched flex builds unpatched flex. <pder>very nice. now bison, then bash? <stikonas>nsb had newer flex first which I'll try to skip <stikonas>pder: so bison, and then only minor adjustments to bash <stikonas>we just need to add one rm command, update README, etc... <warren>Does anyone know if Guix as shipped on Debian bootstraps from Debian's toolchain and results in a binary identical Guix? <warren>I have never used Debian, but I want to package Guix for Fedora. They would require anything that ships in Fedora to be buildable from Fedora's toolchain. <warren>bootstrappable from Fedora's toolchain <vagrantc>what part are you wondering about binary identicalness? <warren>vagrantc: Is the Guix in Debian built entirely from source or does it rely upon some base binaries? <vagrantc>the binary it ships is definitely not binary identical to any guix binary in guix, but it can produce (at least as good as guix can) any package <vagrantc>warren: it's built from the Debian toolchain, but can bootstrap a guix just as good as guix itself can <warren>wouldn't it be possible for Debian toolchain -> guix -> guix builds itself again to be identical to upstream guix? <warren>I thought that was the raison detre of guix? <stikonas>warren: not everything in guix is reproducible <warren>the parts that matter are reproducible? <vagrantc>it sets up an environment the maximizes the chances of reproducibility, and does some work to make that happen <warren>the ultimate goal of wanting guix is to make static linked binaries of something else like Bitcoin Core. If the end output is identical between any guix that's good enough. <vagrantc>warren: there was some recent thread on guix-devel about building guile (and ostensibly guix itself) reproducibly <vagrantc>warren: it's plausible that Debian's guix, guix's guix and some future Fedora's guix could all produce an bit-identical bitcoin (since bitcoin has gone to some lengths to be reproducible) <stikonas>well, some people might use guix for bootstrappability instead of reproducibility... <vagrantc>and yeah, this might be more on-topic for #guix or some reproducible builds channel :) <stikonas>vagrantc: Debian's guix uses upstream's guix bootstrap binaries anywya <stikonas>so most likely they would result in the same bit-identical software (in those 90% or so reproducible packages) <vagrantc>stikonas: no, the guix in debian is only built from tools in Debian <vagrantc>stikonas: i would know, having disabled the hundreds of tests that depended on the bootstrap binaries :) <stikonas>vagrantc: but do you edit hashes in bootstrap.scm? <vagrantc>stikonas: no ... the first guix pull will then use the bootstrap binaries <stikonas>yeah, so basically build process for buix packages will be identical <vagrantc>warren: same basic requirements for Debian packages (at least how I packaged guix) <warren>vagrantc: OK I will take a crack at this, I will likely have questions. <vagrantc>warren: had to disable a *lot* of tests that assume network access and access to the bootstrap binaries <stikonas>well, in any case it's not full bootstrap :) <stikonas>warren: we'll hopefully bootstrap bash this weekend, still a while to go until guile is bootstrapped <warren>vagrantc: would guix accept commits to have an official offline bootstrap mode or something? <vagrantc>warren: you'll be terrified, er, interested in debian/patches/ :) <vagrantc>warren: don't see why not, and consider me an ally in nudging for it :) <vagrantc>i suspect the full-source bootstrap is going to be infeasible for debian; shipping ancient versions of various core packages likely frowned upon <warren>stikonas: i'm curious what the meaning/significance of bootstrap bash and bootstrap guile means? <stikonas>vagrantc: you don't have to ship everything :) <vagrantc>stikonas: well, to do it in the archive, we do <vagrantc>stikonas: everything built in debian needs to have it's dependencies packaged... <stikonas>warren: we are trying to build everything from 737 byte shell and 357 byte hex assembler <stikonas>warren: this is what we call full source bootstrap (for now ignoring POSIX kernel)... so from just those 2 binaries and a lot of source code we try to build proper toolchain <vagrantc>maybe an exception for bootstrapping specific versions would be allowed, especially if the binaries were in non-standard paths or something like that <vagrantc>though then i wonder about security implications of building a foundation on software that has known security vulnerabilities... <vagrantc>(e.g. old/ancient versions of toolchains) <stikonas>it's not like current debian was not bootstrapped on those <vagrantc>but this audience in this channel may be a bit easier to convince than ... all of debian, for example :) <vagrantc>the case can be made ... but ... things move slow in Debian :( <stikonas>well, you can make gcc-bootstrap package that has no dependencies and includes all bootrapping steps as part of build process but does not install insecure binaries... But I guess such big frankenstein build would also be frowned upon <stikonas>vagrantc: in any case you can't install everything <stikonas>vagrantc: some of the packages are non-redistributable <fossy>gforce_de1977: ah, yeah, kernel 2.4 will never compile on new compieer <vagrantc>might be possible to get exceptions, too <stikonas>in live-bootstrap those non-redistributable packages are not a big problem since we compile from source and don't distribute them <fossy><bauen1> this discussion reminds me, didn't tccboot use some tools to pregenerate / preprocess files and make them usable with tcc ? <stikonas>in fact they only live for a few seconds until they get overwritten with good packages <fossy>it precompiles some things from the kernel sources to build the iso <fossy>but, the kernel it kexecs into is completely from source <fossy>I am not using any of the tccboot code, so that eliminates the precompiled code, all I used tccboot for was finding the required files and arguments to tcc for linking :D <fossy>vagrantc: you clearly know a lot more about this than me, but one issue I envisinge is security flaws, distros dont like using these old things because they have soooo many cves <fossy>(I.e. why I am hesistant to provide network access during the bootstrap) <vagrantc>i mostly know the distro policies, but you might likely know more in regards to technical details :) <stikonas>well, maybe if we firewall off incoming connections then risk is managable? <stikonas>fossy, vagrantc: curiously, live-bootstrap has newer bzip2 than my main system! <fossy>stikonas: hehe, maybe a few things lol <fossy>firewall is probably doable, but you need an uptodate firewall <stikonas>fossy: hmm, I'm trying to play with bison now, got a few ugly warnings... or maybe errors, not sure yet <stikonas>/after/bison-3.4.1/bison-3.4.1/src # flex scan-code.l <stikonas>fossy: yep, those seem to be critical errors... <stikonas>I wonder if I need newer flex or better libc <fossy>I bet its newer flex, can you try in chroot mode <stikonas>well, Im already trying to build it with busybox <stikonas>flexdef.h:63: error: include file 'netinet/in.h' not found <stikonas>fossy: hmm, newer flex seems to require much more powerful C library <stikonas>although, let me see if it can run anything <stikonas>fossy: so we can postpone newer flex and bison for a bit <stikonas>although, it would be really good if we can get better C library <stikonas>is this sed breakage that you occasionally saw? <warren>For the purpose of fedora's guix it would be adequate if it is bootstrapped from Fedora's toolchain where the end result is identical to Guix bootstrapped elsewhere. I'll take a look at the binary diff afterward to see if it matters. <warren>It would be cool if we could bootstrap all the way from the 737 byte shell and 357 byte hex assembler on Fedora. <fossy>stikonas: hm, no, i'll just rerun it <fossy>stikonas: altho we need bison for autools i think <fossy>unless gcc or something dumb needs it <stikonas>well, we can either try to get autotools/binutils/gcc/glibc and then biosn <stikonas>or alternative would be to try musl, but I had some problems with it... <fossy>well, we can just download each of the autotools packages, binutils,gcc, glibc, take a look at them, and see if they have *.y <xentrac>in the perl4 era it was common to bootstrap perl without perl --- but not without yacc and not without Configure <stikonas>well if you use Configure then you don't need perl <stikonas>the problem is that Configure is generated with perl <stikonas>xentrac: so we have a really old heirloom yacc parser, and slighly newer flex 2.5.1 lexer <xentrac>right, that's why I mentioned Configure <xentrac>(though IIRC at least some Perl versions had a lower-case configure generated by autoconf as an alternative) <xentrac>but I didn't know how to use Unix in those days so I could easily be wrong <stikonas>well, autoconf itself doens't need perl, but automake does <stikonas>/usr/bin/automake-1.13: Perl script text executable <stikonas>so everything depends on how good yacc is... <stikonas>hopefully I didn't cripple it too much :D <xentrac>so, a thing I want to mention about yacc <xentrac>is that bottom-up parsing is actually pretty easy <xentrac>no, yacc guarantees linear time for the resulting parser <xentrac>but if you take a very simple backtracking approach, parsing is pretty trivial <stikonas>fossy: yep, bash just compiled in rootfs.sh chroot after I removed rm y.tab.c y.tab.h <xentrac>Prolog DCGs do this top-down, but they don't handle left recursion <xentrac>the problem with the very simple backtracking approach is that in the worst case it's exponential time <fossy>stikonas: i'll wait for the CI rerun to finish, see if it succeeds, but you can open another PR on the top <stikonas>ok, I think it will have all 3 commits but doesn't matter... <fossy>yeah, then when i merge it, it will go down to 1 <stikonas>luckily I called my dir files without reading DEVEL.md :D <xentrac>but it occurs to me that maybe you could bootstrap yacc successfully --- maybe even modern Bison --- with a very simple yaccalf kind of thing that uses the exponential-time approach <stikonas>xentrac: biggest problem with bison is that bison needs better C library... <stikonas>if we have better C library, I think we are done <stikonas>first of all tcc -ar is not coping well... <xentrac>because in that case you don't actually care what the worst case is; you only care how long it takes yaccalf-built bison to build bison <fossy>stikonas: ohhh i have a solution for that <xentrac>hmm, interesting, I didn't realize that! I thought the biggest problem with bison was that it was written in bison <fossy>so, instaed of unified sources, we can do unified ars <fossy>tcc -ar lib1.a a.o b.o c.o d.o e.o f.o g.o <fossy>tcc -ar lib2.a h.o i.o j.o k.o l.o <stikonas>(also we have m4 and I have bash locally <stikonas>make and m4 are both from GNU, not older versions <xentrac>so the obstacle is now building glibc without yacc, or yacc without glibc, and nyacc and archaeological-yacc aren't good enough? <stikonas>anyway, bash is really useful to have now... <stikonas>we probably need to create /proc file system for it to work too... <xentrac>sure, interactivity is verboten in a reproducible build process anyway <xentrac>although it can be helpful when you're trying to debug it ;) <stikonas>yeah, would ne nicer to debug with our own bash and not injected busybox <stikonas>especially, since my busybox also ships other tools... <stikonas>and I want to test our bootstrapped GNU sed and not busybox'es built-in... <pder>what is missing from mes libc that is needed for bison? <stikonas>but I think I had problems even with newest flex <stikonas>yeah, regex.h is definitely needed for flex 2.6.4 <Hagfish>i think i've missed my chance to use this meme: weird flex, but okay