<fossy>stikonas: the fix was quite trivial, it was one define <fossy>however, we don't have an interactive shell until we can mount /dev <fossy>(this will be the same for dash) <fossy>wait looked at what, bison or dash <stikonas>that seems to use some symbols that we don't have <stikonas>I thought after tcc we will be able to move quite fast <fossy>i doubt we will be able to compile musl easily ***cbaines_ is now known as cbaines
<fossy>i'll take a quick look at musl <stikonas>musl might also help with those heirloom lex and yacc <stikonas>well, I haven't finished looking but they use at least quite a few incldues that we don't have <stikonas>but maybe I can get it from somewhere else <fossy>stikonas: we have getopt in libgetopt.a <fossy>you just need to link with -lgetopt <stikonas>so musl I was trying to build with CC=tcc ./configure --target=i386 <stikonas>(configure can be replaced later, I don't think it is essential <stikonas>but then I got ./src/internal/syscall.h:237: error: identifier expected <stikonas>oh, sorry, yacc was actually complaining about optarg, but maybe we can still find it <OriansJ>and TCC should be able to handle that <stikonas>ok, I've got passed compile stage after patching some stuff... Now linking <stikonas>not sure how functional or even working binary will be... <OriansJ>all this tells me I need to find time to work on the M3 solution <fossy>lmaoooo old bash + new configure script = autoconf has an absolute fk you moment <fossy>autotools absolutely *requires* /dev <fossy>mkbuiltins is a shell script <fossy>so we would have to either reimplement it in C or otherwise rewirte it for kaem <stikonas>ok, I think I managed to build yacc by basically replacing all wide functions with normal <fossy>idk why youd want that but whatever <fossy>furthermore dash has a .y file too <fossy>hm, not in newer versions tho? <stikonas>ok, that yacc is too weak to build bash parse.y <stikonas>or maybe I don't know how to use yacc... which is more likely <stikonas>hmm, I think all in all, coreutils is probably the easiest next thing <stikonas>given that you already have some experience building it <stikonas>although, I don't think we can just mount /dev <fossy>stikonas: no, we have to do the full bootstrap, yacc is too weak <stikonas>well, hopefully that patch didn't break it... <stikonas>unless you want to look at it while I sleep <fossy>if you want to push to a branch thatd be great <fossy>i'll try and get it into a makefile <fossy>i'll also merge bash, given that dash seems to be a PITA <stikonas>although, we haven't reallly bootstrapped bash yet <fossy>what if we did like -Dwchar=char <fossy>yeah ik, but for the wide stuff <fossy>would be much easier to read <justinmoon>Hey. I'm looking for an explanation of all steps involved in building guix from hex0. Thanks. <fossy>it is really not very simple <fossy>stikonas: (directed at justinmoon) <fossy>stikonas: i'll use your patch first <stikonas>justinmoon: guix takes quite a few cheats to get from hex0 to there <fossy>then if the build succeeds i'll try replacing some hunks with -D <stikonas>fossy and I are discussing some of these issues now <fossy>justinmoon: for the lower mescc-tools-seed steps, basically <fossy>hex0 compiles hex1 which compiles hex2. then hex2 compiles M0 then M1 then M2. M2 compiles M2-Planet, which is the cross platform C compiler. it recompiles better versions of hex2 and M1 <fossy>then from M2-Planet we can build things like a scheme interpreter (mes) <stikonas>(which does a few cheats but it's better than anybody else right now) <stikonas>e.g. stuff like ./configure scripts are not really source... <justinmoon>Perfect. Is guile-bootstrap the full guile implementation that can build guix? I understand mes scheme is less capable than guile? <stikonas>justinmoon: guile is scheme implementation <stikonas>it's full implementation unlike mes which is partial <stikonas>e.g. mes can't run those tools from that graph (bootar, gash, etc...) <stikonas>and since we don't have guile, we don't use it in live-bootstrap project either <xentrac>rain1: thanks for the planckforth link! it seems more perspicuous than most tiny forths <xentrac>fossy: flex, bison, automake, and autoconf are frequent let's-check-in-build-artifact offenders <xentrac>stikonas[m]: I don't think GNU was designed for our kind of auditability-focused bootstrapping at all; rather the contrary, they believed that if a programming language wasn't even good enough to write its own compiler in, it was a toy <xentrac>so we should count ourselves lucky that they wrote as much in C as they did, which was largely a matter of encouraging people on proprietary Unices to adopt their tooling <xentrac>fossy: autoconf configure scripts are designed to run on proprietary Unixes (and VMS!) that don't have bash installed, so any reasonable bourne shell ought to be enough, although maybe that's slipped a bit in the last 20 years <xentrac>stikonas[m]: every package is a struggle, but progress is astoundingly fast now <vagrantc>any wild guesses as to why mescc-tools started failing on Debian in version 1.1.0 ... seems to be Kaem/test/test14/ ... which just calls printf <xentrac>(running autoconf itself I think might require bash, though, even if its configure output runs okay in SunOS 4 ksh or whatever) <vagrantc>my main guess is the user's shell being /bin/sh vs. /bin/bash, or /bin/sh pointing to /bin/bash vs. /bin/dash <vagrantc>but i haven't figured out how to trigger it with any of those combinations <vagrantc>although "talking" it out is giving me some ideas to try :) <xentrac>vagrantc: hmm, nothing obvious comes to mind <fossy>good day vagrantc, let's see <fossy>vagrantc: how do i access the logfile <vagrantc>oooh! i've seen it fail with a local build for the first time <fossy>vagrantc: well, i wrote these tests <fossy>vagrantc: do you have a logfile of the failure <vagrantc>but that's the build log ... not much information in there <fossy>but you said you just reproduced it loally? <vagrantc>$ LANG=C /usr/bin/kaem -f "test/test14/kaem.test" <vagrantc>hiprintf: warning: ignoring excess arguments, starting with '' <vagrantc>but when i build the package normally, it works fine... <fossy>are the printf versions the same? <vagrantc>but something in the reproducible builds environment triggers it <vagrantc>my first hunch is an issue with /usr/bin/printf vs. a bash printf builtin <fossy>how would kaem be calling a bash printcf builtin <fossy>it is def calling /usr/bin/printf <fossy>dont think that i made kaem strip empty tokens <fossy>not sure how that explains the difference tho <vagrantc>still haven't figured out the difference between a working environment and a non-working one... <fossy>what debian version is reproducible-builds running on vagrantc ? <vagrantc>the host is buster in all cases, same as buildd.debian.org, the chroots are their respective releases <vagrantc>so, i'm seeing it in an environment where i have stdin/stdout ... <vagrantc>coreutils hasn't been updated since october, and I only uploaded a few days ago, so that is essentially the same version <vagrantc>oooh, got different tests to fail from reprotest... <vagrantc>LANG=C ../bin/kaem -f "test/test14/kaem.test" <vagrantc>+ [ test/test8/proof: ɹ = test/test8/proof: OK ] *vagrantc tries without crazy locales <vagrantc>built fine without varying locales ... which is unfortunate, i was hoping to use reprotest to triage what is breaking the other issue <vagrantc>same chroot, built from interactive shell and Kaem/test/test14 fails, from automated build process succeeds... <vagrantc>well... i'll /o\ a bit more and move on to other things :/ <vagrantc>fossy: you're gonna love this: -hiprintf: warning: ignoring excess arguments, starting with '' <vagrantc>+hiprintf: warning: ignoring excess arguments, starting with ‘’ <vagrantc>the difference appears to be the use of '' vs. ‘’ <vagrantc>it does prefix with LANG=C ... but i think there are other variables that can affect the locale and override lang <vagrantc>LANG=en_US.UTF-8 ./bin/kaem -f "Kaem/test/test14/kaem.test" <vagrantc>hiprintf: warning: ignoring excess arguments, starting with ‘’ <vagrantc>LANG=C ./bin/kaem -f "Kaem/test/test14/kaem.test" <vagrantc>hiprintf: warning: ignoring excess arguments, starting with '' <vagrantc>now for a magic trick, since they're called with LANG=C <vagrantc>LC_ALL=en_US.UTF-8 LANG=C ./bin/kaem -f "Kaem/test/test14/kaem.test" <vagrantc>hiprintf: warning: ignoring excess arguments, starting with ‘’ <vagrantc>the test expects the UTF-8 locale variant <vagrantc>because the error message that's part of the test includes different content <justinmoon>Oriansj: thanks for the graphic! really helps :) <OriansJ>vagrantc: I think I should figure out what is causing the behavior to be different that what bash would do given that input and fix it. <vagrantc>yeah, the other option would be to fix the printf error output :) <OriansJ>it is easier to fix kaem than printf <vagrantc>why does the test explicitly capture the stderr ? <vagrantc>you want to actually check that it catches the error correctly? <vagrantc>at any rate, i've done my part, come up with a workaround, look forward to a "correct" resolution :) <OriansJ>well test/results/test04-output would be the one checking if it catches the error correctly <OriansJ>as test 14 is just supposed to test end-of-line escapes according to note in the header <OriansJ>vagrantc: please test the latest mescc-tools to verify that I have addressed your issue. ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<stikonas[m]>foosy: do we need m4 before bash. Those bison bootstrap steps had it there, although I don't know why... <malina>I am literally doing my first musl bootstrap at the moment and just did bash, after doing m4 and bison or maybe bison was after bash <fossy>stikonas[m]: i believe bison makes use of m4. <fossy>gio: can you confirm if m4 is needed for bison/flex bootstrap as implemented in nbs? <malina>but m4 is part of most gnu automake tools and what not. you can make do without but I suspect it can get a bit hairy without.. , after all, you are trying to bootstrap into a stand alone thing so you want gnu tools <malina>unless of ocurse you intend to bootstrap something more posixly or so <malina>i did bash/bison just now. in that order. <xentrac>did you use the prebuilt grammar C file that ships with the bash source, but which is rebuilt by bison when you change the grammar? <gio>(I did this stuff years ago, so I don't remember all the details without checking the source code again!) <malina>after cross compiling from glibc into musl with gcc, and erm i did m4, expect , bsd-curses (I think or maybe I still used ncurses.. "musl" will be my totally experimental sandbox :D), <stikonas>fossy: by the wya, have you started looking at yacc? or should I look at it while you sleep? <gio>Technically that is autogenerated, that's true. I leave it for a future improvement to cherry pick which of those lines are really required and make the file more auditable. <gio>Though it's not the kind of file that would make an audit complicated. <stikonas>well, we were mostly adding those defines to CFLAGS for now... <malina>although the next step, I did toybox instead of coreutils, so I expect from now on, things will start to break pretty randomly :p and I will go again <stikonas>you need bison to build bash from source... bash source contains "non-source" y.tab.c, so we need to generate that first <xentrac>I defy anyone to modify y.tab.c by hand to update bash's grammar <xentrac>or to spot a number that's been changed from its correct value <stikonas>and I guess distros don't rebuild y.tab.c before compiling bash... <xentrac>they certainly could; they'd have to if they have their own patches to the .y <stikonas>oh yeah, I understand that they could, they already have bison installed, it's just that they don't... <stikonas>hopefully .y file is at least reproducible. although, it probably depends on which version of bison you use <xentrac>and possibly, though hopefully not, on which version of libc it's linked against, on which CPU architecture, on which kernel, and compiled with which compiler and which compiler options <fossy><~anton> unfortunately we don't have the old user data as deepend deleted the vm already <fossy><stikonas> fossy: by the wya, have you started looking at yacc? or should I look at it while you sleep? <fossy>I spent some time writing a readme <fossy>Tomorrow im going to push a restructure too (kaem files in a diff location etc) <stikonas>since I already looked at yacc a bit yesterday <stikonas>but yeah, we definitely need some restructuring <stikonas>fossy: oh, you last commit doesn't copy all makefiles into sysa <stikonas>well, I guess you'll restructure everything tomorrow anyway <stikonas>probably the simplest fix for no is to remove "." before globbing in get_file function <stikonas>yeah, bash-2.05b-builtins.mk and common were not copied <malina>(btw, ye bison isn'tneeded, my bad, i should have known sigh, Just happend ot be doing some old scripts). <OriansJ>gio: Proper documentation is always a love letter to yourself in the future. Be kinder to yourself and take the time to write one. <stikonas>janneke: I have a question regarding mes c lib. It looks to be GPLv3, isn't it? <stikonas>I'm worried that this would cause problems with static linking <stikonas>i.e. if we link mes libc with non-gpl v3 application <stikonas>yeah, but unless it's LGPL, I think I'll be stuck <stikonas>well, maybe that's the reason to build musl libc early... <stikonas>janneke: maybe that's not critical, since we are not distributing those binaries... <janneke>static or dynamic linking makes no difference, btw <OriansJ>stikonas: it doesn't apply to non-distribution where you have the source code <stikonas>yeah, so I guess we can still do bootstrap... <OriansJ>and we shouldn't be distributing bootstrap binaries anyways <stikonas>we'll discard those temporary executables <OriansJ>as only the bootstrap seeds and the final binaries might be distributed. <OriansJ>if needed janneke and I can make M2libc LGPLv3 and get it to cover the bootstrap libc needs <stikonas>well, I'll have some non-distibutable binaries for now <janneke>maybe a better solution is to ask upstream to relicene their code gplv3+-compatible? <stikonas>janneke: I think no chance of that happening <janneke>well, if upstream does not want to play nice, should we give in on our free software prinicples? <stikonas>well, yacc is only needed to bootstrap bison <janneke>anyway, we need to make the right choice when the need arises <janneke>any specific package that you are thinking about right now? <OriansJ>janneke: well no one here will be asked to compromise their free software prinicples. <OriansJ>Everything must have source code and the 4 essential freedoms <stikonas>those are needed to bootstrap flex/bison from GNU <stikonas>yeah, either you consume final binaries as end user, so you still have 4 freedoms <janneke>stikonas: well, this could be a great way to raise awaress within GNU about bootstrappability <janneke>it's not our problem if bison cannot be bootstrapped <janneke>or only with non-gplv3+ compatible package <janneke>it could be a feature to address this problem to GNU and the bison maintainers <janneke>OriansJ: in general the advise is against using the lesser gpl <janneke>for strategic purposes, we could make that choice, but i would not make it lightly <stikonas>well, in any case we don't distribute combined CDDL and GPL stuff <OriansJ>janneke: well M2libc is just a C library <stikonas>well, glibc was LGPL I guess for exactly that reason <OriansJ>and I quote: The most common case is when a free library's features are readily available for proprietary software through other libraries. In that case, the library cannot give free software any particular advantage, so it is better to use the Lesser GPL for that library. <OriansJ>M2libc is the bare minimal C library with nothing special inside. <OriansJ>janneke: everything else is GPLv3+ (except blynn-compiler which is only GPLv3) <janneke>OriansJ: right, so let's see where this goes <OriansJ>indeed. We will leave M2libc as GPLv3+ for now and if it becomes an issue we convert M2libc to LGPLv3+ and have everything that is a problem use it <xentrac>stikonas: no chance of that happening today <xentrac>but I think the ownership of yacc in the last 25 years has belonged to AT&T, Novell, Caldera, SCO (the real one, not the evil one), Sun, and Oracle <xentrac>so in five years it might have a new owner and they might be more open to relicensing <OriansJ>xentrac: well Oracle grade evil is easier to deal with. Just pay them enough money and their license it anyway you want. <malina>I have a hard time imagining a ful bootstrap chain as this as anything but open sourced, for one. GPL curbs freedoms to deal with the posit of "evil, greedy businesses and people". Again, can't really see that being a thing for 'source strapping', so licensing shouldn't be particularly interesting here.. I mean.. what.. "Try our totally verifiable strapping from source source, for only 10.99, mind you, it's a closed binary". lol. you can try <malina>licensing a binary/ies from the bootstrap, but truth is, the method to deduce any binary from such bootstrap is hard to license. (as in, a specific solution you can licence). it will never prevent someone to decide to make another solution with prior steps and so whatever they want. which after all, is the point of somehting free, you say? it's like yacc, lex , flex and bison. none of the parsers or so which are generated are bound by the <xentrac>hmm, do you think Larry Ellison is motivated by profit? I thought he was motivated by power and winning <xentrac>it's possible that I just don't know anyone who has enough money to bribe Larry Ellison <xentrac>but that such a quantity of money might exist <OriansJ>xentrac: think of him like a lawn mower. when given a choice between $xK and $0; he chooses money everytime. <xentrac>I'm not sure that's necessarily a practical solution to the problem, though. Oracle is seeking US$8.8 billion in damages in Google v. Oracle, and if they prevail, they miight be in a position to file similar lawsuits against anyone who has ever implemented, say, the Unix API or Oracle extensions to SQL <xentrac>it might take a similar sum of money to get them to see yacc as worth releasing from their arsenal <xentrac>(if they really are yacc's owner, which I'm not totally sure of) <OriansJ>xentrac: worst case, someone here rewrites yacc in Haskell <OriansJ>xentrac: or C or even assembly if they want <OriansJ>we will figure it out like we always do <OriansJ>Nothing is going to stop the bootstrap from happening <OriansJ>The worst that happens is it gets slowed down by something <xentrac>actually another approach that might be adequate might be to use simpler, less efficient parsing algorithms. not sure if that can be call-compatible <xentrac>but I mean it's quite straightforward to do general context-free parsing with backtracking for example. it just has an exponential-time worst case <OriansJ>and gash-utils should have sed as well <OriansJ>They are all supposed to run on mes.c <xentrac>malina: it's entirely plausible to imagine, for example, BAE Systems producing a proprietary derivative of our bootstrapping toolchain. they would of course need to provide the source to their customers, but they could want to forbid those customers to redistribute it --- or, worse, to modify it. So strong copyleft is doing its usual share of work in this case <OriansJ>so is gash if I remember correctly (correct me if I am wrong janneke) <xentrac>(I'm currently doing work with a subcontractor to BAE Systems, and it is a great deal of headache to deal with them; they are quite ruthless) <OriansJ>xentrac: military contractors are never fun to work with and for obvious reasons. <xentrac>that hasn't been my experience in the past at all <xentrac>but certainly many of them would love to sell their special security sauce to security-hungry clients with deep pockets and an appetite for long-shot research that might pay off <OriansJ>In the words of the movie Pentagon wars. In baseball the losing team isn't killed. <OriansJ>correction: In baseball, the losing team isn't killed by their opponents. <stikonas>OriansJ: gash currently doesn'trun on mes.c <stikonas>OriansJ: that's why we didn't use it in live-bootstrap... <stikonas>at the moment we are still using kaem for scripting <stikonas>although we have now switched to Makefiles for building <stikonas>makes building a bit nicer than writing kaem commands... <stikonas>non-interactive bash is close too but we need to sort out bison first <stikonas>well, I think I'll first try to get that AT&T yacc working <stikonas>well, if anybody knows nyacc, feel free to join too :) <stikonas>that might be nicer solution in the long term <OriansJ>well janneke should know nyacc as he uses it in MesCC <stikonas>janneke: do you have any idea how hard would it be to bootstrap flex/bison in nyacc? <janneke>stikonas: i'm really only using nyacc, the c-parser was fully written by mwette <janneke>i suspect that yacc is a lot simpler than c, though <rain1>pretty much everything is accidentally turing complete <bauen1>rain1: or a half-complete implementation of common lisp - someone i don't know the name of ***roptat_ is now known as roptat
<deesix>OriansJ, the 'TMP' branch of my M2-Planet repo updates the M2libc submodule and uses the standard naming that comes with it; in the test scripts. If you think it's OK, feel free to cherry-pick 89829fc38eba16fcbd09d0eb5f1e8a04a76c7e47 <deesix>OriansJ, sorry. It turns out it's not OK. I'm preparing a replacement commit. <deesix>OriansJ, please use e9553d5f1344752d31e157c33f1cc790843d24c6 _instead_. Sorry for the inconvenience!! :( <deesix>The change is about test0101, creating some x86 proof, which uses the renamed ELF. ***civodul` is now known as civoudl
***civoudl is now known as civodul
<deesix>I found it during unification, as those cross-arch situations needs some extra care. <fossy>many distributions link together zfs on computer side, I.e. it is compiled and installed by the end user as we are doing <stikonas>and we are not installing, we already discussed this with other people here... <stikonas>fossy: anyway, I briefly tested that yacc binary and something is broken... <fossy>bauen1: does bash/make run on myunix or does it need too many syscalls <stikonas>it complained that token on line 3 is unrecognized keyword <stikonas>I also tried building yacc with gcc and it still failed but quite a bit later <stikonas>anyway, the patch does break more things <stikonas>maybe it's possible to fix that patch... <stikonas>fossy: bash segfaults, but I guess you know that <fossy>stikonas: yes in an interactive shell <fossy>not if u use it to run a script <stikonas>we can probably do it in a couple of lines <fossy>I patched the tty check out which allowed normal scripts to succeed <OriansJ>deesix: the revised commit has been uploaded <stikonas>but I think I'll try to write a few liner c app that created /dev/tty <deesix>OriansJ, I see, thank you very much. <fossy>stikonas: heirloom yacc is cddl 1.0 yeah? not 1.1? <stikonas>fossy: anyway, it's not just missing tty... <stikonas>actually even with /dev bind mounted from host system it crashes <stikonas>I think it refuses to run even empty script... Only bash --help and bash --version work <stikonas>well, I can try to build lex but it probably also needs all those wchar functions <fossy>I ran a script inside live enviroment using the bash...