<stikonas>hmm, something goes wrong with bison on stage 2... <stikonas>I think it will work but we need to update m4 <OriansJ>stikonas gforce_d11977 when one has issues with reproducibilty. It is best to do the following process: a fresh git clone [to rule out local residual generated files], record every command issued [to check for skipped setps], get relevant system details such as OS, versions of qemu, Amount of RAM and CPU core count etc [To find something you can experiment to duplicate the cause and solve the issue.] <OriansJ>The phrase "I can't reproduce" is a red flag of we might be missing something important and need to know why they are having the issue, even it ends up being bad ram or failing hard drive. <stikonas>OriansJ: well, that's because gforce_d11977 was runnning a kernel with allnoconfig <stikonas>and this late in the bootstrap I think we are starting to rely on more features... <stikonas>it's just that we haven't dealt with kernel yet <stikonas>but in the future I think we'll compile new sometime after tcc, so we can start using more kernel features now <OriansJ>stikonas: and we recorded what sort of error it produced and have steps to duplicate should we wish to address it in the future? <OriansJ>A known_issues.org might be helpful should someone else hit the same problem in the future. <stikonas>gforce_d11977: is bisecting which kernel option is needed now <stikonas>which is the first application that we couldn't build with mes libc and required musl <OriansJ>stikonas: an inch of progress towards landing on the moon is miles better than imagining that you are already there. <Hagfish>far till gcc? what are the main milestones between here and there? <stikonas>Hagfish: bison (fairly close now)->get ./configure scripts working (that needs mainly gawk and rebuilding coreutils, i.e. build more utils which we couldn't build with mes libc) and then big one is perl... (to get automake/autoconf working). After that it would easy <stikonas>(maybe will rebuild more stuff once we have binutils) <stikonas>but stuff is parallelizable... can get there quicker if more people help :) <Hagfish>thanks for the info. i forget that all these low level tools are so tightly welded together <stikonas>and it would be much harder if not for all the work other people did before <stikonas>especially work on Guix but also gio's bison-bootstrap <Hagfish>i was really excited about gio's asmc, but didn't realise he'd worked on bootstrapping bison too <pder>stikonas: with the new tcc + musl, there still seems to be issues with floating point numbers. printf("%f", 1.2) or any other value prints 0.0 <pder>I tried rebuilding core-utils but ran into a few issues. Maybe the preprocessor defines need to change for the new libc <gforce_d11977>stikonas: it can be worth checking why a configure script wants 'gawk'. often they only do thing like 'echo foo bar | awk print $2' <gforce_d11977>for the record: still bisecting the reproducibilty/kernel-symbol thingy, but i am near! 8-) <gforce_d11977>(my wife is complaining that the computer runs to whole night and is loud!) 8-)))) ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<gforce_d11977>fossy: stikonas: OriansJ: that was easy, after having run nearly 60 livebuilds, a found the culprit: ofcourse (!) this is needed: CONFIG_MODIFY_LDT_SYSCALL=y <gforce_d11977>i make a last testshot in i386 and x86_64 mode, will upload to logs and make a PR for live-bootstrap <stikonas[m]>pder: yeah, some cflags might have to be different. And if it's easier, maybe we can build just missing binaries? <fossy>stikonas: i'm going to reusify tomorrow <fossy>it makes doing this a breeze <fossy>pder: the preprocessor defines without a doubt need to change <stikonas>fossy: ok, I'll try to get newer m4 in the evening... <fossy>btw im sorry ive dropped off a bit on doing stuff, im back at school now so sorry if things move a bit more slowly on my enx <stikonas>I think from now on things should go a bit easier (maybe with the exception of perl, but hopefully it is doable) <stikonas>OriansJ: we only use static binaries for now <OriansJ>stikonas: that might be true but I can't imagine anything good besides a dynamic linker which would use the modify_ldt syscall <Hagfish>might be an interesting talk to watch on sunday <bauen1>isn't there some talk about mes too ? <stikonas>not sure about fossy, but I would prefer downloads to be cached in more permament location, /tmp gets erased after reboot... <stikonas>although, I can override location using env variable even in your PR... <gforce_de1977>stikonas: the other approach would be e.g. cache=~/.cache/live-bootstrap <stikonas>yeah, then we don't need to export env variable every time <deesix>Not long ago it used ../sources or something like that, but that was removed recently without a reason in the commit. <deesix>Maybe that path is not available on every situation? Anyway, it's great to see it back, gforce_de1977! <deesix>Only accepting PR (on github I guess) is another pain point for some contributors. The submodule thing being another one. (my 2c) <gforce_de1977>fossy: deesix: i understand your opinion, maybe fossy can accept things via email? also a short "pastebin" with your changes can help <gforce_de1977>is a native english speaker around? Is the correct sentence: "pull yourself up on your boot straps" or "out on your.."? <siraben>gforce_de1977: I've heard it as "pull yourself up by your bootstrapps" <siraben>"by" because the bootstraps are used as a means to pull oneself up here <gforce_de1977>siraben: but you are not a native speaker, so that does not count 8-))))) <siraben>who says I'm not a native speaker? lol <siraben>I've known English (and Thai) my whole life <gforce_de1977>siraben: from your website it looked like you are from thai...yeah, but english is official in thai? <siraben>Long story but I used to live in Canada when I was little so I learned English first, then when I moved back I studied in international schools where English was the language of instruction (teachers were native English speakers too). <gforce_de1977>siraben: ok, sorry, that does count, so you are 1000 times more a native english speaker that I 8-) <siraben>(maybe I should put my TOEFL score (118/120) on my profile) /s <Hagfish>(to confirm, as a monolingual english speaker, the generic phrase is "to pull oneself up by one's bootstraps", and is used like so: "you need to pull yourself up by your bootstraps") <gforce_de1977>siraben: i'am from germany, but dont know my TOEFL score (was around 2001 i guess) 8-)))) <bauen1>lol, this conversation reminds me that there was still some kind of english test i wanted to take so i can get the "english speech level" (Niveaustufe) that is basically native speaker <bauen1>tbh figuring out this system is more complicated than learning english <stikonas>last time I had to take english test was before I started my undergrad... Didn't have to take anything since... <bauen1>stikonas: i mean it's optional, but nice to have <stikonas>well, maybe... I just meant nobody was asking me... <gforce_de1977>stikonas: without knowing you, when you write/read everyday some english words, you are better than 90%... <pder>stikonas: If the plan is to build bison-3.4.1 like in gio's bison-bootstrap project, it looks like it needs m4 1.4.6 or later <stikonas>yeah, well, only 1.4 in master, 1.4.4 in my PR <stikonas>pder: by the way, do you have those errors that you saw in coreutils? <pder>Do you know the path to get awk? <stikonas>I unpacked it and it litered my working directory <stikonas>so it should be just a matter of running make <stikonas>pder: then we also need at least uname sort and expr from coreutils <pder>Let me take a look at coreutils- I made a little progress rebuilding with tcc + musl <stikonas>and that will start letting us to run at least some configure scripts <pder>beware of a strange bug I noticed with pipes or redirecting output. I noticed it is sometimes truncated. Same result piping to tee <stikonas>so far I haven't noticed anything like that <pder>I was trying to build bwkawk. And the command maketab awkgram.tab.h outputs more to the terminal than it does with redirected to a file <stikonas>anyway, I think gawk should be easy. As far as I remember, the only problem was .y file there <OriansJ>gforce_de1977: To pull one's self "out" by one's bootstraps is the term for when one is to escape a bad situation such as poverty. To pull one's self "up" by one's bootstraps is the term for when one is to engage in an impossible task such as flying by pulling up on one's bootstraps. So out would be most applicable to the work being done here. <xentrac>not familiar with this "out by bootstraps" phrasing ever <xentrac>but I suppose von Munchhausen was pulling himself out of the swamp <OriansJ>in more fun news, M2-Planet might be taking advantage of qemu-userspace to simplify the problem of cross-platform testing via EMULATION flag <OriansJ>It'll reduce how much I'll have to bug yt to do testing. <deesix>Isn't an override enough? I did that during my first steps with aarch64. <deesix>... and more recently with armv7l on armv6l. <deesix>Some bugs in the emulation casused some problems, but more recent versions of qemu solved them (without introducing others, let's hope). <OriansJ>deesix: Well we only do require a few dozen working instructions <OriansJ>and yt can always sanity check with real hardware. <deesix>OriansJ, why is the override not enough? <OriansJ>The rest I have hardware for testing (except knight ^_^) <OriansJ>deesix: because aarch binaries do not run on x86 without an emulator <deesix>I still test AArch64 on hardware before the final steps of my branches. <deesix>OriansJ, the override makes it hit the emulation, without any more changes. <deesix>What's the new flag bringing to the table? <OriansJ>you mean make detection of override variable being set to enable emulation? <deesix>I mean that export GET_MACHONE_FLAGS="--override aarch64" make the scripts run the binaries, that the system knows how to emulate with qemu. <OriansJ>well I was thinking of the cases where AMD64 hardware could run x86 binaries natively and potentially other combos like like; hence allowing the pretend to be architecture and enabling emulation being separate but I can see your prespective. <deesix>If the system can run them natively, then emulation does not hit, I guess. <OriansJ>you don't even need to prefix qemu-aarch64 if you have qemu-user installed <gforce_de1977>in German, this is a similar saying, "pulling yourself out of the mire" <gforce_de1977>but i'am not sure if the origin is the Baron Münchhausen, or if Baron Münchhausen just takes this already known sentence.... <gforce_de1977>OriansJ: the münchhausen picture maybe can be used in bootstrapping slides, instead if yogurt 8-))) <xentrac>gforce_de1977: it is not entirely clear, historically speaking <OriansJ>deesix: I guess I learned something cool today about qemu-userspace. once installed, the kernel spawns qemu with the correct architecture and the program and arguments as arguments to qemu. No code changes actually required <Hagfish>"Like the famous Baron von Munchausen, the persons affected have always travelled widely; and their stories, like those attributed to him, are both dramatic and untruthful. Accordingly, the syndrome is respectfully dedicated to the baron, and named after him" <Hagfish>interesting that the syndrome is named after the same person, but because it was named by a British guy who didn't understand umlauts, the spelling of the syndrome is slightly different <Hagfish>no wait, the fictional character doesn't have an umlaut, but the person he was based on did have one <OriansJ>really short version: history has editors and they make mistakes and steal like crazy <OriansJ>I wonder what it would take to add knight-posix to qemu-userspace <xentrac>OriansJ: is there a way to get qemu-user to set LD_LIBRARY_PATH in an architecture-dependent way? I always end up either setting it by hand or linking statically <OriansJ>xentrac: don't know because I only generate static binaries <OriansJ>everything from hex0 to MesCC is 100% static binaries <OriansJ>heck, we don't even use environment variables until MesCC (although kaem does support them to a degree) <OriansJ>also, I just started playing with qemu-user for the first time 45 minutes ago. So I am still getting a feel for things <OriansJ>but if I had to guess one could use get_machine in a script to set LD_LIBRARY_PATH in a per architecture basis and then execute the program which would inherit that environment variable. <stikonas>everything using MesCC is also 100% static binaries <OriansJ>stikonas: you are absolutely correct as it would require libc support for dynamic linking which MesCC doesn't have <OriansJ>but TCC with a more convential libc should have. <OriansJ>interesting qemu-user doesn't hook kernel exec on armv7l <stikonas>OriansJ: yeah, tcc with more conventional libc should have but at least tcc with musl is a bit flaky (I think until you have binutils) <stikonas>and I think once you have binutils you are almost done with bootstrapping GCC <fossy>I don't mind having a download cache but the previous implementation was less than amazing <fossy> gforce_de1977s implementation is pretty good, I don't want it to go to ~/.cache though, it should be in the repo <fossy>so either back to sources or another name <stikonas>it would also be nice to add some checksumming <OriansJ>xentrac: you could write a script and hook it in /proc/sys/fs/binfmt_misc to set the environment variable and then spawn qemu-$(arch) <stikonas>pder: hmm, newer m4 is a bit harder to build :(. I guess it should still be possible but I am hitting more issues <OriansJ>just manually added support to armv7l to run aarch64 binaries with: update-binfmts --install qemu-aarch64 /usr/bin/qemu-aarch64 --magic '\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00' --mask '\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff' <OriansJ>although be careful as it is an easy route to enable a rather nasty rootkit via binfmt_misc <fossy>stikonas: i plan to do that after that PR is merged. <fossy>on the topic of checksumming <fossy>OriansJ: did you say you had a crc32 impl in m2-planet <stikonas>we also have modern hashes in coreutils, although, that's a bit later <fossy>stikonas: yeah, i plan to use this impl for earlier stages, and then sha256 once we have coreutils <stikonas>fossy: by the way, did you have a chance to look at flex PR? <fossy>stikonas: i'll do that right now <fossy>OriansJ: what's the crc32_tab array? <fossy>ah, how'd you calculate/make that? <stikonas>there is something on wikipedia about that table <OriansJ>with assembly for the same thing that would be relatively trivial to convert to M1 assembly if one wanted CRC32 checksums before one is done with building M2-Planet <OriansJ>again not converted yet but something MesCC or TinyCC should be able build <fossy>stikonas: fine, one thing, coudl you please document the use of flex-tmp somewhere, as the use of that is not completely clear? (README.md probably?) <stikonas>well, I just took that makefile (with minor modifications) from nbs <stikonas>i.e. compile it's own scanner with itself <fossy>stikonas: could you add a comment to that effect in the makefile (from nbs with minor modifications) <stikonas>or you'll just keep the comment in makefile? <fossy>stikonas: yeah, afaict it goes like this, old flex + scan.l -> scan-tmp.c, scan-tmp.c + other .c -> flex-tmp, flex-tmp + scan.l -> scan.c, scan.c + other .c -> final flex <fossy>but the copyright comment i'll do later <fossy>i'm gussing that the scan.c made by old flex isn't good enough for absolutely everything <stikonas>or it might be just safeguard (like gcc recompiles itself) <stikonas>pder: oh, if you can, try to build uniq too <fossy>just put something like we recompile scan.l using the new version of flex to remove any buggy artificats from the old flex <fossy>hm, looks like i need to update mescc-tools-seed for this crc32 impl, maybe i'll just do that anyay <fossy>can you merge the first and last commits " Add flex 2.6.4. " " Update flex build process in README. " <fossy>then i'll merge (no need to wait for ci again)