<fossy>stikonas: hm, ok, you can just leave it then <fossy>pder: i have an alternative solution <fossy>we don't need the fixed up libc until coreutils <fossy>so we can just recompile mesc libc with patch once we have it <fossy>oh, that's what stikonas said lol <fossy>it's not that hard actually once you decipher the Makefile <fossy>because it uses completely internal headers <fossy>so no -D's really needed thank goodness <stikonas>fossy: so I guess we insert linux at some point before bash? <fossy>i'm putting it directly after patch <stikonas>pder: by the way, if you do libc recompilation, please do in a separate kaem file, so that it is easy to rip off once we update mes <fossy>still haven't figured out it if i want to make a real drive or another initramfs one <fossy>reduces requirements of host kernel <fossy>but i think we can use heirloom one maybe <stikonas>well, depends on how hard it is to compile proper one <fossy>hey bauen1 will myunix maybe support kexec or something similar <stikonas>fossy: bauen1 eventually wants to have some modules <stikonas>so we might be able to compile kernel-side kexec <fossy>well we can figure that out when we get to it <pder>so we do an additional rebuild of libc after bzip2 and before coreutils? <stikonas>so that other binaries also get updated qsort (in case they use it) <xentrac>I thought Savannah.nongnu.org had binaries too, OriansJ? without having to go the gnueval@ route <OriansJ>xentrac: I am taking what I was told to get binaries in bootstrap-seeds onto savannah <stikonas>well, it might get more expotion as GNU project anyway <OriansJ>So if it means I need to sing and dance to get this done, so be it <xentrac>but not savannah.nongnu.org I think? <fossy>pder: i was thinking next to tcc-patched.kaem <fossy>mes-libc-patched.kaem just after it or something <xentrac>OriansJ: kexec is in its essence just an unconditional jump to some new code, which is about 10 bytes I think <OriansJ>xentrac: as I understand it savannah.nongnu.org will not accept binary blobs at all. <xentrac>but you need some way to load that code, some kind of binary format support, etc. <xentrac>oh really? I guess I must have misunderstood that <OriansJ>either that or they rejected my submission for some other reason and then just told me a lie. <OriansJ>If it takes copyright assignment to the FSF and a 50page textinfo manual to get this done, I'll only have to do it once. <OriansJ>Everything else is pure source and they can't complain about me depending on a Gnu project. <OriansJ>although they will probably bitch about the git submodule forest I have created. <xentrac>so if there's a policy against hosting binaries on savannah.nongnu.org it's recent and I can't find it <OriansJ>fossy: yes; although I probably should rename it to stage-posix per janneke's recommendation when i upload it savannah. <xentrac>there is a policy against hosting non-GNU projects on savannah.gnu.org <OriansJ>maybe they are just enforcing it on new projects <xentrac>maybe you got a reply from a new volunteer who doesn't understand the policy correctly <xentrac>the FSF tends to have a lot of volunteer churn because Richard is abrasive <vagrantc>rms is no longer part of FSF, last i heard <pder>stikonas: does the patched qsort make any difference with heirloom yacc> <stikonas>I guess I need to rebase on top of your branch <stikonas>by the way, should we comment our blynn-kaem for now until it's not really used for anything? <stikonas>but keep everything in the repository... <OriansJ>xentrac: look it isn't about binary releases but binaries as source; checked in as source and git commited as source <OriansJ>as when one does git clone bootstrap-seeds, they are downloading every binary for every commit <OriansJ>when someone updates the hex0 source file, they are updating the binary too in that exact same commit <xentrac>OriansJ: oh, I see, that makes some amount of sense <OriansJ>it is up to all other bootstrap-seed developers to verify that the binary checked in and the source correspond using any means outside of bootstrap-seeds. Be it sed+xxd or a custom C program or etc <xentrac>I mean dataexplorer and lzip aren't doing that <OriansJ>vagrantc: so that there is no build process <OriansJ>no need for init, shell, or ANYTHING <xentrac>I wonder if it would be adequate to check in the hex dump in xxd format <stikonas>vagrantc: those two binaries should be enough to build EVERYTHING in userspace... (except for initial POSIX kernel that runs those binaries) <OriansJ>xentrac: I guard its contents like an angry dragon. <OriansJ>stikonas: don't forget the NATIVE tree that doesn't even depend upon a POSIX kernel <stikonas>yeah, indeed. Although, I haven't tried it yet <vagrantc>i guess the mescc-tools tests were significantly changed for 1.1.0 ? never had any locale issues triggering test suite failures before ... :/ <OriansJ>stikonas: let us just say floppy disks and a boatload of manual steps <deesix>vagrantc, looking at the logs you linked... it seems that our 'LANG=C foo' line is not run as one line. <OriansJ>vagrantc: or Debian's reproducible build system is checking for LOCALE changes now <vagrantc>OriansJ: it has definitely been checking locale issues since i started getting involved 5 years ago <OriansJ>vagrantc: or it is just my fault for introducing another bug <vagrantc>there may have been a shift as to exactly which locales are checked, but not anything hugely significant <vagrantc>and the earlier 1.0.1 builds in debian actually had *most* of the patches in 1.1.0 applied <vagrantc>my workaround for the tests in Kaem/ appears to have worked, at least, but i'm guessing LANGUAGE is the new culprit <OriansJ>vagrantc: regardless for the reason it stopped working, I want mescc-tools to be 100% reproducible <vagrantc>e.g. it used to fail on the 1st build, and not it succeeds on the 1st build and only fails on the 2nd build <OriansJ>and will do what is required to address it <vagrantc>i'll see if i can figure out a simple reproducer like i did for the first issue <OriansJ>vagrantc: I just need your help checking that I didn't miss anything. <deesix>vagrantc, I mean, these two lines: <deesix>++ sha256sum -c test/test8/proof.answer <vagrantc>deesix: i don't think LANG is the issue, as in the example, LANG=C is exported in the build environment <vagrantc>i think it is the other variables, notably LANGUAGE <vagrantc>since in the build where it is failing LANG=C and LC_ALL=C are exported in the environment <vagrantc>when LANGUAGE is unset, it builds fine now <vagrantc>e.g. the 1st build in the reproducible builds tests <OriansJ>well I have a patch that works around the issue specified in the test <OriansJ>but I am wondering what other locale environment variables might interact despite doing that <vagrantc>what i also found odd is different tests failed on the different architectures (where there is a different LANGUAGE= value set) <vagrantc>OriansJ: i don't think LANGUAGE can be set to C ... or can it? <vagrantc>well, works sounds good enough for me :) <vagrantc>i guess my other workaround would be to export LANGUAGE=C ... <OriansJ>you can reproduce by doing export LANGUAGE=nl_BE:nl before doing make clean test; see it fail with the same error, apply the patch and see it now successfully complete <OriansJ>and i we find another locale environment variable that causes trouble, we know where to stick it <vagrantc>any reason not to export them for the whole build? <OriansJ>vagrantc: because localization is sometimes useful and I am only turning it off then it absolutely required. <vagrantc>fair enough ... i may be more shotgun style to workaround in Debian <vagrantc>e.g. in Debian i can assert that it's reasonble to build with a specific locale <xentrac>OriansJ: I think localization should be turned off by default for a reproducible build. i.e., LC_ALL=C <xentrac>because among other things the C locale changes a lot less often than the en_US.UTF-8 locale or any other <vagrantc>yes, when attempting to *ensure* a reproducible build, you want to specify the same environment. If you want to find potential reproducibility issues, you vary as much as you reasonably can <OriansJ>xentrac: well the reproducible build of mescc-tools that matters is done by M2-Planet <stikonas>pder: no, patched qsort does not help with yacc <vagrantc>right now debian only is only doing the find-the-bugs style reproducibility tests ... though hopefully that will change soon <OriansJ>and M2-Planet is 7bit Ansi Ascii only <stikonas>vagrantc: you should bootstrap Debian from hex0 :D <OriansJ>In theory you could put unicode in the /* comments */ but outside of that it'll complain loudly <vagrantc>would be really cool to bootstrap it and still come out with identical binaries, of course :) <fossy>is there any way for debian to be built from outside debian currently <OriansJ>vagrantc: it'll be easier after fossy and stikonas finish the path to GCC <vagrantc>fossy: i think arm64 was initially bootstrapped on ubuntu, if i recall correctly ... though that isn't much of a difference <stikonas>instead of downloading from archive, just run live-bootstrap <fossy>good luck with thtat one ever happening <OriansJ>vagrantc: if you can find a way to get M2-Planet to produce non-reproducible output, I will be stunned and probably give you money. <vagrantc>there is someone who does cross-architecture bootstrapping of debian, which has some similar problems, though is pretty different in many ways (e.g. you start with a working cross-compiler) <stikonas>well, if we can get live-bootstrap to automatically bootstrap useful system, then from there it's not too hard to bootstrap distros <OriansJ>good thing M2-Planet is cross-platform too <fossy>stikonas: debian being one of the not so easy ones rn <stikonas>that's why I think I might try to get Gentoo instead... <fossy>interestingly, it's the smaller distros that are easier to bootstrap <vagrantc>debian's bootstrapping origin is lost to time and history ... there were bootstrap seeds at various points, but nobody knows what they are <xentrac>fossy: yeah, that makes a lot of sense <fossy>i still haven't demistified debian's build system yet, but once i fugre out how exactly that works, i can probably make something work <vagrantc>guix and i assume nix have their bootstrap binaries at least recorded, which makes it all easier <vagrantc>fossy: happy to answer questions about debian's build system ... though it's hairy enough can't promise great answers :) <fossy>vagrantc: just FWIW, do you have any reasonably up-to-date resources for me to figure out how i can build a package from a source repo <fossy>is it correct that each package has its own repo <fossy>there's no monorepo that i just havent found right <stikonas>I wonder if live-bootstrap can reproduce guix bootstrap binaries bit-by-bit... <vagrantc>yeah, every package is essentially it's own project <vagrantc>i haven't heard of this live-bootstrap thing till now ... link? <vagrantc>i presume you mean to boot into an environment to start the bootstrapping process? <stikonas>yeah, it boots into live-environment with 1090 bytes of binaries <vagrantc>fossy: it's a little crufty now, but i usually used "debmake" to set up the initial templates <stikonas>vagrantc: basically even more extreme than guix bootstrap <vagrantc>fossy: or look at one of my recently packaged guile packages in debian, they pretty much demonstrate the minimum of what you need <fossy>vagrantc: okey, i'll take a look <stikonas>vagrantc: and we don't allow anything pregenerated, no running of ./configure scripts, no pregenerated bison parsers <vagrantc>fossy: it's basically a handful of files in debian/ ... if you're lucky debian/rules can be a one-liner <fossy>do you just run dpkg-buildpackage to build these things <stikonas>so you don't need to reboot into real hardware if you don't want <vagrantc>i remember civodul worked on a proof of concept to do something sort of like that for guix ... though with a larger set of bootstrap binaries <fossy>guix does this but with guile <fossy>and not as extreme definitions of "source" <vagrantc>it was one of my wild-and-crazy-offhand-ideas that someone actually went and implemented :) <stikonas>vagrantc: so now we are trying to get bash... <stikonas>it's almost done, just need to get bison :D <fossy>the whole thing is nowhere near done <vagrantc>but bootstrapping from an initramfs reduces the set of potentially leaked binaries <fossy>i tried extending mescc-tools-seed <vagrantc>you know exactly what you have and can't accidentally call something else :) <fossy>the amount of GNU stuff that hardcodes path is annoying <stikonas>vagrantc: and also leaks from filesystem like /dev/ <vagrantc>whoah, mescc-tools has been building successfully on debian gnu/kfreebsd-i386 ... still fails on -amd64 <malina>consider fefe's dietlibc too, as a route. I would suspect it's easier to get going than musl. <OriansJ>"Identifier: captures_build_path_via_assert" WTF??? <vagrantc>OriansJ: that identifier *might* be miscategorized and/or from an older version ... it definitely was capturing the build path at some point <vagrantc>OriansJ: i think the build path issue was fixed by 7f3ce4278770f42da06d9dddab9d0f8148998e2 <vagrantc>although requires setting CFLAGS -fdebug-prefix-map=$buildpath=. (or -ffile-prefix-map=$buildpath=. ?) and a compiler that supports those flags <OriansJ>huh, would have expected xz to compress 500GB of /dev/zero better <OriansJ>needs 69M but renaming to drop .xz extension in need allows compression to 11K <vagrantc>OriansJ: i think a git syntax error ... :) <vagrantc>OriansJ: git show 7f3ce4278770f42da06d9dddab9d0f8148998e2 <OriansJ>I guess it is time to get ./bin/get_machine ${GET_MACHINE_OS_FLAGS} --OS operational <OriansJ>vagrantc: but I am the upstream source <vagrantc>OriansJ: "pathspec '7f3ce4278770f42da06d9dddab9d0f8148998e2' did not match any file(s) known to git <vagrantc>command. hah. we're all full of typos. :) <vagrantc>or i cut-and-pasted the hash wrong somehow ... wtf... <OriansJ>with a git clone, git remote add and git fetch as the only input commands? <vagrantc>37f3ce4278770f42da06d9dddab9d0f8148998e2 *vagrantc also cut-and-pasted, but still managed to typo <OriansJ>vagrantc: git gives no shits but does what exactly you tell it <OriansJ>even if what you told it was absolutely batshit insane <vagrantc>anyways, sorry for the confusing ... that commit is what allows passing -f*-prefix-map=$buildpath=. to fix the build path issues <OriansJ>well yes, it inherits environmental CFLAGS <vagrantc>but without that, something embedded the build paths <vagrantc>ok, and now i wait for mescc-tools to build on the reproducible builds infrastructure and hope everything is ok again :) <OriansJ>vagrantc: and I have a solution for kfreebsd-amd64 *vagrantc is kind of surprised kfreebsd-i386 worked at all <OriansJ>Linux emuulation mode being enabled maybe? <vagrantc>can't rely on that being enabled, but maybe it was for that particular build machine <OriansJ>don't worry the fix will solve it forever <OriansJ>and just shortly before I finished the patches <OriansJ>which by the way I just uploaded vagrantc <OriansJ>now setting the environment variable GET_MACHINE_OS_FLAGS will allow you to get get_machine to return any OS string you want. <gio>stikonas: I don't really know how to help you. It works for me without any special care, I'd say. Unfortunately I don't have much time right now to help with your case. <stikonas>gio: anyway, I think we now why it works for you <gio>Ok, good luck finding that! <stikonas>which is not too far, but I'm hitting some issue that I don't know how to solve... <gio>stikonas: Don't know, but I guess I would just try to work out the appropriate byte sequence and write that one with .byte or something. <bauen1>fossy: stikonas: kexec / module loading is probably easy enough to add, just a bit of work to setup paging and copy to/from the right location and call an init hook ? <bauen1>might be a bit more complicated if you need to do some elf loading, but nothing world ending <bauen1>but if i want to, then i'll do that after the first io syscalls are working correctly <OriansJ>looks like I need to rethink how I am going to support FORK and WAITPID in stage0's vm --POSIX-MODE <OriansJ>deesix: actually getting vm to create a core image and spawn a seperate vm instance to run that core image rather than just punting as was previously done in stage0's vm to enable a proper POSIX fork for mescc-tools-seed for knight <OriansJ>which would allow all of the M2-Planet tests to have a knight implementation. <OriansJ>I guess more of a problem of choosing to make stage0's vm have a Linux compatible --POSIX-MODE or actually figuring out a proper MMU design for Knight and Porting Linux to run on it. ***nckx[2] is now known as nckx
<stikonas>pder: so the other mes libc bug can be seen in seq... <stikonas>although, it wasn't completely clear from the code what might be going wrong... <stikonas>it might be a good idea to compile in debug symbols in live-bootstrap by default for now... <deesix>OriansJ, the 'TMP' branch of my M2-Planet repo has three commits, with --exec_enabled removal, full switch to new-style mescc-tools flags (no more --BaseAddress, --BigEndian, --LittleEndian) and a bit of indentation clean-up. Let me know if your current work would conflict. I can redo/rebase later if you prefer. It's only tested on x86 but the change is the same everywhere. <OriansJ>deesix: thank you so much for this. I'll take care of testing on AMD64 and armv7l. With a good look over AArch64. <OriansJ>armv7l and AMD64 pass happily deesix <OriansJ>since none of the checksums actually changed, we know that it all binaries still work, the only bit that could break in these commits are the tests <deesix>Yep. All good on armv6l (with override). <OriansJ>deesix: have you seen the latest mescc-tools variable yet? <deesix>OriansJ, the get_machine one? No, just read about it here yet. <OriansJ>GET_MACHINE_OS_FLAGS will be part of Release_1.1.1 to allow [ "$(./bin/get_machine ${GET_MACHINE_OS_FLAGS} --OS)" = "Linux" ] to restrict tests to hosts OS which will be able to run the binaries should we wish to properly port M2-Planet new more Operating systems. <OriansJ>correction: ["$(get_machine ${GET_MACHINE_OS_FLAGS} --OS)" = "Linux" ] <OriansJ>which will allow export GET_MACHINE_OS_FLAGS="--override windows" or any other host OS we want to pretend to be. <deesix>Makes sense. Please, watch out for flag style :) --os maybe? <OriansJ>I guess I'll have to add support for that to get_machine <deesix>As for the three commits... if it does not conflict with your actual work, I'm happy with them, cherry-pick as you wish. <OriansJ>deesix: thank you I'll merge it in after I patch in support for --os in get_machine <OriansJ>deesix: --os flag is now the new standard for get_machine and --OS is still preserved for the next 3 releases per tradition. with Warnings of use starting after next release <OriansJ>and your commits have been merged deesix. Thank you for your help ^_^ <deesix>OriansJ, if --OS is so new... and I guess there're no uses yet(?), do we really need the deprecation routine? *deesix checks if that was part of a release <OriansJ>deesix: --OS isn't new but --OS support overrides is new <OriansJ>which is a response to vagrantc's kFreeBSD-amd64 build failures but kFreeBSD-i386 success <OriansJ>So that uname spoofing will create a test record of said spoofing to allow faster detection of Linux emulation on *BSDs <OriansJ>a704d133a18c0ac68f573a6b0b55386917f54d85 to be precise when it was added so since before Release 0.7.0 <deesix>I was there, yes... from 0.6.x era it seems. <deesix>Yes, in fact I misunderstood what's new. <OriansJ>deesix: easy to be mixed up and make mistakes with all these changes being done in parallel <OriansJ>I like to think of these changes as me starting to clean up past mistakes that I made while rushing. <OriansJ>a boatload of technical debt remains but I am grateful for having someone as careful as you deesix to help bring them to my attention. <deesix>I'm glad to help. Please, do not hesitate to stop me if you detect unproductive nitpicking or think it's better to delay something. <stikonas>fossy: should tcc support flow properly? <stikonas>I was debugging seq and it has some crazy behaviour <pder>stikonas: I am taking a look at seq and there is indeed something broken <stikonas>pder: so this left me completely puzzled <pder>first is a float or double? <pder>what if first is a float? <stikonas>that was going to be my next test, but we started talking <pder>if first is a double does printf("First is %lf\n", first); change anything? <stikonas>the strange thing is that we are not even parsing something from command line <stikonas>it's directly assigned in the source code <stikonas>otherwise I would blame string to number conversion... <stikonas>although, in any case, yacc doesn't have doubles... <fossy>is this what is causing the issue <stikonas>well, if we can't build yacc with mes libc, and musl is a bit problematic too, maybe we can extract some bits from musl... <fossy>i'm not convinced wchar is the problem <fossy>yesterday was linux kernel shenanigans <stikonas>I think that -DNO_SYSCALL_TLS was causing crashes <fossy>(oh hey, not using glibc? broken compilation for you! even though we don't use anything from the libc's headers in the end!) <stikonas>(and TLS I think is thread local storage) <fossy>so many things have __GNUC__ if-guards <fossy>and defining it assumes compiler things that we dont have