IRC channel logs

<OriansJ>fossy: I was planning to do a release once I completed the M2libc conversion. Which I should get done this weekend, wife and son willing of course.

<pder>yes, same here. I just found the m4 git repo. The history might help me understand how to get the build working

<pder>OriansJ, I started converting blynn-compiler over to use M2libc. The initial stages are working with pack_blobs, but it fails pretty early on after that, and I havent figured out why

<OriansJ>pder: if any of the files are over 20MB I could be to blame

<OriansJ>as I #define BUFSIZ 0x140000 /* 20MB */

<OriansJ>(and yes #define works in M2-Planet now to a degree)

<pder>No, its actually very early. ./bin/vm --raw blob/root -pb bootstrap -lf generated/parenthetically -o bin/raw_l

<pder>Oh, I wonder if the #define might be tripping me up

<OriansJ>definitely

<pder>If I have #define CELL_SIZE 1\n//CONSTANT CELL_SIZE sizeof(unsigned) - what is CELL_SIZE?

<OriansJ>1 outside of bootstrap-mode

<pder>I bet thats it.

<pder>So I should also be linking with libc-full.M1?

<OriansJ>yes

<OriansJ>and let me double check on the #if you would probably want to ensure M2-Planet proper CELL_SIZE

<OriansJ>hmm it looks like yt didn't include an _M2_ variable for #if to flag on; I guess I am going to need to add that and possibly support for -Dfoo=1 sort of things

<stikonas>pder: also keep in mind that whenever I used alloca.c on musl (on older m4), it was breaking flex bootstrap, so for now I kept m4 before musl

<pder>stikonas: interesting, i'll keep that in mind

<stikonas[m]>Maybe alloca defines with tcc were similarly broken

<stikonas[m]>To regex

<pder>Thanks, OriansJ: the CELL_SIZE #define seemed to be the problem

<OriansJ>pder: #if defined(__M2__) will be working on the next commit

<pder>great, thanks!

<OriansJ>pder: commit is now up

<OriansJ>probably should add support for -D foo=1 now that I made it easy for myself

<OriansJ>and support for -D var and -D var=val has been added to M2-Planet

<fossy>stikonas[m]: i didn't bother with readline bc guix didn't use it and it didn't really matter (no interactive at the time)

<fossy>OriansJ: ah, ok, very cool

<OriansJ>and knight-posix now is on M2libc; all that remains is x86

<gforce_de1977>fossy: stikonas: I added checksumming: https://github.com/fosslinux/live-bootstrap/pull/30/commits/2de4a40b79d84f7f552262d8e9b59f329468e92b - somethings is fishy with the PR (needs rebasing?) but you get the idea and can give comments

<gforce_de1977>sorry, PR is https://github.com/fosslinux/live-bootstrap/pull/30 (now including the checksums)

***mephista is now known as spicy_icecream

<stikonas[m]>fossy: well, guix can't build readline anyway because mes libc is not sufficient

<janneke>stikonas[m]: in practice you're right, but it's somewhat the other way around

<janneke>the mes c library was created to be the absolute minimum required to bootstrap mes

<janneke>then, it was enhanced to support bootstrapping tcc

<janneke>and later, it was enhanced to bootstrap gcc-2.95

<janneke>it's a *feature* that it cannot build much more ;-)

<stikonas>janneke: oh ok. Well, we are thinking of trying to go directly to gcc-4 in live-bootstrap

<stikonas>now that we have more complete C library

<janneke>yes, the mes c library will need to be enhanced once we start targeting risc-v or powerpc

<janneke>we'll want to skip gcc-2.95 and target gcc-4

<gforce_de1977>janneke: i want to encourage you to use this picture in your slides: https://en.wikipedia.org/wiki/M%C3%BCnchhausen_trilemma

<janneke>gforce_de1977: thanks, that's a nice image ;-)

<janneke>maybe next time; the video had to be made two weeks ago

<gforce_de1977>janneke: thats ok 8-)

<fossy>janneke: well we used mes c library to go to musl

<fossy>janneke: but imo that scope is fine for the most part, as you said

<janneke>fossy: hmm...

<janneke>why musl, what a weird choice?

<janneke>eh, i don't want to sound overly critical

<janneke>anything is "allowed" to "get things going"

<fossy>janneke: musl was chosen because it has a simple build system

<fossy>i have an explicit goal of not using pregenerated configure scripts

<fossy>musl's is readable and handwritten so we used it until we can get to glibc

<fossy>glibc (obviosuly) uses autotools so we are still working on that autotools stack

<janneke>aha, right

<janneke>there are many considerations and things that need to be fixed, of course

<fossy>and as you are saying mes c library is just the absolute minimum, since gcc is so large it happens to mostly work for other pieces of software ;D

<fossy>oh, yeah, of course, it's not cut and dry

<fossy>as i was saying to stikonas earlier musl isn't the most compatible of things from all time even though it is compliant, and especially back in the 00's when gnu software in particular was by no means standards compliant (relying on extensions etc)

<stikonas>well, so far musl works alright for us even for old software

<stikonas>it usually needs slightly different CFLAGS but that's expected

<janneke>as long as you're on linux, yeah

<stikonas>yeah, maybe if you try hurd it will be more problematic

<janneke>obviously that's ultimately the goal for guix

<janneke>(and for mes)

<OriansJ>well that is the best thing about bootstrapping work, only the trivial to replace pieces ever hit the *DONE* stage, everything else will continue to change and evolve towards better solutions.

<siraben>anyone working on something like live-bootstrap but formalized in Nix?

<siraben>I have some components (kaem, mes, stage0, mescc-tools-seed, mescc) with corresponding Nix expressions if that is useful to anyone

<rain1>no but that would be cool

<siraben>It's in the `pkgs/` folder in blynn-compiler

<rain1>maybe mention it in the nixos community

<siraben>I need to move it out into its own overlay at some point

<siraben>Upstreaming it to Nixpkgs would be interesting as well.

<OriansJ>which reminds me; I really need to find time to harmonize the M1 assembly in M2libc and M2-Planet with MesCC to it can Leverage

<OriansJ>M2libc

<OriansJ>and thus enable to solve MesCC Porting problems in advance and giving M2-Planet a more C standard Library.

<OriansJ>and if the libc license continues to be a problem for live-bootstrap, we can relicense M2libc to LGPLv3 rather easily

<janneke>OriansJ: that sounds really backward; mes c lib is rich enough for the guix bootstrap, and m2-planet compatible?

<civodul>siraben: some are working on something like live-bootstrap, formalized in Guix :-)

<civodul>joke aside, it'd be great to see another distro follow that path, and Nixpkgs is an obvious choice

<siraben>Hehe, I'm aware of the Guix work, which is great.

<civodul>i guess what i mean is that closer collaboration might be more fruitful

<siraben>Right. Diversifying and formalizing the process would definitely help.

<civodul>the process is formalized in Guix, but of course it's not "diversified"--having one thing production-ready is already quite some work

<siraben>side note: I actually tried Guix before Nix but because it's a GNU project they are very intent on not having unfree software and I have a Late 2013 MacBook Pro, so Wi-Fi didn't work among other things

<siraben>I like that they use an actual programming language though instead of inventing a new one like Nix did

<civodul>it's not "they", it's you and i :-)

<civodul>but okay, i see what you mean

<civodul>Guix proper doesn't provide non-free software, it's a choice for you to make

<janneke>siraben: from a bootstrappable or security perspective, i'd say any non-free software is out?

<civodul>personally, i think that reproducible builds and bootstrapping don't matter much if one ends up running opaque binaries, though

<civodul>yeah

<siraben>janneke: of course but I was talking about from my personal experience getting it to run on the hardware I have.

<janneke>similarly, linux should be "out", i guess -- which is why i frowned upon musl...

<janneke>siraben: sure, in hindsight it's always easy to suggest picking hardware that supports free software

<siraben>I probably could make Guix work now that I know a lot more about building things but am time constrained

<janneke>fwiw, i have installed atheros wifi cards in some computers, to run linux-libre; or otherwise use a cable

<civodul>same here

<civodul>in the end it's a choice and it's a tradeoff

<civodul>whatever the practical choice, i think it's good to have a horizon

<gforce_de1977>janneke: OriansJ: because you both are here: can anyone explain why the name is 'kaem'? is it just a mixed 'make'?

<OriansJ>gforce_de1977: well I named it

<OriansJ>to address the need to remove make from my build process. Thus to pay respect to the tool it replaced, I named it after make

<gforce_de1977>OriansJ: thank you, just want to include it in the docs

<stikonas>siraben: I actually replaced wifi chip on my laptop...

<stikonas>so now only bluetooth doesn't work with linux-libre

<stikonas>which I can live without

<gforce_de1977>here also: AR93xx/ath9k with 450mbit/s

<stikonas>I just saw openwifi cards too with open hardware design, although, you still need proprietary software to compile and flash verilog into chip...

<stikonas> https://github.com/open-sdr/openwifi-hw

<OriansJ>siraben: well some people here take the trust problem more seriuosly than others. For example I am running on a libreboot x200 and others have coreboot or vendor provided firmware

<OriansJ>The point is everyone chooses what they are comfortable with

<OriansJ>Some people need non-guix things on their computer and that is their personal choice

<OriansJ>We tend to recommend more freedom respecting choices by default but that is because then we have an ability to make things better going forward

<OriansJ>Personally I like to think of it as iterating towards a better solution rather than ever being done.

<civodul>yeah i think the user freedom and security requirements are largely aligned

<siraben>OriansJ: The next laptop I use I plan to have 100% libre software

<siraben>though it may be a few more years until something finally breaks in this 2013 MBP

<OriansJ>siraben: buy what makes you happy. We would accept Windows/reactOS/DOS Users here too if they would code to help us achive the dream of bootstrapping all of the things.

<gforce_de1977>regarding GLBIC + musl, here is an interesting comparison: https://www.etalabs.net/compare_libcs.html

<gforce_de1977>sorry GNU-LIBC

<gforce_de1977>what is new to me, that GNU-LIBC can not work on no-MMU?

<OriansJ>gforce_de1977: not even remotely surprising; GNU early on explicitly does not work on 8 or 16bit processors or systems without an MMU

<OriansJ>althought on the Bloat comparison chart M2libc+M2-Planet kicks the shit out of everyone on Smallest static C program. test0000 for x86 => 135bytes

<OriansJ>and I finally finished measuring the average needed write buffer for a function that writes to disk every newline: 29 characters, so I'll be dropping from 20MB down to 4KB and doing malloc instead of calloc to make it a hair faster.

<OriansJ>I guess I use alot of newlines in my output

<civodul>janneke, rekado_: amazing that the GWL Q&A ends up revolving around bootstrapping!

<civodul>i didn't expect it in an HPC track :-)

<gforce_de1977>civodul: is there an recording/link?

<gforce_de1977>OriansJ: no-MMU does not mean 8/16bit, e.g. m68k/68000 is 32bit but has no MMU

<civodul>gforce_de1977: recordings will be at https://fosdem.org/2021/schedule/event/guix_workflow/

<rjek>ARM6 and ARM7 had no MMU

<rjek>(ARM610 and ARM710, on the other hand, did.)

<rjek>ARM2 and 3 were dependent on external MMUs

<gforce_de1977>rjek: good point, superH also

<janneke>civodul: yeah, that caught me off-guard -- small world or something

<OriansJ>civodul: completely missed that I guess.

<OriansJ>gforce_de1977: fair but I remember an explicit choice made for glibc not to support machines of those properties to reduce development efforts.

<OriansJ>rjek: well internal vs external MMU isn't something a User Space programmer should ever be able to notice and even a kernel programmer in well designed ISAs wouldn't even care to note.

<OriansJ>Now lack of an MMU however is quite noticable to both userspace and kernel space.

<OriansJ>all M2-Planet and below binaries assume a fixed loading space into memory, which on a system without an MMU would conflict with any other program that used that same memory block.

<OriansJ>So kaem and hex0 would get into each other's shit before hex1 was even done beig built.

<OriansJ>On a system with an MMU, we are fine but should we need to run on a MMU-less system, changes would need to be made (mostly in the hex2 implications for the later stages but custom elf-headers for the earler stages)

<OriansJ>janneke: and you are absolutely right that MesCC's libc is more advanced than M2libc and that for MesCC to use M2libc now would be a big step back.

<OriansJ>However, my goals are not for this week but for addressing the long term needs of bootstrapping; so that we can share efforts in regards to porting and bugfixing.

<OriansJ>yt, deesix (or anyone else who wishes honestly) any objection to me changing M2libc's license to LGPLv3 now?

<Hagfish>would it make sense to dual license it?

<janneke>OriansJ: by all means, feel free to roll your own M2lib instead of contributing to mes lib c

*janneke remains terribly puzzled

<OriansJ>Hagfish: it is possible but LGPLv3 includes the option to just be straight GPLv3

<OriansJ>probably should be LGPLv3 or later to allow for later versions of the GPL as well.

<OriansJ>janneke: well mes_libc isn't designed for M2-Planet compatibility nor is it something I can just git submodule without bringing in all of Mes

<OriansJ>now I certainly can import functions already completed in mes_libc into M2libc to save effort

<OriansJ>Plus by being seperate it enables seperate licensing to match the needs of the less compatible parts of the bootstrap such as the CDDL pieces, which LGPLv3+ would be able to be used with and still enable distribution (as opposed to straight GPLv3 which would be a distribution problem for the binaries)

<Hagfish>good thinking

<stikonas>yeah, and in guix distribution problem is actual problem unlike in live-bootstrap

<stikonas>since guix builders build it and distribute to all users

<OriansJ>One more argument to ponder https://www.gnu.org/licenses/why-not-lgpl.html does M2libc contain a feature that is not readily available elsewhere? Personally I don't believe so but I would love to understand if someone believes differently.

<stikonas>well, that's why glibc is under LGPL and not GPL...

*fossy is confused

<stikonas>fossy: ?

<stikonas>I was just commenting to OriansJ

<fossy>M2libc == libc for m2-planet, mes c library == libc for mes, no?

<stikonas>yes

<fossy>Therefore M2libc != mes c library

<stikonas>mes lib c is probably using constructs unsupported by M2-planet

<fossy>yeah...

<stikonas>fossy: any chance to review https://github.com/fosslinux/live-bootstrap/pull/31 ?

<stikonas>it's probably straightforward

<fossy>OriansJ: the entirety of m2libc provides features not readily avaliable elsewhere (a proper libc for m2-planet)

<fossy>yeah shortly

<stikonas>although, I don't know if M2libc will be enough for those cddl components

<fossy><janneke> similarly, linux should be "out", i guess -- which is why i frowned upon musl...

<fossy>what is wrong with linux as a kernel used in the bootstrap?

<fossy>particularly linux libre

<fossy>its a bad bootstrap kernel, sure

<stikonas>anyway, if mes libc gets good enough, we can just keep it...

<stikonas>but as of now, mes libc is not enough to bootstrap bison

<stikonas>or autotools

<stikonas>and would old glibc be any more hurd friendly than musl?

<fossy>stikonas: huh? the only purpose for m2libc is m2-planet c, not "real" c, which we need for post tinycc

<stikonas>yeah, that's what I thought initially, although, OriansJ mentioned CDDL components

<fossy>m2-planet/m2libc is significantly smaller than mes/mes c library

<fossy>cddl only matters for redistribution..

<stikonas>yeah, although, lex and yacc doesn't need that much

<stikonas>yeah, I know

<stikonas>but e.g. guix needs redistribution

<fossy>oh, sure

<stikonas>because builders build those packages

<stikonas>and people download

<fossy>but guix has different goals to us and hence a different path

<stikonas>for live-bootstrap we are fine

<stikonas>yeah...

<fossy>well, us being live-bootstrap :)

<stikonas>yeah...

<vagrantc>it is possible for guix to mark something as not substitutable, if there's a strong argument for it

<vagrantc>and then it will rebuild from source when needed...

<fossy>janneke: I am intrigued into the goal of being run on hurd. Is there a specific bootstrapping/foss reason for it, or is it more an alternative type thing?

<stikonas>pder: by the way, did you have any luck with m4?

<OriansJ>fossy, stikonas: well the plan for M2libc is to create a standard C library which both M2-Planet and MesCC can use which will be powerful enough to enable all of the pieces required to get a proper bootstrap done

<OriansJ>at their core both M2libc and Mes_libc is M1 definitions wrapped in C

<fossy>why not unify m2libc and mes c library if thats the goal

<OriansJ>So cooperation between the two projects becomes possible if I just harmonize M2-Planet's M1 output to match the DEFINEs in MesCC

<OriansJ>fossy: that is the officially idea for M2libc, although I haven't put in nearly enough time to get it to that state

<stikonas>although, cooperation might be a bit hard if licenses don't match

<stikonas>you can move code only in one direction

<OriansJ>Now that M2-Planet supports #if defined(__M2__) and the new bootstrap.c library (to eliminate the cc_* bootstrap problem); there shouldn't be an issue for MesCC and M2-Planet to share the exact same libc (as I only need to change out a handful of strings to make M2-Planet use MesCC's DEFINEs)

<OriansJ>right now mes_libc is GPLv3+ and M2libc is GPLv3+

<OriansJ>So harmonization and import from mes_libc is just a matter of human effort

<OriansJ>as we can literally do #if !defined(__M2__) around whole functions that M2-Planet can't support.

<OriansJ>Which would help MesCC port to new architectures as M2-Planet does porting to new architectures faster and help M2-Planet port to new operating systems as MesCC beat M2-Planet to hurd support.

<OriansJ>Think of M2libc as just another overly ambitious project of mine that I have massively failed to put enough time into; (until it actually gets up to speed)

<OriansJ>(as if I have anything other than overly ambitious projects ^_^)

<mihi>OriansJ, I hope you "accept" Windows users even if our shared dream of "bootstrapping all of the things"™ is not achieved :-)

<mihi>janneke, I'm also surprised in your interest of using Hurd for the bootstrap. For once, as someone who toyed around with Hurd distros for a few years now, I would not trust it any data remain unmangled just due to filesystem bugs. I run into a situation where the rootfs is corrupted beyond repair every few months, good that I have multiple layers of snapshots. And also the fact that you need extra tools to

<mihi>generate some of the microkernel (MiG) is not the best choice for a bootstrap kernel for me.

<mihi>(Unfortunately, apart from one pathological case, I was never able to reproduce any of those filesystem corruptions. Only thing I can say that they tend to happen more often if the filesystem is mounted/unmounted more often, or in other words the system is rebooted more often)

<OriansJ>mihi: I was raised to accept people for who they are, not who I want them to be.

<mihi>:)

<fossy>stikonas: what's the purpose of scan.lex.l in flex-2.5.11? it was cleaerly not written by us, can we make it a patch or smth

<stikonas>fossy: scan.l is too complex for lex, so it was simplified

<fossy>ah, i see

<fossy>can we maybe do a cp + patch

<stikonas>gio manually simplified it

<fossy>oh ok

<fossy>so it's gios work i'm looking at here

<stikonas>patch from scan.l to scan.lex.l?

<stikonas>yeah

<stikonas>I just started with tarball of upstream flex