IRC channel logs

<fossy>yes, i'm considering after 1.0 that only reproducibility would need to be tested rather than constant updates of checksums, and then update checksums on a semi-regular basis

<fossy>(for each change)

<stikonas>anyway, we won't have that many more packages

<fossy>this is true

<stikonas>though we should probably add ncurses and readline

<stikonas>and sam_ mentioned that there is a better implementation of bc

<sam_>yeah, gdh's

<sam_>much smaller

<stikonas>so I think we can replace current one which depends on ed

<sam_>sci-calculators/bc-gh, https://git.gavinhoward.com/gavin/bc/

<stikonas>at some point I should try to bootstrap gentoo from live-bootstrap

<stikonas>now that we have python 3.11 and gcc 12 it shouldn't be too hard

<sam_>:)

<stikonas>install it from stage0 tarball :D

<oriansj>nice

<roconnor>kaem's unset seems to unset a whole lot of things.

<roconnor>so what I believe is happening is that collect_command is always leaving a token whose value is NULL at the end of the token stream.

<roconnor> https://github.com/oriansj/mescc-tools/blob/master/Kaem/kaem.c#L813

<roconnor>and here, unset will unset an environement variable if the value of the token to be unset is NULL

<roconnor>thus the command "unset prefix" will unset the prefix environment variable, and then unset the first enviroment variable.

<roconnor>which explain why my "LIBRARY_PATH" is getting unset, and why it doesn't get unset if I add an "A=whatever" environment variable before the LIBRARY_PATH.

<roconnor>I'll see about PRing a patch to just ignore unsetting NULL values.

<muurkha>I thing 0x8048000 derives somehow from Xenix or something ridiculous like that

<muurkha>*think

<stikonas[m]>fossy: are we now over CI time limit?

<janneke>stikonas[m]: you were talking about m2-planet built mes on x86_64, right? that hasn't worked yet, indeed

<stikonas[m]>janneke: no, I was using self-hosted M2-Planet from stage0-posix

<stikonas[m]>Oh but yes, I was trying to build mes-m2 with it

<stikonas[m]>(I misread your question initially)

<stikonas[m]>It shouldn't be too hard to get it working...

<stikonas[m]>But there are still quite a few missing things

<stikonas[m]>I did clean up some of the x86 scripting that I have though

<stikonas[m]>to make testing on amd64 easier...

<janneke>ACTION recovered the missing kaem.run --bootstrap patch and pushed wip to gitlab

<janneke>"<stikonas> It shouldn't be too hard to get it working..." why do you say that, do you have specific ideas?

<janneke>i guess we could build an x86_64 version via gcc using --with-bootstrap and diff / bisect M1 output, but that could be quite some work yet

<stikonas[m]>Hmm, I thought there isn't that much x86 specific bits in mes

<stikonas>I guess I can try adding those patches that I had in mind, try to build it and perhaps there will indeed be more problems later on...

<janneke>stikonas: hmm added some debug printing

<janneke>it could be that mes-m2's malloc fails, but itoa also fails (works fine with mes-mescc on x86_64)

<janneke>ACTION goes on to look at itoa

<janneke>stikonas, oriansj: i pushed https://gitlab.com/janneke/mes/-/tree/wip-m2-planet-x86_64-bug

<janneke>the top commit exposes two bugs in m2-planet 0.10.0 for x86_64

<janneke>(or it shows something that i'm doing wrong)

<janneke>the toplevel commit message explains what's going wrong

<janneke>fa88e282 REMOVEME x86_64 M2-Planet bug: scaffold/bug.c scaffold/bug.kaem

<janneke>something seems to be going wrong with accessing globals on x86_64 (works fine on x86(

<stikonas[m]>I'll check when I'm back home

<janneke>stikonas, thanks -- no hurry!

<janneke>ACTION inserts an extra patch with new m2-planet (0.11.0) defines

<janneke>stikonas: headsup, in the introduction of the kaem.x86_64 build

<janneke>c70f5191 XXX DRAFT x86_64: build: Cater for M2-Planet.

<janneke>(looking at the commit i just remembered)

<janneke>that i've been struggling with _start, first trying to use a copy from the mes lib c _start (like i did for x86), but then sort of settling for a _start copied from M2Libc as it just segfaulted -- no idea why

<janneke>it seems that there's a confusion of calling conventions on x86_64

<stikonas>yes, M2libc might be using different calling convention...

<stikonas>though I thought it just pushes things onto stack when calling function...

<stikonas_>janneke: so I can at least build mes-m2

<stikonas_>argh, no, I somehow built x86 version..

<stikonas>must have used wrong kaem file...

<stikonas>ok, I also get segfault now with mes-m2

<stikonas>I'll see if I can figure something otu

<stikonas>out

<stikonas>hmm, for some reason gdb does now show any debug info...

<stikonas>it should show function names

<stikonas>let me see if I can figure this out first

<stikonas>otherwise it's very hard to debug

<stikonas>ok, I know what's going on with debug info

<stikonas>will fix it...

<stikonas>hmm, still no debug symbols... hmm...

<stikonas>ok, there are two problems...

<stikonas>janneke: I've fixed a couple of issues that prevented debug info from showing up

<stikonas>could get my patches from https://git.stikonas.eu/andrius/mes/commits/branch/wip-m2-planet-x86_64-bug

<stikonas>on unrelated now, I'm a bit confused how blood_elf_flag=${blood_elf_flag:-} works in kaem but oh well, it seems to have the right value

<stikonas>--64 is now passed

<stikonas>and at least I can now see that segfault happens in gc_init() function

<stikonas>yeah, it's probably either malloc or itoa in that gc_init

<stikonas>hmm, atoi seems to be working fine...

<stikonas>janneke: so I don't think it's either malloc or itoa...

<stikonas>might be that M2-Planet chokes on some advanced stuff

<stikonas>though strange how it works on x86...

<stikonas>anyway, I've found out that in src/gc.c::gc_init() g_cells has nonsense value

<stikonas>oh, maybe it is malloc

<stikonas>malloc returned something strange there

<[exa]>good evening all

<[exa]>I just kindof discovered this whole thing so I'll hang around. As a basic question, is there any plan for a rv64 port?

<stikonas>[exa]: good evening

<stikonas>[exa]: we have riscv64 port of stage-posix

<[exa]>oh nice

<stikonas>not the whole thing though, mes port is not complete

<stikonas>janneke: so I think brk pointer already has nonsense value

<stikonas>[exa]: so there is riscv64 submodule here https://github.com/oriansj/stage0-posix/

<stikonas>and if you run e.g. bootstrap-seeds/POSIX/riscv64/kaem-optional-seed

<[exa]>I was kinda wondering there, because the rv64 instruction set looks strictly more simpler than the knight instruction set, except for the ugly registers and ints torn into pieces, so how to approach that thing most easily...

<stikonas>ir should build you some tools including M2-PLanet

<stikonas>[exa]: riscv64 instructions were quite nasty...

<stikonas>but we have done it

<stikonas>the problem is that all the constants (immediates) are encoded in a horrible way

<stikonas>so it's quite hard writing first few binaries

<stikonas>until we have code to deal with that

<[exa]>yeah the lovely 10-12 splits

<stikonas>yeah...

<stikonas>once you are passed that, it becomes reasonably nice

<stikonas>[exa]: anyway, you can try running that stage0-posix-riscv64 thing

<stikonas>it will build a few simple tools in the end

<[exa]>oh cool

<[exa]>anyway this is really great, thanks all for contributing :D

<[exa]>stikonas: btw how do you deal with the instruction encodings in the end? (just finished reading the seeds, I expect the actual code will be in the M2-... stuff

<[exa]>)

<oriansj>[exa]: we encode the logic in M1 and hex2 in mescc-tools and after that you don't need to generally think about instruction encoding.

<stikonas>[exa]: these are the instructions that are encoded https://github.com/oriansj/M2libc/blob/main/riscv64/riscv64_defs.M1

<[exa]>ah it's in M2libc, I failed to click through all the way there

<[exa]>thanks a lot!

<stikonas>there is also a copy in stage0-posix-riscv64

<stikonas> https://github.com/oriansj/stage0-posix-riscv64/blob/bc7a08e9f54095df352de2404574f5ad283971ee/riscv64_defs.M1

<stikonas>and this is basically everything we need and more

<stikonas>it's a bit uglier for x86

<[exa]>...I gave up on x86 opcodes :D

<stikonas>e.g. compare to https://github.com/oriansj/M2libc/blob/main/x86/x86_defs.M1

<stikonas>you have to encode the whole "line"

<stikonas>muurkha said it's somewhat nicer in octal form but oh well...

<stikonas>anyway, for risc-v those definitions are basically complete and don't need any changes

<stikonas>(at the expense of more complicated early binaries)

<muurkha>risc-v opcodes are annoying to read in any base

<stikonas>yeah, risc-v might be easiest in binary...

<stikonas>though binary is hard to read due to very long numbers

<[exa]>base-conditionalriscvpermutation helps

<[exa]>I understood they did it because of cable wiring but man, the price for a few gates...

<stikonas>I was using this template that I created to help with encoding... https://github.com/oriansj/stage0-posix-riscv64/blob/bc7a08e9f54095df352de2404574f5ad283971ee/Development/immediate_transformations.txt

<stikonas>(these are in binary)

<muurkha>[exa]: it's not so much the money or area cost of the gates but the latency cost

<stikonas>so I was basically converting hex to binary, applying those permutations from that txt file and converting back to hex

<muurkha>adding more muxes to instruction decode slows down your achievable clock

<[exa]>aaaaaanyway, I'm here mostly because I was playing with tiny macrolanguages too, basically trying to offload a lot of overhead from tools to a simple thing that just preprocesses nice prolog-like syntax, and was thinking about bootstrapping the thing with lisp too oneday (perhaps with a bit of codegolfing, good macros are good)

<[exa]>and then yesterday I spot this project... well I'll see if it fits somewhere. :D

<muurkha>tiny macro languages like GPT/m6?

<muurkha>uh, GPM

<muurkha>tiny macro languages can be pretty powerful, but they are tricksy

<[exa]>GPM?

<[exa]>ah finally search success

<muurkha>sorry, the general-purpose macro language that M6 was a copy of

<muurkha>by Christopher Strachey

<[exa]>uuuh cool

<[exa]>found it in gnu m4 history manual

<[exa]>no actually more towards making the syntax more humane than expanding a lot of stuff. The target was lisp which can expand much stuff itself anyway

<muurkha>the m4 manual says GPM fit into 250 machine instructions, presumably on the Titan

<[exa]>yeah that's a nice feat

<muurkha>which is 1536 bytes

<muurkha>but I haven't found a reference for that claim in the contemporary literature

<muurkha>you might be interested in qfitzah, which is an experiment I've been trying

<[exa]>I was writing a prolog the other day and realized the usual operator machinery can be extended there, e.g. by allowing juxtaposed literals (like with lisps and haskells) and more complex operators (aka definable parentheses) and there seems to be an ugly load of uses for that when writing transpilers

<muurkha>it's a term-rewriting language, like Q, Aardappel, or Mathematica, using S-expression syntax

<muurkha>definable parentheses is a pretty interesting idea! is the idea that parens are just like data?

<[exa]>ha I stumbled upon that one sometime already, cool

<muurkha>on what?

<[exa]>on qfitzah

<muurkha>oh neat

<[exa]>browser tells me I've been on that link but it might have been quite a bit of time

<muurkha>my original writeup is https://dernocua.github.io/notes/term-rewriting-micro-interpreter.html

<[exa]>probably because of googling stuff for that prolog

<[exa]>yap that one

<muurkha>my prototype implementation in i386 assembly is http://canonical.org/~kragen/sw/dev3/qfitzah.s

<[exa]>ha you might like some dirty trickery from lambda calculi implementation

<muurkha>I bet I would

<muurkha>example input for qfitzah: http://canonical.org/~kragen/sw/dev3/example.qf1

<muurkha>I need to fix a bug in the implementation that keeps you from just redirecting the input

<muurkha>right now the implementation depends on each read() call returning exactly one line of input

<muurkha>so http://canonical.org/~kragen/sw/dev3/Makefile does while read line; do echo "$$line"; sleep 0; done < example.qf1 | ./qfitzah

<[exa]>oh that's nicely declarative

<muurkha>and the executable is 1K!

<muurkha>smaller than GPM on the Titan :)

<muurkha>but not nearly as small as stikonas's hex0_riscv64 that was inspiring me, sadly

<muurkha>a drawback of the qfitzah S-expression design is that there's no reasonable way to write a list

<muurkha>you'd like to be able to use input or output syntax something like [3, 1, 4, 1, 5, 9] that you can then iterate over, but as you can see, the closest I've gotten is

<muurkha>(Cons 3 (Cons 1 (Cons 4 (Cons 1 ...))))

<[exa]>ok well that's the tool I'm writing

<[exa]>:D

<[exa]>and I originally wanted it to make minikanren a bit less eye-forking experience, this would be a valid usecase

<muurkha>one way out is syntactic sugar, like in Prolog where that's equivalent to [3 | [1 | [4 | ...]]]

<[exa]>you need to parse it somehow, from the parser you get out as something like [](','(3, ','(1, ','(4, ...))))

<stikonas>hex0_riscv64 was actually originally written by ekaitz

<stikonas>though I've reworked it a bit later

<muurkha>but that's kind of a special case, and adding special cases to the grammar bloats the interpreter

<stikonas>but the other riscv64 binaries are by me...

<[exa]>muurkha: yeah decoupling it to another tool is the pint

<[exa]>*point

<muurkha>stikonas: yes, but what inspired me was when you got it down to 392 bytes

<muurkha>but it occurred to me that maybe I could take the approach taken in Haskell, the lambda-calculus, and SKI-combinators, where A B C D E is syntactic sugar for ((((A B) C) D) E)

<[exa]>muurkha: btw good parsers unfortunately aren't small, the logic you need to generate there is roughly equivalent to all cuts in the grammar, which grows a lot

<muurkha>dunno, the S-expression parser in qfitzah is uh

<[exa]>muurkha: with the operators I simply had it as a special case that prevents errors, see here https://gitea.blesmrt.net/exa/prlg/src/branch/master/app/Parser.hs#L317

<muurkha>335 bytes currently? including some stuff that isn't used about reading integers

<[exa]>(btw the above thing is not the one that should fit into <1kB :D )

<muurkha>interesting, you're writing the parser in the rewriting language itself? are you going to compile it or interpret it?

<muurkha>it's pretty

<[exa]>this is a playground project

<muurkha>xs@(_:_) means "xs, which should also be a pair of two things I don't care about"?

<[exa]>basically the thing for bootstrapping was supposed to be a nice language input (like the one parsed here) + some simple backend for spewing out the output in some way, probably lisp

<[exa]>yeah xs@(_:_) means "it's a cons but that's all we care about"

<muurkha>right

<muurkha>I'm curious why you're doing a Prattish sort of thing rather than a PEG

<[exa]>basically if there's something in the queue that didn't trigger the previous operator handling, it's a literal-ish thing, so I just smuggle the empty "operator" there and let it handle the rest

<muurkha>if I'm understanding the general idea

<[exa]>yeah well, for PEGs you'd need to make the rules and priority rules and everything everytime the user changes the parsing with a new operator definition

<muurkha>oh, yeah, I didn't realize the operator precedence was run-time redefinable

<[exa]>you can do stuff like `:- op(500, xfy, +)`

<muurkha>right, like Prolog

<[exa]>it's brutally powerful

<muurkha>this syntax seems nicer than Prolog tho

<[exa]>btw the other thing in this one is that I cloned the vienna abstract machine, it has a nice methodology for compiling the patternmatches

<[exa]>in this one the only extra thing is that the 'a b c' parses

<muurkha>you seem to have made the same decision as in Qfitzah about distinguishing constants from variables via caapitalization? but then I'm puzzled why `ho` is lowercase

<[exa]>that's the prolog convention

<[exa]>X is variable, x is basically a lispy 'x symbol

<muurkha>except in Qfitzah I did it backwards

<[exa]>yap kinda haskell way

<muurkha>yes, but in ho ops vs xs@(_:_) I assume all of those are variables?

<muurkha>is do-notation an indentation-sensitive monadic bind thing like in Haskell?

<[exa]>non-operators, there's no distinction yet between atom semantics at that point

<[exa]>re do-notation, that was one interesting thing I wanted to have done reasonably without the indentation tricks

<[exa]>btw this doesn't bootstrap itself! :D

<[exa]>(yet :D )

<muurkha>I was wondering how that worked :)

<[exa]>no it's just a haskelly prolog to play with lowlevel prolog impls

<muurkha>so are you doing the prologgy backtracking thing?

<muurkha>that makes parsing super easy

<[exa]>yeah the prology parsers work with this, I don't have any them handy but it can be done in the prolog

<[exa]>this is the "stdlib" with the implementation of --> https://gitea.blesmrt.net/exa/prlg/src/branch/master/inst/prelude.pl#L73

<[exa]>in that you can write normal prolog grammars (PEGs)

<muurkha>PEGs are not normal prolog grammars

<[exa]>afaik the way they parse is pretty close

<muurkha>PEGs have a worst-case linear-time parsing algorithm, Prolog DCGs are exponential-time

<[exa]>DCGs is the normal name, but the style is PEGs basically

<[exa]>wait really?

<muurkha>not in the usual case, and you can manually insert cuts if it's becoming a problem

<muurkha>but yeah they're pretty similar

<[exa]>what's the linear-time PEG algorithm?

<[exa]>that's new to me

<muurkha>Packrat

<[exa]>packrat isn't linear time

<[exa]>(or is it?)

<muurkha>it is, that's the whole point

<muurkha>PEGs implicitly insert a cut at the end of each production, whichh is what allows Packrat to safely memoize the parse

<[exa]>doesn't it have like a yuge constant derived off the grammar size?

<muurkha>yes

<[exa]>ah ok that would explain it

<[exa]>( btw I found this one nice https://github.com/lukehutch/pikaparser )

<muurkha>although it's still good enough to be usable in practice

<[exa]>aaaaaaaanyway, the plan here was to have "a nice language" -> lisp with kanren -> this should be able to bootstrap self

<[exa]>btw if I read right the stage2 lisp should be able to support minikanren, right?

<muurkha>to bootstrap itself?

<muurkha>I think so, yeah, though I've only ever implemented microkanren

<[exa]>the step between micro and mini is a few more macros, nothing harsh tbh

<[exa]>good

<[exa]>very good

<[exa]>nvm I need to have some sleep, thanks a lot for discussions (and for working on this! certainly the most enlightening find in the last few months :] )

<muurkha>for working on what?

<muurkha>sleep well!

<stikonas>janneke: I think I have some idea what is going wrong

<stikonas>janneke: look at the global list in M1 file: https://paste.debian.net/1278960/

<stikonas>that would work fine on 32-bits

<stikonas>as %0 is 32-bit number

<stikonas>now we need to figure out what causes this miscompilation

<stikonas>oriansj: any idea?

<stikonas>the line in mes is char *__brk = 0;

<stikonas>I wonder if somehow assignment immediately tries to put value there

<stikonas>but does not pad it

<stikonas>oh, there are actually two labels called GLOBAL___brk...

<stikonas>I guess one comes from include/mes/lib.h:extern char *__brk;

<stikonas>still, that sounds like M2-Planet bug

<stikonas>and that's where the bug is https://github.com/oriansj/M2-Planet/blob/master/cc_core.c#L2741

<oriansj>yep, we would need just a little bit more logic to deal with sign extension.

<oriansj>but only for 64bit architectures

<stikonas[m]>Yeah, I'm trying to print more %0

<stikonas[m]>Tried register_size / 32 - 1

<stikonas[m]>But that crashed my system

<stikonas[m]>I guess register size was in bytes

<oriansj>or you could just do: if((AMD64 == Architecture) || (AARCH64 == Architecture) ||(RISCV64 == Architecture))

<stikonas[m]>Oh yeah and need to check sign...

<oriansj>or just not allow any negative numbers

<stikonas[m]>well, I'll see once my laptop reboots...

<stikonas[m]>register_size is more universal...

<stikonas[m]>Rather than listing all arches

<stikonas[m]>Anyway, those are minor details, the main thing is that I figured out why malloc in mes crashed

<stikonas>oriansj: negative numbers in global assignment are already causing an error...

<stikonas>lib/linux/malloc.c:27:Received - in program

<stikonas>yes, they are not handled in in global_assignment

<stikonas>so I guess let's not add them at this point

<stikonas>in fact no expressions are allowed there

<stikonas>just non negative numbers

<stikonas>good enough for now...

<stikonas>oriansj: https://github.com/oriansj/M2-Planet/pull/54

<stikonas>(cc janneke ↑ )

<stikonas>I suspect we'll have more issues...

<stikonas>but let's fix this first

<stikonas>argh, actually this padding thing I added is little endian...

<stikonas>maybe I should do if arch instead...

<oriansj>merged

<stikonas>oh, I was thinking of reworking it...

<stikonas>hmm

<stikonas>or shall we leave it like this then?

<stikonas>I realized that this will be buggy on 64-bit big endian arches

<oriansj>we don't have big endian 64bit architectures yet

<oriansj>we can fix it then

<stikonas>could you at least add a comment?

<stikonas>otherwise we'll forget

<oriansj>sure

<stikonas>actually in case of mes, this would have still worked

<stikonas>as mes has global assignment = 0

<fitzsim>oriansj: were you able to try debootstrap'ing a ppc64be rootfs?

<oriansj>fitzsim: I ran into a segfaults but I'm figuring it out (slowly)

<stikonas>$ ./bin/mes-m2 -c "(display 'Hello,M2-mes) (newline)"

<stikonas>Hello,M2-mes

<stikonas>that's a good sign

<oriansj>not merged

<stikonas>doesn't guarantee that mescc will run

<oriansj>^not^note^

<stikonas[m]>I'll merge it to stage0-posix and update changelog...

<stikonas>ok, pushed updated stage0-posix and live-bootstrap

<stikonas>retested mes-m2 on amd64

<stikonas>Hello world indeed works now but mescc still causes segfault

<stikonas>and I have no idea how to debug that...

<stikonas>well, let's see if janeke has any new ideas tomorrow

<stikonas>strace shows that crash happens after module/mescc/mescc.scm was read...

<stikonas>oriansj: should https://github.com/oriansj/M2-Planet/issues/18 be closed?

<stikonas>it seems that you have pushed some fix

<oriansj>yep and I asked melg8 to review and close if I fixed it to their statisfaction.

<oriansj>but I guess I could close it as all of those cases are covered.

<janneke>stikonas[m]: that's great, thanks

<janneke>ACTION can reproduce the fix, for mes but is somewhat puzzled that scaffold/bug.c still crashes

<stikonas[m]>janneke : Maybe that's another bug?

<stikonas[m]>After all mescc still crashes for me

<muurkha>aw

<fossy>stikonas[m]: yes, we're over CI time limit - i'm working on that rn

<janneke>stikonas: mescc works for me, are you sure the segfault isn't because libc.a isn't found/built yet?

<janneke>stikonas[m]: you could try something like

<janneke>MES=bin/mes-m2 ./pre-inst-env mescc -S scaffold/hello.c

<stikonas[m]>janneke: I was trying to use mescc to build libc...

<stikonas[m]>janneke: still we need these patches too https://git.stikonas.eu/andrius/mes/commits/branch/wip-m2-planet-x86_64-bug

<stikonas[m]>Otherwise debug sections don't work

<stikonas[m]>(I couldn't fork your repo on gitlab, it said I had no permissions for that)

<janneke>stikonas: right, thanks!

<stikonas>janneke: ok, so I was running "/usr/bin/mes-m2 -e main /usr/bin/mescc.scm -D HAVE_CONFIG_H=1 -I include -I include/linux/x86_64 -c lib/linux/x86_64-mes-mescc/crt1.c"

<stikonas>but I can confirm that ./pre-inst-env works...

<stikonas>at least with scaffold/hello.c

<janneke>good, but weird -- something tricksy still going on then

<janneke>possibly the MES_ARENA/MES_STACK settings?

<stikonas>anyway be back a bit later, going to eat breakfast...

<stikonas>I had

<stikonas> +> MES_ARENA=20000000

<stikonas> +> MES_MAX_ARENA=20000000

<stikonas> +> MES_STACK=6000000

<janneke>ah, well; enjoyf your breakfast

<janneke>(i found the #if __M2__ in src/gc.c suspicious, you'd think we might need something 64bit-specific)

<janneke>almost lunch time here!

<stikonas[m]>Well, I got up a bit later today, it's bank holiday today, so no need to go to work and also I debugging that mes-m2 crash last night...

<janneke>ah, now that sounds nice and relaxed

<stikonas>hmm, that ARENA_SIZE is multiplied by sizeof (scm) later...

<stikonas>so maybe it's fine

<janneke>right

<stikonas>ok, I think this one is the problem in my scripts, so not mes or M2-Planet issue

<stikonas>I was using $ARCH in one place which is stage0-posix arch name instead of $MES_ARCH

<stikonas>and it was passing amd64 instead of x86_64 to mescc

<janneke>do'h

<janneke>yeah, it's pretty brittle and delicate with all these variables

<stikonas>anyway, I'll fix my scripts and retest

<stikonas>but I think we'll be in the same position now

<stikonas>scaffold.c would compile

<janneke>but mes-0.25 won't build with m2-planet-0.10.0 anyway

<janneke>\o/

<janneke>(for x86_64, that is)

<stikonas>definitely

<stikonas>we'll have to ask oriansj for a new release before mes-0.25

<stikonas>do we have any ETA for mes-0.25?

<janneke>no, not really

<stikonas>that's fine

<janneke>well, this x86_64 was one of the biggest blockers

<janneke>so yeah, no hurry

<stikonas>well, I don't expect many stage0-posix changes, and it's probably reasonably ready for release

<stikonas>janneke: some good news, mescc is working now and started building libc

<janneke>\o/

<stikonas>ok, my libc kaem script will need adjustments

<stikonas> https://paste.debian.net/1278975/

<stikonas>but that's normal for each mes release...

<fossy>go 23

<fossy>haha

<stikonas>seeing some build warnings or errors now, but somehow build continues

<stikonas> https://paste.debian.net/1278976/

<janneke>stikonas, yeah, unfortunately that's "expected"

<janneke>someone should have a look at that sometime

<stikonas>anyway, we are already in a much better position than 2 days ago

<stikonas>janneke: another thing, if mes-0.25 will need m2-planet 0.11 on x86_64, then we don't need old defines for backwards compatibility

<stikonas>(you might still want them for x86 though)

<janneke>yeah, i guess i'll want to remove them right after the release

<janneke>the mes-0.25 release

<fossy>./go 14

<fossy>again.. twice in one day

<stikonas>ok, mes-m2 can build mes

<emilytrau[m]>fun working on another novel full bootstrap, you can find and fix things that fall through the cracks! ♥️ https://github.com/fosslinux/live-bootstrap/pull/287

<fossy>:D

<stikonas>emilytrau[m]: can you fix reuse linter please

<fossy>erm

<fossy>that's not the PR'

<fossy>s fault

<fossy>that's something else going on

<stikonas>oh indeed

<stikonas>I just looked at logs

<stikonas>and installation fails

<stikonas>and indeed PR doesn't add any new ifles

<stikonas>files

<fossy>regardless, emilytrau[m], could you please add a SPDX-FileCopyrightText header at the top of that file for your changes?

<emilytrau[m]>easy :)

<janneke>emilytrau[m]: hey, great to see you here!

<fossy>not a clue in the world why it said reuse not found. rerun worked..

<emilytrau[m]>hihi! 👋 janneke

<janneke>o/

<stikonas>emilytrau[m]: by the way, longer term I'm planning to drop that patch from yacc if we get gash working on mes

<janneke>emilytrau[m]: you're getting some thorough reviews, and also some help, so it seems?

<emilytrau[m]>found it because for the nix toolchain i want to go a more direct route without musl like guix, but to run ./configure scripts we need our own awk! y'all guix folks have a lot of fun with gash and gash-utils we can't use (yet!) 😆

<stikonas>yacc is a bit problematic with meslibc due to licensing (GPLv3+CDDL)

<fossy>mm, gash on mes would allow a bunch of things to be short circuited

<stikonas>so once we have gash working on mes, I'm thinking of using gash to run musl's configure script before we have bash and yacc, and then build yacc with musl (hence all those wide char removal patches won't be necessary)

<stikonas>fossy: indeed...

<emilytrau[m]>fixing yacc just removed a musl -> m4 -> flex -> bison -> awk dependency chain for me ahahah

<fossy>stikonas: btw, in the next ~week i'm going to chuck up a 1.0 roadmap on github - we can talk abt this a little bit more in detail later, but are there any blockers that come to mind for you?

<stikonas>no real blockers...

<stikonas>just some improvements that would be nice to have

<fossy>we will have to decide whether to wait for kexec fiwix -> linux or reboot for 1.0 (most of reboot is docs but either way, sysa->sysc isn't really in a releaseable state rn imo)

<fossy>i guess in a few weeks we will know how much work fiwix -> linux is though

<emilytrau[m]>janneke: lots of great changes! i'm super happy with the improvements and hopefully it can land soon. after that i "may" have a secret feature branch with much much more :D https://github.com/emilytrau/nixpkgs/tree/trusting-trust-next

<fossy>i'm thinking roughly ~2 weeks => branch off for 1.0 (so big features can land in master but 1.0 can be stable); june => more extensive testing + RC; mid july => release

<fossy>i am intentionally making the timeline a bit longer than i think will be required. i would really like 1.0 to be in a place where someone whos never touched bootstrapping before can run live-bootstrap with minimal issues

<stikonas>yeah, it does look a bit long

<janneke>ACTION looks an emilytrau's secret branch

<janneke>ah, because you don't have guile(-bootstrap), you're taking the live-bootstrap path, sorta

<janneke>that makes sense! i would have thought you'd simply add/keep busybox to/in the binary seed, initially

<emilytrau[m]>where's the fun in that 😤

<janneke>hehe, true

<emilytrau[m]>that said i am very excited to take the hatchet to much of it all once gash supports mes

<janneke>hopefully some of this can be cleaned-up again, after we have gash-on-mes, as fossy mentioned, but that will take some time still

<janneke>hehe

<stikonas>well, live-bootstrap tries to remove all pregenerated files, which makes bootstrap path quite a bit longer because we have to get to Autotools before we can run ./configure scripts

<janneke>stikonas, yes; we haven't looked at incorporating that in the guix bootstrap at all yet

<janneke>an amazing effort and contribution!

<stikonas>one could also imagine something between guix and live-bootstrap paths where guile-bootstrap is not used but e.g. we bootstrap stuff like tar gzip using untar ungz...

<janneke>yes!

<stikonas>anyway, in distros at some point you have to drop condition on no-pregenerated files...

<emilytrau[m]>janneke: some of the ugly nix-isms we have to deal with starting without a shell hehe https://github.com/emilytrau/nixpkgs/blob/cc8ee0925d9eb05a9a9197cbbe3b6b667b32bf02/pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/stage0-posix-x86.nix#L4

<janneke>ouch! nice hack tho

<emilytrau[m]>stikonas: i've been avoiding using gnu-tar and gzip as much as possible and surprisingly untar and ungz can get you pretty far! one of the only places i need gzip is with bash-2.05b.tar.gz, ungz isn't able to handle it but haven't looked into why

<stikonas>emilytrau[m]: oh you basically had to rewrite stage0-posix kaem files...

<stikonas>yeah, those gnu-tar and gzip packages are very old

<stikonas>I have briefly looked at updating tar, but meslibc needed extra functions for that

<janneke>stikonas: have you kept track of that?

<stikonas>emilytrau[m]: can't you use kaem-optional-seed in that stage0-posix-x86 file?

<stikonas>or were there some issues?

<stikonas>it's probably somewhere on bootstrappable logs

<stikonas>though those are hard to search

<stikonas>janneke: let me see if I can find anything locally

<janneke>the (meanwhile gone stale again) wip-gcc4 branch adds quite some extra mes lib c functions, iwbn to make sure gnu-tar and gzip are also supported

<emilytrau[m]>no environment variable substitution until full-kaem. nix needs env-vars to pass in a path to the output folder of the derivation

<stikonas>I see

<stikonas>yes, original kaem has no environmental variables

<emilytrau[m]>just had to get creative :p

<stikonas>well, we are not addign env variables to kaem-seed :)

<stikonas>that would really blow its size

<emilytrau[m]>the other way to do it which i didn't think about till i saw roconnor implement it is to do stage0 all the way up to mescc-tools-extra in a single derivation/build step at which point you can copy files/read env vars and install the outputs!

<stikonas>emilytrau[m]: oh and back to yacc, is there a way to exclude it from getting into nix substitutes?

<emilytrau[m]>is the resulting output binary /bin/yacc subject to unfree/unredistributable license restrictions? otherwise for the source code yes we can mark it as non-cacheable

<stikonas[m]>yes, is is non redistributable (until we get gash)

<stikonas[m]>yacc is CDDL licensed and mes is GPLv3

<stikonas[m]>so it's basically ZFS on Linux type of issue

<stikonas[m]>with gash I can build musl first (which is MIT if I remember correctly)

<emilytrau[m]>hmm 🤔 actually that should still be ok since yacc won't be in the end result runtime output where caching is important

<emilytrau[m]>and whenever you need to make a change that requires yacc at build time, that machine can then fetch and build yacc itself

<stikonas>or you can add a step after musl to rebuild yacc (that 2nd build would be redistributable)

<stikonas[m]>anyway, hopefully this will resolve itself in the medium term with gash

<emilytrau[m]>at some point up in the chain we'd end up at busybox and distribute that i'm guessing

<janneke>love it how we try to chisel out a nicer bootstrap path all the time

<stikonas>yeah... And recently we were able to get rid of binutils 2.14

<stikonas>that will help us a lot with bootstrapping new arches (e.g. riscv)

<janneke>indeed

<janneke>it never ceases to amaze me how much more we can accomplish if just one more person joins the effort

<emilytrau[m]>Looking at the caldera license again I think it might be able to categorise as "unfree redistributable"? nix has a mechanism for that (designed for eg. nvidia drivers). would you agree?

<emilytrau[m]>oh sorry i might have mixed up CDDL and caldera. It seems that nixpkgs currently treats CDDL as a free/redistributable license

<stikonas>CDDL is free but is not GPLv3 compatible

<stikonas>it's explicitely mentioned here: https://gplv3.fsf.org/wiki/index.php/Compatible_licenses#Common_reasons_for_incompatibility

<emilytrau[m]>ah i see now thank you. i will look into marking yacc as such in nixpkgs

<stikonas>though some people claim different things...

<stikonas>hard to tell since I'm not a lawyer

<stikonas>still, safer to assume that is is not redistributable for now

<mihi>emilytrau[m], another bootstrap path from stage0-posix to gcc 4.6.4 without musl or guile/gash would be <https://github.com/schierlm/FullSourceBootstrapFromGit> - but it does not care about pregenerated files in tarballs.

<emilytrau[m]>my plan for nix, which is somewhat different to the guix/live-bootstrap path, is to take the most direct approach to bash4 + any gcc. from there we can hook into nixpkg's stdenv infrastructure and the rest of the package set with minimal extra maintenance. the path after that might be a bit longer but in my head the less packages to write/maintain should be worth it

<janneke>sounds fair

<janneke>for guix, i started out with the simplest path i could hope to get merged

<janneke>it's much easier to stepwise improve on it once it's in

<emilytrau[m]>for that reason i'm going for a bottom-up approach. can more easily get small additions merged in and do a big swap over at the end :D

<roconnor>bash

<roconnor>Sorry I mean to say that ungz works on bash-2.05b. It's bzip2 where ungz seems to fail.

<roconnor>bzip2-1.0.8.tar.gz

<stikonas>which can now be avoided with unbz2

<emilytrau[m]>huh that's odd. i'm using the main branch of stage0-posix/mescc-tools so maybe there was a change that caused my bash to not decompress 🤔

<roconnor>my mescc-tools-extra seems to be upto date: ec53af69d6d2119b47b369cd0ec37ac806e7ad60

<emilytrau[m]>ah it seems the stage0-posix sub module hasn't been updated in a while

<stikonas[m]>It's only missing m2libc updates

<stikonas>oh, the one in stage0 repo indeed has not been updated in a while...

<stikonas>janneke: which branch of tcc is 0.9.26 is the newest?

<stikonas> https://gitlab.com/janneke/tinycc/-/tree/mes-0.25?ref_type=heads ?

<janneke>stikonas: i've been using mes version numbering to keep track with tinycc

<janneke>but haven't upgraded beyond mes-0.23 in guix

<janneke>the newer branches dabbled in arm and risc-v support, without all too much success yet

IRC channel logs

2023-04-30.log