IRC channel logs
2022-12-31.log
back to list of logs
<stikonas>hmm, there might still be a bit of an issue even if we use #__uefi__ define to select binaries that we run... <oriansj>well the only way #ifdef blocks don't work is if we copy UEFI binaries onto POSIX systems and the reverse but then they wouldn't run anyway <stikonas>yeah, exactly, that's why I later said it's fine <oriansj>stikonas: I was thinking more about the --os flag and tell me what you think of this idea: the default is bare metal (aka no __${OS}__ at all) and --os becomes required to build on Linux/UEFI/etc <stikonas>hmm, we can do that too, but we don't have much baremetal support... <oriansj>well if make --operating-system required, then building natively and building cross-platform becomes the exact same thing <oriansj>or we can do what every C compiler does and just compile the default to be the host and hope people read the documentation when it is time to cross compile <fossy>ahhh, stikonas[m], i figured it out <fossy>in my dotfiles i have umask 007, so it had that permission <fossy>and somehow that made its way all the way through to the tarball xD <fossy>(re: the musl diff, if that wasn't clear) <stikonas>sigh, something is seriously broken in M2-Mesoplanet #include parsing <stikonas>oh, maybe not, maybe I just need to reorder includes... <stikonas>hmm, since M2-Mesoplanet includes file as soon as possible, removing other includes from uefi.c should help <stikonas>and i might need to do something a bit ugly, define some structs before #includes... <oriansj>or just put the struct in its own file and #include it <stikonas>anyway, I'll sort it out after the lunch <stikonas>I can see some minor bugs but in general stuff works, sha256sum.efi sha256sum.efi works <stikonas>unbz2 works, untar fails to create some directories <stikonas>hmm, maybe even untar worked, just complained for some reason (possibly repeated attemts to create dir) <stikonas>oriansj: thanks, I'm preparing M2libc change now <stikonas>I think I'll keep that one struct before #include if you don't mind <stikonas>otherwise we need to add even more files... <oriansj>well if the struct is generally useful (aka tied to operating system functionality) it belongs in an includable file but if it is just specific to a single program, then yes that is fine. <oriansj>but requiring it prior to an include just smells wrong. <stikonas>stdio.h includes stdio.c in __M2__ so I guess I still have to do it before that include in stdio.h <oriansj>well in theory the struct defines should only be in the .h files <stikonas>but with M2libc we might have to update some M2-Planet build commands <oriansj>well the big problem is if any files are compiled by cc_* too <stikonas>otherwise I would have seen an error earlier in the build <stikonas>with this I can run stage0-uefi up to the end <stikonas>(I only haven't done final sha256sum check) <stikonas>it's much nicer now that uefi stuff is in arch independent directory <stikonas>otherwise we would have to deal with both uefi and arches <stikonas>ok, thanks, now I just need to generate sha256sum.answers <stikonas>which is not as easy as running make Generate-amd64-anwers on posix :( <oriansj>stikonas: amazing news about completing stage0-uefi for amd64 ^_^ <oriansj>I am so glad you finally finished the last few steps <oriansj>janneke: is there anything you can do to get mes.c running on uefi? <stikonas[m]>Still need to fix POSIX stuff, almost done with M2-planet tests <stikonas>oriansj: for mes.c I think one of the paths would be if somebody improved M2libc to be able to build mes.c <stikonas>and that would probably let us run mescc <drakonis>this one has a ocaml based bootstrapping compiler <mihi>stikonas, congrats for finishing stage0-uefi :) <stikonas>still there are some bugs, but main things work <mihi>do you regret having started it? <stikonas>no, I don't regret, but I thought it will be done quicker <stikonas>but in the process M2-Planet got improved too <stikonas>and if somebody decides to do it for e.g. aarch64, it should now be much easier <mihi>as an unrelated note, I finally found time to fix step 5 of my full source bootstrap from git repository. It only builds gcc, but getting the repo free from non-text files was harder than I anticipated (and frustrating so that I stopped working on it quickly) <stikonas>oh yeah, we do have a lot of frustrating problems in #bootstrappable <mihi>and if rickmasters / rick-masters now decides to port his kernel to uefi, he should have a lot easier time and does not have to write everything in hex2 asm :) <stikonas>I think he actually wrote it in hex0 asm <mihi>tbh I did not look at it, only from what I remember from the logs. <stikonas>but builder-hex0 has other advantages, it's lower level and does not assume file system stuff <mihi>I assume it relies on BIOS calls? <stikonas>it's filesystem is basically a stream, name, size, data <stikonas>I have briefly looked at it, but not in super great details <mihi>so at least the BIOS calls need to be replaced by EFI BootServices calls, maybe wrapped in context switches (depending how much the memory model of his kernel is different from uefi's) <stikonas>well, it might be useful if we can't reach fiwix on uefi <mihi>that is the big question. If we can shoehorn everything up to fiwix compilation in UEFI environment, we don't need it. <stikonas>as builder-hex0 actually provides POSIX runtime envronment <stikonas>M2libc only provides source compatibility <stikonas>but syscalls and PE headers are not POSIX compatible <mihi>I assume we won't be able to make tcc compile against M2libc <mihi>or at least, if we were able, getting builder-hex0 to work on UEFI is probably easier <stikonas>tcc can still only produce POSIX elf binaries <stikonas>hmm, there are some issues with updating stage0-posix :( <stikonas>mkdir.c:47:strlen is not a defined symbol <oriansj>stikonas: well we would only need tcc to build the most minimal of POSIX kernels (assuming we don't have M2-Planet or MesCC do it <stikonas>oriansj: no, you also need tcc to build tcc <stikonas>it's unlikely the the very first tcc can build kernel <stikonas>though who knows... rickmasters would know better <stikonas>but I think the first tcc builds are very limitted <oriansj>even written in M1 is a huge improvement to writing in hex0 <stikonas[m]>Still, it's good to have hex0 version of builder-hex0 too <oriansj>well the only thing an operating system might need that M2-Planet doesn't do well is jump tables <oriansj>drakonis: I was looking at the language you posted earlier (https://github.com/austral/austral ) and it looks relatively easy to parse. So the hardest bit of it to bootstrap (if we wanted to use it lower in our stack) is to do the type system. <oriansj>and the rant: https://eev.ee/blog/2016/12/01/lets-stop-copying-c/ seems spot on for several of the points (disagree on the ! bit [but maybe because I believe in syntax highlighting making that difference very obvious]) (a strong disagree on the multiple returns bit (sure it is nice to use but it either results in untyped lists or the compiler writer having to add an implicit custom struct for every type)) <oriansj>although if one wanted a better low level bootstrapping language than C; I can suggest that a compiled typed Lisp would probably be the ideal. (easier to parse and much stricter types, not to mention easier to write tools to formally validate correctness) <oriansj>(define foo (lambda ...)) (define bar types:uint32) <oriansj>a few clean up details and you'd have a language one can learn in 5 minutes and would be a powerful competitor for C <oriansj>(define types (module ...)) (define uint32 (type ...)) <oriansj>(define types (module .... (define uint32 (type ...)))) [if I wanted to give an idea of the style involved] <oriansj>I guess I should write that up in case someone is interested in actually doing a typed lisp to replace the C in the bootstrap <oriansj>now I think I need the most cruel perspective possible on what I got wrong. <oriansj>tell me what I missed, did completely wrong, or could do better <oriansj>and then I guess I should probably implement a working version. <oriansj>(first in C, then in itself and then finally in assembly) <drakonis>oriansj: that would most certainly be very good. <oriansj>drakonis: well in bootstrapping in general, yes but it is bigger than Mes, so at that point it kind of misses the point <oriansj>and it appears the #lisp community has some nice suggestions for improvement <oriansj>well I think that is enough tweaking for tonight (I need sleep); hopefully more people will have suggestions to make it a better bootstrapping lisp <stikonas>I'll probably have to fix M2libc before I can update stage0-posix <stikonas>I think that commit that fixed UEFI broke something in POSIX... <stikonas>possibly need to add some early function declaration <oriansj>I can't see how the UEFI fix broke POSIX <stikonas>hmm, x86/artifact/M2-Mesoplanet-000000 does not contain string.h at all <stikonas>(I also noticed that I accidentally removed #if x86_64 guards on UEFI, but that's unrelated) <stikonas>oriansj: it's 41a04d675314766fd278c1f385e28c54027e9f8c that broke POSIX <stikonas>oriansj: can you reproduce that strlen error if you update all stage0-posix submodules? <stikonas>M2-Planet --file ../x86/artifact/M2-Mesoplanet-000000 --output ../x86/artifact/M2-Planet-000000 --architecture x86 --debug <stikonas>mkdir.c:47:strlen is not a defined symbol <stikonas>though it's only for mkdir, so some other programs build fine <oriansj>you are missing a -f ./M2libc/stdio.h <stikonas>I thought for mescc-tools-extra M2-Mesoplanet would find it <oriansj>stikonas: that is an M2-Planet command, not M2-Mesoplanet <stikonas>have you pulled in stage0-posix-x86 and other submodules? <oriansj>hmmm M2-Mesoplanet never imported string.h <oriansj>well it appears it exists even if M2-Mesoplanet is built with GCC <oriansj>./bin/M2-Mesoplanet -E --architecture amd64 -f ../mescc-tools-extra/mkdir.c -o foo <oriansj>but the #include guard is causing string.h to not be included <oriansj>so something earlier is #including <string.h> and is inside of a block which is not being imported <oriansj>I think we need to create a prototypes file just for UEFI <oriansj>it is either that or I do a major rewrite of how M2-Mesoplanet expands C macros <oriansj>well, yes at that point we will have a C compiler at feature parity with TCC ;-p <oriansj>which honestly would be very nice but it ultimately becomes a question of what level of complexity can we implement in M2-Planet before we exceed our ability to debug what we have created. <oriansj>and C macros is a language more complex than the rest of the C language <oriansj>(as it is both is and is not white space sensitive) <oriansj>in some ways I think C would have been a better language if they just stuck to the M2-Planet C subset <oriansj>once environmental variables got mixed in, then it only got more and more complex <oriansj>if one thinks about it; there is no reason one couldn't just do if(AARCH64 == ARCHITECTURE) {asm(..);} else if(AMD64 == ARCHITECTURE) .... ELSE {#ERROR("unsupported architecture")} <oriansj>then one doesn't even need #if, #ifdef, #ifndef or the like <oriansj>assuming you move the ugly special case into separate files <oriansj>(and the the different architectures disappear when one does dead code elimination) <stikonas>I guess historically people wanted to upload more computation to be compile time <oriansj>well deadcode elimination was compile time <oriansj>and it wasn't publish about until 1981 and C was created in 1972; so they might not have even known about it <mihi>oriansj, re <https://logs.guix.gnu.org/bootstrappable/2022-12-31.log#062301>: I don't get what the memory model of this language is, and how it avoids the need of garbage collection. Also, it is unclear to me, how (if supported) open arrays (i.e. arrays of a size not known at compile time) can be declared and defined, and how you would define a record type that contains one ore more pointers to <mihi>itself (like a linked list or tree). <mihi>And if you don't plan to add runtime support for S-expression lists, neither a macro preprocessor, I would advise against S-expressions and rather write a(b, c, d) instead of (a b c d). But probably that is just my personal taste. <oriansj>mihi: it is automatic garbage collected language but implementing the garbage collection isn't required for bootstrap implementations. (DEFINE foo (ARRAY variable)) [It just requires that variable is defined and evaluates to an integer type] (DEFINE list (RECORD (POINTER list next) (POINTER ANY payload))) [Probably should add those bits to the documentation] <oriansj>mihi: although part of me thinks ANY is a bad idea <oriansj>and one can also change the size of an array by just assigning it a new block (SET! foo (ARRAY variable)) <oriansj>thinking about it I can probably eliminate the allocate <muurkha>for bootstrap compilers you can get away with only deallocating when the program exits if you have gigabytes of RAM <muurkha>#ifdef is simpler to implement than dead code elimination and is sometimes used for things like adding fields to structs which dead code elimination can't <muurkha>agreed that the preprocessor is pretty messy; textual macro systems usually are <oriansj>muurkha: well one can work around that by just having different definitions in different files <oriansj>muurkha: well one can get away with only deallocating on exit if one has tens megabytes of RAM and a language that knows to put things on the stack rather than the heap <stikonas>that's what we do in stage0-* too most of the time... <stikonas>in stage0-posix we rely on kernel to free up the memory <stikonas>and in stage0-uefi we deallocate the whole allocated block just before exit <oriansj>now to send out my new gnupg key (signed by my old one) <oriansj>ok, new 2023-2024 gpg git keys for me are now up <oriansj>as well as the usual public gpg key servers <stikonas>oriansj: I'm a bit confused how you want to put string.h stuff into a separate header <stikonas>otherwise programs that don't include string.h at all (catm is one example) wouldn't build on UEFI <stikonas>and including only string.c doesn't help... <stikonas>I could possibly copy the whole implementation of those functions into UEFI header <stikonas>it somewhat ugly but maybe we have to live with it <stikonas>tested it on both stage0-uefi and stage0-posix <stikonas>thanks, I saw, double checkin stage0-posix now <stikonas>I suspect we might also have similar problems with some other headers <stikonas>but right now nothing outside uefi uses it <oriansj>I can remove that problem by being exact on what actually is needed by UEFI <oriansj>(probably by making a UEFI_bootstrap.h) <stikonas>and probably we can then just include it from uefi.c <oriansj>which should prevent UEFI from having further impacts on POSIX <stikonas>oriansj: what about if we completely disentangle standard includes and extra includes (such as uefi/unistd.c or amd64/linux/unistd.c) ? <stikonas>and then make M2-Mesoplanet inlclude another file with arch/os specific includes based on --operating-system and --arch <oriansj>well we don't need full duplicates for UEFI <oriansj>only the handful of core functions actually used <stikonas>well, quite a bit of stuff from string.c though <oriansj>and something I am going to have to be very careful with. <oriansj>should I expand bootstrappable.c's strtoint to also support scheme format octal/hex/binary string formats? <muurkha>oriansj: yeah, Ur-Scheme puts a lot of things on the stack, but I think probably even if it put all its local variables and everything on the heap it would still be able to compile itself <oriansj>muurkha: well given a 64bit address space and enough RAM, I don't doubt it <muurkha>I think you could probably do even in32 bits <oriansj>for something as complex as Ur-Scheme, potentially for something as complex as Haskell => nope <muurkha>OTOH stack-allocating data for a lazy language like Haskell is a lot more complicated <muurkha>I did do a combinator-graph-reduction compiler for the λ-calculus a few years back, but I did it in JS, so I got GC for free <oriansj>(always be free to share you work ^_^ ) <oriansj>hmmm quasiquote, unquote and unquote_splicing do you think they would be worth the complexity? <oriansj>lets try without, I can always add it in prior to doing the assembly version <muurkha>qq is pretty great for macros, but I'm not sure how highly to value macros, not having enough experience with Lisp <oriansj>well line macros are great for assembly <oriansj>hygienic macros have the potential to be useful (assuming they are rarely used and with considerable care) Reader macros however seem like a demon temptation box that scrambles everything so no one knows what the source code actually says <muurkha>I meant Lisp-style macros that provide new special forms, not string macros or reader macros <oriansj>fair enough (thank you for clarifying) <muurkha>I don't know what to opine about hygienic macros vs. syntactic closures vs. Common-Lisp-style conventional macros except that evidently there are smart people who know more about macros than I do who think that each of the three is the best option <muurkha>C-style macros pack an enormous amount of expressiveness into a tiny package; IIRC the GPT macro preprocessor was something ridiculously tiny like 300 instructions. but it's also an insanely bug-prone package, which is not great for making a trustworthy bootstrap <muurkha>because bug-prone also means bugdoor-prone <oriansj>well every macro type has a use case which it is the best choice. Just like how different scoping systems have cases where they are the best choice. But I'd argue, that it is all people trying to justify their preferences. <oriansj>but we have studies and know lexical scope and hygienic macros are much easier for humans to correctly interpret the code and accurately prediction resulting behavior. <muurkha>I haven't seen the studies comparing hygienic macros with other kinds of Lisp macros <muurkha>the contra on the hygienic-macros side is that they're orders of magnitude more difficult to implement than CL-style macros <muurkha>(also you need to be able to poke holes in the hygiene sometimes, but that's easy to add) <rillian>does kaem (the programme name) stand for anything? <rillian>I noticed it's an anagram for 'make' <oriansj>rillian: it stands for the users. /tron and in spoken form it is pronounced "common" as its job is just to provide the functionality common in every shell. (aka the spawning of programs with arguments) <oriansj>I have yet to figure out a clever backronym for it yet (feel free to pitch names) <drakonis>oriansj: how does it feel that more languages now have bootstrapping paths? <oriansj>drakonis: I'm always excited to see progress <stikonas>maybe it's worth just cherry-picking 2 working commits?