<oriansj>stikonas: well you could always switch from sysa to stage4->stageN <oriansj>samplet: could you either point to more information about that tar extension or perhaps a mescc-tools-extra patch to help support that? that way we can remove that bug entirely. <oriansj>rickmasters: well hex2 and M1 can be leveraged to find errors in hex0/hex1/M0 programs but unfortunately debugging your implementations of hex0/hex1/hex2/M0/cc_* is something even debug logic wouldn't help too much... although gdb support for your kernel might be a bit much to expect. (Hence why I was originally thinking of doing the kernel in C first to figure out basics before doing the extra painful assembly work) <oriansj>but you really beat me to that punch and seem much better at kernel work than I am so I am really excited to see your progress (it is quite awesome) <stikonas>I didn't bother with gdb for stage0-uefi work either <rickmasters>honestly i wouldn't know how to implement gdb support for a kernel whether written in C or otherwise <oriansj>fair, it seems to involve a good bit of complexity <rickmasters>it traditionally required gdb running on another computer using a serial port to talk to a special gdb handler in the kernel <rickmasters>i did debug a couple issues by breaking into the qemu monitor and using its debug facilities <samplet>Is mescc-tools-extra implemented in M2? <stikonas>so we have a few extra goodies like better preprocessor and #includes <samplet>Cool. I learned a fair bit about M2 while hacking on Mes. It’s developed quite a bit since the version Mes is using AIUI. <oriansj>and the C behavior is generally pretty standard for the bits we do implement <stikonas>but yes, M2 has grown quite a lot in features <stikonas>though adding more features to M2-Planet is increasingly hard <oriansj>but it was never intended to be a single step from cc_* to tcc or GCC but only the subset of features needed by mes.c <oriansj>So some badly planned short-cuts were used. <oriansj>redoing it with an IR and separate architecture specializers would have produced more efficient code and a much easier path to adding functionality. <samplet>In brief, the GNU extension works by having a “fake” file with typeflag 'L' or 'K' (for long name and long link respectively). Then, the name of the next file is the contents of the “fake” file. <samplet>I read my own code so no one else has to. :) <oriansj>but at this point doing that work would basically cause us to skip mescc entirely and eat about a year of dedicated effort. <oriansj>so not something that would push the bootstrap forward into a new area so much as make future work on the transistion to tcc simpler <oriansj>and it looks like a new person finally posted some questions about stage0 <oriansj>that reminds me, I probably should do a hex0 monitor for UEFI as one can't assume a text editor in a UEFI implementation can they? <oriansj>just requires a tweak to the read/write behavior and using keyboard input to populate 2 files (all keypresses and the built binary) <oriansj>it is absolutely terrible with no editing functionality nor backspace functionality but it'll only cost a small handful of bytes <theruran>I have a question... how can I boot into stage2 LISP in QEMU? :) <oriansj>theruran: well stage2 lisp is currently only written in knight assembly, So only if you are running a knight instruction set architecture emulator inside of qemu as qemu doesn't not yet support emulation of knight yet. <oriansj>ultimately for non-bare metal bootstrapping, building of mes.c provides for a much better lisp experience <theruran>ohhh is that why it is slow to boot up using the given VM? <theruran>I am looking for a path that does not touch C <oriansj>I would strongly suggest you taking a look at Ur-Scheme <theruran>oriansj: so there is a Knight assembler written in hex2 or ??? <oriansj>it is great and its author is here too. <oriansj>well the knight assembler is written in C, M1 and hex2 <theruran>bronze-age-lisp is written in x86 (32-bit) and I am wondering how much work it is to convert it to 64-bit and what needs to be done <theruran>it is pretty clever. I doubt I could make an improvement <oriansj>looking at it, it appears to do structs the manual way (multiple %define lines) but that is fine, one doesn't need to leverage gas's support for structs. ***muurkha_ is now known as muurkha
<oriansj>So to convert to AMD64, you'll need to tweak the syscalls as the calling convention and syscall numbers will be different. And adjust the struct sizes to reflect the bigger int size and possibly a few max_int/min_int corner cases in the code <oriansj>oh and the univeral replacement of E* with R* <oriansj>well going from x86 to AMD64 was engineered to be a smooth transisition. <oriansj>you literally could run x86 binaries in AMD64 and only make changes if you want 64bit integers and 64bit pointers. <theruran>when I try to build it, I get a linking error saying its for i386. and I stopped there <oriansj>well there might be an assembly directive you could just change to fix that <oriansj>as usually when you build a 32bit binary on a 64bit architecture, it should error with a message like: (Error: operand type mismatch for `pop') and require you to do something like: as --32 $Source.S -o $Binary.o && ld -m elf_i386 $Binary.o -o $Final_Name to work around that if there wasn't a directive <theruran>I accidentally ran build.sh instead of quickbuild.sh so it will be awhile <oriansj>on nasm the building of a 64bit binary is usually: nasm -f elf64 -o $Final_Name $Source.S <muurkha>oriansj: I'm glad you enjoyed Ur-Scheme <oriansj>muurkha: well it helped me improve my mes-m2 rewrite a great deal and it gave me the dream of just compiling mescc and having something fast build tcc <muurkha>I mean Debian Sid treats it as a separate architecture <theruran>nasm --help says: elfx32: dwarf ELFx32 (x86-64) dwarf (newer) [default] <muurkha>oriansj: great! my life has thus had some benefits <theruran>I thought that meant running 32-bit ELF on 64-bit system <oriansj>muurkha: x32 is AMD64 with 32bit pointers and a few more special things that I don't remember off the top of my head <muurkha>and it might have more! since I'm not dead yet <muurkha>oriansj: yeah, and I'm guessing that's what -gelfx32 does on nasm <muurkha>I mean it's possible that -gelfx32 builds a 32-bit binary but it would surprise me <theruran>OK I will try a few things once it's done building. very slow for some reason... <oriansj>muurkha: you are awesome and I love your work it teaches me things I haven't seen yet or perhaps a new fun perspective on it. <oriansj>theruran: well 18,269 lines of macro heavy assembly isn't an easy task, especially if one tries to be too clever by half <theruran>it's generating the assembly from Kernel LISP code running on Klisp <oriansj>hmm: klisp mold.k "$@" > build/out.asm where does that root klisp binary get built? <theruran>oriansj: it is another Kernel implementation based on the Lua source (C) <oriansj>so generated assembly because they never wanted to put in the effort required to write it actually in assembly??? <oriansj>stikonas: you are now assigned push/commit access to stage0-posix-amd64 and the invite has been sent <muurkha>the intuition behind Qfitzah is that it ought to be easier to write compilers in a term-rewriting language, and a term-rewriting bootstrap interpreter ought to be pretty easy to write if it doesn't have to be fast <muurkha>but so far qfitzah only runs really simple programs <theruran>I assume it's because LISP programmers don't like writing anything that's not S-expressions ;) <theruran>I did find a note in TROUBLESHOOTING doc though: LD='ld -m elf_i386' ./build.sh <muurkha>I didn't know elf_i386 was even a valid -m <muurkha>the benefit of S-expressions is that it saves you from having to write a complex parser <theruran>OK it linked, but running it immediately gets an "ERROR: Internal error." along with some hex addresses <muurkha>like, even with a parser generator, it's easy for the parser for a programming language to run to several pages of code <muurkha>and if you're writing the parser in assembly, it can be dozens of pages of code <muurkha>yay! hopefully you can figure out what the addresses are <muurkha>the S-expression BNF is, like, sexp ::= "(" sexp* ")" | atom <oriansj>muurkha: I'd have to disagree on the saving you from having to write a complex parser bit as mes.c has a *WAY* more complex parser than M2-Planet does <muurkha>that's adequate (if annoying) to express any tree structure <oriansj>so the parser can be changed how it reads bytes and how it parses them <oriansj>without reader macro support you are absolutely right, pure S-expression only is only about 77 lines of assembly <muurkha>the S-expression reader in Qfitzah is a little longer than that but not much <muurkha>it's a little hard to tell because it's mixed in with support for strings and integers that isn't actually being used yet <muurkha>oriansj: did I already show you Meta5ix? it makes me wonder if it might be worthwhile to do a parser generator very early on <oriansj>pure S-expressions are so simple and honestly I wished more languages used them, the problem is the temptation for reader macros support is too strong <oriansj>muurkha: I don't believe so as I usually make copies of everything you share <muurkha>because in 18 lines of Meta5ix you already have a Meta5ix parser, but to get that to bootstrap you need some kind of bootstrap interpreter <muurkha>a friend of mine also got his own reimplementation of META-II to be able to generate Python code, but extremely far from idiomatic Python code <muurkha>META-II out of the box is a little awkward about indentation; it's designed for 01960s assembly language syntax <muurkha>but it turns out Python is a lot more forgiving of indentation if you put your entire program into a single gigantic list comprehension >:-) <oriansj>looks almost like C code generated by M1 <muurkha>about Forth, the parser in those 15 lines of Forth took me almost an hour because it took me a long time to debug <muurkha>I thought it would be a lot faster but I'm not that great at Forth <oriansj>if it makes you feel any better, it took me weeks to do FORTH.s and even after that we were still finding bugs in my work for about a month after. <muurkha>yeah, I don't remember how long it took me to write StoneKnifeForth <muurkha>and it wouldn't be surprising at all if SKF had bugs in it, though it might be hard to tell <muurkha>bug report: compiler considers foo and far equivalent. state: closed, working as designed <muurkha>I think the right way to think about Forth is not as a small programming *language*, because it's easy to do a better language <muurkha>but rather as the simplest way to get interactive debugging, memory dumping, I/O port probing, etc., up and running <oriansj>more reasonable than what I saw today: User locked out due to password expiration while they were on vacation attached is properly filled out form with management approve to get the password reset. Response: User account has been succefully been deleted; if you require access please follow the proper process for account creation. <muurkha>with Forth, instead of a programming language you get an interactive shell, a sort of debugger, and a sort of IDE all for the price of a small programming language. unfortunately, considered as a language alone, the language is extra special bugprone <muurkha>so you spend more time debugging, which I am not a fan of <muurkha>also it has terrible traps for new programmers <oriansj>well FORTH has a niche where it is great, the biggest problem is those who love FORTH seem to mistakenly believe it belongs outside of that niche and well sadly they are wrong. <muurkha>well, you can do anything with it, same as assembly <oriansj>I can even imagine loving to use FORTH but I know how easy it is to write a C compiler in Assembly and C is just better than FORTH for making useful things <muurkha>I can't disagree with that from my experience, but I wonder if my experience is colored by having written hundreds of thousands of lines of C and only hundreds of lines of Forth <muurkha>maybe someone who has written hundreds of thousands of lines of Forth can make useful things with Forth more easily than I can with C <oriansj>well yes, the more familiar you are with a language the more you like it. <oriansj>especially when you learn how to do something neat in one language and you try to replicate it in another language and you just can't and it seems so stupid <muurkha>I mean, C is pretty awkward for a lot of things <muurkha>yeah, that would happen a lot trying to move from Forth to C, because Forth has general compile-time metaprogramming and C doesn't <oriansj>like how in assembly you can do micro-stubs but it is impossible to do in all higher languages. <muurkha>even gas macros are a lot more powerful than the C preprocessor <oriansj>well gas when abused is more flexible than FORTH <muurkha>unfortunately for Forth, nothing is more flexible than Forth ;) <oriansj>but the more freedom one has to change the language to fit their own tastes, the more worthless the language is for cooperation. <oriansj>the rewriting of what the language is while you use it can very quickly exceed the ability for anyone to know what the heck the line noise is and even what language it was written in. <muurkha>maybe, I don't know how to cooperate with people <muurkha>but that sounds like a problem that could be managed <muurkha>I mean Emacs Lisp is pretty flexible and everyone is contributing to this single humongous program <oriansj>muurkha: you start with talking usually and then a couple lines of code being different but providing some value to your life and then it grows from there <oriansj>emacs isn't 1 program, it is literally tens of thousands <muurkha>it doesn't have readmacros though, just regular procedural macros <muurkha>it feels like tens of thousands of programs, but they're all running in the same interpreter at the same time and calling each other <oriansj>well elisp makes a program literally 1 line of code in some cases <oriansj>dozens more often, hundreds rarely and thousands only when there are multiple developers <oriansj>and debugging when it goes bad is impossible <muurkha>I mean they all have read/write access to each other's global variables <muurkha>usually debugging elisp gone bad is pretty easy <muurkha>although there's some seriously hairy stuff in c-mode <oriansj>and can hook into each other's internals <muurkha>hard to say two elisp packages are separate programs when one is defining advice for the other <muurkha>fortunately advice is more often used for end-user customization\ <oriansj>emacs bankruptcy is the only thing that really saved emacs from going off the rails <muurkha>anyway so elisp makes me skeptical of the Python thesis that cooperating with people benefits from the language being inflexible <muurkha>even though I don't know what cooperating with people actually does require <oriansj>people who stick close to the core generally are good without much trouble but the second you go towards the deep end, giving up and starting from scratch is sometimes the only option. <muurkha>but the last time someone gave up and started from scratch was in 01984 <muurkha>everyone's been using Richard's GNU Emacs since then <oriansj>if I can create a construct in the language where you need to spend a week figuring out what the f&%k (*&*#&$&^#$*&^*!***&^&*: means; then you just can't work with another person. <muurkha>well, maybe you can if they don't do that ;) <oriansj>but that requires a strict culture in guix limiting what one can do in guile <oriansj>and why few projects in guile grow past one developer which doesn't have that culture <muurkha>more static types would probably help with reading other people's code <oriansj>it is either the language enforces a common standard to reading comprehension or the community has to make hard rules that enforce a common standard for reading comprehension. <oriansj>muurkha: the haskell donut might disagree with that <muurkha>when the community makes those rules, it can make exceptions to them <muurkha>which is a thing you have to do at the semantic level too <oriansj>very true and the more the community resists that temptation to do exceptions the better <oriansj>much like having a standard astyle for C code <muurkha>on one project I'm working on we are discussing whether we should use longjmp() to handle stack overflows, for example <muurkha>longjmp() is not the sort of thing you want to use lightly <muurkha>not in a program where you care a lot whether it works or not <oriansj>indeed, in fact it isn't even used once in all of stage0 or even mes.c <oriansj>I need to get some sleep. Talk more tomorrow ^_^ <oriansj>I love that bootstrapping is even there for once, gives me hope that people might put just a little more thought into these sorts of things. <siraben>Given that Apple is also undergoing a very large change in their instruction sets to ARM it makes sense they care about bootstrapping. <unmatched-paren>pabs3: Huh? "This proposal specifically plans to not have a "pure C++" Swift compiler any more. <pabs3>unmatched-paren: see the "Concrete Build Process" section, it starts out with a C++ compiler <pabs3>hmm, perhaps I misread it :/ <unmatched-paren>4. Build optional Swift bits with the minimal stage 1 compiler. Note that these bits may not be fully optimized because the stage 1 compiler may lack some optimizer passes. ***attila_lendvai_ is now known as attila_lendvai
***zv_ is now known as zv
<theruran>@oriansj> well the knight assembler is written in C, M1 and hex2 <theruran>I don't understand. looking at stage0 and looks like lisp.s only needs M0-compact <stikonas>but it's not used later, so you can basically forget it <theruran>hi stikonas - but isn't this lisp.s theoretically capable of doing something useful? such as implementing M1 or another assembler for x86_64? <stikonas>theruran: yes, theoretically it is capable <stikonas>theruran: but cc_x86 is not that hard to write <stikonas>yes, it is about 16 KiB binary rather than 2 KiB for M0 <theruran>well, I might work on it. or at least I am trying to understand the paths available. <theruran>I want to obviate C in the bootstrap path for Scheme/Kernel <Irvise_>theruran: in the end you will need C. So were it comes, either early or later, should not matter too much imho. <Irvise_>theruran: the kernel will need it... Yes, it is not bootstrapped early. But in the end, you will need C. And luckily we have TCC, which is fairly competitive. <stikonas>though it still have to be something compiled ***pgreco_ is now known as pegreco
***pegreco is now known as pgreco
<muurkha>we might be able to do better than C for a kernel bringup language with the additional 40 years of experience we have now *vagrantc notes the use of "might" <muurkha>C has a lot of advantages; most of the worst mistakes were fixed in the early 01980s, it's not that hard to compile, and lots of people have experience reading it <muurkha>in a vacuum obviously the relative precedence of the bitwise operators and the equality operators is wrong, for example <muurkha>but you can't change them now. the best you could do is give a compile error if someone tries to compile code that depends on that precedence order <stikonas>yes, C' easy readability is big advantage <muurkha>it sucks at higher-order programming, it has no pattern-matching, Golang interfaces are usually better than raw function pointers, stronger type checking would help catch a lot of errors, and C's arithmetic coercions are totally a mistake <muurkha>by "higher-order programming" I mean things like qsort(), map(), and filter() <stikonas>yes, but golang has to be written in someting <stikonas>for bootstrap language you don't want pattern-patching or all extra features <muurkha>yeah, I didn't mean Golang is an alternative <stikonas>it has to be as small as possible, so that it is easy to implement in assembly <muurkha>for Qfitzah I'm trying having *only* pattern-matching, it seems to be working okay <stikonas>for 2nd compiler, pattern-matching might be useful <muurkha>yeah, clearly you don't want pattern-matching in hex0 <muurkha>but pattern-matching is a pretty useful feature for compilers <stikonas>anyway, I'm slightly sceptical about replacing C stuff with scheme. Not opposed to it if it works <stikonas>but it's harder for most people to maintain than C <oriansj>theruran: well you could do a kernel in Assembly like L3 and do everything above that in scheme without trouble but if you wanted to do a kernel in Lisp, Common Lisp is a much better plan than scheme. <oriansj>honestly the biggest strength of C for bootstrapping is struct and local variables and there are assembly languages that support those and are certainly good enough for the things one would want assembly for if you didn't care about portability. <oriansj>it is literally all of the functionality that most people say lisp brings (things like map, reduce, fold, etc) but very quickly you discover very little lisp code out there actually could run on this lambda core. Reader macros and tail call support somehow seem to be the things people actually want about lisp, not the s-expressions and first class functions. <ekaitz>oriansj: tail calls depends on the lisp you use, also continuations... but yeah <stikonas>but C has an advantage that almost everybody (who might care about bootstrapping) knows it <stikonas>you'll find plenty of people who have not seen scheme <oriansj>ekaitz: completely fair, although doing a common lisp in assembly while bootstrapping seems impossible of a task for a single individual even if they were willing to dedicate a few years of hard work doing it. <stikonas>hmm, I wonder whether I should implement full hex2 for UEFI <stikonas>though perhaps it is simpler if for code is in sync with posix version... <oriansj>stikonas: that is a very good question as absolute addresses are something to avoid right now in UEFI <oriansj>and cc_* can be done without them. But we will need to do some tweaks for M2-Planet <stikonas>well, I'll see, it might be just simpler not to remove it <stikonas>since I'm mostly basing UEFI version on POSIX version <stikonas>rather than writing the whole thing from scratch <stikonas>no binary changes, but I fixed some comments in hex0 files <stikonas>best to keep it in sync with stage0 repo <oriansj>thank you for being awesome stikonas, really impressive UEFI progress <stikonas>once we got hex0, it's PE32 header and kaem-minimal working it's much simpler <oriansj>especially when you realize that your work is making it possible for me to figure out how to make M2libc treat UEFI no differently than a POSIX system <oriansj>and thus no code changes would be needed after M2-Planet