<stikonas>you mentionde that I need to just resolve conflicts, although, you didn't give me green tick, so I don't wnat to merge myself then <OriansJ`>Then one writes a floppy loader in hex0 and then one need only replace the system calls in stage0-posix to bios calls and you are done. <OriansJ`>basically one loads the whole source floppy into RAM at a known memory address and writes the result into a known memory address and dump the whole thing to a floppy disk when done. Which works fine as none of the binaries from hex0 to M2-Planet are even close to the size of a floppy disk and once you have M2-Planet, just write a minimal posix in the M2-Planet C compiler subset. <OriansJ`>you can get 90% of the work done by just porting stage0-posix to DOS or CPM. (Which I will do once I sort out the mes-m2 business) <stikonas>but arguiably bios calls aren't really better than posix calls <fossy>nah, bios calls are better than posix calls <fossy>they are closer to the metal <OriansJ`>stikonas: well it depends very very heavily on the size of the BIOS. For example original IBM 5150 PC bios was four 2kB ROM chips (so 8kB) <fossy>which removes a layer of abstraction and hence one of the seeds <fossy>but either way using posix calls requires running a kernel on top of the bios <fossy>it just adds another layer to whatever is below <OriansJ`>fortunately porting to CPM/DOS should get stage0 to 90% of running on x86 bare metal <OriansJ`>but first I have one heck of a mes-m2 puzzle to solve. <OriansJ`>1) why the heck does identical binaries "assert fail: string->type == TSTRING" on one system but successfully compile on another 2) get mes-m2 into a form that GCC can compile <stikonas>OriansJ`: live-bootstrap now merged your suggestion of not rebuilding mes <fossy>stikonas: from my initial testing i *think* we should be able to use metaconfig for recent perl <fossy>thankfully there is one perl dev who is receptive to using metaconfig in distros so i'm following debian's path on using metaconfig <stikonas>yeah, I was more hopeful once you said Debian uses it <fossy>yeah 5.10 isn't possible because of git/cvs history <fossy>oh, yeah, that's quite possible <stikonas>that's something that's good to solve anyway <fossy>i'm guessing that metaconfig will create a Configure that != Configure.orig <fossy>but should be reproducible anyways <stikonas>or maybe some distros (debian or guix solved it) <fossy>the script isn't all that complex <stikonas>do you know what perl is needed to run recent metaconfig? <fossy>i think it's use 5.004 or smth? <stikonas>well, I think perl 5.8 is definitely possible but if we can get to something new with Configure, that's better <fossy>the only reason i dind't jump straight to new is because regen.pl uses a bunch of modules introduced in 5.8 <fossy>well i have 5.10 done without modules <stikonas>oh that's better than I might have expected <fossy>it is very messy, you have to have exactly the right vesion of metaconfig, but metaconfig itself isn't bad software :P <fossy>just the process of obtaining and using it on a codebase <stikonas>well, autotools have some compatibility problems too <fossy>cause the tarball with all the metaconfig stuff for perl contains two versions of metaconfig, and then there's an external version, and another one tagged for that release.. <stikonas>although, it's more forgiving if you use different version <fossy>i chose debian's ver of dist <fossy>dist being the package contianing metaconfig <stikonas>hmm, I should probably try building libunistring at some point <OriansJ`>originally I thought the mes-m2 failures had to do with the systems being virtualized but then when I tried a guixsd it worked just fine. So now I am looking at the possiblity that mes-m2 currently only runs correctly on systems with guix installed. <stikonas>hmm, somehow mes-m2 uses something from outside? <stikonas>we also need to resolve other intermittent errors in live-bootstrap... <OriansJ`>stikonas: it feels that way that is why I've been trying to do a fully isolated build of mes-m2 to rule that out but I have been unable to get it to work in virtual machines that don't have guix installed. <OriansJ`>and the command that produces the error is of course the example in the README <OriansJ`>I'm going to carve out a few hours tomorrow to try to reduce what exactly in guix corresponds to the difference in behavior in hopes to reduce it down to something smaller. <OriansJ`>it is: ./bin/mes --no-auto-compile -L module -e main scripts/mescc.scm -I include -v -S scaffold/exit-42.c -o foo.S <OriansJ`>running it via kaem with --init-mode produces the segfault on running systems <OriansJ`>ok I've got a minimal set of env variables to expore <OriansJ`>env variable is GUILE_LOAD_PATH=/home/$username/.guix-profile/share/guile/site/3.0/ <OriansJ`>setting to GUILE_LOAD_PATH=/usr/share/guile/site/2.2/ works as well on the known working system <OriansJ`>stikonas: does export GUILE_LOAD_PATH=/usr/share/guile/site/2.2/ make it work for you? <OriansJ`>now to fix the README to reflect this new information <stikonas>but that sounds like something is loaded from there? <OriansJ`>looks like setting it to: GUILE_LOAD_PATH=. works as well <OriansJ`>so I think mes just segfaults when it tries to read an ENV variable that isn't there. <OriansJ`>stikonas: absolutely, janneke we have a bug report for you! <OriansJ`>it is the (setenv "GUILE_LOAD_PATH" (string-append dir ":" (getenv "GUILE_LOAD_PATH"))))) <OriansJ`>getenv will return #f which isn't a string and the rest goes from there <OriansJ`>and the fix is in mes-m2 along with a fix the for the double ctrl-d behavior in mes <fossy>janneke: also, can one theoretically build mes c library using tinycc? <OriansJ`>fossy: looking at the libc files indicates provided you don't use the ones with the M1 asm but the binutils asm; then yeah it'll probably work <janneke>OriansJ`: okay thanks -- yeah, GUILE_LOAD_PATH must be set anyway to find nyacc, of course <janneke>there is no form of check for any of that, and that's bad ***lukedashjr is now known as luke-jr
<janneke>fossy: the mes c library is built by tinycc in the bootstrap, yes. there is no other c library for tinycc to use, right? <fossy>janneke: oh yeah duh what am I saying I literally wrote a script where mes c library was built by tinycc <janneke>fossy: ah, well you may not realise how much effort went into "can theoretically build" to "can build well enough for a full gnu/linux bootstrap" <fossy><janneke> fossy: ah, well you may not realise how much effort went into "can theoretically build" to "can build well enough for a full gnu/linux bootstrap" <fossy>the latter is a significant tier above the first, the amount larger a libc needs to be to be able to thrill the role mes libc does is much larger than that to build tinycc, if thats what you mean <fossy>I have a reasonable idea of that from live bootstrap <gio>fossy: No, it doesn't work yet. I still believe that it's not far from working, but I haven't been able to touch it for two years. <bauen1>stikonas[m]: no, i restarted the run, i'll try to save the log the next time it occurs <gforce_d11977>gio: like bauen1 said: "we just need enough money to send everyone in this irc room to a remote island for a few months, along with an empty computer and a dozent books about x86 then the problem will solve itself <gio>If you have a way to give me money to work on that, no need for the remote island! :-P <bauen1>these are the kind of issues comparing a few runs with make --debug will reveal too <stikonas>bauen1: that's one of gforce_d11977's documented bugs <OriansJ`>bauen1: it is easy to miss things now that progress seems to be speeding up. <OriansJ`>janneke: the only places where GUILE_LOAD_PATH is referenced is: mes/module/mes/guile.mes and mes/module/mes/boot-0.scm so I don't think it needs to be set for nyacc <OriansJ`>grep -iR GUILE_LOAD_PATH shows that quite quickly in mes-m2 (which contains a copy of nyacc) <bauen1>yeah, i have to rebase on master every few days now lol <OriansJ`>I'd even bet money that janneke and civodul never would have thought it would have been possible to bootstrap GCC without pregenerated files but here we are, we literally did it. <OriansJ`>but the community has been slowly growing, getting new perspectives and solutions to hard to crack problems. <OriansJ`>I would have never expected someone to show up and say hey I made a preprocessor for M2-Planet but it happened and now M2-Planet allows janneke to do things like #if defined(__M2__) to paper around the limitations of M2-Planet and #if __MESC__ to write a single library without having to have seperate files for everything <stikonas>hmm, strange, there should be /bin/sh symlink <stikonas>gforce_d11977: so basically Argument "" isn.t numeric in bitwise and is the main remaining issue <stikonas>gforce_d11977: yeah, I mean not output pipesize.h, but use our fixed pipesize.h file <stikonas>not sure why you see those intermittent issues more often though <fossy><gforce_d11977> gio: like bauen1 said: "we just need enough money to send everyone in this irc room to a remote island for a few months, along with an empty computer and a dozent books about x86 then the problem will solve itself <gio>A lot of source code, I hope! <gio>Like in The Matrix: "Guns, lots of guns!". <bauen1>and just maybe a bit of food and water ***ChanServ sets mode: +o rekado
<NieDzejkob>I see there's been some discussion on my 2klinux repo. Yes, there's a working FAT driver, and a bare-metal FORTH that runs in ring0. re: bison generated parsers — none of that, I started because I was dissatisfied with the "purity", so to say, of the current approaches, which generally assume a kernel. As others have noticed, the C compiler part is basically nonexistent, as the project ended up on the backburner <NieDzejkob>Since last touching the project I've realized that as it is, 2klinux is quite hostile to introspection, as all the source code is simply compiled/interpreted/ran immediately on boot, without user control. I think I'd go for something where a REPL and/or text editor comes in earlier. <bauen1>^ the introspection part is also something that live-bootstrap is somewhat lacking currently <bauen1>i also think that some arm chip is probably a lot easier to truely bootstrap, yes most have some form of binary blob in a rom to load the "bootloader", but it's a step forward that is very easy to achieve (the initial bootstrap will be highly platform dependens anyway) <NieDzejkob>I think I'd start with something that 1. is driven with the keyboard or serial port 2. lets you write and load stuff 3. fits in 512 bytes, as you can't meaningfully do less on x86. Whether that lets you fit a forth or only some kind of monitor (cf. https://github.com/niedzejkob/ember). <bauen1>truely in this context just means without reliance on a kernel (or uefi which is basically a whole operating system) <NieDzejkob>Well there's the question of whether you want to do this with commodity hardware or not <NieDzejkob>If not, why stop at an arm/riscv chip? Go grab a bag of 74xx chips. <NieDzejkob>and then use *that* until you can compile coreboot, I guess? <bauen1>NieDzejkob: my current plan would be to add an sd card reader / writer to my 65c816 sbc, which can be "bootstrapped" using dip switches, use it to write the initial monitor for the arm sbc to the sd card, copy more code to the sd card, then boot the arm sbc (which has an uart, so hook up your trusted terminal), from the monitor load / review / compile more source code <bauen1>NieDzejkob: my goal is mostly to reduce the amount of blobs involved, you probably can't quite avoid them, while staying cheap and main stream enough that other people can "easily" follow my steps <bauen1>how exactly you populate the initial sd card probably depends on your paranoia <bauen1>NieDzejkob: true, you could also write the initial stage (until you have something like dm-verity) to an eeprom and have the arm sbc boot from that <gef>> the initial bootstrap will be highly platform dependens anyway) <gef>it doesn't have to be, as regards the bootstrapping: using some kind of OISC (single command) emulation layer could allow a generic approach, although the I/O aspect is more of a question. <gef>OISC make FORTH very affordable and as long as you can feed a checksum, hash or some similar control function, you can then boot from untrusted media without too much worry <gef>(i.e. calculate cheksum before passing on the execution) <bauen1>gef: imo the initial bootstrap is just platform dependent parts, a hypervisor, an sd card driver, irq driver, memory management, uart driver ... ; by the point at which a generic approach will buy you a lot you'll probably already have the ability to load additional code from a storage medium <gef>I think we shouldn't have to talk about irq & memory management, until the moment a kernel gets fired up. And even then, it should be possible to arrive somewhere far in the bootstrap process before they become a strict necessity. <gforce_d11977>NieDzejkob: hello and thanks a lot for your explaining words: dumb question: what means "backburner" exactly? is this some polish slang maybe? <xentrac>bauen1: what are the unavoidable blobs involved? <bauen1>gef: it depends on what requirements your storage, i/o and isolation mechanisms (if you want any e.g. kernel / userspace) drivers have mostly, but yes, you might get away with never freeing memory if you have enough etc.. <bauen1>it's all about trade offs once again <xentrac>gef: an OISC is easy to implement but pretty painful to program without an assembler, macro processor, etc. <xentrac>I suspect something like Brainfuck might be about the right level <gef>SBN4 is a beauty (for regex, interpreters and compilers) but a kernel pushes it a bit too far. However, for anything in user space that regards the subject of this channel, it is should be sufficient. <bauen1>xentrac: harddriver controllers, etc... ; but most importantly some sort of bootloader in a rom somewhere that starts the processor, i don't think you can avoid that one easily <xentrac>gef: is SBN4 "subtract and branch if negative 4"? <gef>xentrac: right. Different level than BF and all that <xentrac>I mean BF or something slightly less annoying might be the right level to replace the OISC <bauen1>you can however choose a path that gets you running code that involves almost none or only rom blobs in the loading chain, and assuming that the rom blobs are small enough you might even call them hardware (which in theory you also need to verify) <gef>I've done various experiments and among OISC choices I haven't found something that beats it, so far. <NieDzejkob>gforce_d11977: english idiom, I believe. "not being done now, but left to be considered in the future" -- cambridge dict <xentrac>bauen1: you should be able to get a 65c816 running without a big rom blob I'd think? certainly you can with a 6502 <bauen1>xentrac: yes, but that thing won't run linux <bauen1>xentrac: i'm assuming that you want to do a bootstrap on mostly easy / cheap to obtain hardware <bauen1>xentrac: if you have the resources for risc-v build by hand from your own gates, then that's the way to go <xentrac>maybe building with a pick-and-place robot and just verifying it by hand would be better ;) <bauen1>xentrac: but if you have to choose between an arm sbc with a 32kb (more like 16kb) bootrom and an x86 with a 100mb uefi implementation + intel me + bootrom, i think arm has a lot more potential <xentrac>you don't have to trust the hard drive controller as long as it doesn't have access to scribble all over your memory and you have some way to verify data you get from it <gef>xentrac: sharp, that is the case. Not even the loading functions, you'd just need to trust the checksums and that your state of memory is undisturbed <gforce_d11977>NieDzejkob: thanks bauen1: yes, do the bootstrap on 6502/C64 + 16MB REU with bankswitching would be interesting. friend of mine has now 3 years time and is an expert in 6502 assembler, maybe i can convince him to join the project 8-) *NieDzejkob is more of a Z80 person himself <bauen1>technically my 65c816 sbc is cheating a bit by using an atmega to emulate an 16 byte bootrom and uart lol <xentrac>gef: there's still the question of where you get the checksums, and the checksumming code, from; I wonder if you could use XXTEA as a secure hash function? <xentrac>a 16-byte bootrom is a thing you can quite reasonably build out of discrete logic <gef>z80, 6502, x86, 68*: i've been there. For bootstrapping, sbn4 won me over, because it allows for one single approach across all platforms. IMHO, the checksum function has to be adjusted to it. I would avoid a specific CPU checksum, so that it is easy to cross check multiple CPUs in parallel. <gforce_d11977>xentrac: NieDzejkob: ...'my 65c816 sbc, which can be "bootstrapped" using dip switches' ... how can you feed data into your 65c816 SBC? pix? <NieDzejkob>I've actually built a ROM out of around 100 diodes before... <NieDzejkob>not for any bootstrapping-related topic, I just needed a ROM with a capacity so small that using the available chips felt wasteful <xentrac>NieDzejkob: and a decoder to drive the word lines, I suppose? <NieDzejkob>even though it would've been cheaper, faster, and take less space with an eeprom <NieDzejkob>xentrac: yeah, I think that part was a 74xx demux <xentrac>oh, heh, decode on output rather than iput <xentrac>you can get by with fewer diodes if you use a sparser encoding with more one-hot fields <xentrac>which also tends to result in simpler logic downstream: more wires, less gaets <xentrac>if you use a CMOS mux/demux then the unselected outputs are just tristated, but unfortunately that doesn't save you the diodes <xentrac>well you can avoid diodes on one word I guess <xentrac>gef: a thing about subleq and similar machines is that they tend to be kind of more complicated to implement in hardware than machines that have "more instructions" <xentrac>because to actually implement a subleq-family CPU you need to cycle through a longish series of states within each instruction <NieDzejkob><stikonas> it seemed to me that stuff like busybox is not easy to build </stikonas> it shouldn't be that hard if you just take the sources and don't insist on using the original build system <stikonas>it probably depends on how good libc you have... <gef>xentract: I've implemented sbn4 years ago in just 12 lines of microcode, very sparse and easy. But it requires ALU subtraction (for which I used an EPROM actually) <stikonas>well, I'm just saying that because we briefly looked at other shells in live-bootstrap <stikonas>but in the end fossy have built bash first <xentrac>gef: yeah, and you need a microcycle counter to index the microcode store. I think you can get by without any such state, just the program counter, if you're doing a harvard architecture? <xentrac>I mean basically you replace the microcode that implements sbn4 with the program you really want to run <gforce_d11977>stikonas: i feel/hope that if the bootstrap works, that there will be a battle to shorten the path (e.g. using other shells or toolchain implementations) <xentrac>if you really want to minimize the number of gates you can do your subtraction bit-serially but then you need a "microcycle counter" again to count the bits <gef>xentrac: there are several opportunities for optimisations... actually, I'm not trying to minimize gates per se, but rather have something that is both usable and understandable (therefor auditable and bootstrappable) <stikonas>gforce_d11977: I'm not sure if that will shorten the path, you'll have to build bash sooner or later, you'll have to build bison, sooner or later. You might be able to get rid of autotools and perl, but then you have to rewrite build systems of binutils and gcc <xentrac>yeah, there's always a tradeoff between having a machine code that's easy to debug, and having a machine code that's easy to implement <xentrac>I think there's a pretty wide range in that tradeoff space where some kind of RPN is the optimum <xentrac>I suspect that the 12-bit words and the two-word stack were probably both over-aggressive; 16 bits and three or four words are probably better tradeoffs <xentrac>A 16-bit memory space is enough to support a reasonable self-hosting system with an IDE, as demonstrated by Turbo Pascal for CP/M; by contrast, the 12-bit PDP-8 mostly ran code cross-compiled on a larger machine <xentrac>Calculus Vaporis needs four microcycles per instruction. if you make it a Harvard architecture and give it a separate index register to drive the address bus during data fetches I think you can eliminate the microcycle state machine? <xentrac>but it was just my first amateur effort. it's probably possible to do a lot better <xentrac>it has 7 opcodes, and my sketch of the instruction decoder was about 20 NAND gates <gforce_d11977>stikonas: the busybox ash can be compiled, so that it behaves in some ways like a bash, maybe all the tools can work with that instead of using "the real thing". but bash/ash/sh alone is not the only issue, having a busybox or a similar multicall binary approach can *maybe* make things easier <stikonas>gforce_d11977: not sure about it being much easier. yes, you might be able to substitute bash with busybox sh but how much does that win... <stikonas>if you want actual multicall binary, you still have to build a lot of things, so similar complexity to coreutils <stikonas>but in any case no-pregen bootstrap needs at least gcc which means we need bison, that means we need m4/flex/non-toy libc and some shell <stikonas>that's already significant part of live-bootstrap <xentrac>stikonas: is there a reasonably up-to-date dependency diagram somewhere? <xentrac>that describes our current critical path from nand to tetris, or at least stage0 to firefox <bauen1>xentrac: you need to code / review the initial steps by hand, up until your secure hash function, then you can reuse binaries / code from earlier reviews if you know their hash, and remember the hash of any additional code you reviewed, etc... building your own hash tree (if it is tampered with or you loose it you'll have to do a lot of reviewing again) <bauen1>gforce_d11977: you could use uart or a dip switch + led interface to feed data in/out of my 65c816 sbc <bauen1>stikonas: you don't need bash if you adjust later code not to need it, so you can push bash further down the line, which might result in less dependencies (e.g. you now don't need a specific version of the autotools) ; so there is some benefit in investigating these things <bauen1>stikonas: and perhaps make the bootstrap easier to understand / reduce bugs, maybe <bauen1>xentrac: i've talked a bit about securing the "progress" of a bootstrap, how you can build such a hash tree and what isolation mechanisms a kernel could provide to make your life easier <bauen1>xentrac: in the end it kind of depends on what action you want to take when you find a bug in the bootstrap, and if you can even exclude the possibilitie that it has contaminated later stages in some form <bauen1>maybe i'll do a bit of a demo using SELinux to show how the kernel can be used to protect the integrity of earlier bootstrap "stages" from the later, and also marking code as untrusted until the user has reviewed them and marked them as trusted ... <stikonas>bauen1: but we don't use any dependencies for bash <stikonas>xentrac: OriansJ` and me made some diagram <stikonas>posibly autotools can be simplified a bit <stikonas>then you need to rewrite all other build systems <xentrac>because they all use autotools, you mean? <stikonas>for early stuff we wrote manual makefiles <xentrac>is the dot source for this available somewhere? <bauen1>stikonas: then it's less of an issue, but we still have 2 versions of bash that are build, so it might be better to replace / eliminate one of them <stikonas>because readline uses termios.h, so need better libc <stikonas>well, unless libmescc gets significantly more functions <stikonas>but that is just moving complexity elsewhere <stikonas>round bash in the middle of the diagram is just for convenience it's bash-2.05b that is on the left <bauen1>also regarding the spdx copyright headers, how many line changes constitues a change that would need me to add my own name with SPDX-FileCopyrightText: ? <bauen1>since i'll be touching most scripts, but with relatively minor changes (add a few env vars, an additional mkdir here and there) <stikonas>bauen1: feel free to add if you want your name <stikonas>especially here, it's better to keep licensing info precise 
