<damo22>OriansJ`: would it be helpful if radare2 supported your ISA? i can add it in <damo22>is the idea for stage0 to have its own abstracted ISA so you can more easily port it to any arch? <OriansJ`>damo22: actually the idea of the Knight ISA was to keep me and everyone else honest; aka an architecture that can be implemented in TTL and thus no hardware backdoor could exist <OriansJ`>as for adding radare2 support for the architecture; that would be rather cool to see ^_^ <damo22>how is the knight ISA related to the c compiler? <damo22>i am still struggling to follow the source <OriansJ`>and by manually porting it to multiple architectures, I can prove that the general bootstrap path is valid for any sane architecture <OriansJ`>damo22: the knight ISA is just one of the ports of the general bootstrap process of stage0 <OriansJ`>hex0-monitor->hex0-assembler->hex1-assembler->hex2-assembler->M0-assembler->cc_x86->M2-Planet->... <damo22>so stage0 could target knight or x86 or ... <OriansJ`>up next are the armv7l and aarch64 ports <OriansJ`>then when I get Power(PC) hardware, I can do that port too <damo22>i am curious to know if radare2 could be used to help create new stage0 ports <damo22>because they already have asm and disasm <OriansJ`>damo22: honestly radare2 has been extremely helpful in the writing of M2-Planet's generated output <OriansJ`>(rasm2 -a x86 -b 32 'shr ebx, 2' to figure out the M1 encoding for that instruction) <OriansJ`>and they even got the byte encoding correct, even though the official ARM documentation got it wrong <damo22>i ported radeon atombios instruction set to r2 <OriansJ`>and it would be amazing to have another disassembler written by someone clearly much better than me support knight <OriansJ`>I honestly punted so hard on the identification of strings in High_level_prototypes/disasm.c <damo22>is the self-bootstrapping property of M0 something related to the ISA? <damo22>like does it depend on the ISA or can it work for any ISA if you port it <OriansJ`>M0 is ISA neutral as a specification; provided your ISA can be encoded in bits <OriansJ`>for example lets say the bit pattern is 110111 0 1 111 00101 <OriansJ`>you can do DEFINE ADD 110111 and DEFINE INDIRECT 0 ... and then simply write ADD INDIRECT MEMORY R7 ... <OriansJ`>or if you prefer octal or hex or honestly anything else it does support it <OriansJ`>thus each architecture is a unique subset of M1 which maps to the instruction encoding of that architecture <OriansJ`>Thus far we support hex, octal and binary with alignment padding <damo22>where are the defines for all the arches so far <damo22>is the encoding for x86 for example, something arbitrary that you decide in advance, or is it the actual op codes of that architecture? <OriansJ`>damo22: the actually instruction encoding is specific to the architecture but the name and the manner of encoding is entirely arbitrary <OriansJ`>for example one could change the M2-Planet M1 encodings for x86 to better match the architecture <OriansJ`>aka instead of LOAD_IMMEDIATE_rdi; they could find a set of DEFINEs such that they could write LOAD_IMMEDIATE RDI and LOAD_IMMEDIATE RSI ... by only adding defines for the register names <OriansJ`>I only spent the time required for optimial encoding of knight because I wanted to make it a beautiful example of what each architecture could have <OriansJ`>if DEFINES with octal or binary works better; then it is clearly a better choice <OriansJ`>So that one could radically change the DEFINEs and update the lines in a file and still get bit for bit identical binaries <damo22>it depends on the values of the instruction set <OriansJ`>yep, some instruction sets change the encoding of register names depending on the instruction operating on it <OriansJ`>which make ADD R0 and SUB R0 impossible; but ADD_R0 and SUB_R0 will still work <OriansJ`>So M0 is flexible to fit any programmer's preferred way of interacting with the instruction set, regardless of how insane it is <OriansJ`>(I still haven't figured out the correct way to support RISC-V's immediates yet however) <OriansJ`>the problem is however that M1 only supports contigous immediates <OriansJ`>aka %-1 will be 32bits long and ~-1 will be 24bits long on armv7l and !-1 will be exactly 8bits long but there is no way to do ADD %-1 which breaks the %-1 into pieces <OriansJ`>and deals with the bit/byte order if needed <damo22>if the immediate is shorter than 32 bits, does it make sense to truncate it? <damo22>is there such a thing as opposite of sign extension? <xentrac>you can truncate the most significant bits <xentrac>that's reliably the inverse of sign extension <OriansJ`>it doesn't even inspect beyound a handful of characters <OriansJ`>it knows # and ; are line comments, :labels, %&$@!~ pointers/numbers and DEFINE and that is it <OriansJ`>if you want to write something crazy like DEFINE POP_EAX 58 and then write POP_EAX %42; it will simply output 58 2A000000 for x86 <xentrac>(I think i386 is a bit more readable in octal than in hex, which I think I've said before) <OriansJ`>xentrac: and M1 supports octal but I haven't found the time to write a good x86 defs file yet <damo22>what about some prefix that can be applied to operators that specify that the result needs to be sign extended/reduced to some fixed width? <OriansJ`>damo22: the leading character !@~% indicate the fixed width <xentrac>for StoneKnifeForth I took the approach of writing just raw machine code sequences in decimal, without trying to write any kind of an assembler <xentrac>I think StoneKnifeForth contains about 30 bytes of machine code as a result <OriansJ`>if you want little endian immediates: pass --LittleEndian or if you demand big endian immediates: --BigEndian <OriansJ`>if you need alternate bit endianess, it supports that too <xentrac>(I should emphasize that this is not intended as any kind of criticism of the M0 appraoch) <OriansJ`>xentrac: I appreciate criticism that I can do something about <xentrac>but this isn't any kind of criticism, either actionable or not <OriansJ`>xentrac: well M0 does support 'no touchy' <OriansJ`>which means you can put anything between '' and M0 will assume you know what you are doing and will pass it through without any alteration in anyway <xentrac>in SKF the things that are written in machine code in the source are syscall, switching stacks, LITERAL, CALL, JZ, jnz, -, <, !, @, c!, part of write(2), stack initialization, and Getchar <xentrac>which was enough to get it to compile itself successfully <OriansJ`>so I would have done those lines in M0 as '135 246' and '91 90 89 205 128 135 236 195' <xentrac>the idea was that that was enough to compile enough of a programming language to write a compiler for a higher-level language in it, in which I could write a decent assembler <xentrac>perhaps the price of getting obsessed with minimality over practicality <xentrac>it's smaller than almost everything! it sacrificed everything on the altar of smallness <xentrac>except the ability to compile itself into a working i386 ELF executable <xentrac>(and other programs that don't need facilities that are too much more complicated) <OriansJ`>and I wanted people to understand everything and sacrificed size to get it <xentrac>that's probably smaller htan the SKF binary, I forget <xentrac>yeah, in retrospect my bet that making SKF super small (in the sense of simple) would more than compensate for any added difficulty in understanding each line of code did not really pay off <OriansJ`>xentrac: honestly, it still is a rather impressive FORTH <OriansJ`>but like every FORTH thus far, none of them really did what they promised with bootstrapping <xentrac>I wonder what the FORTH/ADHD connection is ;) <xentrac>in the same way that Ur-Scheme isn't really a Scheme without user-defined macros, StoneKnifeForth isn't really a Forth without user-defined IMMEDIATE words <xentrac>it's probably more accurate to describe it as a Forth-like language than as a Forth <xentrac>I find it's sort of easier to write things in assembly than in Forth. I tend to have less bugs in assembly and it feels easier to understand the code <xentrac>it takes a hell of a lot more code in assembly though <xentrac>I suspect that it takes a lot of discipline to use Forth properly, maybe more discipline than I have, because there's a constant temptation to refactor <OriansJ`>xentrac: actually; it might depend on if what you are trying to do is the FORTH way of doing things <xentrac>I see stage0/forth.s is an indirect-threaded-code Forth <OriansJ`>FORTH is a stack centric world and if you want to do something outside of that you need to do one hell of a messy dance to do it <xentrac>and I think that's where a lot of my trouble with Forth came from actually <OriansJ`>the state of the stack needs to be known and respected at all times <xentrac>what it needs to be in practical terms is obvious <xentrac>the state of the stack needs to be obvious at all times <OriansJ`>one wrong push or pop and the entire pile of nested utility falls apart <xentrac>right, and that is a kind of bug you can't have in C or Lisp <xentrac>but the particular trap the stack gets you into is that you have three or five or seven things on the stack and so you forget what they are <xentrac>and that happens because you're afraid to use variables <xentrac>and, at least in my case, that happens because you think the variables are global and therefore toxic <xentrac>yeah, .s helps you figure out where your program went wrong <OriansJ`>reepca did one hell of a good job improving stage0's FORTH and providing such a useful FORTH library <xentrac>but it doesn't help you design the program so you don't have stack bugs <xentrac>so, a better way to use the stack is to use it just for argument passing, expression evaluation, saving values in the rare cases where you're doing recursion, and maaybe one local variable <OriansJ`>and in the end, writing a C compiler in assembly turned out to be easier than writing a solid FORTH <xentrac>basically the stack is the minimal way to add expression evaluation to assembly language <xentrac>but I still find Forth more bug-prone than assembly, and I suspect that's from lack of experience <OriansJ`>actually the stack is where you save locals that you don't want to stomp over <xentrac>your Forth programs will be full of stack-effect bugs <xentrac>don't think of them as C global variables, think of them as C static variables <OriansJ`>such a nightmare to solve correctly on a FORTH you wrote yourself on an architecture that no other FORTH supports <xentrac>did you get an interactive REPL running, btw? maybe I should try running it and find out <OriansJ`>because it is more important that the steps in stage0 be doable on arbitrary new hardware than for me to do it slightly faster once <OriansJ`>./bin/vm --rom roms/forth --memory 1M --tape_01 stage3/intial_library.fs <OriansJ`>it enabled some solid testing of the FORTH but there was always this fear in the back of my mind about edge case bugs that I hadn't spotted yet <OriansJ`>in the end, C compilers are so much easier to test and build in assembly <OriansJ`>I spent longer writing that FORTH than I spent writing cc_x86 <OriansJ`>So now when someone says bootstrapping should be done in FORTH or Lisp, I roll my eyes and say "Show me one"; I've done them all and from experience FORTH was the worst of the 3, Lisp ends up being much easier in C and C ends up only requiring a rather simple Macro assembly and a trivial debug_list function that you can write in 5 minutes <OriansJ`>I even leave it in my cc_*s because it is essential if you wish to extend that C compiler <OriansJ`>There is just one small problem with that path though <OriansJ`>No scriptable shell to execute the steps <OriansJ`>I still need to find the time to write it in assembly, M1 and hex0; so that one can use it as the init on a posix to build everything from source with no external dependencies <xentrac>I mean basically kaem is just the minimal scripting of the Unix kernel needed to launch programs, plus a bit of syntax <OriansJ`>and when compiled by M2-Planet it is a 16,246 byte binary (that I could probably get down to 1/6th that in assembly) <OriansJ`>and the one thing that surprised me is that cat is essential to all bootstraps (if you don't want insane duplication) <OriansJ`>so much that I hand wrote a variant of cat that removes the need for > filename in a script such that the shell would not have to support that construct <OriansJ`>basically it is limited by the read/write bandwidth of the posix kernel it runs on <OriansJ`>a single 14 instruction loop that pulls upto 1MB of data per cycle <OriansJ`>I could have easily changed it to 1GB but I didn't want it to run into problems for posix systems with less than 1GB of memory <OriansJ`>it ends up being faster than GNU or BSD cat <xentrac>it is surprising that cat is essential <xentrac>Ur-Scheme ended up being only about 1600 lines of Scheme, most of which is really assembly in Scheme syntax <OriansJ`>oh and even as a valid elf it is only 215bytes big <xentrac>of course Ur-Scheme would add to the Scheme bootstrapping problem rather than help solve it <xentrac>it's kind of amazing that ELF is almost 200 bytes of overhead <xentrac>SKF generates a stripped-down ELF based on the breadbox stuff <xentrac>the ELF overhead in httpdito is actually almost 1000 bytes when built with gas <OriansJ`>I simply handwrote custom elf-headers in hex <xentrac>even with objcopy -S -R .note.gnu.build-id <xentrac>yeah, that's what I did for SKF, except I did them in decimal instead <xentrac>but I did it in a half-assed way and you did the hell out of it :) <xentrac>haha, I just understood the name of "blood-elf.c" <OriansJ`>I had to support the binaries MesCC was generating and janneke did an amazing job figuring out the fiddly bits in ELF I didn't know about <OriansJ`>So he gave me a rough sketch and 20 minutes later we had everything we needed <OriansJ`>in fact MesCC introduced alot of really interesting ways of doing things like: %ELF_end>ELF_base <OriansJ`>straight up measure the length between 2 labels <OriansJ`>janneke needed it and wrote the first version and then I later made it generic <OriansJ`>It is amazing what can be done when other people help you explore thoughts you previously would have never considered <OriansJ`>% means write a 32bit relative number, a is the base label name, > indicates that we want the relative difference in addresses and b is the label name for the target <xentrac>right, that's what I was understanding <xentrac>yeah, it's wonderful when other people help you think :) <OriansJ`>It was janneke's success in Lisp and my pain of writing one in assembly that finally convinced me that Lisp is much easier to do in a C than assembly or below <OriansJ`>If push came to shove, I probably could shoe-horn the current mes-m2 into M2 buildable state in a weekend but honestly I want to make mes-m2 a golden standard for an easy to understand lisp <OriansJ`>I want every feature of it so well understood that for generations collage professors will pull up it's source code and say "read this and you will understand how a modern lisp works" <OriansJ`>I however still need to figure out a bunch of tiny details that janneke wrapped into it. <OriansJ`>for example, how macros tie into the variable definition format (as I wish to simply the variables from a hash table to a simple linked list) <xentrac>have you seen Neel Krishnaswami's micro-lambda-calculus compiler? <xentrac>I find compilery stuff a lot easier to write in OCaml than in C or Lisp or especially assembly <xentrac>I haven't managed to write a tiny-ML compiler or interpreter in any of those languages though <OriansJ`>I could probably translate that into C but the runtime dependency on llvm is kinda big for a bootstrap <xentrac>Neel just posted it in a thread about another tiny OCaml compiler that did depend on LLVM <xentrac>as you can see it generates a simple textual quasi-assembly language for a simple register machine, which happens not to be the LLVM virtual machine <damo22>i feel that what you guys are doing is going to be revolutionary for seeding an OS from scratch <xentrac>it's definitely pretty straightforward to translate into C or assembly, if you handwave the GC at least, although doing it in C might involve a certain amount of void* <damo22>what id like to see is a hardware implementation, so you can buy off the shelf, a tool for bootstrapping any compiler <xentrac>the 'a in the declaration of exp is a type variable, like a C++ template parameter, so you can have an expr exp or an exp of some other kind, although in this case it's only explicitly instantiated with expr <OriansJ`>libresilicon, FPGA, TTL, individual transistors or relays? <OriansJ`>well one can already buy an FPGA and make an i386 core run on it <xentrac>has someone done that? the i386 is pretty big <xentrac>and I didn't know there was an opencores implementation yet <OriansJ`>iCE40 FPGAs even have a RISC-V cores availabe and a Free software stack <OriansJ`>xentrac: yeah, we found it a yearish back <OriansJ`>someone interested in helping in hardware showed up and we asked if they would get it ported to the iCE40 family <OriansJ`>they vanished and we haven't heard a thing from them since <OriansJ`>yeah, just give me a minute to find it again <xentrac>Calculus Vaporis should fit in an iCE40 with no trouble, since it has about 1000 gates in a bit-parallel implementation, but it's probably a bit too constrained <xentrac>the J1A is an entirely practical CPU to put in an iCE40 though <damo22>what about a tool that has a USB port and takes source code in one stream and outputs a binary stream that is the compiled version <OriansJ`>917 LUTs for regular and 2019 LUTs for the set required for Fedora <xentrac>damo22: so the thing is that for the trusted box to increase your assurance, it needs to prevent attacks under some security model <xentrac>damo22: if your threat model includes "attacker has control over my laptop" then compiling the code inside a USB stick plugged into the laptop doesn't help; the attacker can just replace the correct compiled version output by the USB stick with a broken or backdoored version <xentrac>damo22: if your threat model, on the other hand, does not include "attacker has control over my laptop", then you might as well just compile the code on the laptop <xentrac>adding the USB stick doesn't help either way <damo22>that has a minimal environment for accepting source and compiling it <damo22>a coreboot payload is just a standalone ELF binary <xentrac>replacing your BIOS with a BIOS that is known not to have backdoors and believed not to have security holes does indeed protect against threat models including "attacker has corrupted BIOS" and "attacker knows unpatched BIOS vulnerabilities" <xentrac>and then you could indeed use the laptop's keyboard and screen to look at the code you're compiling, assuming the rest of the hardware that can interfere with that is also trustworthy <xentrac>OriansJ`: the PicoRV32 repo seems to talk about running it on Xilinx FPGAs only, not the iCE40 <xentrac>the kexec case involves trusting the pre-kexec kernel as well <damo22>the coreboot payload is probably more suitable <xentrac>that is, if an attacker has backdoored your pre-kexec kernel, they can leverage that into a successful attack on the post-kexec kernel as well <damo22>you could run it as a secondary payload <damo22>so if your main machine has a huge disaster <xentrac>OriansJ`: the Icicle repo looks like the real deal though. that's awesome <damo22>you can boot into the rescue bootstrapper <xentrac>damo22: that's an interesting idea! more as a system recovery tool than as a way to get trustworthy reproducible builds <damo22>the point being, you can boot from zero and verify the source code, then save a binary gcc toolchain to disk <xentrac>so the difficulty with a dual-payload boot ROM on modern hardware is that once you boot, say, Linux, it is possible for Linux to rewrite the boot ROM <damo22>i know, but unless you have iomem=relaxed option, the SPI is locked down in most cases <OriansJ`>or the fact modern hardware has firmware in the CPU that is measured in MB <xentrac>damo22: the handling of iomem=relaxed depends on the kernel; if the attacker has control of the machine they can replace the kernel <OriansJ`>which makes it rather trivial to embed something malicious in the CPU microcode that you'll never spot <OriansJ`>or the fact that there are single transistor attacks against modern lithography <damo22>if you dont trust the x86 architecture, youve got bigger problems <xentrac>yeah, although those are somewhat more difficult attacks to launch <xentrac>any random person who finds a zero-day RCE on your machine can get root and rootkit your kernel <OriansJ`>I hope to have answers to all attacks when people ask me about them <xentrac>they will have a much harder time adding malicious transistors to the CPU <damo22>well i still think coreboot payload isnt a bad idea <OriansJ`>a zero-day RCE when your default iptables rule is drop; certainly is a harder to find bug <damo22>you shouldnt bootstrap the whole GNU ecosystem off x86 <OriansJ`>damo22: agreed, someone could simply start with a stage0_monitor as a coreboot module <OriansJ`>xentrac: especially with Intel ME's ethernet sniffer vulnerability <damo22>but for many security models for your own purposes, coreboot payload would probably suffice <OriansJ`>damo22: I think it would be better to say, one shouldn't depend upon x86 to bootstrap GNU <OriansJ`>damo22: I view coreboot payloads as a convience or a floor for people who generally trust their hardware <OriansJ`>xentrac: (Intel ME directly can read ethernet frames from the ethernet socket and thus be hijacked even with perfect OS software running) <OriansJ`>damo22: I work for the State of Michigan as a Computer security and reliability engineer <xentrac>I was thinking more about things like buffer overflows in libjpeg though <OriansJ`>and Intel's ME has been a rat's nest of remote vulnerabilities <OriansJ`>like one could use a self-signed cert to gain access and then use ME to root the host <xentrac>note that damo22 just successfully got me to feed a possibly malicious PNG to Firefox's PNG decoder without ever sending my machine any packets directly <xentrac>so if there were another vulnerability in libpng... <OriansJ`>xentrac: life is simpler if you take the acceptable compromise strategy <damo22>on my thinkpad, i have wiped 95% of the ME firmware <damo22>i think its stuck in a loop waiting for its ARC kernel <damo22>but im a lot happier to use it knowing that it doesnt have a network stack <damo22>i was funded to free up 2 chipsets for libreboot <damo22>i now work in a hospital doing genomics <OriansJ`>damo22: take it the libreboot drama got too bad? <damo22>yeah, i posted my opinion on that and stopped helping that project <damo22>at one point we had a libre fork of coreboot called librecore <damo22>it lost traction and focus, no one had time to work on it <OriansJ`>damo22: if it makes you feel better; I only get to do this fun stuff in my free time <damo22>OriansJ`: youre doing great with it you must be dedicated <OriansJ`>a couple hours a night and a few hours on the weekend when my wife doesn't demand my time <damo22>i do audio stuff, like recording <damo22>ive never heard of forth until yesterday <damo22>my friend roped me in to play a music gig <damo22>so i need to practice for that too <OriansJ`>damo22: you did the Gigabyte GA-B75M-D3H coreboot port? <OriansJ`>I owe you a thank you and a beer for that bit of work <OriansJ`>and if you ever feel a desire to do more of that work let me know (I can send some free hardware your way) <damo22>the hardest part was ram init for sandybridge, which i did about half of <damo22>autoport can do most of the related boards with that generation <OriansJ`>and no RAM, only L1+L2+L3 cache until it is done <OriansJ`>and RAM initialization is entirely architecture specific <damo22>if you store too many variables, the whole cache evicts and you wonder why the log stops <xentrac>the cache part doesn't sound that hard, although I guess it's somewhat complicated by the chip powering on in 8086 mode <OriansJ`>(unless the architecture defines that it is done automatically by hardware) <damo22>the jedec start up sequence is complex for DDR3 and is really hardware specific how the memory controller is wired up and controls the lines <OriansJ`>It always seemed like such a cheap move forcing the bios to intialize RAM when the chip itself could have done it <xentrac>that sounds a lot harder than only using registers ***OriansJ` is now known as OriansJ
<damo22>we used qemu in a special mode that does IO through a serial port direct to an external motherboard to execute the real firmware blob one instruction at a time <OriansJ>hardware initialization is not for the faint of heart <damo22>so you encode the instruction and send it over serial to a monitor that executes the instruction on the cpu and has all the real side effects <damo22>intel did a reasonable job at making hw resilient to killing <damo22>except a thinkpad T60 that EC died <damo22>if raminit was easy like setting a few registers to turn it on, we'd have sooo many more coreboot ports <damo22>but its now in the hands of google and they are turning it into a blob wrapper <dgpv>Got the idea how to save this while on the plane yesterday, so had to implement, even if I said that I will pause working on this <dgpv>I say because the idea came yesterday, this is all part of yesterday's hacking session :-P <damo22>which channel would i go to ask for help with expresscard peripherals? 