***attila_lendvai_ is now known as attila_lendvai
<rickmasters>It may have seemed like I disappeared but I've been slowly working away on kernel development. <rickmasters>What are the exact requirements for the kernel for each stage of bootstrapping? <rickmasters>I think it makes sense to understand this in order to answer the next question: <rickmasters>As last reported, the current builder-hex0 kernel can build mes-m2 and run Hello World. <rickmasters>However, it was a bit of fools gold - it could not run mescc to compile C code. <rickmasters>So, I took on the task of figuring out what it would take to build mes using mes-m2 <rickmasters>Unfortunately I was detoured when stage0-posix stopped building due to a existing <rickmasters>Debugging the kernel and then mes-m2 was an arduous task because the problems surfaced as memory corruption. <rickmasters>But after many weeks, I've resolved the issues and functionality necessary to build mes! <rickmasters>I haven't tried to run mes yet. It just finished building after around 36 hours! <rickmasters>I have been using builder-hex0 for this investigation but I am not determined to commit those enhancements. <rickmasters>I know oriansj favors moving to a higher language immediately for further kernel development. <rickmasters>One could argue that builder-hex0 should have stopped development after building M1 or M2. <rickmasters>However, enhancing builder-hex0 might mean the difference between developing next in <rickmasters>C99 versus M2 and I may have a different opinion on that. <rickmasters>Anyway, since I'm still having fun I think I'm going to see what it takes to build tcc. <rickmasters>I'm hoping for better luck than I had with mes but we'll see... <stikonas>rickmasters: so that's mes built with mescc.scm? Impressive. Although 36 hours is a long time :(. But it is what it is <rickmasters>yes, mes build with mescc.scm using live-boostrap sysa/mes-0.24/mes-0.24.kaem which compiles over 150 C files <stikonas>hopefully it runs, it should be fairly similar to mes-m2 though it might be using more syscalls... <rickmasters>I'm hoping so, it's essentially the same program as mes-m2 <rickmasters>i also spent a long time reworking boot2now to support multiple stages, sort of cacheing the previous stage in my primitive script/file system <rickmasters>I've also been reading here the interest in bootstrapping on UEFI - it does seem more modern and I'm also interested... <rickmasters>I'm wondering if there are open implementations of UEFI? <stikonas>I did start reading up a bit on UEFI api, so fell free to join me <stikonas>I think UEFI binaries would run at close to POSIX speed <rickmasters>i'd like to join you on UEFI but not sure how much time i'll have right away <stikonas>in the evening I can publish what I have, right now it's just hex0.c high level prototype and build/run scripts to kick off qemu with it <rickmasters>i'm really not sure why mes-m2 on builder-hex0 is so frickin slow. I can't see how the kernel can slow it down so much since mes-m2 is doing the heavy lifting <stikonas>rickmasters: UEFI supports calling another image and blocks until return <stikonas>input/output and argument parsing are also different from POSIX <stikonas>and that's basically it (at least until much later) <stikonas>well, at binary level ELF is not used but at C level these 3 differences are the only ones <rickmasters>yeah, builder-hex0 would have been a lot easier if I could have hacked up stage0-posix to change the OS interface... <stikonas>and they'll mostly be the same between all apps <stikonas>well, basically with UEFI I think we can skip kernel and let UEFI act as a kernel <stikonas>and then just right kaem, hex0, hex1, hex2, etc.. <rickmasters>that could probably be made to work up until make-based builds <rickmasters>for UEFI calling another image - is it LoadImage() you are referring to? <stikonas>I guess it's LoadImage() followed by StartImage() <rickmasters>the stage0-posix executable are all fixed to the same base-address (and not relocatable) so I'm wondering if UEFI images need to be relocatable... <rickmasters>if so that would require extensive changes to stage0-posix <rickmasters>I initially tried to put stage0-posix executables at different addresses to simplify the kernel but when I realized that kaem launches kaem I gave up on that <stikonas>rickmasters: it will have to be a fork of stage0-posix, I guess stage0-uefi <stikonas>since at binary level format is different. It uses PE binaries rather than ELF <rickmasters>i'm not sure how much work it would be but probably not trivial. I'm pretty sure it can be done though <stikonas>ok. I haven't look at asm or binary level yet <stikonas>well, C prototype for hex0 was not too hard <stikonas>took me a bit to get familiarized with UEFI api <stikonas>quite a bit can be taken from stage0-posix <stikonas>and linkages will be different, I guess in hex0.hex2 prototype we can take quite a bit from stage0-posix <rickmasters>you know, I might explore porting builder-hex0 to UEFI, not forking stage0-posix, as an alternative but I don't know if that makes sense yet without more research <stikonas>though it might be harder than porting hex0/kaem to UEFI <rickmasters>ok, just skimming the link you sent, it seems there is a EFI Byte Code Virtual Machine, is that in your radar? <rickmasters>did the hex0 prototype you wrote in C get compiled to x86 or the EFI byte code? <stikonas>but if we can write UEFI byte code and have it cross-platform, maybe that's an option too <stikonas>UEFI also includes a virtual machine specification based on a byte code format called EFI Byte Code (EBC), which can be used to write platform-independent device drivers, but not UEFI applications. As of 2015 there has been limited use of EBC. <stikonas>sounds like we can't use it for applications <rickmasters>stikonas: stupid question: how do you pass options to your hex0 UEFI application? is that a machine "BIOS" setting? <stikonas>rickmasters: right now I use UEFI shell but that can be replaced with kaem-minimal once it exists <stikonas>and kaem-minimal does not need any options <stikonas>UEFI shell can be accessed from boot menu <rickmasters>oh, UEFI has a shell. how convenient! lol (PC BIOS sucks) <stikonas>rickmasters: well, I think uefi shell is just an application written on top of UEFI <stikonas>well, UEFI is much closer to full OS than BIOS <stikonas>anyway, I think kaem-optional-seed will do the job later, but for now UEFI shell is good for development <rickmasters>my 4K trs-80 color computer had a better standard interface (a BASIC prompt) than BIOS (none) <stikonas>but UEFI is 64-bit (at least most systems ship with that) <stikonas>and later stages of x86_64 bootstrap is still a bit lacking <rickmasters>so some UEFI implementations do not support 32-bit? most? <rickmasters>it appears to be an option, but modern systems skipped it? <stikonas>hmm, I'm not sure... I thought 64-bit UEFI can run 64 bit apps and 32-bit UEFI can run 32-bit UEFI apps <stikonas>but outside VM I think you'll struggle to find 32-bit UEFI <rickmasters>i wonder if there are 64-bit UEFI boot loader applications which can boot 32-bit kernels <stikonas>I guess that might be fairly simple UEFI application, it should load kernel something in the memory, possibly shut down UEFI services and jump to kernel address <rickmasters>well, searching around it appears to be possible with some shims but its unnatural and there are issues with it <rickmasters>anyway it looks like 64-bit UEFI is the appropriate target <stikonas>well, maybe somebody will be more inspired to fix mes/tcc for 64-bits <stikonas>right now motivation was lower since 64-bit linux kernel can run 32-bit apps <stikonas>though not without some problems, live-bootstrap has Y2K38 problem <rickmasters>it does make me question the long term relevancy of 32-bit bootstrapping in general, and that includes builder-hex0 :( <rickmasters>similar to BIOS, it's like historical more than practical <stikonas>and it's nice to have something lower level than UEFI <stikonas>it might be good example if somebody really needs to implement full baremetal bootstrap <stikonas>you baasically need to take builder-hex0 and replace bios calls with something that interacts with hardware, i.e. some small driver, say SPI <stikonas>it will probably be a few times bigger than builder-hex0 <stikonas>but way smaller than any UEFI or BIOS implementation <oriansj>well we have until 2038 to sort all of that out <oriansj>and UEFI bootstrapping looks like it might be fun once I figure out how we are going to enable relocation with hex2/hex1/hex0 generated files <oriansj>and a stage0-UEFI port sounds cool (along with UEFI enhancements for M2libc) <oriansj>and rickmasters you are probably right getting a better C99 compiler before we start doing a kernel in C might end up saving us a good bit of time. <oriansj>especially in light that xv6 would become a buildable base for our work <stikonas>hmm, not sure where to publish my repo... <stikonas>maybe I'll push to my server for now, it has github oauth2 login enabled... <stikonas>to test run make in Development dir, click F2 and select UEFI shell <stikonas>oh you might need to also install mtools <rickmasters>oriansj: thanks. not sure i've seen xv6, i'll have to take a look <rickmasters>stikonas: thanks. that'll be a real time saver to start playing around with UEFI <stikonas>I haven't added any README yet, but merge requests are welcome <stikonas>hex0.c is fairly low level, it includes efi.h but that is struct and typedef only includes <stikonas>so in assembly it will be just some offset <stikonas>oriansj: hmm, yes, relocation might be a problem... <stikonas>I guess for hex0 it's not a problem since it's just hex code anyway <oriansj>stikonas: well absolute addresses can be resolved at runtime for the mallocs but the addresses for globals remain a problem unless they are malloc'd too <oriansj>So there is some complexity needed but not anything actually *hard* if anything the lack of absolute addresses for the lower steps forces us to avoid a few earlier and worse design ideas <oriansj>To the github problem, well if I am setting up a server with a wiki and a domain we control, why not go the distance and add something to host git repos and email too <Hagfish>"NOTE: we have stopped maintaining the x86 version of xv6, and switched our efforts to the RISC-V version". hmm, that's a bit unfortunate, especially as there are open PRs against the repo <oriansj>Hagfish: actually that means it is ripe for someone else to take over the lead for xv6-x86 <oriansj>just maintain feature parity with risc-v xv6 and focus on clarity and you are golden <Hagfish>yeah, it would be a pity to depart from the original goals of being an OS to teach students concepts <Hagfish>a teachable OS is an auditable OS, and teaching isn't complete if it doesn't include bootstrapping :) <oriansj>learn everything you ever wanted to know about compilers, libraries and operating systems (and probably hardware later too ^_^) <Hagfish>that might take more than a year for someone who is completely new to those concepts, but it would be a year well spent <Hagfish>and a good teacher could probably trim down the course to just focus on the highlights, and to inspire the student to go away and learn more on their own <stikonas>oriansj: mallocs are not an issue I think <stikonas>oriansj: we just reserve memory_pool from UEFI <stikonas>or maybe the functions is called allocate_pool <oriansj>stikonas: we can still an idea from knight and just use a register as a malloc pointer <oriansj>ok and I bought a domain and paid for it to be registered for the next 10 years: bootstrapping.world <oriansj>next up is I am going to setup a server with sshd and doku-wiki and then something to provide a public git service <oriansj>Then probably getting accounts setup for the people here <oriansj>so if anyone has a good suggestion for a self-host web-git service let me know <oriansj>say wiki.bootstrapping.world for the wiki and git.bootstrapping.world for the public git repos and files.bootstrapping.world for the tarballs/videos <oriansj>unless someone thinks we need something more complicated than just a folder with files <Hagfish>that sounds like a nice logical setup <Hagfish>it's worth checking what the best practices are if you're hosting an sshd server on the public internet <Hagfish>presumably only allowing keys, not passwords <stikonas>oriansj: I've only tried gitea and gitlab, but gitlab is pain to install <Hagfish>i got the impression that sshds are a big target, but maybe the real threat is just that you get swamped by bot connections <stikonas>with password authentication disabled, they are harmless <stikonas>I had to disable new user registration on my gitea instance (though they can still login via github) <stikonas>but my server is at home on residential connection... <stikonas>so now that oriansj has non-residential server, it's probably a better place <oriansj>Hagfish: port knocking solves that pretty quick <oriansj>basically if you don't knock the server correctly, no sshd for you <Hagfish>yeah, i used to think that knocking was security by obscurity, but i guess it does counter a real threat in an efficient way <oriansj>easiest way to set that up is randomly generate the sequence and email those who need access the sequence <Hagfish>yeah, that doesn't sound like too much overhead (although in theory you'd want to regular rotate that "key") <stikonas>well, port knocking is in principle extra password <stikonas>so it's some kind of 2 factor authentication <oriansj>Hagfish: if you can guess a 5 number sequence (in the port range we use for knocking which is a couple thousand) and get it right in 3 guesses, then yeah rotation will be needed frequently. However if not, we can just black list any IP that hits the first port number more than 3 times but fails to hit the second port with its next packet. <oriansj>and do a yearly rotation as part of a scheduled maintance task <oriansj>also one needs to remember obscurity is a tool of security but you are absolutely right that it shouldn't be the only tool. As passwords and certificates and passphrases are just obscurity and if the bad guy gets infinite guesses, they will get in. So rate limiting and lock-outs after repeated failures and just straight dropping of traffic from all non-approved IP addresses limit the odds of someone guessing right. <oriansj>if someone has a zero day against some service you use (and there are lots of those out there), your only safe play is to drop all traffic from parties you don't trust <oriansj>right now, I have enough remotely executable "zero day" (but honestly like day 1000+ but Oracle still hasn't patched yet) vulnerabilities to say if you have an Oracle database hardened to STIG levels that I can connect to over the network, I am able to own that box in 3 minutes but add a simple firewall rule that just drops all traffic from IP addresses that aren't trusted and boom that risk drops down to zero (until I get a zero day <oriansj>ideally public facing systems are nothing but a simple proxy but as I do have to budget within the limits of my wife, less than optimal security might have to occur. <oriansj>and that system should get wiped/reset daily <oriansj>as that system just has the proxy setup and just the ability to directly access to the actual servers on the same ports it proxies <oriansj>no credentials or even extra access but services which flag if new files or processes are created or if checksums of the process memory changes; those flags result in the systems being instantly wiped clean, notifications being sent out and a dump to investigate. <oriansj>and for extra fun, because user passwords and port sequences are generated at setup (and sent out); we don't need git-secrets for the configuration of the servers. Anyone would be able to build identical services with the same users if they so wanted. <oriansj>we need only add public keys for the people which we use to encrypt the emails with the credentials which are generated. <oriansj>(adding usenet/mailing-list and other extras might be an option if anyone wants to do that) <oriansj>and as I don't think I can convince her to let me buy multiple servers nor do the $50/month per 1U hosting cost for dozens of servers; we might have to just setup a backup.bootstrapping.world that anyone can mirror the data and everyone can know what needs to be restored if things go sideways ***ChanServ sets mode: +o janneke
<stikonas>hmm, I can't see why EFI byte code wouldn't work for applications... <stikonas>though I don't know how widespread it is... <stikonas>although it looks like EBC might be going away... So probably best not to target it <oriansj>or maybe someone will figure out a trivial VM which fits in 2KB and runs on bare metal providing the most minimal functionality we need in a portable way and it'll enable stage0 porting to new hardware <oriansj>getting a subset of knight is quite possible in 16KB <oriansj>I'll need more practice with kernel writing first before I'd probably do that <oriansj>but first, I think I'll just make our wiki a series of static pages with a git repo behind it until I come up with a better idea <stikonas>well, there are brainfuck VMs that fit even in smaller amount and probably one can modify them to support file operations necessary for bootstrapping. But the problem is that then machine code is probably easier to read than BF code