IRC channel logs
2021-05-15.log
back to list of logs
<stikonas>melg8: I guess because it's simple enough and does not require anything labels <stikonas>in principle we don't need it until hex2 <stikonas>alternatively, I guess one can detach elf header from hex2.. and use cat to reattach it... <stikonas>OriansJ: any reason why we are not doing that for hex2 but are doing it for M0? <OriansJ>stikonas: because it is one of those easy jobs that got left on the floor for anyonne who wanted to improve stage0-posix <OriansJ>because if all that remains is hard tasks, it becomes difficult to start contributing to stage0 and feel empowered to make it better. <OriansJ>That is why I work on the annoying/hard tasks and leave the simpler and easier work for other people. <stikonas>ok, I guess that's good, although, maybe there should be some easy task list, so that people would know... <OriansJ>stikonas: fair but writing that list unfortunately landed in my easy pile... <OriansJ>So knowing if spotting those things would be hard is a bit of a meta problem for me. <OriansJ>little tweak here, little bug there, spelling mistakes all over the place <melg8>Hi, in Release_1.7.0 of M2-Planet - there were folder functions. but i dont see it in master, where it is now? <OriansJ>melg8: all library functions have been moved into M2libc <OriansJ>(to reduce duplication and simplify the task of updating the library of functions neeeded by multiple programs) <OriansJ>also instead of using M2-Planet function names, M2libc is the start of using standard C functions instead of intensionally different functions; in effort to simplify porting efforts to M2-Planet <gforce_d11977>melg8: but patching date is crap, we should patch the underlying functions (output monotonic raising values) starting at e.g. 01-01-2000 <stikonas[m]>And date binary might not be the only source of date... You can just call libc functions that date uses <melg8>OriansJ if i build kaem with hex2 vs catm of ELF-i386-debug and than hex2-0 should it differ? and if yes - why? <OriansJ>melg8: there should not be a difference if you -f $file1 -f $file2 .. -f $fileN or catm hold $file1 $file2 .. $fileN and pass them to hex2. now there might be a difference between hex2 written in hex1 and hex2 written in C <OriansJ>also hex1 doesn't support %label>label calculations <OriansJ>nor &label (that is only included in hex2) <melg8>Thanks! another question: how hard would be to add to kaem functionality of creation directory? i need that so i can copy not just single executable out of whole build, but whole /bin/ dir ? <OriansJ>well if the sys_mkdir is in M2libc (which it is in $arch/Linux/sys/stat.h) you need only include it in your build and call the function with the correct arguments. <melg8>that's nice, thanks! inside of guix - copy-file "M2-Planet/test/common_x86/x86_defs.M1" "x86/x86_defs.M1" - this is some external tool used to copy stuff around? <OriansJ>melg8: guix leverages gash-utils if I remember correctly. Which is basically core-utils written in scheme and run on guile which is orcastrating the build. <melg8>but guile is not build from source as part of bootstrap but just "is" in system from binary? <OriansJ>melg8: that is one of the issues with the current guix bootstrap that will be addressed by live-bootstrap when it builds the guix bootstrap seed. <melg8>don't you know is nix and guix .drv format differ? i've read about guix implementing some kind of interpreter for it's derivations, so it can manage build from "inside" of bootstrapping process? <OriansJ>melg8: well I can't speak to the inner details of guix itself as I don't hack on that but there are certainly people here who do (like civodul and janneke) who probably would be able to answer that question or point you towards people who would know. <OriansJ>just like the POSIX kernel in live-bootstrap is another seed issue that will later need to be addressed. <OriansJ>basically there will be "hand holds" of shrinking sizes as we approach an absolute bootstrap from "nothing" <OriansJ>guix was guile+linux+bios+microcode; live-bootstrap is posix+bios+microcode and what comes next will probably be bios+microcode and then solving those as well. <OriansJ>melg8: binary blobs that are used to make the work easier while working down the bootstrap. Like in rock climbing. <OriansJ>Linux+guile made the guix work far easier and enabled a great deal of progress in simplifying the bootstrap. <OriansJ>Now just Linux is being used to keep the bootstrapping work simple as we work to eliminate the need for the guile blob in the bootstrap to guix. <OriansJ>Then we will let go of the POSIX blob when we are able to take the next step down the rockface of bootstrapping to zero. <OriansJ>it is a very iterative process with some duplicated effort but that is the cost of addressing various people's baseline of trust. <melg8>again, at what stage posix strikes? when build stops be "kernel free"? (if we talking not about stage0-posix, but about stage0 at least)? <OriansJ>melg8: yes stage0 is bare metal only and stage0-posix was created for those people who trust their POSIX kernel (be it Linux, BSD*, etc) but wanted to bootstrap everything else. <OriansJ>just like there will be a stage0-CPM or stage0-DOS should someone need that sort of trust foundation. <melg8>i'm between two different directions of personal interest) paranoia tells me do work on developing baremetal, and love for nix build system tells to at least reproduce guix advances for nix) even though i would try live-bootstrap path to it, not requiring nix to use guile for it <OriansJ>melg8: well tcc is a bottle neck for bootstrapping GCC and we need a kernel for it to run. but a POSIX kernel written in the M2-Planet subset would solve that problem entirely (bauen1 is the kernel person if I remember correctly) <OriansJ>but you should work on what feels the most fun so you don't get burned out. <OriansJ>melg8: not unless I turn it into a kernel <OriansJ>So all of the steps would have to be manual until you get to after a kernel being built. <melg8>manual? can't they be encoded inside of c code at least? <OriansJ>melg8: think insert floppy disk and boot computer that doesn't have an operating system installed. <OriansJ>one could automate the qemu bootsteps but that wouldn't actually be on bare metal now would it? <melg8>yea, but for development that would be nicer, than.. you know, ci running robotic arm with floppy disk each time - to air gaped machine from 90-s <OriansJ>melg8: well the development is going to be rather limited as only the read/write/init functions need any change. <OriansJ>init simply sets what is written to and read from (and any memory setup required) <melg8>i mean until this kernel stuff is done - it would be nice if it tested not by hands <OriansJ>read just simply reads 1 byte from the input source (be it file, serial port, etc) <OriansJ>write just writes 1 byte to the output destination (be it file, serial oort, etc) <OriansJ>melg8: probably, that is why the CPM/DOS port will probably be the easiest half-step <OriansJ>and anyone is free to do that work if they so desire. the blueprints and standards are readily available. <melg8>it would require alot of guidance for me, just to get started) never developed kernel stuff and very little c++ baremetal <OriansJ>as hex0, hex1, hex2, M0, M1, cc_* and M2-Planet behaviors are all universal be it on bare-metal, CPM/DOS, POSIX/Linux/*BSD, Windows, VMS, etc <OriansJ>melg8: I am more than happy to guide someone through the work if they desire to do so <melg8>okay, if i fail with nix attempt, will know that i have plan B what to do) <OriansJ>melg8: if you are failing with nix, it means I didn't give you the help you needed to better understand the pieces. <OriansJ>or I needed to add something to stage0-posix to address those problems <OriansJ>both of which are something that I will do <melg8>i mean, now it looks like it can be done, and even maybe by me (even with my limited knowledge of nix) but there is alot of nuance stuff, like - build system, people seems don't like large rebuilds, than live-bootstrap which i take as base for this - now not adapted for all arches which nix supports. If i want it to not just be home project for me <melg8>in nix, but push to nixpkgs - that can be alot more challenging <OriansJ>as there shouldn't be anything magical needed to get nix to build stage0-posix (if you can run a shell stage0-posix will build) <melg8>i'm starting with kaem. and go from there. i've managed to build 1 package with c-kaem. Now only missing useful part is kaem to be able to make dirs, so i don't need to rerun same process 5 times to get 5 binaries out of it. <OriansJ>well it'll be simple to add mkdir to kaem <melg8>1 thing i've encountered is - that nix builds stuff in /build directory... and all bits of sources are in /nix/store/ and... default runner (kaem in this case) can't (or can it?) just copy everything from this /store paths to set it in single unified directory. So i rewrote whole build script to use evaluated expressions from nix, so it builds <melg8>using /store paths instead of relative <melg8> +> /nix/store/v00nmlmnj0qsdc6nprm68hxq4p6hfygp-bootstrap-seeds/POSIX/x86/hex0-seed /nix/store/v00nmlmnj0qsdc6nprm68hxq4p6hfygp-bootstrap-seeds/POSIX/x86/hex0_x86.hex0 hex0 <melg8> +> hex0 /nix/store/v00nmlmnj0qsdc6nprm68hxq4p6hfygp-bootstrap-seeds/POSIX/x86/kaem-minimal.hex0 kaem-0 <melg8> +> hex0 /nix/store/6y3ys70mn4sf5a8snyl8bis1ivzvj7bb-source/x86/hex1_x86.hex0 hex1 <melg8> +> hex0 /nix/store/6y3ys70mn4sf5a8snyl8bis1ivzvj7bb-source/x86/catm_x86.hex0 catm <melg8> +> hex1 /nix/store/6y3ys70mn4sf5a8snyl8bis1ivzvj7bb-source/x86/hex2_x86.hex1 hex2-0 <melg8> +> catm hold /nix/store/6y3ys70mn4sf5a8snyl8bis1ivzvj7bb-source/x86/ELF-i386.hex2 /nix/store/6y3ys70mn4sf5a8snyl8bis1ivzvj7bb-source/x86/M0_x86.hex2 <melg8>but maybe that just my lack of knowledge - and it can be done with nix and kaem to prepare single directory <melg8>i'll show script to nix guys so they maybe guide me <melg8>and ... for the record - sha256sums are same as in live-bootstrap <melg8>and now i'm using catm to copy stuff out of build <gforce_d11977>another run, another problem 8-))) tar-1.34 checksum is wrong, here the logs and the graph: <OriansJ>melg8: bootstrap-seeds doesn't require a build, they are just taken as built and done. <melg8>yes, so when i had gcc + coreutils in client side of things - that was 1 download, and if now it is from kaem to coreutis through live-bootstrap - it's like n-times longer <OriansJ>and stage0-posix doesn't use an external bootstrap-seeds but leverages a git submodule to grab a specific version and everything is done via relative paths, so if you just cd to the target directory in nix prior to the build starting, everything will be done in there <OriansJ>and yes stage0-posix is multi-arch and builds from hex to a minimal cross-platform C compiler and live-bootstrap goes from their to bison and GCC <gforce_d11977>("a minimal computer: just a CPU with a serial link and a crystal") <OriansJ>gforce_d11977: I don't think it'll work that good for bootstrapping. one would want 1 input and 1 output as it is simpler than dealing with 1 input/output. (and require less memory too) <OriansJ>gforce_d11977: well serial ports or usb or etc <OriansJ>basically one would want a single device to write the output to and a seperate device for reading of input so that everything could be done in as little memory as possible <OriansJ>less transistors and room for potential lithography compromise attacks to alter the output. <gforce_d11977>ofcourse you are right, this "tiny m68k system" was just sooo cute. taking this more serious - and hopefully some university will bring it up - would need a lot if thinkering.... <OriansJ>also the 68K is far from a simple CPU in comparision to the RISC-II done at Berkley <OriansJ>it certainly is "cute" but not entirely practical in its current form. <gforce_d11977>you are right, this beast already has a monitor-implementation onchip <OriansJ>also we would want a system where we could replace individual chips with a trace of individual gates that implement the same functionality. a la Monster 6502 <OriansJ>which would work for a simpler CPU architecture with far less microcode requirements. <OriansJ>a 68K cold-fire subset certainly would be possible. <OriansJ>of course this is the sort of technical bit that would only be really solved by someone actually doing the work to prove it one way or the other. <gforce_d11977>there was this project where a guy implemented on an 8bit-microcontroller an 32bit ARM-emulator and run Linux on it: this works when you just have enough RAM, so sticking RAM an a monster 6502 and give it a year or 2 to run a live bootstrap would be a nice ARTWORK, e.g. as an museum installation 8-))) <OriansJ>gforce_d11977: well an 8 bit processor with a 16bit address space couldn't really properly emulate a 32bit CPU <OriansJ>not to mention the 256byte stack limit... <gforce_d11977>OriansJ: ofcourse it can. if you have enough time for emulating it. <OriansJ>gforce_d11977: well one would have to create a 32bit state machine which supported writing to a 4+GB disk on a system that couldn't even with GEOS access past 20MB <gforce_d11977>"everything which cannot be done in assembler have to be soldered" (or similar) <OriansJ>gforce_d11977: soldering on more memory chips doesn't solve the problem that they can't actually be addressed with the address pins on the address bus. <gforce_d11977>just read the posted link, dmitry has done it (and some other guys worked on similar stuff) <OriansJ>gforce_d11977: dmintry did a 32bit arm emulator on AVR which had a 22-bit program counter and 64 banks to switch to. While the 6502 only support 16bit Program counter and 4 banks to switch to (16 if one uses the 6509) <OriansJ>so 28bits of address space is plenty to simulate a sub 256MB process <OriansJ>but let us just assume that is enough memory. You'd need a disk controller for the 6502 that supports a 32bit address space. <gforce_d11977>OriansJ: you are thinking too low level. just emulate a program counter with more bits. An reading from/writing to a e.g. 1TB-SD-card (which itself includes untrushted code) can ofcourse be done with 8bits. you "just" have to write a wrapper. again: do not try to optimize that, it is slow anyway... <OriansJ>gforce_d11977: I don't know of any SD-card interfaces for 6502 systems nor any disk systems for 6502 systems that support for than 24bits of address space. <OriansJ>so it would have to be a custom built disk controller and you'll be limited to 1 32KB page of memory at at time (assuming you can squeeze all that functionality into 32KB) <OriansJ>as the data you banked in would be in the same address block you would be executing. <OriansJ>also the 256byte stack limit for the 6502 means you would have to not use the stack for your emulator either; which makes for even more complex of a coding problem. <OriansJ>So assuming you had a custom 32bit (or bigger) hard drive interface; a 6509 with 1MB of RAM and had the time&&skill required to implement an emulator in under 60KB (which also includes its entire stack space, 1KB needed for just holding the register values of the emulator and all state tracking) <OriansJ>and assuming that disk controller could write to the pages that are not banked in. <OriansJ>and all the mapping between the instructions and the opcodes <OriansJ>unless you used seperate pages for the various opcodes and their implementations that you had to bank in and out for every single instruction. <OriansJ>but all of this depends on a Disk drive that doesn't exist yet and has a bunch of unique properties. <OriansJ>or what would be far simpler is just take the monster 6502, expand the PC and stack to 32bits and then this entire bit would be trivial <OriansJ>drop the number of registers down to 16 and it might end up taking less gates than the monster6502 <OriansJ>the Turing limitation is always about the size of the addressable memory space. If you have infinite amount of RAM anything is possible but if you only have 4bits of address space, you can't simulate shit. <OriansJ>So shoe-horning in a 32bit disk drive would give you enough address bits if you are clever enough to simulate what you wanted but with any standard 6502 disk drive, it is impossible without a human playing floppy disk pager for the simulator. <civodul>melg8: Guix uses pretty much the same .drv format as Nix (yet they're not interchangeable in practice) <melg8>btw, who is David (Nix), - and where is source for nix?) <vagrantc>i love it when people take my wild crazy ideas made in jest and actually implement them :) <vagrantc>xwvvvvwx: i think you are being asked about ... haven't heard from you in a while ... *vagrantc waves to xwvvvvwx <gforce_d11977>OriansJ: i have to think about it. implementing the emulator in 60k is indeed needed... <OriansJ>gforce_d11977: well sweet-16 fits in 300bytes because thankfully 6502 does have a 16bit PC and one doesn't need to simulate via disk paging and 1byte opcodes only and 32 zero page addresses for the 16 registers (R0-R15) <OriansJ>but to support the 1KB required to support arm, x86 or AMD64 registers +scratch registers; you wouldn't be able to leverage the zero-page instructions like woz did <OriansJ>so limited to indexed absolute instructions (3 bytes) per <OriansJ>and assuming only 5 instructions per primitive operation (and zero bytes spent on decoding). (4 single byte writes and a jump instruction to return.) to implement the over 4000 unique ARM opcodes; you already blew your 60KB budget and you didn't even add the complex state bits required for paging out to disk <OriansJ>So you are either going to have to do an arm subset or do a programming paging implementation for the various instructions you wish to implement. <OriansJ>so lets assume you figure all that out (entirely possible given enough effort) <OriansJ>and go with the 6509 to get the ability to use a sweet 1MB of RAM if you use 64KB pages but then you'll have duplicate the registers between the pages which requires scratch writes out to disk or you could use 32KB register banks but be limited to 512KB <OriansJ>and you'll have to allocate 1 of them for the RAM of the virtual machine you'll be paging out to disk. <OriansJ>But the biggest 6502 floppy drive only had 800 KB of storage capacity to work with and the biggest 6502 hard drive only had 32MB of capacity. <OriansJ>so you'll need 1310 floppy disks that you would have to be visually alerted to insert and remove <OriansJ>or an external robot to handle the swapping of floppies and all the control routines required. <OriansJ>So it is possible to do gforce_d11977 (assuming external user floppy paging or robotic floppy paging) and some extremely clever and tight code implementing only an essential subset of ARM or x86 or AMD64 <OriansJ>So anyone who manages given all of that will have achieved something seriously difficult and impressive even if the emulator takes up 480KB in RAM <OriansJ>with bank switching potentially every 10 simulated instructions (10 millisecond delay) but leveraging a blazing fast 20Mhz 6502 core; you are looking at best case of a 2Khz simulated CPU speed. (floppy disk I/O latency will be a major factor) <OriansJ>So 347.22-ish days boot time to linux <OriansJ>assuming zero bit flips or data integrity errors occuring in that time period. <xentrac>hmm, why would an emulator take up 480KB? <OriansJ>with half of your 16 banks being used just to store your 256KB instruction lookup table according to the dmitry.gr link you provided. <OriansJ>xentrac: ARM is far from a simple state machine with over 4K unique instructions <xentrac>oh, because you're implementing all the opcodes <OriansJ>xentrac: using only 8bit instructions <xentrac>the usual i386 registers are 4-byte EIP ESP EBP ESI EDI EAX EBX ECX EDX, right? 36 bytes on the zero page? I think that will fit in 1K <xentrac>I feel like 99% of i386 instructions executed are probably covered by about 30 opcodes and 5 addressing modes <xentrac>of course if the other 1% requires you to swap a floppy disk that's ... still not okay <gef>oriansj: "only an essential subset of ARM or x86 or AMD64" - after some tried experimentation of mine, my conclusion is that the essential subset is the SBN4 int32 architecture. This would be enough to arrive up to M2-planet without big hurdles. Getting a posix kernel working though, is a different game. Also, banking isn't such a big deal. Amstrad's CPC series had its ROMs overlapping all the RAM banks. If you program consciously around it, the <gef>overheads aren't excessive. <OriansJ>don't forget scratch registers for immediates, address to bank register mapping, etc <xentrac>if you're on a von Neumann machine you can JIT-compile each basic block into a JIT cache, which would probably give you more like a 20× slowdown (20 6502 instructions per i386 instruction) than a 10000× slowdown <OriansJ>gef: so you want to emulate a 64bit architecture to run the emulation of a 32bit architecture? <gef>* essential subset -> I mean the essential subset across _all_ hardware (architectures) and software (languages. Yes, I mean that sbn4 (https://en.wikipedia.org/wiki/One-instruction_set_computer#subneg4) is the minimal layer to even bootstrap from within a *computer language*. It's emulation is a triviality - for most languages it is just an oneliner. <xentrac>JIT is less doable on an AVR where erasing a block of NAND takes milliseconds <gef>32bit should be the right model, since it can be emulated efficiently from architectures from the 80s onwards (and some before, z80 wouldn't be to bad due to efficient 16 bit register arithmetic) <xentrac>oisc is fun but I suspect you can do better with a misc-like approach or Wirth's RISC (or MIPSter's RISC-U) <melg8>@OriansJ when we build kaem from sources for the fist time, where <stdio> header comes from? i dont se it in M2 call <OriansJ>melg8: #include ... is a null op in M2-Planet because it is meaningless in bare metal <OriansJ>So asking where stdio isn't the question but the stdio functions that are used; which always must be in the -f $files listed in the build <OriansJ>gef: in an absolute turing sense yes that might be true. In the oh god imagine having trying to an ARM emulator in OISC instructions; I am not gonna touch that. <OriansJ>melg8: -f M2libc/x86/Linux/sys/stat.h probably <OriansJ>melg8: yes, unless C prototypes are used in the files. <melg8>and what are the rules for order? i should add this file before file where i want to use it? <OriansJ>melg8: left to right like if you used cat. So imagine cat $file1 $file2 .. $fileN > $input and you passed only that code to the C compiler <melg8>why mescc-tools/stage0-posix/m2-planet - all have master, and m2-libc - main?) <OriansJ>melg8: github changed the default from master to main <OriansJ>and it wasn't a detail that mattered to me <melg8>now i cant make universal download function)) <OriansJ>if you make a master branch for M2libc, I'll merge it and then you'll have it 