IRC channel logs

2021-06-12.log

back to list of logs

<stikonas>and it's just that single file
<stikonas>everything else builds fine
<stikonas>I guess if we try to first build mes (skipping libc+tcc) then it would work...
<stikonas>but that would increase build time
<stikonas>although, with current build times of live-bootstrap, it wouldn't be too bad
<oriansj>it would be ironic to have to build mes with mes to compile qsort.c
<oriansj>*to^to just^
<stikonas>oriansj: I'm trying to narrow it down to see what it doesn't like in qsort.c
<stikonas>it's happy with qswap function
<stikonas>so must be either qsort or qpart
<stikonas>we do have a working mes checkout though, the PR I have is able to compile some other mes
<stikonas>oriansj: it doesn't like #ifdeds
<stikonas>#ifdefs
<stikonas>or rather #if 1
<stikonas>if I remove those and just keep what's in #else branch it works
<stikonas>oriansj: actually, it's not #if but the C++ style comment that follows
<stikonas>/__x86_64__
<stikonas>oriansj: maybe remove it?
<oriansj>or convert into /* __x86_64__*/
<xentrac>uhoh, NieDzejkob, orange website alert: https://news.ycombinator.com/item?id=27477599
<xentrac>oh I guess you already saw it
<NieDzejkob>yeah, a friend of mine told me a moment ago
<NieDzejkob>I love how they immediately went "this generalized statement in the introduction doesn't exhaust all the nuances", even though it's not even the main topic of the post :D
<oriansj>NieDzejkob: nice write up
<NieDzejkob>thanks
<oriansj>I love seeing new options for our bare metal root of trust ^_^
<xentrac>NieDzejkob: what, about the kernel?
<NieDzejkob>yeah
<oriansj>xentrac: kernel probably would be a couple steps down the line
<xentrac>I think that's an example of what I was just complaining about in https://news.ycombinator.com/item?id=27478804
<xentrac>oriansj: yeah, maybe. do we have some kind of rough outline of what system calls are needed for different stages of the bootstrap?
<oriansj>xentrac: yes: exec, open, read, write, close and waitpid/wait4pid are all that are needed until you have permissions on your filesystem then you also need chmod. Which will get you to the end of stage0-posix (except for mes-m2 with needs more)
<drakonis> https://news.ycombinator.com/item?id=27477230 sidenote: orange site has noticed guix as well
<drakonis>brace for impact
<oriansj>anyone who thinks they can write a kernel without a kernel, needs to be forced to use a toggle switch board instead of a keyboard for a week to learn the difficulty of the task of writing software without the loving help of a stable operating system.
<oriansj>building a kernel without a kernel can be done but development is a whole different question.
<xentrac>also fork, right?
<oriansj>xentrac: depends if your exec spawns a seperate process or not but yes if your exec doesn't spawn a new process, you would need fork as well.
<xentrac>because I don't think waitpid will be very useful without fork and exit
<oriansj>xentrac: you are right, I forgot about exit again.
<xentrac>you could implement exit by execing the shell the way PDP-7 Unix did but that would involve a lot of changes to user-level code ;)
<xentrac>I think you're exaggerating the difficulty a little; it's quite practical to do self-hosted development on a variety of control-program-and-monitor-type systems with no separate kernel, of course including CP/M itself, but also MacOS pre-X, MS-DOS, Windows 3.1, F-83, Apple ][ DOS, ProDOS, the Commodore 64 "KERNAL" which wasn't, and so on
<xentrac>also Oberon
<xentrac>I'm not saying it's not painful but it's not in the same ballpark of difficulty as using a toggle switch board instead of a keyboard ;)
<oriansj>xentrac: fair but I always hated that developing on DOS ment you had to properly save both sides of a code change on external storage to ensure you didn't lose work (or your recovery state) in the event the program crash took out the filesystem too
<oriansj>or had to have a seperate test system to test any builds you do.
<xentrac>yeah. there was an industry of breakpoint boards for PCs in the 01980s and 01990s so you could press a button that wasn't on the keyboard to freeze the CPU
<xentrac>corrupting the filesystem was surprisingly rare
<xentrac>but something as simple as an infinite loop in Windows 3.1 was a pain in the ass
<xentrac>and also, as we saw last night, in NieDzejkob's miniforth ;)
<xentrac>breaking out of an infinite loop with an interrupt is sometimes necessary in those conditions, but it means that your memory state is potentially corrupt, and if you have an iloop bug that happens when interrupts are disabled it can be a real pain to track down (one of the benefits of the breakpoint boards)
<xentrac>things like Oberon and BASIC interpreters mostly avoided that kind of problem because a buggy program couldn't corrupt arbitrary state, and a keyboard interrupt couldn't interrupt the BASIC interpreter's internal operations
<xentrac>on the other hand being able to reboot to a usable prompt in a second or two made a lot of that kind of thing less painful
<NieDzejkob>oh, I was just wondering how it was that the crashes in DOS didn't disrupt the filesystem, and now you mention that the answer is that they did :D
<oriansj>NieDzejkob: if you can get a FAT filesystem and the handful of needed POSIX syscalls needed to hit TCC, we could bootstrap Linux.
<oriansj>(or ext2 if you prefer it)
<oriansj>and I would just like to point out M2-Planet+cc_x86 can be built to run on bare-metal and cc_86 can be easily written in any language.
<stikonas>oriansj: hmm, I think tcc also has // comments... So even if we fix qsort, there it breaks later...
<Melg8[m]>can somebody explain for me - how in this implementation - program could manipulate registers or it can't? and can program just execute some binary values for processor? like if someone have C compiled hello world as payload?
<NieDzejkob>this implementation being?
<xentrac>NieDzejkob: DOS didn't have a disk cache by default, so you could take floppies out of the drive without warning the computer you were going to do that
<Melg8[m]>that's about this https://github.com/NieDzejkob/miniforth miniforth
<NieDzejkob>so how I can manipulate CPU registers from the level of Forth?
<xentrac>oriansj: does stage0-posix require real concurrency or can it be faked in the way MS-DOS faked pipes by writing the output of all but the last process to temporary files?
<Melg8[m]>NieDzejkob: yea
<NieDzejkob>that's rarely necessary, but when it is (like to call int 0x13 and request a disk operation), I write a wrapper in assembly and put it in memory
<NieDzejkob>you can actually write a very nice, small assembler in Forth
<NieDzejkob>heh, every time I stop to check whether what I've just transcribed works, it works, but when I don't stop to check, it never works
<NieDzejkob>this time I forgot the @ in : dp 0 [ dup @ 2 - ! ] ;
<xentrac>haha oops
<Melg8[m]>so what it would take to this miniforth bootsector to be full quine? so to rebuild itself from assembly in which it was written?
<Melg8[m]>i mean - now you have host system with make/python doing the job. But even if this was not the case. what it would take for booted sector + some files to reassemble itself from this listing https://github.com/NieDzejkob/miniforth/blob/master/boot.s
<NieDzejkob>as is, quite a lot, because I didn't optimize for that and I depend on python
<NieDzejkob>however, if I had a reason to do so, I could probably assemble it by hand hex0-style in an evening
<oriansj>xentrac: stage0-posix effectly has no concurrency. kaem and kaem-optional forks are just waitpid until exit and then look at exit value
<oriansj>NieDzejkob: M1 would be a great deal faster
<oriansj>as you could just create DEFINEs that match the assembly instructions and be done
<oriansj>and all of the relative and absolute addressing would be taken care of for you
<oriansj>(hex0 is effectively saying you never plan on making changes without a *REALLY* good reason to do so
<oriansj>)
<xentrac>oriansj: neat, that makes it sound pretty achievable
<oriansj>xentrac: we achieve the unachievable around here ^_^
<xentrac>does that mean we can't achieve building without a kernel then? D:>
<oriansj>So something xv6 level could certainly be written in M2-Planet or even assembly should someone be willing to put in the time.
<oriansj>xentrac: the knight work is pure bare-metal
<oriansj>stage0-posix is when you have a POSIX kernel you trust
<NieDzejkob>oriansj: or I could rewrite it in a Forth-style assembler :P
<NieDzejkob>ax bx movw-rr, si pop,
<oriansj>NieDzejkob: if you think that will be fun, I encourage you to follow your dreams ^_^
<xentrac>oh speaking of hex, NieDzejkob, a couple of byte-saving tricks I've found useful to avoid the two-byte sub al, "A" - ("0" &~0x20) - 10 and the following four-byte imul
<xentrac>1. use octal
<NieDzejkob>1. no
<NieDzejkob>:P
<xentrac>2. use the hex digits 0123456789jklmno instead of 0123456789abcdef
<NieDzejkob>right now the number base is configurable. what luxury!
<xentrac>it is!
<xentrac>also shl bx, 4 is only 3 bytes instead of 4
<xentrac>my argument for octal is that the i386 instruction encoding is significantly less unreadable in octal
<xentrac>(aside from being less code to implement)
<NieDzejkob>the imul could be 3 bytes too, it uses a 4-byte encoding purely for storing-variables-in-instructions reasons
<xentrac>oh, that's what you meant about keeping yasm from using the 1-byte immediate
<NieDzejkob>I think I expanded upon this much more in my blogpost
<xentrac>I guess either of those two alternatives also saves you the cmp/jbe pair
<xentrac>leaving just a 3-byte and bx, 15
<NieDzejkob>hmm, I don't think there's much use to be had from bytes saved at this point
<xentrac>I'm sure you could find things to do with the luxury of extra space :)
<NieDzejkob>I suppose I could make write-to-disk a builtin with enough bytes
<xentrac>yeah!
<NieDzejkob>I suppose that's what I'll try to do if I mess it up this time too
<xentrac>one thing about bootOS that bugged me: if its files were a whole disk track in size instead of a single sector, it seems like it would be significantly more powerful?
<Melg8[m]>Now... imagine bootstrapping form real source... Next level of it would be - paper manual in which all needed information (maybe like a wiki style) contained to produce bootstrapped system by the real assembler - human programmer.
<xentrac>and that would just be a matter of passing a different AL to BIOS interrupt 0x13 service 2 or 3
<xentrac>I think?
<xentrac>I mean it wouldn't cost any bytes, and only a fraction of a second added on to a roughly-one-second operation
<oriansj>Melg8[m]: compiling/assembling is a job best suited to machines; humans make errors quite frequently. So anything they build should be tiny.
<xentrac>(8 sectors seems like it should be safe)
<NieDzejkob>xentrac: well, the author does have a thing for bootsector-sized programs
<xentrac>he does, quite an impressive one
<Melg8[m]>@oriansj - i imagined this https://www.reddit.com/r/programming/comments/9x15g/programming_thought_experiment_stuck_in_a_room/ experiment, but where "stucked in a room" doesn't have internet or previous knowledge of how to bootstrap anything on machine. How big of a wiki it would require to taught this person how to do it. Even if assumption is that every piece of compilcated source code like linux kernel - is already on the disk and doesn't
<Melg8[m]>require to be manually typed in.
<xentrac>Melg8[m]: aah, that's an interesting problem, isn't it?
<xentrac>NieDzejkob: but 4KiB is enough for a pretty complete programming system, I think. like Apple Integer BASIC or some small Forth IDEs with screen editors and jump-to-definition
<NieDzejkob>xentrac: ideally a recursive wget ran on my blog will be complete one day :P
<xentrac>Melg8[m]: if all the bootstrapping source code is on the disk you only need to walk the person through enough knowledge to write and debug the initial binary seed?
<Melg8[m]>should civilization have manual about how to bootstrap itself?) not like we use this legacy infrastructure to do this new infrastructure - but cleaned up way from knowledge to working systems)
<xentrac>NieDzejkob: ?
<NieDzejkob>ah, meant to tag Melg8[m]
<xentrac>oh, haha
<Melg8[m]>@xentrac that would be nice)
<Melg8[m]>NieDzejkob: "without internet" )
<xentrac>Melg8[m]: you might be able to get it down to 512 bytes or so ;)
<Melg8[m]>why we care about bootstrapping? it's not about most concise form of program representation - because than - just use already built kernel + gcc + full software stack etc. I think part of this is human being can understand and verify what really happens, can create mental model of what pc would do, why it would do it, and generate some tests for that behavior to prove to themselfs that program is not malicious and it's doing what it
<Melg8[m]>"supposed" to do
<oriansj>Melg8[m]: well all of the pieces needed to be known to bootstrap everything could fit in a 300 page book assuming one didn't also have to teach assembly and C programming as well.
<oriansj>For example should one consider a Dr Stone type world, the steps required to essential parity are few but full software parity would take decades (if lucky)
<Melg8[m]>@oriansj oh no... now i need to watch this Dr Stone thing....
<oriansj>hint youtube-dl works on crunchyroll
<oriansj>(for those who are datahoarders too)
<oriansj>But assuming that sort of by scratch, I'd build a knight architecture instead of x86 as although it would be more work initially, it makes less work when one needs to expand the instruction set and less work arounds for bad architectural design decisions. (Plus having a 32bit processor with 16 General purpose registers +PC in less transistors than a 6502 would be cool)
<oriansj>plus FORTH would be much easier to implement as every register can be a stack register (hint push r0 r15; pop r3 r15 => using register 15 as the stack pointer push r0 then pop into r3)
<oriansj>(ret r14 to use register 14 for the return stack)
<xentrac>Melg8[m]: I'm interested in it partly as an avenue to diagnosing and fixing errors and exploring alternatives; if the whole stack is incomprehensible and unmodifiable, then bugs are just things we have to cope with, and we can never rethink the overall architecture
<Melg8[m]>yea... now we will create "the room" series of mental experiments... like (homo sapiens with knowledge of English + The Book + a little bit of sand + clay + some copper ore... lets goo
<xentrac>but if the whole stack is comprehensible and explicable, then we can try alternatives for any piece of it
<oriansj>Melg8[m]: well I have a good book on basic lithography but the creation of the chemicals needed at the required purity would be one heck of a problem to solve. So having a chemist of Senku's skillset would basicly required.
<xentrac>yeah, but Senku's chemistry ability is as unrealistic as Tsukasa's fighting ability
<Melg8[m]>can somebody advise me a book/or site/wiki on x86 assembly language? because to undestand any of this... i need one) so i could play with that ideas myself?
<xentrac>I'm ... probably not the best person to answer that
<oriansj>Melg8[m]: depends if you want a reference or an introduction to assembly programming
<oriansj>if it is just reference Intel publishes x86 manuals which (despite having errors) are a complete reference for x86 instructions, their encoding and how they work.
<Melg8[m]>i know C++/C but don't know anything about assembly - that was under my "abstraction level" for years. But i would like to understand and maybe implement myself some of this bootsector programms
<xentrac>even How To Make Everything skips over a lot of the details you'd really need. the Primitive Technology YT channel and blog I think is a lot better at showing all the necessary information
<xentrac>Melg8[m]: oh, you're going to really like this! assembly is much simpler than C
<xentrac>well, not the entire Intel manual
<xentrac>but there's about 32-64 instructions that cover the vast majority of software
<xentrac>you don't need to understand PSHUFB and CPUID to write compilers or bootstraps
<oriansj>If it is an introduction to assembly programming: links to oscar toledo's books are here: https://github.com/nanochess/bootOS and he does a really good job
<xentrac>there are several thousand instructions you don't need
<oriansj>but if you prefer to learn a simpler form of assembly, knight and stage0 are really quick and easy to learn.
<oriansj>all of the essential x86 assembly instructions can be found here: https://github.com/oriansj/M2libc/blob/main/x86/x86_defs.M1
<NieDzejkob>is Senku a Dr. Stone character?
<oriansj>there is a similar list for armv7l as well https://github.com/oriansj/M2libc/blob/main/armv7l/armv7l_defs.M1 and aarch64 https://github.com/oriansj/M2libc/blob/main/aarch64/aarch64_defs.M1 and a complete instruction list for knight can be found: https://github.com/oriansj/M2libc/blob/main/knight/knight-native_defs.M1
<xentrac>NieDzejkob: yes, Tsukasa is another
<oriansj>NieDzejkob: yes he is the main character who clearly is impossibly bright
<xentrac>right, but everybody in the series is impossibly something
*NieDzejkob has added dr stone to their watch later list a few screenfuls of chat ago
<xentrac>(despite being impossibly bright, he tried casting iron in a wooden mold)
<NieDzejkob>gaah, spoilers :P
<NieDzejkob>(probably a minor one)
<xentrac>yeah, I don't think that's a spoiler. you'll have to tell me after you see it ;)
<oriansj>xentrac: Have you read the Gingery Lathe (and related) book yet?
<oriansj>its bit on sand casting I find quite interesting.
<oriansj>correction it was: Build Your Own Metal Working Shop from Scrap Book 1
<oriansj>Build Your Own Metal Working Shop from Scrap. Charcoal Foundry
<xentrac>I've only paged through it, a few times. the first time, I had no idea what he was talking about
<xentrac>ways? gibs? saddle?
<oriansj>xentrac: it makes much more sense if you had taken shop class.
<xentrac>yeah. or watched hundreds of hours of YouChube videos about machining, as I have since then
<NieDzejkob>must. resist. nerdsnipe.
<oriansj>NieDzejkob: the problem about bootstrapping, it attracts a very specific set of interests.
<xentrac>this is the opposite of specific
<oriansj>xentrac: it is all very specific to the problem of what happens if you don't have what you need and need to make it from limited supplies.
<xentrac>yes, but that's a very general problem. I mean for a broad definition of 'need' everyone is always in that situation
<oriansj>xentrac: the difference is the people here generally have a *NEED* to do something about that potential future.
<NieDzejkob>okay, got to *that point* again. Anyone wanna sanitycheck my new implementation of the int13 wrapper? push si; mov si, packet; mov ax, bx; mov dl, DISK; int 0x13; mov bx, ax; pop si; NEXT
<oriansj>NieDzejkob: and if your nerdsnipe was going to be a good youtube channel to learn machining, throw it my way. I'll add it to my youtube-dl backlog.
<xentrac>in the last half-hour we've discussed boot-sector programming, Japanese children's TV, 1970s minicomputer instruction sets, youtube-dl's breadth of support, Forth implementation techniques, datahoarding, assembly-language tutorials, sandcasting and other forms of metal casting, macro processing, and engine lathe design
<oriansj>xentrac: that seems fine to me
<xentrac>I would say not just fine but excellent
<xentrac>but not specific
<NieDzejkob>day like any other, no?
<oriansj>NieDzejkob: exactly
<oriansj>It is one of the reasons I love everyone here. You people are amazing, fun and interesting
<oriansj>very few people would take even the slightest interest in me writing a C compiler in assembly, let alone doing a 4 compiler speedrun.
<NieDzejkob>read-block didn't crash this time ^_^
<oriansj>Heck, hacker news ignored the x86, AMD64 and AArch64 ports of stage0 posix and only the knight native one got any interest.
<oriansj>NieDzejkob: nice
<NieDzejkob>annnd got it https://mobile.twitter.com/NieDzejkob/status/1403530592105476102
<NieDzejkob>(pure picture link: https://pbs.twimg.com/media/E3pXHJgWUAAYmcR?format=jpg&name=large)
<siraben>Is there an interpreter for the G language?
<siraben>Thinking more about it, a simple language for bootstrapping is interesting
<siraben>especially if it has little undefined behavior
<siraben>oriansj: is there a spec for the subset of C that m2-planet compiles?
<gio>siraben: I am not aware of any G interpreter. Though I would not say that G has little UB: it is just a little bit higher-level than assembly and not portable. Whatever you touch, you're responsible for what you're doing.
***smartin1 is now known as smartin
<oriansj>siraben: I can't say there is a formal spec for M2-Planet but there is a list of primitives that it supports and there are known issues in it relative to the C standard (which with work could be reduced or eliminated)
<oriansj>although I probably need to update it to include the C preprocessor support that yt added: https://bootstrapping.miraheze.org/wiki/Stage0
<oriansj> https://github.com/oriansj/M2-Planet/blob/master/known_issues.org
<oriansj>Aside from the handful of known wrong behavior, M2-Planet is effectively the core of the C language that simply forces people to be honest in what they #include in their outputs
<oriansj>The -D foo=1 option was added to make janneke's life easier for isolating M2-Planet specific behavior in mes.c development
<xentrac>siraben: are you talking about the g language that labview uses
<siraben>xentrac: oh, I was referring to the bootstrapping language
<siraben>oriansj: ah, that's good
<siraben>I plan to go through Verifiable C this summer https://softwarefoundations.cis.upenn.edu/vc-current/toc.html which might be relevant
<xentrac>not familiar
<siraben>xentrac: someone posted a link to the G language earlier
<xentrac>there are three listed on https://en.wikipedia.org/wiki/G_programming_language
<xentrac>is it one of the those
<siraben><Melg8> siraben: here you go https://gitlab.com/giomasce/asmc/-/blob/master/G_LANGUAGE.md
<xentrac>aha thanks
<xentrac>regarding our occasional conversation about how CMOS obsoleted TTL some time ago, I recently found a discarded TI dot-matrix printer from, I think, 01984, with a lot of discrete logic chips in it
<xentrac>two 74LS374s, a 74HC174, a 7406, a 74LS245, a 74LS32, a 74189, a 74LS74, a 74LS373, a 74S138, and so on
<xentrac>the "HC" chip is the same one you'd buy today, and it's CMOS rather than TTL; the "S", "LS", and letterless versions are various TTL families
<xentrac>so the crossover where TTL became useless except for niche applications was around 01984 or 01985
<xentrac>I suspect that for things like dot-matrix printers it happened a little bit later than for general-purpose computers, because computational power was less important, low-power operation was unachievable, high speed was unachievable, and it was necessary to control large amounts of current
<xentrac>I guess computational power is kind of the same thing as high speed in the context of discrete logic ;)
<xentrac>people usually say CMOS has better noise immunity than TTL, but I'm not sure that's true when we're talking about inductively coupled current noise, which I think is maybe a bigger concern in a machine like a printer
<ekaitz>xentrac: afaik the using CMOS is mostly because of its lower power consumption
<xentrac>nope, that was true before 01975
<xentrac>but by 01985 CMOS wasn't just lower-power but also across the board faster
<ekaitz>oh
<ekaitz>I'll investigate a little on that later, thanks for bringing it!
<xentrac>also, as I said, it has better noise immunity. and it doesn't have a fanout limit like TTL, although your CMOS circuits do get gradually slower as you increase fanout
<xentrac>also CMOS tolerates wide variation in power supply voltages, which gives you a lot of flexibility in circuit design, though early CMOS processors like the 8008 (introduced 01972) had a problem where they required more power supply voltages than TTL and so were less flexible instead of more so
<xentrac>but for example this 74HC174 can operate anywhere from 2 V to 6 V, and like TTL it only needs two power rails. but TTL versions like the 74LS174 require their Vcc rail to be in 4.5 to 5.5 V
<oriansj>siraben: the g language discussed previously was gio's custom designed stack language;
<oriansj>it is one of the downsides of single letter programming language names, usually dozens of other people claimed the same letter already and only the most popular version tends to be found.