IRC channel logs

<oriansj>rickmasters: your kernel looks at the path and if it is /dev/hda1 it just dumps straight to disk right?

<oriansj>so I wouldn't need to do gft-create -o foo.img and keep it small enough to fit in memory right?

<oriansj>so I could do something like --disk-size 20GiB and just dump 20GiB of disk blocks straight onto the disk right?

<stikonas>fossy: by the way, why do we have all that link stuff in separate file rather than in tarball?

<stikonas>ok, found another bug with packages...

<stikonas>when we touch symlinks, the destination file is affected

<stikonas>rather than symlink

<stikonas>newer touch has -h option, but not in the old coreutils...

<stikonas>oh, I guess that might be the reason why links are separate... touch was not fixing their timestamps, so tarball packages were not reproducible

<rickmasters>oriansj: the kernel runs the main script on the internalshell, then if the file /dev/hda exists, it is flushed to disk starting at the first sector

<rickmasters>oriansj: so, it does not happen immediately

<oriansj>so it only does the write when the sector is full right?

<rickmasters>oriansj: i believe so, and i have a TODO to fix that, but i have also noticed that it appears to work anyway, not sure why

<oriansj>would a close also cause a write to disk?

<oriansj>that way if I make sure to close at the end, then we can be certain everything is written.

<rickmasters>so, the write to disk is at the very end of builder-hex0.hex0 so take a look, it actually writes the last sector even if not full, so no bug, no TODO

<rickmasters>... just checked

<rickmasters>oriansj: no close does nothing :)

<rickmasters>so i answered your question inaccurately, it doesn't write when you've filled a sector, you can write to a memory file all you want, it flushes at the very end

<rickmasters>oriansj: it flushes right before the kernel terminates / reboots

<rickmasters>to code has the label ok_exit:

<rickmasters>*the code

<oriansj>oh I was asking relative to what you mentioned yesterday about bootstrapping a proper filesystem at the end instead of just flushing it to the disk

<rickmasters>so, the kernel only flushes /dev/hda to the beginning of the disk, partition based writes (/dev/hda{1,2,3,4}) are not implemented, not sure it was needed

<oriansj>I was thinking of when you suggested: catm /dev/hda1 newfilesystem

<rickmasters>so, writes to /dev/hda just creates a memory file with that name, at the end it looks for that file and flushes it

<rickmasters>right, I sent a link that outlined the idea of writing to partitions, but it said it was unimplemented, but its not hard

<rickmasters>i already have a routine to read partition locations from the partition table (i read source from the location of partition 4 in the partition table)

<rickmasters>that's why i kept the partition table in the MBR, so it could be used to read/write different locations

<oriansj>ok, so it'll be a bit before you could have that implemented. Here is what I am doing and planning: gfk-create --disk-size -f file1 -f file2 ... -f fileN -o /dev/hda1 and writing 512bytes at a time the entire partition. with the optional flag --master-boot-record $file to setup everything.

<rickmasters>it wouldn't take long (an hour of so) to support writing /dev/hda{1,2,3,4} (if any of those are written by a process) to the corresponding partition locations

<rickmasters>the partition locations would need to be set in the MBR for that to work

<rickmasters>the MBR could be supported by writing one sector to /dev/hda (which represents the entire disk)

<oriansj>well that isn't an issue as we can use M1 to create a new MBR file and include it in the write to /dev/hda1

<rickmasters>you could set the first partition to start at sector 1 if you want

<rickmasters>(that's not typical I think, but up to you)

<oriansj>I was just going to have 1 partition per disk with the bootloader in the MBR which would support reading GFK and prompt for the filename of the kernel

<oriansj>so that the creating of the Linux image would just be written to a new disk so if anything goes wrong we still have a functional disk to redo things

<rickmasters>so, normally the MBR is not considered to be in a partition, the partition table is in the MBR

<oriansj>yes the MBR is just sector 0 of the disk

<oriansj>with a special 55AA at the end of the block

<rickmasters>there is no sector 0, in CHS disk geometry, sectors are one based

<oriansj>valid point

<oriansj>I was thinking in terms of LBA but ok

<oriansj>CHS to LBA mapping isn't something I tend to keep in my head but i appreciate the correction.

<rickmasters>yeah, if you specify sector 1 for partition 1 you'll write to the MBR location

<rickmasters>but, /dev/hda already writes to the first sector so i'm not sure you need partition support

<oriansj>I don't need partition support

<rickmasters>so i think what you need is already implemented

<oriansj>only the ability to do a write straight to disk in 512 (or 4096byte) blocks

<oriansj>as writing 20GB of data can quickly fill up memory

<rickmasters>ok, not sure if we're connecting, since /dev/hda is kept in memory until the end you won't be able to create that big of a file

<oriansj>I think we are slightly missing what the other is expressing

<rickmasters>but, immediate writes to /dev/hda is doable, i already implement that for reads

<rickmasters>i know how to do that

<oriansj>ok and the writes done to create the binaries are only in memory so the write to disk wouldn't accidentially overwrite the binaries right?

<rickmasters>correct

<rickmasters>you could overwrite unprocessed source files though

<oriansj>true

<rickmasters>you could make sure the source is processed before writing or you could move the source past where you need to write

<oriansj>but we could order the src blocks so that the source files we want to preserve onto our next filesystem would be at the end

<rickmasters>that's easy, just change partition 4 in the partition table and dd to that location in build.sh

<oriansj>ok, good so this plan could work.

<rickmasters>i may need to implement close so I know went to write the last block or you could make sure to write in sector multiples

<rickmasters>some of the stage-posix0 executable don't seem to call close, iirc, so not sure about that

<rickmasters>i.e. catm /dev/hda helloworld

<oriansj>yeah, I'll have to fix those

<rickmasters>they presume exit will close everything, i think, but my kernel does nothing

<oriansj>I'll make sure all programs call close before exit

<oriansj>but M2libc does call __kill_io prior to exit; which probably could be modified to call close afer doing fflush as well

<rickmasters>this would be more than an hour of work, what time frame are you hoping for?

<oriansj>rickmasters: no rush as I am still writing gfk-create but should be done by next week

<rickmasters>oh sure, no problem

<littlebobeep>So can someone explain how building a kernel with M2-Planet works? This is for stage0 bare-metal bootstrap right (not stage0-posix)? So you get up to M2-Planet stage and build a kernel, how do you start using it, do you need to reboot?

*littlebobeep is interested in avoiding a Linux binary dependency to bootstrap

<oriansj>littlebobeep: well rickmasters provided builder-hex0 which is a POSIX capable of running all of the stage0-posix steps

<oriansj>and can be built by a hex0 program fitting in the MBR

<oriansj>So we can use M2-Planet/M2-Mesoplanet to build a POSIX kernel written in C that should be powerful enough to run all the steps needed to build Linux

<rickmasters>littlebobeep: builder-hex0-mini -> builder-hex0 kernel transition is implemented with a reboot, but i've been thinking of redoing that to eliminate the reboot

<littlebobeep>Where can I see what builder-hex0 does?

<oriansj>and yes reboots will happen when transistioning to the C kernel as well

<littlebobeep>oriansj: I do not know what you mean by saying it "is a POSIX", it is a POSIX what?

<oriansj>littlebobeep: https://github.com/ironmeld/builder-hex0

<oriansj>short version the syscalls that stage0-posix needs

<rickmasters>littlebobeep: yeah, the Makefile does it

<oriansj>littlebobeep: POSIX is a kernel standard. In this case builder-hex0 supports the syscalls that stage0-posix needs and just enough extra functionality to get the ball rolling

<rickmasters>if you pass no-reboot to qemu it exits on reboot, then the build script extracts the new kernel from the disk image and dd's the kernel into a new image

<littlebobeep>oriansj: so how did stage0 bare metal work WITHOUT builder-hex0?

<rickmasters>well, anyway its a combination of the build*.sh scripts and the Makefile, you'll probably need to study it a bit for it to click

<oriansj>littlebobeep: by reading and writing to external storage (punched paper tapes honestly)

<oriansj>and a boatload of reboots

<littlebobeep>oriansj: wow so stage0 cannot use HDDs or SSDs or what, I never fully understood it compared to stage0-posix

<oriansj>littlebobeep: it could in theory, I just haven't done so yet.

<oriansj>the major point is that all of the steps could be implemented with or without a kernel and will all produce the exact same results

<littlebobeep>oriansj: Ahhh you mean some random Linux binary compiled on a non-properly bootstrapped system with stage0-posix produces verifiably same results as all-source bootstrap from stage0? You would need to do both to confirm this

<oriansj>littlebobeep: well yes there is still a great deal of work remaining to audit buil

<oriansj>der-hex0

<oriansj>but I am in good faith assuming it behaves as specified and if I am wroong we just fix it using what we hzave learned from it

<oriansj>^hzave^have^

<rickmasters>orianjs: i want to improve the comments and write more up about it. auditing it good

<rickmasters>*oriansj

<oriansj>I encourage good comments

<littlebobeep>oriansj: So did you actually use paper tapes with stage0?

<muurkha>inb4 "no"

<stikonas>rickmasters: for audit it would be good to have higher level prototypes where possible

<stikonas>but that can be done later

<stikonas>if you don't have them

<stikonas>you probably saw in stage0-posix we have C, Gas, M1, hex2 prototypes. Although, maybe not everything is applicable here (i.e. C)

<rickmasters>it's native hex0. one thing that's good is that its hard to hide much in 3K and also pull off a posix kernel

<rickmasters>:)

<littlebobeep>So will rickmasters' code become part of stage0?

<stikonas>it's a separate repository

<stikonas>I guess it could be used instead of stage0

<stikonas>stage0 was doing those tapes...

<stikonas>which you can't easily do with x86 hardware

<rickmasters>anyone can copy or fork it wherever they'd like, happy to help

<littlebobeep>I don't know how to use tapes or punch cards haha but oriansj is talking about them like they used them....?

<stikonas>well, you punch holes (bits in tape), feed it into machine

<stikonas>that's stdin

<stikonas>and machine punches holes in stdout tape

<muurkha>littlebobeep: the great advantage of punched tape or punch cards is that you can see your digital data

<rickmasters>funny, i actually took a class in early 80s and punched cards w/FORTRAN code, although i don't even remember what we did with them

<muurkha>it isn't mediated by a computer which could conceivably be lying to you

<muurkha>there is a ppt command in bsdgames (the Debian/Ubuntu name of the package) which will draw you ASCII art of a punched paper tape. echo 'hello littlebobeep' | ppt

<muurkha>(the output is too voluminous to paste here

<muurkha>)

<muurkha>the package also contains a morse command to produce morse code and a bcd command to produce punched cards

<muurkha>punched cards are kind of a shitty medium

<rickmasters>goodnight everyone

<muurkha>night!

<dongcarl>oriansj: Oh? What's the stance on bribes? Crypto people love to talk game theoretical attacks, out of band ones even more so haha

<rickmasters>stikonas: I've identified some reasons why mes-m2 does not work on builder-hex0.

<rickmasters>stikonas: 1. As we already determined, the disk image would need to be increased to support the source.

<rickmasters>stikonas: 2. It appears that giving qemu more memory is required, but I am not sure what is sufficient.

<rickmasters>stikonas: 3. The hex2 --base-address for mes-m2 is different than stage0-posix. Easily changed.

<rickmasters>stikonas: 4. I had mistakenly left the kernel file memory in a location that conflicts with the mes-m2 process. Fixed.

<rickmasters>stikonas: 5. There are file links in the mes tar file. The untar command from stage0-posix ignores them.

<rickmasters>stikonas: 6. mes-m2 uses access() to look for modules. builder-hex0 returns success, but the file is not there.

<rickmasters>stikonas: Making access work properly could be done, but it would take significant work and I think its more

<rickmasters>stikonas: appropriate to direct efforts toward a new kernel written in a better language.

<rickmasters>stikonas: 7. Evaluating any (display "string") command (for debugging) crashes mes-m2. Unresolved.

<rickmasters>stikonas: There are probably many more issues yet to be discovered that would block running mescc.

<rickmasters>stikonas: Finally, I also noticed that there is an issue with environmental vars - only 32 are passed to mes-m2.

<rickmasters>stikonas: This may be a kaem limit/bug - I am not sure. But my linux system has so many variables, mes-m2 does not

<rickmasters>stikonas: get any of the ones you put in the after.kaem file. Turns out, they are apparently not needed

<rickmasters>stikonas: but this is an issue that should porbably be tracked down further.

<rickmasters>stikonas: Figuring all this out was a long slog and I'm going to have to move on to other priorities. :)

<stikonas[m]>Yeah, looks like too many problems

<stikonas[m]>Some are easy but others are indeed not trivial to fix

<rickmasters>its not trivial in hex0. even hex-2 would be a lot easier

<stikonas[m]>Yes, I fully agree

<stikonas[m]>Jump calculations are painful

<rickmasters>there are a lot of viable paths. i think oriansj is planning on writing a kernel... ?

<stikonas[m]>Yes, kernel in M2 dialect makes a lot if sense

<rickmasters>i'm happy to just improve builder-hex0 comments and documentation, or port to hex2? that's fairly easy, but anything further might overlap with someone else

<stikonas>well, just having better comments and documentation would be fine. But I'll let oriansj comment on this as he is more involved in this work

<rickmasters>by the way, i don't know the history of the scheme detour, but has anyone explored going from M2 to more capable C dialects then tcc, etc?

<unmatched-paren>rickmasters: i don't understand why we detour to scheme either, but i don't think there is any other more capable C that m2-planet can build

<unmatched-paren>***YET*** :)

<unmatched-paren>someone was looking into cproc (achaninja, i believe)

<unmatched-paren>but they built it from tcc

<unmatched-paren>if we extended m2 enough we could probably go straight to either tcc or cproc (cproc would probably be better)

<unmatched-paren>since it supports riscv64

<stikonas>rickmasters: mes and mescc predates M2-Planet

<stikonas>and tcc is even harder to build than mes

<stikonas>and it took some time to build mes with M2-Planet

<stikonas>and M2-Planet is somewhat simplistic compiler which originated from high level prototype of cc_x86 (that was written in assembly)

<stikonas>so some more advanced C features might be harder to add to M2-Planet

<stikonas>unmatched-paren: as for riscv64, it's possible to add it to mes

<stikonas>at least mes-m2 is easy to build for riscv64,

<stikonas>it's mescc that is a bit harder to port

<rickmasters>stikonas: well, m2 -> mes.c + mescc works, so there's that.

<rickmasters>stikonas: but just perusing mes-m2 code ... its difficult to approach, the C is terse and it requires deep scheme understanding

<stikonas>yes, that's why I'm not so familiar with mes, I am not good at scheme

<stikonas>I can tweak some stuff in mes libc (which is in C) but not really in mescc

<rickmasters>the intersection there is limits the audience to only the most dedicated and patient developers (in the world)

<rickmasters>not 70%, its like .1% tops

<rickmasters>not saying the C path is a walk in the park either, the tcc code is also difficult

<rickmasters>these are just my first impressions and I haven't explored all the options, sorry if this sounds critical

<rickmasters>i can see the other side of the argument - a scheme interpreter can be small and scheme builds itself up to real power from a small core

<stikonas>well, 70% probaly refered to stage0-posix only...

<stikonas>anyway, scheme is there historically

<stikonas>maybe at som point there will be parallel bootstrap paths

<stikonas>but right now that's the only thing we have

<stikonas>and it's better than nothing

<stikonas>2 years ago we had no path at all

<stikonas>and 2 weeks ago we had no kernel

<rickmasters>yes, a working solution is infinitely better than no solution

<rickmasters>i'm sure these kind of "why don't you just ..." questions have been asked many times, sorry, i was just trying to draw out some opinions. thank you for being patient with me

<Hagfish>they're great questions, and potentially educational for others of us who weren't aware of the answers

<stikonas>yeah, nobody is offended here, questions are encouraged

<stikonas>I did ask quite a few questions a couple of years ago too

IRC channel logs

2022-05-16.log