IRC channel logs
2022-12-24.log
back to list of logs
<pabs3>bunnie would be *the* person to link up with around hardware stuff <oriansj>well bunnie is more about shifting the industry in the correct direction and using products that everyone has access to. Which is a step in the right direction. <stikonas>but now we have support for structs on stack and M2-Planet notices that we are passing 64-bit value on x86 as function argument <stikonas>ok, I think that's a bug in mes, we need to define pointers there <stikonas>I'll create a patch, but update will be blocked on new mes... <Hagfish_>wow stikonas, you're finding all these intricate details (well... bugs), but that's actually reassuring that the code is staying understood (and understandable) as it grows, rather than taking on too much technical debt <stikonas>Hagfish_: it's not me that found those but M2-Planet... <stikonas>well, I added that additional check to it... <oriansj>best to put sanity checking in code and let the software do the hard work for us <Hagfish_>yeah, that has a nice multiplicative effect. the more software, the more checks :) <oriansj>hence the obsessive amount of require statements in everything I do <stikonas>I was scared that I messed something up in M2-Planet... <oriansj>stikonas: messing things up is a great way to learn and make things better ^_^ <oriansj>also if worried about breaking functionality, add tests that cover the functionality you need to keep working <Hagfish_>pder: i love the unbz2.c, but it does make me wonder if there should be a unit test for it (and maybe a check of code coverage). forgive me if that's a cheeky question, because i don't want to look a gift horse in the mouth, and i'm really impressed by your contribution! <oriansj>well, I am going to start fuzzing it and will probably be reporting segfaults for pder to fix <oriansj>(everything after (and including) M2-Planet gets fuzzed to reduce the number of user facing bugs) <Hagfish_>does fuzzing also give you a coverage report? <Hagfish_>i think there's just something in the back of my mind which bugs me about unreachable code, even though that's the least likely code to be a problem :D <Hagfish_>maybe i'm more worried about code that only gets exercised under weird edge cases. that's probably more likely to cause real problems <stikonas>Hagfish_: no, I think fuzzing can't do that <oriansj>Hagfish_: or uses the source code to speed up hitting code paths <stikonas>and if you get segfault, it will be in accessible sections <Hagfish_>i guess that adding coverage checking to a fuzzer is as easy/hard as adding it to a test suite <oriansj>and segfaults found within 3 seconds <oriansj>shouldn't take too long for pder to find the root and fix <fossy>rickmasters: that's great (re: fiwix changes)! <rickmasters>fossy: It's going to take more than a day to get the commits in but I'm working on it. <fossy>rickmasters: no rush, i would just love to see it at some point :) <oriansj>rickmasters: thank you for doing just such a great job ^_^ and we love seeing your progress <stikonas>rickmasters: yes, no rush, it will take some time to integrate unbz2... We need janneke to merge my fix. And then we need to wait till the next mes is released... Well, maybe we can patch it manually with "replace" program <Hagfish_>i don't think anyone put those commits on their Christmas list to Santa, and anyway, i trust rickmasters to produce them more than i'd trust Santa's elves, so i'm sure the commits are worth waiting for :) <stikonas>argh, another issue will be that M2-Planet now outputs new style defines... <stikonas>and mes needs to be updated for that too <Hagfish_>i hope the problem isn't looking fractal, where each change triggers at least one more change somewhere else <rickmasters>I've been in discussions with the Jordi, the author of Fiwix to push directly upstream. <stikonas>we just need to add both sets of defines to mes for transition <stikonas>rickmasters: oh, that might speed things up <Hagfish_>is there a policy/precedent for how long those transitions last? <Hagfish_>i think i remember an example before of keeping backwards compatibility for one version, or something, but i might be misremembering <Hagfish_>thanks for the links! i love reading those <fossy>precedent/policy can be rewritten as best fits the situation <fossy>yes, typically 0.xx versions do not need to be backwards compatible at all (breaking changes can be introduced at any time) (with semantic versioning) <stikonas>anyway, I think I have managed to patch mes to work with new M2-Planet <oriansj>and having hundreds of different copies of M1 DEFINEs isn't a problem at all <stikonas>I'll probably do it tomorrow as I'll go to sleep and no time to check checksums <oriansj>(assuming you don't define something more than once) <stikonas>medium term we should add new defines and fix mes <stikonas>longer term we can remove old defines from mes <fossy>Hagfish_: i just saw that too. <fossy>"Newlib is a C library intended for use on embedded systems" - for live-bootstrap i -seriously- doubt that it would be viable. since ultimately we are building up a full linux system, even using an alternative to glibc such as musl has a wide variety of patches required, i have a feeling the level of patching would be quite large for next to no gain - quite possibly it wouldn't have many <fossy>functions/headers since it appears to be targeted to embedded systems <stikonas>and also musl has very simple build system <stikonas>fossy: maybe you can reply there why we are using musl <fossy>i was just about to do that, stikonas :P <oriansj>Hagfish: I've looked at the Newlib C but ultimately it is fossy and stikonas who get to decide what libc live-bootstrap; and I trust their judgement on this matter is sound. <stikonas[m]>I looked at a few C libraries and musl just seemed easiest <rickmasters>I was hoping Jordi would join this chat for a discussion but I think he it might be 3 in the morning where he is... <fossy>for better or worse, there are reasons 99% of distros use either musl or glibc <fossy>(everything else just doesn't have the spread of features applications use, and developers don't care about alt-libcs) <oriansj>that being said, anyone is free to check to see if any libc they like is a better option and provide patches <oriansj>but I'd bet money that fossy and stikonas choose based on what we could build and would work for the tools we need to build. <stikonas[m]>rickmasters: that's actually further west than me... Though I'm in a later timezone, it's 2 an here rather than 3 <stikonas[m]>So if we ban pregen files, we need to write our own makefile <stikonas[m]>And newlibc actually uses same autogen template stuff as gcc <oriansj>but ultimately it is a question is it worth the effort and who is willing to do that work <stikonas>janneke: I've just sent you another mes/m2 patch to add new defines. <stikonas>pder: I've now pushed live-bootstrap change with unbz2 (but still need to make changes to take advantage of it) <stikonas>pabs3: that muon is interesting. At the moment we didn't really need to use anything with meson (and fossy is almost done with python, so presumably full meson could be build too), but still good to have an option <stikonas>(or muon might become very useful if meson decides to require meson to build itself) <sam_>meson is quite sensitive to those issues fwiw <sam_>i doubt cmake will ever have another impl <stikonas>yes, but hopefully cmake won't remove its bootstrap.sh script either <stikonas>but I think they understand that they should keep it too <oriansj>well it would be quite embarrassing for a build system to become unbuildable. <oriansj>let us hope shame keeps them preserving that build script; otherwise we will once again have to create another build chain in guix to keep it working. <janneke>stikonas: does this works with m2-planet 1.9 and 1.10 too? <stikonas>janneke: defines definitely should work, they'll be just unused <stikonas>and I can't see why the other change wouldn't work <stikonas>and I don't see any more compatibility issues with newer M2-Planet. These two patches on top of 0.24.1 are sufficient <stikonas>janneke: this is basically caused by M2-Planet (from git) now supporting structs on stack and global structs (rather than just struct pointers that you can calloc) <stikonas>(oh and also M2-Planet had learned a bit about (u)int*_t types, so it's a bit more careful with type sizes in assignments <stikonas>fossy: there is a crash in checksum-transcriber if file does not enf with newline <pder>stikonas: great, thank you. <pder>oriansj: I got that fuzzing out.tar.xz but I don't know what do to with it. Do I need the examples folder as well? <pder>I also tried ~/mescc-tools-seed/x86/bin/unbz2 -f id\:000000\,sig\:11\,src\:000000\,time\:5027\,op\:flip1\,pos\:36 -o output.txt <stikonas>pder (cc rickmasters): it will take some time to finish as I need to run live-bootstrap many times but I have now moved tcc 0.9.27 immediately after tcc 0.9.26 and fixing up further things (so far just fixed make) <stikonas>pder: have you tried using those options from that readme file? <pder>no, only because I dont have afl-fuzz installed <pder>And I dont know where the examples directory comes from in that command line <oriansj>pder: the examples directory is just a bzip2 compressed copy of the unbz2.c source file <oriansj>The segfaults will be noticed if you use a GCC build (as M2libc + M2-Planet don't do any sanity checking at runtime) <duplexsystem_>what would need to be changed in boot2now to make a disk image that goes through each previous stage when booted instead of using a binary of the previous stage. I'm struggling to fiqure out how such piping would work <stikonas>duplexsystem_: there was a discussion here a few days ago <stikonas>if you look at IRC logs, you'll see rickmasters describing it <stikonas>basically we need to create ext2 image of next rootfs (so we need to build some program that understands ext2) and then we can launch fiwix kernel <stikonas>fiwix to linux probably needs something like kexec <rickmasters>duplexsystem_: honestly, it wouldn't be easy with the state of boot2now right now <stikonas>rickmasters: by the way, with this live-bootstrap rework that I'm doing now, I don't see the need for using sed to patch "patch" anymore... <stikonas>not sure when it was fixed but patch seems to build and run <stikonas>well, if it stops working on some more complicated patch, I would rather build unpatched patch and then rebuild it. But hopefully it won't be necessary <rickmasters>duplexsystem_: basically you'd need to merge all the scripts that gather the source and you'd need to merge all the compiler steps <stikonas>(right now I have already reached actual bzip2 stage, so should be fairly close now <rickmasters>duplexsystem_: I'm planning on creating an all-in-one build so I'm not sure it would be worth your time <duplexsystem_>well if your merging boot2now into live-bootstrap I can wait <stikonas>yes, ideally live-bootstrap should have this option of building without seed kernel (or rather boot2now) <stikonas>well, at first we can start by adding fiwix as an option <rickmasters>duplexsystem_: merging stages in boot2now and integrating into live-bootstrap are separate work items but yeah, both are planned <stikonas>well, those transitions are not easy, took as a while to even sort out linux->linux transition <rickmasters>duplexsystem_: one thing to note is that I don't have a plan to merge builder-hex0-mini and builder-hex0 <duplexsystem_>my current issue with live-bootstrap is it's use of gcc and friends. nothing against gcc but my resulting system uses clang and I'd like to not include gcc tarballs on disk. I've found a way to get to clang without gcc. I like boot2live because it drops off right where I diverge with live-bootstrap. <stikonas>so one would have to use builder-hex0-mini and then reboot with the new "disk"? <rickmasters>duplexsystem: builder-hex0-mini can only produce a boot image, so yes <stikonas>well, yes, right now boot2now stops at tcc <stikonas>but how will you build clang? what are the main steps? <rickmasters>duplexsystem_: boot2now has a kernel bootstrap - that's the main distinction, not where it stops in my opinion <stikonas>often where it stops is determined by the amount of free time that we have... <rickmasters>you can stop live-bootstrap by just editing sysa/run.sh sysa/run.kaem <rickmasters>or diverge into a new path by just changing the steps in sysa/run.kaem or sysa/run.sh <duplexsystem_>stikonas: from tcc the process is this: tcc -> tendra (ironically also abrevated tcc) -> clang 3.3 -> clang XYZ <rickmasters>boot2now is just running the same code from stage0-posix and live-bootstrap but packaged differently for a different kernel. <duplexsystem_>obviously need to sprinkle in some other tools here and there but that's the general idea <stikonas>never head of tendra, but looking at it, it might work, even has a simple makefile for build system <muurkha>I haven't heard of TenDRA in a long time <stikonas>yeah, you'll need to do some of the same stuff anyway, i.e. bootstrapping bison, build bash, etc... <duplexsystem_>it's the only oss c++ compier I could find that seems simple enough for tcc to compile <rickmasters>duplexsystem_: just to reiterate, unless you really need the kernel bootstrap live-bootstrap probably makes more sense <stikonas>duplexsystem_: well, try to write a build chain with it, it migth very well work <stikonas>yeah, just fork live-bootstrap and start building alternative chain... <rickmasters>you'll get a lot further with a real kernel, like make and musl <duplexsystem_>rickmasters: cool cool. my end goal is to have kernel boot straping but yeah I'll start from the top down. maybe by the time I finish more kernel bootstraping stuff will be finished <stikonas>well, a lot of kernel bootstrapping stuff is somewhat orthogonal <stikonas>live-bootstrap does try to use fewer syscalls at the beginning <duplexsystem_>also once kernel bootstraping is "done" someone needs to voulenteer for BIOS bootstraping. I vote we use punchcards for this :P <stikonas>hmm, I'm surprised that some judges didn't know what orthogonal is... At least in non-technical way. I probably wouldn't expect judges to be able to define it using scalar product <stikonas>duplexsystem_: well, oriansj's stage0 uses tapes, soe somewhat similar <stikonas>and especially on BIOS, we don't use that many of its features, we need input and output and a few other things, but it should be possible to replace those functions with hardware drivers <stikonas>though that will be hardware specific, it won't work on all systems of that instruction set <stikonas>stage0-uefi relies a bit more on UEFI though (in particular on FAT file system stuff) <oriansj>duplexsystem_: well BIOS/Firmware bootstrapping is certainly an area that is still very early and looking for people interested in the work. <oriansj>And it'll be the least portable work we will ever do <duplexsystem_>oh I was mostly joking but that does sound very intresting. I have a decent understanding of x86 firmware but I'm not that confident. <duplexsystem_>but if there is work being done on that I'd be intresting in helping :) <duplexsystem>I just lost internet for a bit. Anyone say anything of importance since my last message? I'd check the logs but they seem to be down/not working for me <duplexsystem>Huh I was getting an error saying to couldn't find the resource when I tried <oriansj> duplexsystem: well, we don't need perfection but a reasonable approximation good enough to get the ball moving forward. <oriansj>you also get to pick the hardware you want to work on or with <duplexsystem>How would you even begin to bootstrap BIOS other than manually flashing it. Or is the plan to manually flash it? <oriansj>duplexsystem: well depends heavily on the hardware you select. <oriansj>punched paper tape reader to EEPROM writer sounds like fun honestly <duplexsystem>Is there any x86 system that supports flashing "untrusted" firmware that isn't a after market firmware mod like coreboot <oriansj>duplexsystem: you don't have to use x86 systems to make firmware for x86 systems <duplexsystem>How so? Would you need to flash the firmware on the device your going to use? <oriansj>I was planning on making knight in hardware and use that to create punched paper tapes for other hardware platforms <oriansj>it on power on reads from tape drive 0 until end of tape, writing contents into memory from address zero up <oriansj>it is read only and I just need to punch the most minimal hex0 editor by hand <oriansj>so the pieces I need figured out include firmware for x86, AArch64, armv7l, riscv-32, riscv-64, etc and a paper tape to EEPROM programmer schematic <oriansj>so, assume a text editor and write in standard assembly (we can convert to M1 and hex0 later) <oriansj>and I have already written the hex0 needed for knight <rillian>oriansj: nice. I wasn't sure if the paper tape references in the repo were metaphorical or not. :) <rillian>I've seen paper tape reader projects. Are there are any punch designs available? <oriansj>rillian: only Cthulhu was a metaphorical; the read is deadpan serious. <stikonas>ok, so I've manged to get rid of the first tcc patch that enables static linking (by addign LDFLAGS=-static). the second tcc patch is actually needed to build musl, so I guess I'll have to rebuild patch tcc 0.9.27 before musl