***Server sets mode: +cnt
<stikonas>mes 0.24.1 is now pushed into live-bootstrap <rickmasters>I've been silent for over a month so I thought I'd provide a status update. <rickmasters>I'm not finding a simple path forward to a new kernel that can be built inside live-bootstrap with tcc on the builder-hex0 kernel. <rickmasters>As a reminder, live-bootstap with builder-hex0 cannot proceed past tcc-0.9.26 right now because it cannot run make, <rickmasters>which reads directories and uses pipes, both of which are not implemented in the builder-hex0 kernel. <rickmasters>Pipes are difficult because it implies multiprocessing but builder-hex0 can only run one process at a time right now. <rickmasters>I think the UEFI path will eventually run into the same challenge. <rickmasters>UEFI provides a file system, which is huge, but I don't think UEFI supports multiprocessing with interprocess communication <rickmasters>(for pipes) out of the box. I found some slides indicating it has been done in some way (by Intel), <rickmasters>but I am not confident that there is a usable open source solution available. <rickmasters>I have spent weeks trying to build a linux kernel using tccboot as a guide but progress has been slow <rickmasters>and the results are sketchy. It is difficult because the original project from Bellard is not really open source <rickmasters>and the only viable fork, seyko2, (mysteriously) required adding hacks to seyko2's tcc fork in order to build <rickmasters>I was finally able to build linux 2.4 without errors but I am still working on trying to boot it to make sure it actually works. <rickmasters>The kernel is too old for qemu to load directly so I have to create a bootable image which is difficult and requires many tools. <rickmasters>Performing a kexec of linux 2.4 from live-bootstrap will be another major challenge because of various differences with modern linux. <rickmasters>I have explored building other kernels as well but they are not very promising for different reasons such as <rickmasters>incompatibilites, limited functionality (xv6) or compiler requirements (tilck). <rickmasters>At some point I may decide to abandon linux 2.4 and port builder-hex0 to C instead. <rickmasters>I could then enhance my primitive file system and perhaps make pipes work. A C based kernel might provide <rickmasters>a way forward for other architectures including x86_64 UEFI. <rickmasters>However, instead of making pipes work we could try to remove pipes from live-bootstrap until linux 4.x has been <rickmasters>built and booted. I don't think that would be easy, but it would probably help the UEFI bootstrap path <rickmasters>get to linux as well (perhaps without an intermediate kernel). <rickmasters>So that's my status. Questions or comments are welcome. <Hagfish>rickmast_: fantastic update, thank you for sharing! <Hagfish>it really feels like a jigsaw puzzle or a sudoku... <Hagfish>all these different constraints pulling against each other <Hagfish>my instinct (or perhaps uninformed wishful thinking) is that someone out there has a missing piece, like a kernel written in C which just happens to support pipes and everything else we need <Hagfish>miracles like that are probably quite rare, and we maybe used up more than our fair share of miracles just getting the kernel work you've already contributed <rickmast_>HagFish: yeah, it would have been nice to just drop in a new kernel. "tilck" was promising but it relied on C11 atomics and other extensions that tcc does not have <rickmast_>but there is a lot of code I might be able to leverage <stikonas[m]>Yeah, I don't expect UEFI work to go easy after stage0 <stikonas[m]>It's not finished but might be quicker that starting from scratch <rickmasters>but it just implements syscalls exit and "test" for debug output <rickmasters>it would be a lot of work, but the skeleton looks good <Hagfish>GPL3 wouldn't be a practical problem, would it? are there other projects with incompatible licences that you'd want to borrow from or link to? <muurkha>I don't think it would be a practical problem <muurkha>but some people don't like it I guess <stikonas[m]>though reaching tcc in UEFI bootstrap will be challenging too... <stikonas[m]>and if we can then have a common kernel (i.e. also compatible with boot2now) then it would be a nice convergence point <rickmasters>if I'm going to write a lot of code, I'll want it to offer my preferred licenses. I respect the rationale of GPL3; its just not for me <rickmasters>I'm torn between making linux 2.4 work, adopting a simpler code base, and writing the code myself right now. <Hagfish>i can't think of any heuristics for making such a decision any easier, and i can't think of any comparable situation i've faced <Hagfish>but maybe someone else here will have had a relevant experience <rickmasters>everyday I poke at the linux black box I think, "I could have written a lot of code by now..." <Hagfish>yeah, that must be a little disheartening <Hagfish>of course, productivity isn't just how much code gets written <Hagfish>the world would be a better place if some code had never been written at all :) <rickmasters>i was able to start a port of builder-hex0 to tcc. I converted some hex to GAS assembly which worked and it could link with C code but I haven't taken it further... <rickmasters>So, that path is viable, I'm pretty sure but I'd prefer to use existing code. Just hoping for a miracle TCC compatible posix/linux compatible kernel to drop in my lap <rickmasters>... with support for multiprocessing, pipes, a file system, etc <oriansj>rickmasters: I like the builder-hex0 written in C and slowly expanded to cover our bootstrapping needs <oriansj>that way as new issues show up; you'll be familiar enough with the kernel to be able to make those changes <oriansj>but the real question is does it sound like fun to you? <rickmasters>probably, but it feels like "Not Invented Here" syndrome <oriansj>inefficencies that are the cost of progress in the desired direction are kind of expected. <rickmasters>If I could drop in a different solution like linux, I'm cool with that and moving on with hopefully time saved, but that dream may be ending.... <oriansj>well most kernel developers don't dream of making their kernel easy to bootstrap <rickmasters>What's tough is that I don't really know how close I am. I built linux with a strange fork of tcc and it might be just about there or it might be a pile of miscompiled crap <oriansj>rickmasters: ok, what if someone else got the builder-hex0 in C started for you? <rickmasters>oriansj: well, that's the fun route, I'd feel jealous :) <oriansj>that way we can experiment on that until we figure out what absolutely is needed to bootstrap Linux <oriansj>and we can either do a build it step or just backport a few tweaks to your kernel written in hex0 and have our success as cheaply as possible <oriansj>with the added bonus of having a C kernel as a 1:1 map to your kernel in hex0 so that other people could understand it more quickly and possibly port to new architectures <rickmasters>the file system needs to be enhanced to support reading a directory and it needs multiprocessing to support pipes <oriansj>well pipes can be done as just a file with separate pointers for the reader and the writer <rickmasters>its the multiprocessing that is tricky - timer ticks and task switching <oriansj>hence do it in C first and make it as simple as possible <oriansj>if it seems hard, prototype in a high level language first <rickmasters>I would port most of the OS to C so that other arches could leverage it <rickmasters>tcc supports C with embedded assembly and even byte sequences (for 16 bit code) so it would be easy to port progressively <rickmasters>oriansj: I appreciate this conversation. I was expecting you to prefer an existing kernel <rickmasters>I've got one more thing to try with linux and then I'll probably change course if that doesn't work... <Hagfish>let us know how you get on. it's always great learning about what you find out and produce <aggi>linux-2.4 x86(64?) didn't boot with qemu? <rickmasters>aggi: tcc produces an uncompressed kernel without the proper boot protocol information qemu is looking for with the -kernel option <aggi>ok. i am looking forward to boot-loading this with isolinux, on real hardware, and hoping for rudimentary support for IDE, generic USB1/2, and VESA graphics <aggi>rickmasters: did you pick the latest v2.4 series kernel version? <rickmasters>at least the way tccboot produces the kernel using raw output. If I build an ELF executable it doesn't have the "PVH" entries qemu wants <muurkha>rickmasters: can you just implement pipes with temp files like MS-DOS did? <rickmasters>aggi: I am working on the isolinux method to boot the resulting kernel, but it's throwaway work because we'll probably use a kexec method eventually <rickmasters>muurkha: the pipes are created by a shell which then forks multiple children with different input and output descriptors - my kernel can't handle that <rickmasters>so we either rewrite the shell or we rewrite the scripts to replace pipes with temp files but consider autoconf generates these scripts so its a rabbit hole <stikonas>so rewriting the shell or make might be easier than writing a new kernel <stikonas>as for rewriting shell, I can't tell but maybe gash would work on single process? <stikonas>and we know that gash can run autoconf scripts <stikonas>hmm, I wonder whether we can reach GCC if we have make that is single process and shell that have pipes workarounded... <stikonas>aggi: what was the name of that make alternative? <rickmasters>stikonas: I think that is worth exploring. the complexity of the shell code worries me <stikonas>but writing custom shell might be way easier than writing custome kernel <stikonas>but there might be other problems in bootstrap chain <muurkha>rickmasters: I was thinking that the shell could maybe run the writer, wait for it to finish, and then run the reader <muurkha>maybe topologically sorting them if the topology is complicated <rickmasters>I've think I could simplify make to work single process, no pipes. it's a bit of a mess but I think I could make it work <muurkha>the case where you can't do that is when (a) your disk is too small for the tempfile or (b) the topology is not acyclic <rickmasters>I think make supports DOS and amiga so it can work in a simplified way <stikonas>let me try to search bootstrappable logs for that other make <muurkha>case (b) isn't guaranteed to work even with real multitasking unless the programs are very careful about how much data they write at a time or use select() <rickmasters>the configure scripts have a lot of echo | sed or echo | grep pipelines. there may be some complicated pipelines but the ones I've seen are pretty simple <rickmasters>BTW, make uses pipes for internal signaling to track jobs I think. again, I think it can work without pipes <muurkha>MS-DOS was pretty slow at this IIRC but I think that's mostly because it didn't have a disk cache <muurkha>I think it uses them between cc1 and gas <sam_>otherwise it dumps to temporary files <muurkha>sam_: actually I think you're right but I had forgotten that <stikonas>tar uses pipes but easy to workaround that too <stikonas>which we actually do early in live-bootstrap <stikonas>since pipes in early builds using meslibc are broken <stikonas>(might be bzip2 only, but there were some issues) <muurkha>stikonas: that sounds like implementing pipes as tempfiles would work fine for tar? <stikonas>bzip2 -d archite.tar.bz2 followed by tar xvf ... <muurkha>I mean I guess you can't spawn the gzip process from the same tar process that is going to write to it, and then have that process wait for gzip to exit <stikonas>rickmasters: so pdpmake does not seems to use pipe <stikonas>might be worth investigating if it can be drop in replacement in live-bootstrap <stikonas>and it's probably easier to build than gnu make <stikonas>though we should check if meslibc supports it <stikonas>but even if not, should be easy to extend <stikonas>though it might not be able to get us past autotools <rickmasters>ok, well, this discussion has me leaning towards porting builder-hex0 to C and seeing how far we can get without pipes. <stikonas>yes, C version (with inline asm of course) would be useful in any case <stikonas>rickmasters: is it hard to rebase boot2now on top of newer live-bootstrap? <stikonas>rickmasters: well, I was meaning to take a look into pdpmake anyway even before this discussion <stikonas>it shoud be much easier to build it than gnu make <stikonas>so one fewer package with complex kaem script <stikonas>gnu make needs various -D flags passed to build it <rickmasters>stikonas: I've updated live-bootstrap in boot2now before and it wasn't that bad. <stikonas>hopefully nothing serious this time either <stikonas>this time there is some early stuff that has changed (mes bump) but hopefully that wouldn't introduce any issues <rickmasters>aggi: the version of linux is 2.4.26 which is the one that Bellard provided patches for in tccboot <oriansj>rickmasters: also we don't know if the path to GCC might be easier than previously expected and manybe of a few months of your brilliant work and we might be able just do a build on a Modern kernel and reboot <oriansj>worst case is you have some good fun, we all learn something new and then we try something else <stikonas>aggi: anyway, it does not build as is with meslibc <stikonas>input.c:5: error: include file 'glob.h' not found <aggi>didn't have time yet to decide and continue with anything <aggi>moving to oasis-linux system integration without GNU-buildsystem (autotools) is a huge task <aggi>and i don't want to continue any system integration work with gentoo-tooling anymore (because python, perl and other issues related to bootstrapping) <stikonas>so GNU Make ships it's own implementation of glob... <aggi>with both linux-2.4/tcc and oasis-linux i fear a hard vendor-lock against X86(64) which i want to avoid <aggi>furthermore oasis-linux is heading towards cproc/qbe AND binutils as toolchain <aggi>if the GNU make version mentioned by stikonas can easily be compiled/linked with tcc this may be good enough, and has the benefit GNU make supports parallel make jobs <aggi>anyway, moving towards oasis-linux together with tcc-toolchain and linux-2.4 integration over there, this will imply enormous development costs for packaging and system integration <stikonas>aggi: it's good enough for some stuff but not the other. It runs fine on Linux kernel but wouldn't run on builder-hex0 or UEFI <stikonas>yes, looks like frippery make assumes a fairly capable livc <stikonas>where as gnu make has a lot of built-in stuff <aggi>the libc-question is another one remaining, at least with oasis-linux the GNU-buildsystem is detangled from binutils, and this could be an acceptable compromise to continue with instead of tcc-assembler <stikonas>yes, but that's a bit orthogonal to what rickmasters was talking about... <aggi>the main hindrance is this still: a _hard_ vendor-lock with any tcc/linux-2.4/binutils/oasis-linux against x86(64), because i consider the availability of a cross-compilation and alternative ISA (aarch32/64) a mandatory acceptance criteria for bootstrapping too <aggi>well, yes, my perspective is the continuation of system-integration/bootstrapping based upon on the preceeding bootstrapping stages of yours (which i appreciate even when i decide to take another route when detangling from GNU-toolchain and GNU-buildsystem)