IRC channel logs
2025-03-21.log
back to list of logs
<fossy>spent a couple hours implementing parallelisation in kaem - intended use is for the mes build in live-bootstrap, will significantly reduce the time on multicore available systems (i.e. not bare metal) <matrix_bridge><Andrius Štikonas> fossy: Any chance you could also try unified libc for mes? <matrix_bridge><Andrius Štikonas> I suspect there is a lot of overhead loading mescc so many times <matrix_bridge><Andrius Štikonas> My main concern is that this breaks buulder-hex0... <fossy>stikonas: def breaks builder-hex0, we already force JOBS=1 for early kernel bootstrap (pre-Linux era) <fossy>if unified libc is faster, there is some N where for JOBS < N unified libc is faster, and JOBS > N multicore is faster <fossy>fwiw, using JOBS=5 gave me at least a 3-4x speedup, so i think unified libc is probably not that much faster <matrix_bridge><Andrius Štikonas> Perhaps we might be able to reach tcc directly too at some point... <matrix_bridge><Andrius Štikonas> M2-Planet is gaining features really fast... <matrix_bridge><Andrius Štikonas> Maybe worth trying using prefix increment in your PR? <fossy>wasn't in l-b yet. so didn't try <matrix_bridge><Andrius Štikonas> Oh yeah forgot that we didn't add it there <fossy>is there a particular goal of all the current m2-planet improvements, or just general improvements? <fossy>either is fine of course, just intrigued <matrix_bridge><Andrius Štikonas> And it needs either new mes release or 2 extra cp commands <fossy>is gtker in this channel haha <matrix_bridge><Andrius Štikonas> I think for now gtker is targeting tcc1_7 from tcc_bootstrap_alt <fossy>tcc_bootstrap_alt is very helpful for more "micro-goals" <matrix_bridge><Andrius Štikonas> Well, so am I right now since I'm on my phone... <matrix_bridge><Andrius Štikonas> Once 1_7 is working we can just bump the goal a bit... <fossy>I remain somewhat unconvinced that we could reach tcc 0.9.27 from M2-Planet without an architectural rework of M2-Planet, but i think that will become painfully obvious if I am correct <matrix_bridge><Andrius Štikonas> I thought it is already hard to add new features, gtker proved me wrong <matrix_bridge><Andrius Štikonas> Lots of refactoring of codegen certainly helped <fossy>yes, some things still seem fairly hard though... like the pointer aithmetic <fossy>codegen was the biggest sticking point for me, so gtker's work there is very good <matrix_bridge><Andrius Štikonas> Pointer arithmetic works for +=, -=, ++ and -- <matrix_bridge><gtker> Currently my goal is tcc_1_7 as an intermediate goal for full/cosinusoidallys new repo at some point. Not sure if that's achievable at all but I'll try <fossy>yeah, it works, but from the PR it didn't seem like an easy addition at all <matrix_bridge><gtker> stikonas: Codegen refactoring is probably what made ++ and -- doable for me. If I would have had to copy paste giant blocks around for each push/pop I don't think I would have gotten something working <matrix_bridge><Andrius Štikonas> Without codegen refactoring reviewing would have been nightmare too <matrix_bridge><gtker> Pointer arithmetic for non compound, non prefix/postfix is going to be tough. I think the problem is that there's also just some bugs in the current implementation of type promotion and what-is-the-type-of-the-current-operation <fossy>the current type handling is the most fragile part of M2-Planet i think <matrix_bridge><Andrius Štikonas> Well, we could always patch full tcc a bit <matrix_bridge><gtker> Definitely, even while working on it directly I was forgetting what was on the stack/in registers 😄 <matrix_bridge><gtker> I tried replacing typedefs with just a direct lookup to the type instead of the current approach and it completely broke the bootstrap because of type promotion, which is pretty weird <fossy>having pointer types be explicit types is also fairly problematic imo <matrix_bridge><gtker> Oh yeah. I'm absolutely dreading having to add casts <fossy>cause char* is an explicit type in the internals of M2-Planet <fossy>yeah, see new_primitive in cc_types.c <fossy>so you can't have char*** right now <matrix_bridge><gtker> It might be necessary to redo how pointers/types interact, but I'm not looking forward to doing it 😄 <matrix_bridge><Andrius Štikonas> In practice, I don't think triple pointers are used that much... <fossy>i think this is most clearly evidenced by having the current type as a global in cc_core.c <stikonas>oriansj: do you have any context on why x18 is used as stack register for aarch64 in M2libc/ <deesix>stikonas, I guess that comes from M2-Planet; I think I wrote something about that in some doc there. <deesix>I guess something about aligment. <ekaitz>stikonas: i shared some idea in mastodon about kaem, i don't know if you saw that. I'd like to know your opinion <stikonas>ekaitz: not yet, just came back online recently <deesix>stikonas, M2-Planet/HACKING was the file. <stikonas>ekaitz: hmm, that would need a bit rewrite of kaem I guess... <stikonas>one could certainly write something more make like <ekaitz>surely, yeah but it could be interesting for development ergonomics <stikonas>fossy is probably a better person to discuss tell about this <stikonas>yeah, it could be nice not to rebuild everything... <stikonas>but it would be hard to get all dependencies right <stikonas>we don't even know exactly what package depends on which packages exactly later in the bootstrap <stikonas>and make with not fully specified dependencies would be even more annoying as builds could fail seemingly randomly <ekaitz>we don't have to check for the files or anything <stikonas>ekaitz: well, the other option of speeing up is parallel kaem <ekaitz>isn't that easier if happens automatically? <stikonas>well, not all kernels support parallel execution... <stikonas>so we want at least an option of turning it off <stikonas>and feel free to leave review comments :) <fossy>ekaitz: the idea you conceive is (i think) a good one, but for a tool orthogonal to kaem <fossy>kaem is really a basic shell at the moment <fossy>i think we need to keep a basic shell (kaem) <fossy>but having a baisc make for building stuff would be good <fossy>well - of course you could do everything using just make if you wanted to. but why? some things are suited better to clear, sequential execution imo <ekaitz>make has sequential execution too <stikonas>but with somewhat awkward syntax I guess <stikonas>gtker: do you think * would be better ot not? <fossy>and i know we could implement ifeq/else/endif <stikonas>so it might actually be faster on modern CPUs <matrix_bridge><gtker> I'm not sure. I think it depends on the other branches as well. I would like to cut the giant elseif down to size but I'm not sure what the best way of doing it would be <stikonas>also make might be quite slow for big monolithic project <fossy>but i don't quite see why we should use a more complicated grammar than we need to early on in the bootstrap <stikonas>it might spend a long time figuring out rules <matrix_bridge><gtker> stikonas: With the -1 trick I'm getting "Unknown type 1" when building the compiler itself <stikonas>gtker: ok, let's merge it like this then <ekaitz>that's true, why even bother in the first steps of the bootstrapping? <stikonas>yeah, but resuming from the middle of bootstrap is annoying... <ekaitz>because it would also require writing the tool in hex0 or something <stikonas>fossy and I created some of the scripting for htat <stikonas>well, we don't need that for kaem-minimal <stikonas>just for later stages where full C kaem is used <ekaitz>oh if we can do it in C it shouldn't be so complex <stikonas>anyway, we do have scripting to resume builds in live-bootstrap but it's mostly after bash... <stikonas>when we bootstrap enough stuff to do packages... <fossy>i think some kind of simple (C) make could be helpful especially for the mes and tcc stages <ekaitz>my goal was to make something that follows the rules but doesn't apply checks, so we can replace it with make for testing, but it doesn't really make what make does <stikonas>but it does need to have somewhat similar syntax as make (else make won't run it) <ekaitz>has to have the same syntax, yeah <stikonas>though I agree that simple (C) make could be used for some stuff <ekaitz>fossy: just rebuilding everything, not looking at the file creation or dates <ekaitz>that simplifies the thing and makes it act as a command launcher <ekaitz>but also, enables using a real make instead <ekaitz>(sorry, i didn't contextualize properly here) <stikonas>maybe let's take a step back a bit. What is the primary usecase? <ekaitz>that could be kind of easy to do... i think <stikonas>I guess you want to skip some of the mes steps <ekaitz>stikonas: mes currently is built by a huge script that has to build many things one by one <ekaitz>specially if you are touching `mes` and you want to test how mescc works <ekaitz>because it would rebuild the stdlib and so on <fossy>perhaps i am being silly, how would this help there? <stikonas>then it won't rebuild everything... in mes <stikonas>hmm, though right now live-bootstrap only builds mes using M2-Planet <ekaitz>and also, now we are maintaining a simple.make file that is kind of the same we have in the build.sh but in a makefile format <stikonas>but is it not more convenient to use standard mes build system (with make and build with GCC) for development? <fossy>does mes (outside of live-bootstrap) not have make? <ekaitz>we do have a simple.make file, but it doesn't build everything <ekaitz>and the normal makefile just runs all the build.sh script <fossy>maybe this is a shortcoming of the mes build system.... <ekaitz>but it repeats the same that comes in the build.sh <stikonas>duplication of various build systems here is a bit annoying but I don't know if there is any better way <stikonas>if once writes simple make in simple C then yes, we could switch to simple.make file <stikonas>instead of maintining kaem script in live-bootstrap <ekaitz>and if we went supercrazy we could write the whole bootstrapping in a make recipe <fossy>we are dealing with a unique situation where we want the robustness of make for development, but simplicity for bootstrapping <stikonas>but that will take very long to resolve rules... <fossy>i still don't think that's a good idea <stikonas>I'm worried about using make everywhere though <fossy>i don't think that would be possible with a "trivial" dependency parser <fossy>we would have many things where we have something like bzip2 -(depends)> tcc and bash -(depends)> tcc which would not work with a simple (dateless) dependency parser <fossy>and we use environment variables extensively which are difficult to manipulate in make <stikonas>the awkward thing would be jumps between kernels/reboots <stikonas>make gcc-14 would e.g. cause a reboot in the middle <stikonas>and recovering from that would be awkward <stikonas>at some point I was thinking whether we could have some dependency rules for live-bootstrap steps <stikonas>so that e.g. build gcc-14 would invoke the right dependencies <stikonas>but since we don't precisely know what depends on what, that's also awkward <fossy>i think we want to document at some point what depends on what <stikonas>if we knew exactly what depends on what, then tsort could create a list <stikonas>fossy: yeah, we might want but how do you determine what depends on what <fossy>huh, i did not know about tsort <fossy>stikonas: the best way i came up with in my head is using an atime enabled filesystem <fossy>so compare atimes before and after a package build <ekaitz>tsort! i didn't know about this either <fossy>should indicate what was used <stikonas>strace might also work though it would be noisy <fossy>i don't love it, but it may work as an initial pass to determine what something uses.... <ekaitz>very interesting thoughts indeed <stikonas>well, at the very least both of you now will now about tsort :) <stikonas>I think originally tsort was created to order functions in a C file, so that early declarations are not needed <ekaitz>it's never a wasted discussion here, i learn a lot with you <stikonas>yeah, it's just that speeding up builds for development is not easy here... <ekaitz>> tsort exists because very early versions of the Unix linker processed an archive file exactly once, and in order. As ld read each object in the archive, it decided whether it was needed in the program based on whether it defined any symbols which were undefined at that point in the link. <ekaitz>and it worked on `lorder`'s output <ekaitz>so lorder made the dependencies and tsort ordered the files <stikonas>probably not very used coreutil these days