***fosslinux is now known as fossy
<fossy>please please note that mes is not currently in the bootstrap <fossy>we need it to work under mes-m2 <fossy>OriansJ: i have a PR coming your way shortly for some basic command line utilities in M2-Planet (cp, chmod are the main ones) <fossy>OriansJ: also, if you have some time, would you mind implementing dup2 and pipe syscalls? <OriansJ>siraben: short version stage0 is the overall project; a universal set of steps to bootstrap GCC with the main repo focused on bare metal bootstraps. With mescc-tools-seed being a POSIX fork with the goal of proving those exact same steps on POSIX kernels so that people can on existing systems verify and produce idential binaries as the same bare metal tools. <OriansJ>as M2-Planet on bare metal behaves exactly the same as on a POSIX or DOS or etc <OriansJ>if a step can't be converted into binary, 1 input file and 1 output file; it means it probably needs refinement. <OriansJ>mihi: if you don't wish to perform those changes, just say so. I'm do them myself when I get some free time this weekend/week. <OriansJ>fossy: I'll do what I can when I get the chance. <fossy>OriansJ: yeah no rush at all <OriansJ>mihi: as a general rule, never do work that you don't think is the right thing. If you think I am wrong. Good; tell me. Understand disagreements are healthy things here. <fossy>I am excited for when we are able to get to the bare metal bit <OriansJ>Burnout is too easy in long projects like this. We have plenty of time to address issues and reach compromises which make sense to all parties involved. <OriansJ>I just got side tracked with the microcode problem in bootstrap trust; went with something that has previously been done in TTL and thus probably could be done again. <OriansJ>In short my ambition for this project is probably a little unhealthy but boy does it involve a lot of fun and cool projects. <OriansJ>How many people get to say they wrote a Garbage collected compacting Lisp in assembly (That they also created from scratch) or a FORTH or a C Compiler in Assembly? <OriansJ>mihi: as a general rule, I am a dumbass who mades bad decisions but ultimately whoever does the work gets to decide the direction of change. Eg whoever is willing to do more work than me gets to overrule me on design decisions. everyone gets to make suggestions and provide perspective but whoever is writing the code that gets things done. <OriansJ>pder and siraben are doing more blynn-compiler work than me; they get to dictate direction. deesix and yt are doing more AArch64 work than me; they decide how M2-Planet/etc are going to work. fossy does more kaem work than me and thus can dictate the direction of kaem. <OriansJ>as a general rule I try to never reject a Pull Request or patch unless I am unwilling to do the work to support it. <pder>OriansJ: I pushed another small change for blynn-compiler. I did some profiling on vm.c and found that lazy3 and others are called a ton. Simplifying that code reduces the bootstrap time about 10% on my machine and I think it is more readable. <siraben>OriansJ: thanks for the clarification on stage0 <fossy>siraben: pder post marginally, we have to go ./bin/x < something > something <fossy>once dup2 and pipe are implemented i can do pipe support in kaem then we are all good <pder>fossy: also it would be a very simple change for me to make the marginally stage and on take input and output arguments so the redirection is not needed <fossy>pder: well, that would be pretty good <fossy>so that when we implement a kernel it dosen't need pipe support <fossy>which adds a really complex layer to IPC ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<OriansJ>pder: thank you for the work; merged <deesix>fossy, so you still need dup2 and pipe? Also, where are cp and chmod needed? <siraben>pder: I will look into minimizing the diff between the stages to make auditing easier <siraben>also, might it be good idea to number the stages? <fossy>deesix: it would certinaly be nice to have those which enables shell redirection in kaem <fossy>cp and chmod are being used in my new project <fossy>which I will make public before the new year <deesix>OriansJ, on the topic of "finally start treating libc seriously I guess in M2-Planet", I find it very weird that we're using files from M2-Planet/test/ all over the makefiles/scripts, in mescc-tools-seed for example. <fossy>an initramfs for linux that bootstraps up to a full linux system taking a linux kernel seed, kaem seed and hex0 seed (and mes seed until blynn or smth is done) <fossy>with ability to easily replace the seed kernel when a better kernel presents itself <deesix>fossy, I read you asked for dup2 and pipe, then again at the marginally conversation (kind of rejecting the idea) and OriansJ (or anyone) going to implement them when the need is not there anymore. <fossy>deesix: shell redirection is a nice-to-have under any circumstances, it helps greatly espeically for things such as debugging and logging, as well as in some earlier compile staged <fossy>such as the first binutils, gcc, bash, etc <fossy>however it is not strictly nessecary now if blynn-compiler will have inputs and outputs as arguments rather than as stdout/in <fossy>i don't really know yet, need to thing more about it <fossy>it might be better to work around a lack of it for sake of a bootstrappable kernel as i said in the marginally conversation <deesix>OriansJ, I was a bit confused about $arch_defs.M1, ELF-$arch-debug.hex2, ELF-$arch.hex2 and libc-core.M1 being also in M2-Planet/test/common_$arch/ (and for AArch64 they were created here originally) but I guess in this case the canonical ones are those at mescc-tools-seed/$arch/, right? Because makefiles/scripts don't use the M2-Planet ones. Am I looking at this correctly? <deesix>fossy, I see; just checking if the request was still open. Thanks for the details. <fossy>janneke: mes is segfaulting for me and i cannot figure out why <fossy>when i run bin/mes-mescc --verbose <fossy>i have tested 0.21, 0.22 and master <fossy>it segfaults attempting to provide a bactrace <fossy>compiled using ./bootstrap.sh <fossy>i have a backtrade coming, sec <janneke>have you tried MES_DEBUG=3, or something? <fossy>strace yields nothing of interest <janneke>it segfaults after it finds an error <OriansJ>deesix: understandably. The test/common_$arch/ was just a quick choice that probably wasn't well thought out. As the concept of making a proper for libc in M2-Planet didn't even seem reasonable. <OriansJ>so technically there is no canonical libc for things being built by M2-Planet <fossy>janneke: ah, yes, --version works. i have reprod this in a number of different settings, but when i acidentally passed --vrebose to mes it happened to do this. os this segfault i am seeing expected behaviour? in this particular case? <OriansJ>as M2-Planet you have no binary libraries, only source code that you explicitly include with -f and thus one could always write their own file library or string library. <fossy>i looked at the relevant files and what seems off is the very large value of x, which expectedly so when indexed in some global array is wayyy off <OriansJ>as the #include's only mean things to GCC but not M2-Planet. <janneke>fossy: it depends on how you define "expected", because it's not desirable...but yeah, i see this too <fossy> now time to track down this other odd segfault which may not be related but might also be me being dumb <OriansJ>and yes right now M2-Planet built with GCC doesn't use the contents of test/common_$arch/ because #include <stdio.h> and #include <stdlib.h> does include all of that functionality <OriansJ>but I guess it is time for me to put in the effort to provide the primitives C programmers would expect of M2-Planet when they #include some standard libraries. <janneke>fossy: you probably want to run mes/mescc using their wrappers in scripts/* <fossy>i am using the mescc wrapper <OriansJ>fossy: if you checkout mes-m2; there is a snapshot of MesCC and nyacc that should work out of the box if mescc-tools is in your $PATH <OriansJ>fossy: it should however I only tested with guile <fossy>janneke: ah found the other one <fossy>which after my small modifications produces this command bin/mes-mescc --no-auto-compile -e main -L module -L mes/module module/mescc.scm --help <fossy>sorry to inundate you with this lol <OriansJ>fossy: that would be if you ran a fuzzer against mes.c <fossy>yeah so 'if (TYPE (x) != TCHAR && TYPE (x) != TNUMBER)' and 'assert (TYPE (string) == TSTRING);' are the two lines causing the segfaults, I think that TYPE() macro has issues <fossy>or the passing of x (whatever that is) around <fossy>anyway i'm signing off for the night. cya all <deesix>OriansJ, sure; M2-Planet ignores #define (or any kind of standard preprocessing) and gcc don't use test/common_$arch/ at all (with our asm()). As for my questions about the 4 files... Is mescc-tools-seed the home the $arch_defs.M1? What about ELF*? <OriansJ>deesix: I probably should figure that out since these files are probably only going to get copied as more things try to be built with M2-Planet <deesix>OriansJ, indeed. Duplication is sometimes required but too much "doing the same fix 5 times" is not funny at all. For context, I'd like to say that I was looking at how to resolve the /two kinds of stacks/ we get from the AArch64 M1 definitions. <OriansJ>Perhaps a M2lib of sorts for projects that are written in M1 or use M2 <OriansJ>A single place for common definitions and libraries. <OriansJ>Where proper C standard functions can be done in C <OriansJ>*or atleast the M2-Planet subset of C we support* <OriansJ>probably should think of making it easy for people to add their own OS if they don't do standard Linux syscalls <OriansJ>let me whip together something for say AArch64 tell me what I did wrong and we can iterate towards something better long term. <deesix>OriansJ, there's no hurry. Take your time to think if you want. Support for other OS is a thing to have in mind, but maybe not a priority. <siraben>I'm not a macOS expert but that would be good to have at some point as well <siraben>OriansJ: I haven't been keeping up this this discussion, what is it? <OriansJ>siraben: a future solution to the copy paste problem with building things with M2-Planet <OriansJ>siraben: notice how blynn-compiler has a copy of files from M2-Planet? <OriansJ>when some bug gets fixed in M2-Planet, one then has to find and fix all the copies too <OriansJ>So create it as a unique repo which can be used as a submodule <OriansJ>That way people who care about the C lib in M2-Planet have one single place for everything mescc-tools and M2-Planet specific <OriansJ>So if someone wants to port it all to barOS which doesn't support Linux syscalls, they need only add a folder for each architecture and provide the their own syscall primitives. <OriansJ>it also allows BSDs to have different struct sizes from Linux for things like utsname <OriansJ>it is very much a raw idea right now. <OriansJ>So I want people to figure out what I am doing wrong. <deesix>Oh, yes... 0202471cef4685b8bb4470d1618a15ab32b131ef of blynn-compiler also catched my eye; not only functions/ but definitions for one arch (scared to see more comming). <OriansJ>it needs some work but the general idea is there. <OriansJ>Everything architecture specific is stripped out and placed in its own $arch_folder <OriansJ>Then we make it a submodule in M2-Planet and anywhere else where we have this sort of duplication and save ourselves a lot of trouble long term. *deesix is in a fight with some "typos" in ELF* comments, but the copy-paste is winning for now :P <OriansJ>well this is going to be a big clean up job with lots of pieces needing love. <OriansJ>But I guess eventually I had to address this sort of problem of my own creation. Again. <OriansJ>and if we do it right, MesCC can also share the work done. <deesix>What would be wrong for MesCC? What to watch out? <OriansJ>deesix: well MesCC doesn't support FUNCTION and does follow #includes and would require us to have matching #defines for each of our // CONSTANTs <OriansJ>but other than that standard M2-Planet code should work great with MesCC. <janneke>the biggest problem with the full source bootstrap was pointer arithmetic <janneke>the FUNCTION thing is merely an annoyance; one cannot mix mescc and m2 code <janneke>other than that, m2-planet with the # and // tricks is brilliant <OriansJ>janneke: one can if one doesn't use M2 features that MesCC doesn't support <OriansJ>function static variables would require M2-Planet to add support but we can cross that bridge when/if we get there. <OriansJ>as static variables would just be a label with a NULL in M2-Planet's output <OriansJ>So adding it wouldn't be hard; as it is just a minimally scoped global variable <siraben>OriansJ: I see. that's a good idea for porting it to other OSes that have different syscall primitives <pder>siraben: I used valgrind (callgrind) with the gcc build, and also compared M1 output with the M2-Planet build <siraben>so performing map/foldr fusion those places should have made crossly.hs faster, but I have not benchmarked it <OriansJ>pder: easy way to determe if M2-Planet is producing problems. <mihi>OriansJ: I agree on what you wrote about who dictates the direction of a project. On the other hand, I don't want to create pull requests where I know that I will create you more work in the future as you are not satisified with how it is done. Especially since all that performance tweaking is only a side quest to the real goal. <mihi>And yes, burnout is a thing, which is why I intend not to do any coding during the holidays (or as little as possible) and instead relax. For me, the holidays is 24th/25th/26th (UTC+1 timezone) <mihi>so if you beat me to it, feel free to change it as you see it fit better. I currently don't see it helpful or necessary to make file.c more bloated and memory hungry for no measurable benefits. <mihi>about M2libc: I like the idea and the general file layout. I would prefer the names to be more consistent between architectures, i. e. M2libc/$(get_machine)/ELF-debug.hex2 being always the debug ELF stub, or if needed M2libc/$(get_machine)/ELF-$(get_machine)-debug.hex2. As you have to change all paths anyway, I don't think that makes it a lot more work, and maybe can share some code in M2-Planet tests. <mihi>about other archs: I would love to see getting M2-Planet and assorted stuff to be ported to x86_64 EFI Boot Services (so that it can be run directly from UEFI shell). But that would require lots of changes, starting from that it does not use ELF but PE/COFF binaries, and syscalls are basically jumps into a giant jump table called the EFI Boot services table, which is a member of the EFI system table which is <mihi>passed as a pointer to the main entry point. <OriansJ>mihi: seems entirely reasonable; enjoy your holiday. <OriansJ>oh that is far easier to do than you think <mihi>but I know that I won't be able to spare enough time to implement it myself in the near future, so I won't want to make anyone else do so either. <mihi>(did I mention that the calling conventions used to call EFI services are different from what is generally used?) <OriansJ>Only the syscalls would need to be replaced and libc-core.M1 tweaked and PE.hex2 would need to be written but then you would be done. <OriansJ>calling convention is a NULL point when you can use ASM() <mihi>(which probably can be fixed by setting up the stack layout by _asm directives, but I did that some time ago and it took me "too long" to get it right) <OriansJ>How M2-Planet calls your read/write/etc doesn't matter as long as the primitives do the transformation. <OriansJ>eg you could call read with the arguments in the wrong order and just have the assembly block fix it and then do what is required on the host. <OriansJ>So it oculd be done in an afternoon by anyone familiar with the host system's conventions and syscall # list <OriansJ>the PE.hex2 is probably the hard part <mihi>if that person also was familiar with the structure of the PE/COFF format (I never looked at it using a hex editor), and with M2-Planet calling conventions :) <OriansJ>as I am not too familiar with the format <OriansJ>M2-Planet doesn't have calling conventions; only architecture specific stack conventions <OriansJ>for example because x86 is a stack that grows towards zero and we push arguments 1 then 2 then 3. It is always base pointer - offset With a return address from the call before the locals <mihi>so the calling convention of M2-Planet is "pass all arguments on the stack". That is also a calling convention :) <mihi>IIRC for UEFI the first 2 arguments are in registers and the rest is on the stack (in reverse order from what Linux does) <OriansJ>mihi: a very lazy one but yes and returns in what your architecture prefers in R0 (x86 => EAX) <mihi>unfortunately, the Phoenix Wiki has been down for a few years now (it was very useful as it had pages for all the UEFI structs and functions), and I don't really want to point to the 10MB large UEFI spec PDF :) <OriansJ>mihi: then encode the knowledge with your code <OriansJ>M2-Planet supports /* block comments */ and M1/hex2 support ; and # line comments <OriansJ>we don't need the full spec to bootstrap <mihi>I wouldn't be able to do it in one afternoon :) <OriansJ>mihi: open, close, read and write is too much for an afternoon? <OriansJ>doesn't even eed to be a proper read or write, just a fgetc and fputc that works on a single byte at a time. <OriansJ>M2-Planet using cc-minimal.c is just a single file read and a single file write. <mihi>more wrapping my head round where to get the correct pointers from that need to be called to open a file. read/write is simple in comparison. <mihi>When implementing it in C it is (tedious but straightforward), but that assumes I'm using a C compiler that does the calling conventions for me. <mihi>not to forget that if you want relative paths (and you probably want them) you'd have to first query the path of the currently running binary, find the last slash, append the relative path name, and pass that path to the syscall. <mihi>there are no chdir or similar syscalls in EFI Boot Services. <mihi>but fortunately it will resolve ../ for you <mihi>but yeah, considering how Computrace has subverted the platform, not the best platform to bootstrap either <pder>Not sure if this is known issue in M2-Planet, but indexing into a two dimensional array does not seem to work. <pder>char getargchar(int n, int k) { return env_argv[n][k]; } does not work <pder>char getargchar(int n, int k) { char *tmp = env_argv[n]; return tmp[k]; } works <OriansJ>pder: I can confirm multidimensional arrays are broken in M2-Planet. <OriansJ>At this point, I think I need to just start tracking these sorts of bugs publicly <siraben>pder: there appears to be a regression, if you try to run the generated C program from precisely.hs it no longer compiles with GCC <OriansJ>add a single ')' in the generated output. <OriansJ>but also that code wouldn't compile with M2-Planet either <siraben>this also means it's a good idea to CI the code generation <OriansJ>siraben: or the CI didn't care that the build failed <siraben>OriansJ: because it was never tested <OriansJ>well running go.sh is the testing or make ALL <siraben>OriansJ: ah, I should clarify that it's not precisely.hs failing to compile but the _output_ of precisely.hs failing to compile <OriansJ>siraben: so it probably would be a good idea to make some tests for precisely to ensure its primitives behave like GHC's primitives <siraben>yeah, there's also a mandelbrot test by ben lynn which we could adapt <OriansJ>That way as we approximate the required functionality for bootstrapping GHC; we minimize the number of regressions. <OriansJ>but yes; leveraging the previously written tests probably is a good idea. <pder>sorry about that, I didnt notice it because I only tested ./go.sh and make <pder>What do you think about adding one more step to go.sh to have bin/precisely compile precisely.hs and recompile? <OriansJ>self-hosting is usually a good test for a compiler <siraben>pder: yes tht would be a good idea as well <siraben>pder: well, actually, I think go.sh should be for building only, we could have a test.sh file for the tests <pder>Speaking of samples, there is also site/q11.hs which is known as the N-queens problem. That might be nice to adapt so it runs on precisely <pder>I pushed the missing ) fix to my master branch. Also I fixed getargchar() so it works with M2-Planet. Now command like arguments work like ./bin/precisely coms <pder>Thanks. I was thinking about the best way to avoid shell redirection in the build since it may not be available. The ffi probably needs to be extended to allow opening files for both input and output, but I think most of that could be done in Haskell. siraben: thoughts? <siraben>pder: Oh yeah could use blynn-compiler as a shell scripting language <siraben>Some form of modules would really help here, hm. <siraben>if we had #include in blynn-compiler, even. <pder>For example in precisely.hs on line 2046 is where main is declared and we parse args. I think we would want to be able to specify an input and output file there. <pder>I dont think that would be too involved. The include function in Haskell would just call into C to open the file and read from that file handle <OriansJ>siraben: well #include is simple to add. just treat it like an expanded macro; that is what the C preprocessor would do. <siraben>OriansJ: right, but we'd have to implement that in Haskell or C? <siraben>Because if we did so in Haskell, would have to open/close files <siraben>and that requires extending the FFI, which we need to do at some point anyway <OriansJ>siraben: either will work but it is more of a question which seems more fun to you. <OriansJ>as opening a file in C can either mean just returning a file handle (eg the number 5) or returning a struct which might contain the contents of the file as a char array. <stikonas>hmm, so how will blynn-compiler be used for bootstrap. At the moment I see it creates something called precisely? Is this going to be used for further more complete haskell? <fossy>stikonas: probably a scheme interpreter on top which then can go to mescc and firneds <fossy>janneke: FWIW i got a bit further by using scripts/mescc.scm rather than module/mescc.scm <fossy>do you know why i might be getting assert fail: TYPE (string) == TSTRING <fossy>success! building mes-0.22 using mescc!!! <fossy>your mes-0.21-skldafjiowaturiewjklngklasdngfklureiowautiorsdjklfjaioetwlekjr (not those letters) didn't work but mes-0.22 works!