<OriansJ>yep; fortunately being a stupid malloc makes it more portable but it is gonna waste alot of space (about 4KB per malloc) <OriansJ>1K mallocs at 4K waste or 4MB of wasted memory net <xentrac>the stupidest malloc just bumps a pointer in a static char array <OriansJ>xentrac: that doesn't work on Linux; unless I do a mssive malloc right off the bat <nimaje>so, correcting OSABI and Flags in POSIX/AMD64/hex0_AMD64.hex0 makes it work and as expected the result generates the same binary (it is diffrent from that binary build with xxd) <nimaje>it seems like POSIX/x86/hex0_x86.hex0 and kaem-optional-seed.hex0 only need OSABI fixed, but I haven't tested that <nimaje>oh, no that xxd generated binary had empty output (should have checked that earlier) <nimaje>ok, I just called it wrong, it works fine <nimaje>(well file says ..., corrupted section header size but that's probably ok) <OriansJ>nimaje: well no section headers at all, so kind of expected <OriansJ>I'm correcting x86 and the kaem-optional too <nimaje>brandelf'd kaem-optional-seed is identical to one generated by corrected hex0 and corrected OSABI in kaem-optional-seed.hex0 <OriansJ>now onto the main show mescc-tools-seed <OriansJ>the order is in: AMD64/mescc-tools-seed-kaem.kaem and then AMD64/mescc-tools-mini-kaem.kaem <OriansJ>so first AMD64/hex0_AMD64.hex0 which should have a identical changes as the bootstrap-seed's hex0 <OriansJ>then AMD64/kaem-minimal.hex0 with the same changes as bootstrap-seed's kaem-minimal <OriansJ>the only difference is this one actually is written in AMD64 and we will need to check the malloc <OriansJ>nimaje: incase you were not sure: make test-amd64 (unless you want to help test the x86 binaries for FreeBSD too using make test-x86) <nimaje>already changes OSABI and ph_flags in all *.hex{0,1,2} files, now I am at ./hex2-0 hold M2 and it says Subprocess error ABORTING HARD <nimaje>I just run ../../bootstrap-seeds/kaem-optional-seed in AMD64 <OriansJ>nimaje: well that error message is from kaem; it means the last program returned something other than 0 <OriansJ>as I believe I had hex2 just allocate a big block <OriansJ>4881C7 00008000 ; ADDI32_to_RDI %8388608 ; Create space for temp [8MB] <xentrac>OriansJ: you can't have a 100MB .bss on Linux? I admit I haven't tried it but I'd never heard of such a limitation <OriansJ>xentrac: that is possible but also is the malloc it all up front bit <OriansJ>nimaje: see if increasing it to 12MB works (warning in little endian order means 0x12345678 looks like 78 56 34 12 <OriansJ>(you'll also want to update the mescc-tools and M2-Planet submodules) <nimaje>to change that from 8MB to 12MB I only need to change 00008000 to 0000C000 there? <nimaje>well, ./hex2-0 hold M2 probably failed because I didn't had the submodules <nimaje>ok and then M2 fails (probably because of the old malloc) <nimaje>pulling M2-Planet resulted in ./cc_amd64 hold M2.M1 failing Unknown type SCM <OriansJ>excuse me as I have to read my son his bedtime story (will be back shortly) <OriansJ>xentrac: his favorite part of bedtime (besides the bottle of milk) <nimaje>now it runs the build to completion <OriansJ>nimaje: nice; so everything on AMD64 builds and runs without issue? <OriansJ>love to see that diff (so I can apply and upstream it) <nimaje>well, it builds; not tested if it runs fine, but as most stuff there is needed to build later stuff that should run fine <OriansJ>well the only parts that need to be tested are those in bin/ <OriansJ>you can copy them over and use the standard mescc-tools and M2-Planet tests on them <OriansJ>The only warning is test0008 depends upon signed shift behavior in hex2; which I need to fix in M2-Planet but you can skip that by using the GCC version of hex2 for the M2-Planet tests <nimaje>what does kaem mean with UNKNOWN ARGUMENT? I gave it a file which just contains /bin/ls <fossy>are you running the latest kaem <OriansJ>if you look at mescc-tools/kaem.c in the function main (it will explain exactly how kaem accepts arguments <fossy>you should just run ./kaem script.sh now <fossy>or ./kaem -f script.sh if you feel so inclined <OriansJ>nimaje: the final kaem binary in mescc-tools-seed is just the C version of kaem.c compiled by M2-Planet <nimaje>I ran ./bin/kaem test.kaem with test.kaem containing a single line /bin/ls <OriansJ>fossy: I think I forgot to revert the kaem extensions (I think I missed it by being too busy) <fossy>wait, why were they being reverted? <fossy>nimaje: try ./bin/kaem -f test.kam <fossy>nimaje: try ./bin/kaem -f test.kaem <nimaje>yes, found that just now, that works <fossy>ok.. weird... i added support for ./bin/kaem file <OriansJ>fossy: I needed to pull your enhanced kaem out of the mescc-tools-seed bootstrap path because it was producing an error, that I didn't feel able to address at the time <fossy>OriansJ: OH right, I see, the minimal kaem for the core bootstrap <fossy>is it a compile time error or a runtime error? <OriansJ>I just never got the chance to revert that <fossy>in mescc-tools on the master branch it's at minimal kaem. <OriansJ>just too much on my plate and it just kept getting dropped <OriansJ>so fossy if it isn't too much to ask, please revert that commit of mine and help me figure out what is going wrong with the enhanced kaem so that people can use it too <fossy>yeah i will try and get to that ASAP <fossy>i'm quite busy IRL with school exams but i'll put it on my list, should be done within the next week or so <OriansJ>fossy: oh I get busy; I've got a 7month old who demands lots of attention and a full time job. <fossy>i hope you're having fun at least with your child <OriansJ>fossy: lots of playtime is great but it really cuts down on my programming output <OriansJ>So I'm only getting a couple hours tops (when wife is on kid duty or kid is asleep and wife doesn't expect couple time) <OriansJ>and this week, I had to burn a bunch of time trying to prove a problem in guix about missing tarballs <OriansJ>so I paid for a digitial ocean ubunutu server; downloaded guix's sources.json. extracted the urls and had wget go to town. Honestly it was far worse than even I would have expected only 7,557 remained out of the list of 74,881 tarballs <OriansJ>of which a smaller subset actually match the checksum in guix <OriansJ>as a few are just html pages (and I couldn't get guix to even pull without substitutes on that server) <OriansJ>now that server is running: while true; do netcat -l 7557 < report.tar.xz; echo "there is a download" ; done <OriansJ>and tomorrow I'll be shutting it down (as hopefully everyone who wanted a copy got the 94MB of logfiles that shows exactly what worked) <OriansJ>from 128.199.4.208 on port 7557 if you too want a copy <nimaje>ok, trying M2-Planet tests with the binaries from mescc-tools-seed results in test/test0000/hello-aarch64.sh being successful and then M2-Planet segfaulting in test/test0001/hello-aarch64.sh <nimaje>but enought for today, I should go to bed, good night <xentrac>OriansJ: whoa, that's a lot worse than my guixstrapping experience <OriansJ>hopefully I'll have some fixes for that by tomorrow <OriansJ>xentrac: mine too; mostly just a tarball here or there that went missing or changed. efraim only has a small list too (that they archived) <OriansJ>So either these are packages that no one builds from source or no one actually uses <OriansJ>or I got garbage data from sources.json and the entire activity was a complete waste of time <xentrac>wouldn't be surprising if there were a lot of packages in guix no one actually uses <OriansJ>xentrac: possibly mass package definition imports to boost package counts? <OriansJ>I think I tried to be too clever again with loops <OriansJ>I really need to stop declaring variables inside loops for M2-Planet <OriansJ>or another more likely possiblity; the people usings those packages are simply updating the definition on their own channel and not bothering to upstream <OriansJ>now to figure out how to fix the shifts <OriansJ>one interesting note knight has arithmetic shifts left and right; then shift zero and shift one left and right (eg sr0 and sl1 will fill in with zero or one on the left or right side depending upon the instruction) <OriansJ>shr generally means fill with zeros but would totally be wrong for certain masking operations (which filling with ones would be optimal) <OriansJ>I really need your help deesix to get shifting right for aarch64 (It'll impact the aarch64 mescc-tools-seed port later) <OriansJ>nimaje: mescc-tools-seed built M2-Planet, I now have in a state where it should sucessfully build every M2-Planet test (except test/results/test0102-knight-posix-binary; which I still need to figure out why) <OriansJ>it is wrong but not in a way that would produce different output from the program nor different behavior from the program <OriansJ>essentially numerate_string is returning the wrong result <OriansJ>or more correctly, not sign extending a 32bit value to 64bits when loading a 4byte integer <OriansJ>(another possible bug in aarch64 we will have to deal with later) <OriansJ>integrating sign extension is ugly; I'll pick it up tomorrow (test0008 will still fail for knight-posix but everything else will work) ***ChanServ sets mode: +o rekado_
***rekado_ is now known as rekado
<nimaje>pulled M2-Planet, now ../bin/M1 -f amd64_defs.M1 -f libc-core.M1 -f kaem.M1 -f kaem-footer.M1 --LittleEndian --architecture amd64 -o hold fails (in mescc-tools-seed) <nimaje>lldb says SIGSEGV: invalid address (fault address: 0x241) M1`WHILE_hexify_string_9 + 161 -> 0x602cbb <+161>: lodsb (%rsi), %al <OriansJ>interesting; ../bin/M1 -f amd64_defs.M1 -f libc-core.M1 -f kaem.M1 -f kaem-footer.M1 --LittleEndian --architecture amd64 runs to successful completion and dumps everything to stdout <OriansJ>why would -o hold result in segfault? <nimaje>it also segfaults here without -o hold <nimaje>but then lldb says SIGSEGV: bound violation <OriansJ>M2-Planet doesn't even have a definition for AMD64 that includes ac <OriansJ>So how did that show up in the instruction stream <OriansJ>hold:4140 :Received invalid other; SHR_rax_cl <OriansJ>mescc-tools-seed's amd64_defs was not updated and the SHR_rax_cl was promptly followed by garbage x86 instructions <OriansJ>FYI, one can use M1 and hex2 to sanity check steps in the bootstrap <nimaje>not sure why git apply says error: corrupt patch at line 17 but it was easy enought to handapply <OriansJ>git diff it into a patch and I'll apply it <nimaje>so with https://0x0.st/iDPW.diff mescc-tools-seed builds here, but I got test/results/test0102-knight-posix-binary: NOT OK for some reason when testing M2-Planet with those binaries <OriansJ>yep test/results/test0102-knight-posix-binary failed but the reason for the failure is 0xFFFFFFFF on 32bit systems is -1 but on 64bit systems is 4+Billion and the range check in primary_expr_number resulting in the following difference in M1 output: https://paste.debian.net/1167580/ <OriansJ>it is a known bug; that I will be addressing later today and assuming everything else came out OK. no issues <OriansJ>if you are up for it we can also do mescc-tools-seed for x86 <OriansJ>(I just need to do some cleanup before we go down that route) <OriansJ>and revised mescc-tools-seed is now up (I needed to update the x86 checksums to match the new mescc-tools+M2-Planet source code) <nimaje>with https://0x0.st/iDZA.diff ../../bootstrap-seeds/kaem-optional-seed in mescc-tools-seed/x86 runs to completion (that mes-m2 update is probably unimportant, I just pulled master in all submodules) <nimaje>with those x86 binaries M2-Planet tests pass, but get_machine has empty output (maybe it had that too for the amd64 version), so no build binaries were run only the checksums checked <OriansJ>well the binaries are what generally are run <OriansJ>so if the binaries are identicial, one doesn't need to run them to know they work <nimaje>but --overrides works, so I will retest with GET_MACHINE_FLAGS="--override x86" and amd64 <OriansJ>odd, just ran make test-x86 and ./bin/get_machine seems to behave just fine on Linux <OriansJ>nimaje: does the gcc built version of get_machine.c work on FreeBSD? <nimaje>where would get_machine.c be? mescc-tools-seed/x86 only has get_machine.M1 if I see that right <OriansJ>mescc-tools is where get_machine.c is <OriansJ>because if that works; it means the issue is in test/common_x86/functions/uname.c and test/common_amd64/functions/uname.c <nimaje>yes, get_machine.c works correctly <OriansJ>you can use cgdb to p* the struct to see if the definition is different in the GCC compiled version <nimaje>freebsd uses 256 for those array sizes <OriansJ>nimaje: and I am guessing the Linux emulation doesn't change the structs when returning them to linux binaries <OriansJ>does atleast ./bin/get_machine --OS return the correct thing? <nimaje>and when I try to step through with lldb I get SIGSEGV: bound violation <OriansJ>nimaje: so char sysname isn't offset zero in FreeBSD's struct? <OriansJ>the struct will be in RAX/eax at the RETURN <nimaje>lldb doesn't seem to like that x86 get_machine, maybe I have more luck with the amd64 version <OriansJ>could the bound violation be because we are only allocating 1000bytes short of the size of the FreeBSD return? <nimaje>well, it segfaults on the first step, so the first instruction of that binary <nimaje>ok, it let's me debug the amd64 version <nimaje>rax seems to be 0x0000000000000000 <OriansJ>so FreeBSD is punting and just returning NULL <OriansJ>ok, what happens if we just replace the calloc(1, sizeof(struct utsname)) with calloc(10000, sizeof(char)); would that get FreeBSD to return something? <nimaje>but uname returns 0 on success, and that result should be in rax (?) and it is given a pointer <OriansJ>right because the kernel populates at the address pointed to in the syscall <nimaje>would the argument be in rax at the start of the function? <OriansJ>as the last pushed argument would still be in RAX at the time of the call <OriansJ>well that would be correct for a linux emulation layer <nimaje>yes, but why doesn't --OS output that? <OriansJ>That I do not have a good answer for <nimaje>oh, the amd64 version does work, should have tested that earlier <OriansJ>and does it return Linux or FreeBSD? <nimaje>so, only the x86 version doesn't work and lldb doesn't want to let me debug it <OriansJ>break on FUNCTION_uname; eax should have 0x804a000 <OriansJ>and after the int 0x80; it should have Linux <nimaje>as said earlier trying to debug directly stops with SIGSEGV: bound violation, maybe lldb just can't debug x86 binaries under linux emulation <OriansJ>nimaje: I think we have achieved enough FreeBSD success today <OriansJ>a fully working AMD64 mescc-tools-seed is more than enough <nimaje>yes, make test works in mescc-tools mes-m2 and M2-Planet with the amd64 binaries from mescc-tools-seed, so they should be working fine <OriansJ>now to fixup the int32 behavior on 64bit systems <OriansJ>(I probably did the aarch64 completely wrong) <nimaje>well, with https://0x0.st/iDNy.diff x86 mescc-tools-seed builds fine and mescc-tools and M2-Planet tests run fine, but mes-m2 segfaults and get_machine doesn't work (M2-Planet test need a lot time, especially if run twice (--override amd64 and x86)) <OriansJ>well I can get the M2-Planet test time down if I make M1 and hex2 faster <OriansJ>and I really should look into the mes-m2 segfault <OriansJ>also M2-Planet runs 176 tests (even only taking 1 second per is nearly 3Minutes) <OriansJ>and ./test/test1000/hello-x86.sh can take 15seconds <OriansJ>and self-hosting takes 3.7 seconds (does this 8 times or twice per architecture)