<OriansJ>nimaje: what if we set the Flags in the ELF header from 6 to 7 as ph_flags: PF-X|PF-W|PF-R = 7 <OriansJ>Linux might simply be marking the page executable as well; while FreeBSD isn't <OriansJ>the ndisasm output will be identical but the readelf -a output will be just 1 bit different (but we would have to update the checksums but oh well) <nimaje>trying to change the flags via objcopy gives me objcopy: elf_getscn failed: Missing or malformed ELF header while other tools working with elf files worked fine <OriansJ>you can change the flags by just editing the elf file and running test/test1/hello.sh <OriansJ>that is probably because these binaries don't have a section table or any section headers (nor symbol tables) <OriansJ>It just has a single program table entry and no null program table entry after it (which slightly violates the spec) <nimaje>yes that works, I didn't find the flags in the elf file, because I looked for ph_flags instead of just flags <OriansJ>So essentially FreeBSD respected the segment flags, marked the page as only read/write and when trying to execute; segfaulted <OriansJ>we probably should fix the flags comment to ph_flags to better match the ELF standard <nimaje>now of course there is a segfault in ./test/test1/proof1 so test/test1/hex.M1 (I think) needs fixing to set right OSABI and ph_flags <OriansJ>but test/test1/hex0.hex0 might need that fix <OriansJ>as the test is: ./test/results/test1-binary < test/test1/hex0.hex0 > test/test1/proof1 and then later ./test/test1/proof1 < test/test1/hex1.hex0 > test/test1/proof2 <nimaje>oh, yes there is also the inputfile, I should have writen test/test1/hex.M1 or test/test1/hex0.hex0 <OriansJ>M1 files don't have any of the ELF pieces, because they are a 2 step build; M1 first converts them into hex2 files; then hex2 usually glues them with the elf header and possibly an elf footer into the final binary <OriansJ>if you notice blood-elf generates the majority of the ELF data for files but requires one to use the debug elf headers <OriansJ>That is how M2-Planet binaries work with objdump -d <OriansJ>The good news is those 2 changes do not break the behavior on amd64 linux (i'll be checking arm shortly) <OriansJ>assuming everything passes, we probably want to mirror these updates in M2-Planet to ensure the binaries it produces work on FreeBSD too <OriansJ>and ARM passes as well; thank you nimaje for helping make mescc-tools more standards compliant <nimaje>I like stuff that works correctly <OriansJ>me too nimaje but it is hard to fix bugs I don't know about or lack a system expressing those bugs <OriansJ>It gives me something easy to fix usually <OriansJ>and please verify the latest commit works for you nimaje <nimaje>you missed OSABI in test/test1/hex0.hex0, not sure if more tests build another binary via their testbinary <nimaje>a quick grep suggests test1 test2 and test7 <OriansJ>easy to know as all elf headers must start with: 7F 45 4C 46 <nimaje>test/test7/hex1.hex1 also has flags as 6 which probably also should be a 7 <OriansJ>nimaje: please double check everything works now <nimaje>tests run fine and print ok for everything <OriansJ>good, so we finally got mescc-tools into a happy state for FreeBSD <OriansJ>up next M2-Planet (this should be much faster) <nimaje>well, install in mescc-tools fails to build kaem with a linker error: kaem.c:288: undefined reference to `get_current_dir_name' <OriansJ>ok, that is easy to fix as that is char *getwd(char *buf); without having to pass the buffer <OriansJ>excuse me as I have to put my son to bed (I'll be back *shortly*) <nimaje>(well, I probably should have gone to bed some time ago) <OriansJ>nimaje: we can always pick this up tomorrow <OriansJ>well atleast the right shift behavior is different than x86's (and GCC's) <OriansJ>right shifting 0xfffffffffffffffc became 0x3fffffffffffffff; were as on x86 (and in GCC) 0xfffffffc became 0xffffffff <OriansJ>So I think I used the wrong M1 definition for right shift for AMD64 (probably should double check armv7l and aarch64) deesix up for helping? <OriansJ>oh my; >> and << has different behavior depending on if the result is signed or unsigned <xentrac>I mean << can overflow which is I think undefined behavior for signed? <xentrac>Java and JS use >>> for C's unsigned >> ***ChanServ sets mode: +o rekado
<nimaje>yes, that patch fixes the kaem build <nimaje>for M2-Planet sha256.sh needs fixing (doing the same as in mescc-tools should be fine) and these test/common_*/ELF-*.hex2 files set OSABI to 0 should probably be 3 <nimaje>and test/results/test0008-amd64-binary segfaults lldb says SIGSEGV: invalid address (fault address: 0xf) -> 0x600210: movq %rax, (%rbx) <nimaje>ok, stepping through test/results/test0008-amd64-binary it seems like the second malloc fails in test/test0008/struct.c for some reason <nimaje>did I understand right that -> 0x600111: movq $-0x1, %rax should only happen if malloc() thinks it failed? (seems like that instruction is line 38 in test/common_amd64/functions/malloc.c) <OriansJ>nimaje: I am glad that the patch works for FreeBSD, now I can close out the mescc-tools work once the patch is up <OriansJ>and patch is up; now to figure out the M2-Planet problems <nimaje>fixing sha256.sh makes the test pass when not running the binaries, fixing test/common_amd64/ELF-amd64.hex2 to set OSABI to 3 gives that segfault, but no idea what the problem is <OriansJ>nimaje: I'm looking into test008 now <OriansJ>and a rough patch is up for the ELF headers and sha256.sh being fixed <nimaje>ok make test without running the binaries passes here <OriansJ>nimaje: could you use lldb to break FUNCTION_malloc and see what FreeBSD returns on the second call? <OriansJ>as it does a syscall with the value zero (to get the current malloc pointer) and then with the amount desired + the returned malloc pointer and only should return an error if it returns a malloc pointer we didn't expect <nimaje>well, I got the syscall results via truss https://0x0.st/iD-d.txt trying to set a breakpoint only gives me WARNING: Unable to resolve breakpoint to any actual locations. <OriansJ>odd nimaje as M2-Planet does create symbols for all labels; so break points for the entry to the function certainly should be possible <nimaje>readelf --syms test/results/test0008-amd64-binary has empty output <OriansJ>0x6000DB would be the breakpoint target <OriansJ>after the syscall, the result in rax is compared against the expected value (which on FreeBSD should have resulted in return -1) and probably did <OriansJ>but because the -1 value wasn't used until after the second malloc; that the error occured <OriansJ>as FreeBSD was returning a full page instead of the 32bytes that the syscall requested <OriansJ>I guess we need a C language lawyer to indicate is M2-Planet violating POSIX or is FreeBSD's rounding up to the page size valid <nimaje>(well, the kernel can only give a process memory in multiples of a page) <nimaje>afaik, posix doesn't define the syscall, only the library functions <OriansJ>well first MALLOC should be returning ZERO on error (first mistake of mine) <nimaje>ok, stepping through that malloc both times it hits -> 0x600111: movq $-0x1, %rax (so it returns -1?) <OriansJ>So one possible fix would only return NULL if the return value is less than the expected value (It'll waste a page for every malloc on FreeBSD) <OriansJ>nimaje: yep both mallocs returned -1 on FreeBSD because the sbrk returned a full page rather than just the 32bytes expected <OriansJ>although that would significantly limit the number of mallocs allowed on FreeBSD <nimaje>well, would that malloc be used anywere other than the tests? I don't think wasting a little bit of memory is an issue if it is only for those tests <nimaje>or is there one doing many mallocs? <OriansJ>nimaje: well the malloc.c files are the standard mallocs used by M2-Planet binaries <OriansJ>but it could seperate the sbrk inline assembly from the malloc workings <OriansJ>which would reduce the number of syscalls to the kernel and just require 2 globals in malloc.c for the tracking of the current brk pointer and the current sbrk pointer <OriansJ>So there may be some code duplication but I should have this ready for testing in a little bit (it is still ugly) <OriansJ>the brk function will be located at 0x6000DB and if everything is correct; it should be hit exactly twice on FreeBSD (once with 0) and the second time with (what FreeBSD returns +32) then when it gets the full page, it'll just allocate out of it until it is used up <OriansJ>(The AMD64 checksums are expected to fail; despite the tests passing) <OriansJ>anyone have questions, comments or concerns? <nimaje>next failing is test0106, it doesn't set the right OSABI when compiling ./test/test0106/cc1 <nimaje>ah and yes with that patch it does make exactly two brk syscalls <OriansJ>nimaje: I'll get to fixing that right now <nimaje>and after fixing OSABI in be_start() it segfaults, so probably also missing execute in ph_flags <OriansJ>nimaje: well building cc500.c and having it successfully self-build is the test <OriansJ>and it appears cc500 also would have the same malloc problem as well <OriansJ>as test/test0106/cc500.c uses void be_start() to hand encode in hex the elf header and several primitives <OriansJ>but that test could be disabled on everything not Linux without harm <OriansJ>nimaje: I look forward to your patch ^_^ <OriansJ>I'll fix the rest of the architectures' malloc while you work on that detail <OriansJ>(this will certainly show up when we hit mescc-tools-seed but memory inefficiency is less of a problem there) <fossy>whats the problem for mescc-tools-seed with this impl <OriansJ>fossy: if you look at the malloc in hex2 and M0; it shares the same definition as M2-Planet <nimaje>well, for that test it shouldn't matter if a little bit memory is wasted, so changing that je to jge or jle should be ok <OriansJ>so it'll have the same error on FreeBSD (we also have OSABI set to 0) <OriansJ>nimaje: mescc-tools-seed doesn't technically have a test because it is a single one stop bootstrap from hex0 to M2-Planet+mes-m2 <OriansJ>but wasting memory for building a 64KB binary shouldn't be too excessive <nimaje>so only counting bytes to find ph_flags in that huge unannotated block <OriansJ>it uses a hand written kaem (737bytes of hex0) and a hand written hex0 to build EVERYTHING <OriansJ>deesix: I might be breaking aarch64; could you double check I didn't do anything too stupid? <OriansJ>nimaje: hand counting bytes is something I had my fill of when I did hex0 andhex1 (hex2 reduces it considerably but still is required) and I am glad M0 only required me to do a lookup table for bytes <nimaje>ok, only OSABI and malloc were wrong https://0x0.st/iDHy.diff as it runs with that and cc1 and cc2 are equal that jle should be right; ph_flags is already 7 (should I have added a /s to the fun line?) <nimaje>no idea why it segfaulted lldb when I tried to step through it (that's why I assumed ph_flags are wrong) <OriansJ>even the slightest detail off and it breaks hard and not in the spot you would expect <OriansJ>and it uses arrays in very unusual ways that are very sensitive to change (hence why I made it an M2-Planet test to ensure arrays were rock solid) <OriansJ>I'll have your patch integrated once I get the malloc changes in place and then merge it all into a lovely commit <OriansJ>only knight architecture will not be updated (because knight doesn't require brk or sbrk at all) <OriansJ>note to self: I really need to speed up M1 and hex2 (as they are now a major part of M2-Planet's test time) <OriansJ>calling fgetc a bunch probably isn't very efficient <OriansJ>especially since fossy helped me get fseek and ftell working <OriansJ>nimaje: patch is up; please verify all tests pass and the checksums match <OriansJ>I just validated x86, AMD64, knight-native, knight-posix and armv7l (I need deesix to verify aarch64) for Linux as all working and correct <nimaje>yes, tests work here and the checksums match <OriansJ>now comes lots of binary fun: mescc-tools-seed <OriansJ>hex0 can be approximated with: sed 's/[;#].*$//g' $input_file | xxd -r -p > $output_file <nimaje>that malloc in kaem-optional-seed.hex0 seems ok