IRC channel logs

2020-10-16.log

back to list of logs

<OriansJ>nimaje: what if we set the Flags in the ELF header from 6 to 7 as ph_flags: PF-X|PF-W|PF-R = 7
<OriansJ>Linux might simply be marking the page executable as well; while FreeBSD isn't
<OriansJ>the ndisasm output will be identical but the readelf -a output will be just 1 bit different (but we would have to update the checksums but oh well)
<nimaje>trying to change the flags via objcopy gives me objcopy: elf_getscn failed: Missing or malformed ELF header while other tools working with elf files worked fine
<OriansJ>you can change the flags by just editing the elf file and running test/test1/hello.sh
<OriansJ>that is probably because these binaries don't have a section table or any section headers (nor symbol tables)
<OriansJ>It just has a single program table entry and no null program table entry after it (which slightly violates the spec)
<nimaje>yes that works, I didn't find the flags in the elf file, because I looked for ph_flags instead of just flags
<OriansJ>my mistake of course
<OriansJ>So essentially FreeBSD respected the segment flags, marked the page as only read/write and when trying to execute; segfaulted
<OriansJ>we probably should fix the flags comment to ph_flags to better match the ELF standard
<nimaje>now of course there is a segfault in ./test/test1/proof1 so test/test1/hex.M1 (I think) needs fixing to set right OSABI and ph_flags
<OriansJ>M1 files never have elf pieces
<OriansJ>but test/test1/hex0.hex0 might need that fix
<OriansJ>as the test is: ./test/results/test1-binary < test/test1/hex0.hex0 > test/test1/proof1 and then later ./test/test1/proof1 < test/test1/hex1.hex0 > test/test1/proof2
<nimaje>oh, yes there is also the inputfile, I should have writen test/test1/hex.M1 or test/test1/hex0.hex0
<OriansJ>M1 files don't have any of the ELF pieces, because they are a 2 step build; M1 first converts them into hex2 files; then hex2 usually glues them with the elf header and possibly an elf footer into the final binary
<OriansJ>if you notice blood-elf generates the majority of the ELF data for files but requires one to use the debug elf headers
<OriansJ>That is how M2-Planet binaries work with objdump -d
<OriansJ>The good news is those 2 changes do not break the behavior on amd64 linux (i'll be checking arm shortly)
<OriansJ>assuming everything passes, we probably want to mirror these updates in M2-Planet to ensure the binaries it produces work on FreeBSD too
<OriansJ>and ARM passes as well; thank you nimaje for helping make mescc-tools more standards compliant
<OriansJ>and patches are up
<nimaje>I like stuff that works correctly
<OriansJ>me too nimaje but it is hard to fix bugs I don't know about or lack a system expressing those bugs
<OriansJ>hence why I love bug reports
<OriansJ>It gives me something easy to fix usually
<OriansJ>and please verify the latest commit works for you nimaje
<nimaje>you missed OSABI in test/test1/hex0.hex0, not sure if more tests build another binary via their testbinary
<nimaje>a quick grep suggests test1 test2 and test7
<OriansJ>easy to know as all elf headers must start with: 7F 45 4C 46
<nimaje>test/test7/hex1.hex1 also has flags as 6 which probably also should be a 7
<OriansJ>fixing that now
<OriansJ>ok and patches are up
<OriansJ>nimaje: please double check everything works now
<nimaje>tests run fine and print ok for everything
<OriansJ>good, so we finally got mescc-tools into a happy state for FreeBSD
<OriansJ>up next M2-Planet (this should be much faster)
<nimaje>well, install in mescc-tools fails to build kaem with a linker error: kaem.c:288: undefined reference to `get_current_dir_name'
<OriansJ>ok, that is easy to fix as that is char *getwd(char *buf); without having to pass the buffer
<OriansJ>excuse me as I have to put my son to bed (I'll be back *shortly*)
<nimaje>(well, I probably should have gone to bed some time ago)
<OriansJ>nimaje: we can always pick this up tomorrow
<nimaje>yeah, good night
<OriansJ>in the morning, let me know if this fixes your kaem build: https://paste.debian.net/1167365/
<OriansJ>I think I found a bug in M2-Planet
<OriansJ>well atleast the right shift behavior is different than x86's (and GCC's)
<OriansJ>right shifting 0xfffffffffffffffc became 0x3fffffffffffffff; were as on x86 (and in GCC) 0xfffffffc became 0xffffffff
<OriansJ>So I think I used the wrong M1 definition for right shift for AMD64 (probably should double check armv7l and aarch64) deesix up for helping?
<OriansJ>oh my; >> and << has different behavior depending on if the result is signed or unsigned
<OriansJ>tomorrow is going to be fun !
<xentrac>mostly just >>
<xentrac>I mean << can overflow which is I think undefined behavior for signed?
<xentrac>Java and JS use >>> for C's unsigned >>
***ChanServ sets mode: +o rekado
<nimaje>yes, that patch fixes the kaem build
<nimaje>good morning
<nimaje>for M2-Planet sha256.sh needs fixing (doing the same as in mescc-tools should be fine) and these test/common_*/ELF-*.hex2 files set OSABI to 0 should probably be 3
<nimaje>and test/results/test0008-amd64-binary segfaults lldb says SIGSEGV: invalid address (fault address: 0xf) -> 0x600210: movq %rax, (%rbx)
<nimaje>ok, stepping through test/results/test0008-amd64-binary it seems like the second malloc fails in test/test0008/struct.c for some reason
<nimaje>did I understand right that -> 0x600111: movq $-0x1, %rax should only happen if malloc() thinks it failed? (seems like that instruction is line 38 in test/common_amd64/functions/malloc.c)
<OriansJ>nimaje: I am glad that the patch works for FreeBSD, now I can close out the mescc-tools work once the patch is up
<OriansJ>and patch is up; now to figure out the M2-Planet problems
<nimaje>fixing sha256.sh makes the test pass when not running the binaries, fixing test/common_amd64/ELF-amd64.hex2 to set OSABI to 3 gives that segfault, but no idea what the problem is
<OriansJ>nimaje: I'm looking into test008 now
<OriansJ>and a rough patch is up for the ELF headers and sha256.sh being fixed
<nimaje>ok make test without running the binaries passes here
<OriansJ>nimaje: could you use lldb to break FUNCTION_malloc and see what FreeBSD returns on the second call?
<OriansJ>as it does a syscall with the value zero (to get the current malloc pointer) and then with the amount desired + the returned malloc pointer and only should return an error if it returns a malloc pointer we didn't expect
<nimaje>well, I got the syscall results via truss https://0x0.st/iD-d.txt trying to set a breakpoint only gives me WARNING: Unable to resolve breakpoint to any actual locations.
<OriansJ>odd nimaje as M2-Planet does create symbols for all labels; so break points for the entry to the function certainly should be possible
<nimaje>readelf --syms test/results/test0008-amd64-binary has empty output
<OriansJ>oh wait, test0008 is pre-blood-elf
<OriansJ>0x6000DB would be the breakpoint target
<OriansJ>I'll get the linux returns shortly
<OriansJ> https://paste.debian.net/1167485/
<OriansJ>after the syscall, the result in rax is compared against the expected value (which on FreeBSD should have resulted in return -1) and probably did
<OriansJ>but because the -1 value wasn't used until after the second malloc; that the error occured
<OriansJ>as FreeBSD was returning a full page instead of the 32bytes that the syscall requested
<OriansJ>I guess we need a C language lawyer to indicate is M2-Planet violating POSIX or is FreeBSD's rounding up to the page size valid
<nimaje>(well, the kernel can only give a process memory in multiples of a page)
<OriansJ>true
<nimaje>afaik, posix doesn't define the syscall, only the library functions
<OriansJ>well first MALLOC should be returning ZERO on error (first mistake of mine)
<OriansJ>hmm brk was removed in POSIX.1-2001
<OriansJ> https://www.man7.org/linux/man-pages/man2/sbrk.2.html
<xentrac>oh interesting
<nimaje>ok, stepping through that malloc both times it hits -> 0x600111: movq $-0x1, %rax (so it returns -1?)
<OriansJ>So one possible fix would only return NULL if the return value is less than the expected value (It'll waste a page for every malloc on FreeBSD)
<OriansJ>nimaje: yep both mallocs returned -1 on FreeBSD because the sbrk returned a full page rather than just the 32bytes expected
<OriansJ>although that would significantly limit the number of mallocs allowed on FreeBSD
<nimaje>well, would that malloc be used anywere other than the tests? I don't think wasting a little bit of memory is an issue if it is only for those tests
<nimaje>or is there one doing many mallocs?
<OriansJ>nimaje: well the malloc.c files are the standard mallocs used by M2-Planet binaries
<OriansJ>but it could seperate the sbrk inline assembly from the malloc workings
<OriansJ>which would reduce the number of syscalls to the kernel and just require 2 globals in malloc.c for the tracking of the current brk pointer and the current sbrk pointer
<OriansJ>So there may be some code duplication but I should have this ready for testing in a little bit (it is still ugly)
<OriansJ>nimaje: please check if this will get malloc working on FreeBSD https://paste.debian.net/1167503/
<OriansJ>the brk function will be located at 0x6000DB and if everything is correct; it should be hit exactly twice on FreeBSD (once with 0) and the second time with (what FreeBSD returns +32) then when it gets the full page, it'll just allocate out of it until it is used up
<OriansJ>(The AMD64 checksums are expected to fail; despite the tests passing)
<OriansJ>anyone have questions, comments or concerns?
<nimaje>yes, test0008 passes now
<nimaje>next failing is test0106, it doesn't set the right OSABI when compiling ./test/test0106/cc1
<nimaje>ah and yes with that patch it does make exactly two brk syscalls
<OriansJ>nimaje: I'll get to fixing that right now
<nimaje>and after fixing OSABI in be_start() it segfaults, so probably also missing execute in ph_flags
<OriansJ>nimaje: well building cc500.c and having it successfully self-build is the test
<OriansJ>and it appears cc500 also would have the same malloc problem as well
<OriansJ>as test/test0106/cc500.c uses void be_start() to hand encode in hex the elf header and several primitives
<OriansJ>but that test could be disabled on everything not Linux without harm
<nimaje>that sounds like fun to correct
<OriansJ>nimaje: I look forward to your patch ^_^
<OriansJ>I'll fix the rest of the architectures' malloc while you work on that detail
<OriansJ>(this will certainly show up when we hit mescc-tools-seed but memory inefficiency is less of a problem there)
<fossy>whats the problem for mescc-tools-seed with this impl
<fossy>?
<OriansJ>fossy: if you look at the malloc in hex2 and M0; it shares the same definition as M2-Planet
<nimaje>well, for that test it shouldn't matter if a little bit memory is wasted, so changing that je to jge or jle should be ok
<OriansJ>so it'll have the same error on FreeBSD (we also have OSABI set to 0)
<fossy>oh right
<OriansJ>nimaje: mescc-tools-seed doesn't technically have a test because it is a single one stop bootstrap from hex0 to M2-Planet+mes-m2
<OriansJ>but wasting memory for building a 64KB binary shouldn't be too excessive
<OriansJ>incase you are not familiar: https://github.com/oriansj/mescc-tools-seed
<nimaje>so only counting bytes to find ph_flags in that huge unannotated block
<OriansJ>it uses a hand written kaem (737bytes of hex0) and a hand written hex0 to build EVERYTHING
<OriansJ>deesix: I might be breaking aarch64; could you double check I didn't do anything too stupid?
<OriansJ>deesix: https://paste.debian.net/1167513/
<OriansJ>nimaje: hand counting bytes is something I had my fill of when I did hex0 andhex1 (hex2 reduces it considerably but still is required) and I am glad M0 only required me to do a lookup table for bytes
<OriansJ>atleast armv7l I know is working: https://paste.debian.net/1167514/
<nimaje>ok, only OSABI and malloc were wrong https://0x0.st/iDHy.diff as it runs with that and cc1 and cc2 are equal that jle should be right; ph_flags is already 7 (should I have added a /s to the fun line?)
<OriansJ>nice
<nimaje>no idea why it segfaulted lldb when I tried to step through it (that's why I assumed ph_flags are wrong)
<OriansJ>well cc500.c is very very touchy
<OriansJ>even the slightest detail off and it breaks hard and not in the spot you would expect
<OriansJ>and it uses arrays in very unusual ways that are very sensitive to change (hence why I made it an M2-Planet test to ensure arrays were rock solid)
<OriansJ>I'll have your patch integrated once I get the malloc changes in place and then merge it all into a lovely commit
<OriansJ>only knight architecture will not be updated (because knight doesn't require brk or sbrk at all)
<OriansJ>note to self: I really need to speed up M1 and hex2 (as they are now a major part of M2-Planet's test time)
<OriansJ>calling fgetc a bunch probably isn't very efficient
<OriansJ>especially since fossy helped me get fseek and ftell working
<OriansJ>nimaje: patch is up; please verify all tests pass and the checksums match
<OriansJ>I just validated x86, AMD64, knight-native, knight-posix and armv7l (I need deesix to verify aarch64) for Linux as all working and correct
<nimaje>yes, tests work here and the checksums match
<OriansJ>excellent nimaje
<OriansJ>now comes lots of binary fun: mescc-tools-seed
<OriansJ>although https://github.com/oriansj/bootstrap-seeds needs to be done first (as they are the first binaries that are wrong)
<OriansJ>hex0 can be approximated with: sed 's/[;#].*$//g' $input_file | xxd -r -p > $output_file
<nimaje>that malloc in kaem-optional-seed.hex0 seems ok