IRC channel logs

2021-10-15.log

back to list of logs

<oriansj>stikonas: minor correction M2libc is under the GPLv3 just like mes-m2 is
<stikonas>oh ok, then no problem with license
<oriansj>as per a previous discussion with janneke, it was thought that bootstrap support is too valuable to use a weaker license.
<stikonas>yeah ok, although bootstrap support is usable even for proprietary compilers as they can bootstrap gcc/glibc and then build on top of that
<stikonas>it's only those early binaries
<oriansj>I was more for lgplv3 to solve the GPLv3/CDDL license issue for live-bootstrap but given it was possible to bootstrap musl early, that benefit seemed mutted.
<stikonas>that have to be GPLv3
<stikonas>that issue can be solved or workarounded later...
<oriansj>indeed
<stikonas>either somebody finishes gash port to mes
<stikonas>and then it's basically automatically solved
<stikonas>or we need to recreate musl's configure file without running configure.sh script
<stikonas>it's a fairly simple script but still far beyond kaem's capability
<stikonas>anyway, having same license as mes libc has a lot of benefit of code sharing
<oriansj>or write a replacement in the mes.c scheme subset
<stikonas>gash port hasn't seen updates since june... https://git.savannah.nongnu.org/cgit/gash.git/log/?h=wip-mes
<stikonas>well, given that we have a fixed environment, replacement can be easily written in almost anything...
***Hagfish_ is now known as Hagfish
<gbrlwck>do i understand correctly that hex0 has an inverted syntax?
<stikonas>gbrlwck: what do you mean by inverted?
<stikonas>I guess what you mean is low endian
<stikonas>it shows bytes in the opposite way from hexdump
<gbrlwck>RD_A2 RS1_SP !16 LD would translate to `ld 16, sp, a2`
<stikonas>although, in the same way as diffoscope
<stikonas>oh you mean M1
<stikonas>it's not exactly inverted
<stikonas>it's what we invented to deal with "fixed word" architectures (as risc-v)
<stikonas>rather than each macro representing some bytes
<gbrlwck>ah, and i originally meant hex2 :)
<stikonas>we decided to add (OR) a few macros into combined macro
<gbrlwck>ahhh
<stikonas>so that is done by . character
<stikonas>and for implementation reasons (so that you won't need to backtrack)
<gbrlwck>so the . in front of a number means: please OR me to the other values with a . in front?
<stikonas>it adds 8 hex characters prefixed with . to the next (and not previous) hex string
<stikonas>yes, but crucially it adds them to the next undotted string
<stikonas>it's much easier to implement that in hex2
<gbrlwck>.00A0 .0B00 C000 => CBA0 ?
<stikonas>exactly
<gbrlwck>nice, thanks for the clarification!
<stikonas>and now the only common thing for all commands
<stikonas>for all instructions is the presense of opcode
<stikonas>e.g. LD or ECALL or ADDI
<stikonas>hence it has to be the last one
<stikonas>the order of everything else does not matter
<gbrlwck>yes, now this makes sense :)
<stikonas>we could have instead done something like
<stikonas>LD !16 RD_A2 RS1_SP NULL
<stikonas>where NULL is defined to be 00 00 00 00
<stikonas>(without the dot)
<stikonas>but then you have to end all instructions with NULL which is a bit wasteful
<stikonas>so this syntax is only used for risc-v
<gbrlwck>so RD_S4 MV (.000A0000 13000000) has the zero implied (because nothing gets OR'd to the first value)?
<stikonas>yes
<gbrlwck>cool
<stikonas>you can skip zero tokens
<stikonas>but on the other hand the position matters
<stikonas>so you have to specify whether it's destination register or one of the source registers
<stikonas>(and some commands, e.g. BEQ take only 2 source registers and not destination)
<stikonas>but it's visible in risc-v spec document
<stikonas> https://riscv.org/wp-content/uploads/2019/12/riscv-spec-20191213.pdf
<stikonas>(page 130)
<stikonas>this dotted syntax is slightly harder to implement in early stages but has the advantage that M1 code looks really similar to GAS syntax
<stikonas>if you look at e.g. AMD64 they use more complicated defines, e.g. DEFINE LOAD64_into_RBX_from_Address_RBX_Immediate8 488B5B
<gbrlwck>so the RISC-V way to do the bootstrapping is also kinda more beautiful?
<gbrlwck>(M0 works now btw)
<gbrlwck>stikonas: what would be the easiest approach to debug the creation of cc_riscv64?
<gbrlwck>kaem-0 now fails at `./riscv64/artifact/cc_riscv64 ./riscv64/artifact/M2-0.c ./riscv64/artifact/M2-0.M1`
<stikonas>well, it's not more beautiful, just different, earlier stages (hex1 in particular) are harder to write
<stikonas>hmm, let me see if I can spot anything
<stikonas>at least it should be the last stage where things could go wrong
<stikonas>btw for cc_riscv64 we just have cc_riscv64.M1 file, no more prototypes
<stikonas>I'll just go over those global registers (s*) and see if any of them are used uninitialized
<stikonas>gbrlwck: I think s7 might be uninitialized
<gbrlwck>you're right!
<stikonas>can you check if that's enough
<gbrlwck>it's not
<gbrlwck>:)
<gbrlwck>is RS1_S9 also not initialized?
<gbrlwck>and RS1_S6
<stikonas>I think s8 too
<gbrlwck>and RS1_S4 (again)
<stikonas>s6 is done on line 95
<stikonas>s4 in 93
<gbrlwck>true
<stikonas>so s7, s8 and s9
<gbrlwck>is MV the preferred way to initialize or should i LUI or does it not matter at all?
<stikonas>MV is good enough for small constants
<stikonas>LUI is needed for large ones
<stikonas>I think more than 2048
<stikonas>LUI is good enough until 31-bit
<stikonas>32-bits require even more work
<stikonas>cc_riscv64 does not implement 32-bit constants, only up to 31
<gbrlwck>cc_riscv64 seems to work now (at least it doesn't crash anymore)
<stikonas>gbrlwck: this is what M2-Planet emits for constants https://github.com/oriansj/M2-Planet/blob/master/cc_core.c#L648
<stikonas>so does the whole stage0-posix chain work now?
<gbrlwck>we're still bootstrapping, but it looks good
<gbrlwck>we're strapping those boots away :)
<stikonas>should be much quicker on real HW as opposed to qemu
<gbrlwck>well
<stikonas>although, I did some optimizations for qemu
<stikonas>so it went down from 15 min to 2 min
<gbrlwck>i read some things that RISC-V emulation is actually really quick?
<gbrlwck>i'm running a `guix pull` now on the side
<stikonas>well, qemu doesn't like early binaries (before blood-elf processes them(
<gbrlwck>and i just upgraded performance by mounting / from the nvme (and not sdcard)
<stikonas>so it inserts extra mprotect and similar syscalls
<gbrlwck>aha
<stikonas>hex2 was particularly affected
<stikonas>so I improved it a bit by moving io buffer to stack
<stikonas>(from "global variable", i.e. label after binary instructions)
<stikonas>and oriansj later ported same fix to other arches
<gbrlwck> https://termbin.com/vbv5
<gbrlwck>sharing because it's so nice ;)
<gbrlwck>oriansj: https://github.com/oriansj/stage0-posix/pull/59
<stikonas>gbrlwck: can you also fix M0 prototypes?
<stikonas>there is M0.S and M0.M1
<stikonas>best to keep them in sync
<xentrac>gbrlwck: RISC-V is significantly easier to write an emulator for than any popular architecture, and it has a few features that improve the performance of naïve emulators
<gbrlwck>stikonas: sure thing!
<gbrlwck>should compiled M0_riscv.S be identical to artifact/M0 ?
<stikonas>no, S file will be different
<stikonas>M0_riscv64.M1 should compile to the same thing
<stikonas>.S file has some extra ELF metadata (section tables, possibly even more debug info)
<stikonas>although, the actual program are (.text in ELF jargon) would be identical
<stikonas>and they should be producing identical output when running on .M1 input
<gbrlwck>what's the difference between hex2-0 and hex2-1?
<stikonas>gbrlwck: hex2-0 is written in hex1 code (so fairly close to machine language but with support for calculating jumps)
<stikonas>hex2-1 is built from C sources
<stikonas>(and supports features needed for any architecture, not just risc-v)
<stikonas>it's hex2*.{c,h} files in https://github.com/oriansj/mescc-tools
<stikonas>C version also has better error reporting, etc...
<gbrlwck>so, i can compile the new M0 with hex2-0 (but this gives a different hash to the version in artifacts), but comiling it with hex2-1 looks like this:
<gbrlwck>root@ubuntu:~/stage0-posix# ./riscv64/artifact/hex2-1 --architecture riscv64 --little-endian --base-address 0x00600000 -f ./M2libc/riscv64/ELF-riscv64-debug.hex2 -f M0.hex2 -o M0
<gbrlwck>Target label ELF_section_headers is not valid
<stikonas>that's because you used debug ELF header
<stikonas>that one would need M1 file processed with blood-elf and appended
<stikonas>ELF_section_header is a pointer to where debug info is stored
<gbrlwck>i see
<stikonas>and riscv32 would need even different headers if somebody does that port
<stikonas>even though almost all instructions are identical to riscv64
<stikonas>and of course we still haven't built mes for riscv64...
<stikonas>so there are a lot of things to look at...
<gbrlwck>so: M0 built iwth hex2-1 produces the same output as in artifact
<gbrlwck>they differ though from M0 built with hex2-0
<gbrlwck>(i compiled M0_riscv64.M1)
<stikonas>hmm, not sure why...
<stikonas>diffoscope might tell...
<gbrlwck>they are nearly the same size
<stikonas>maybe some zeroes at the end...
<stikonas>probably doesn't matter
<gbrlwck>i'll give diffoscope a try, since the machine is *much* faster now
<stikonas>those are prototypes anyway
<stikonas>although, diffoscope might not show too much if sizes are different
<stikonas>it will just say binary files with diffrerent sizes
<gbrlwck>i know; but should hex2-0 and hex2-1 produce identical output?
<stikonas>might be hexdump is more helpful
<stikonas>I think so
<gbrlwck>so we have a bug :)
<stikonas>well, first of all check that hex2-0 succeeded
<stikonas>echo $?
<stikonas>should show 0
<stikonas>M0 and M1 don't produce the same output though
<stikonas>but they would produce the same binaries after hex2 processing
<stikonas>(M0 and M1 produce files with somewhat different newline structures)
<gbrlwck>hex2-0 succeeded
<gbrlwck> https://termbin.com/405q
<gbrlwck>ah, so the hex2-0 version doesn't even produce useful output
<gbrlwck>stikonas: are RD_S4 LUI (in M1) and lui s4, 0 (in GAS) identical?
<gbrlwck>nvm
<stikonas[m]>Yes, they are
<stikonas[m]>Although, both do nothing on its own
<stikonas[m]>lui is mostly useful with non zero immediate constants
<stikonas[m]>gbrlwck: any reason for lui?
<stikonas[m]>And not addi/mv?
<gbrlwck>nope
<stikonas[m]>I think those are safer
<stikonas[m]>Lui would only initialise high bits
<stikonas[m]>Leaving bits 0-11 untouched
<stikonas[m]>Hmm, or does it zero low bits...
<stikonas[m]>Can't remember now
<gbrlwck>yeah, you're right
<stikonas[m]>BTW, you can force push into PRs
<gbrlwck>:)
<stikonas[m]>I saw you closed it
<gbrlwck>addi sp4, zero, 0
<gbrlwck>*s4
<gbrlwck>sets s4 to zero, right?
<stikonas[m]>Well,in gas you can use li
<stikonas[m]>But that's the same thing
<stikonas[m]>li is load immediate
<stikonas[m]>It's a pseudoinstruction
<stikonas[m]>Gas will correctly expand it into addi, lui/addi pair for larger numbers or even more instructions for 32-bit constants
<xentrac>you mean 64-bit?
<xentrac>lui/addi suffices for 32 bits I think
<stikonas[m]>Lui/addi is only enough for 31 bit unsigned
<stikonas[m]>Or 32 bit signed
<gbrlwck>`RD_S4 ADDI` should do the trick in M1?
<stikonas[m]>Yes
<stikonas[m]>In M1 ADDI and MV are identical, I just added two defines for same hex
<stikonas[m]>In Gas I think mv goes without immediate
<stikonas[m]>E.g. mv a0, a1
<stikonas[m]>Vs addi a0, a1, 0
<gbrlwck>updated the PR
<gbrlwck>wait
<gbrlwck>doesn't work yet
<stikonas>you'll need to create a new one after closing
<gbrlwck>is this "linux doesn't initialize registers to 0" thing a linux bug?
<xentrac>I suspect so. if they come from the pre-exec program, that's a potentially serious security leak
<xentrac>but maybe they come from somewhere else
<gbrlwck>they?
<xentrac>the register contents
<stikonas>I was told they don't come from pre-exec program
<stikonas>only some old linux versions had a leak with fp register
<gbrlwck>so register content after exec is undefined?
<gbrlwck>(my machine runs Linux 5.11.0; so not too ancient)
<stikonas>gbrlwck: well, that's what we found out yesterday
<stikonas>in any case if somebody wants to port it to baremetal later, it's better to initialize them
<gbrlwck>stikonas: i just saw that you asked in #riscv yesterday
<gbrlwck>i don't see how one or three instructions more would hurt verifiability
<stikonas>yeah, didn't get that much explanations or reasoning
<stikonas>no, it wouldn't hurt
<gbrlwck>should i post on some kernel mailing list?
<stikonas>I don't know, there is probably some reason somewhere
<stikonas>it's just that we don't know
<stikonas>it might be somewhere in ABI specifications
<gbrlwck>well, we might as well get an answer ;)
<gbrlwck>ok, i'll look into it before posting anywhere
<gbrlwck> https://github.com/oriansj/stage0-posix/pull/60
<gbrlwck>stikonas: could you check if i forgot anything?
<stikonas>looks fine
<gbrlwck>should i/may i include my name in the headers?
<stikonas>I guess you can
<gbrlwck>would that make me a copyright holder?
<stikonas[m]>Copyright is automatic by law, it's not up to headers
<stikonas[m]>Headers are more for information
<stikonas[m]>In law, I guess it would mostly depend on whether contribution is trivial or not
<gbrlwck>lol
<stikonas[m]>Which I guess is a bit of grey area
<gbrlwck>well, depending on judges ruling over whether initializing registers is a trivial thing to do or not
<oriansj>gbrlwck: any code you contribute, you instantly become a copyright holder (over what you contributed).
<oriansj>and your stage0-posix commits have been merged
<gbrlwck>yay
<gbrlwck>so i'll have to excuse myself for the weekend :) have a nice one, all of you!
<oriansj>and make clean test-riscv64 looks good on metal?
<stikonas>well, checksums matched with bootstrapped sha256sum
<gbrlwck>oriansj: i'm giving it a go
<oriansj>it is the ultimate sanity test, short of a new git clone on a freshly imaged machine.
<stikonas>and it also checks that sha256sum functions correctly
<stikonas>since checksums are checked once with bootstrapped sha256sum and once with system sha256sum
<gbrlwck> https://termbin.com/lq2r
<gbrlwck>i think we're good
<oriansj>gbrlwck: thank you, looks great