IRC channel logs

<OriansJ`>akkartik: the current armv7l work is here: https://github.com/oriansj/mescc-tools-seed/tree/master/armv7l/GAS

<OriansJ`>today's addition is cc_x86 written in armv7l assembly

<OriansJ`>I'll convert it to cc_armv7l.S this weekend and start working backwards

<OriansJ`>it was the slowest cc_x86 version I ever wrote (28hours of coding time)

<xentrac>awesome!

<OriansJ`>because armv7l cna't even do 12bit immediates right

<xentrac>aw

<xentrac>so you had to go back and redo a bunch of stuff?

<xentrac>to use constant pools or something?

<xentrac>(I haven't looked at the code)

<akkartik>OriansJ: ack

<OriansJ`>xentrac: well the first thing I had to do was move from a single constant pool of strings and globals and litter them around the functions. then when functions grew too large, I had to shove values inside of the functions themselves.

<OriansJ`>The lack of proper call and return primitives resulted in me having to add over 900 push/pop instructions just to approximate a proper call stack.

<OriansJ`>then the syntax oddities of # for comments but only when it is the first non-whitespace character of the line and @ for comments but only if it isn't the first non-whitespace character of the line.

<OriansJ`>then having to do .balign 4 a shitload because adr doesn't accept addresses that are not aligned to the word size (or worse if you go past 1024bytes away, then you have to be 8, 16 or 32byte aligned)

<OriansJ`>or more often you have to giveup and just do another_stupid_label_3: .word the_thing_you_need and try to stick it within 256 words of the ldr

<OriansJ`>I still think it is absolutely stupid that armv7l doesn't have integer division and floating point hardware is not guaranteed; so have to write/use a function that walks the bits otherwise you'll need more than 32loops to get the actual value.

<OriansJ`>akkartik: sorry about taking so long to respond, it is hard to squeeze freetime with a my son being so active. (mostly giving up alot of sleep to get this stuff done)

<xentrac>yup, sounds like ARM asm

<akkartik>OriansJ: not at all! I'm in the same boat. IRC isn't a fluent medium for me. And California is hot and on fire and smog-filled. All this hasn't made for a productive time. And yet, I'm bored. So here I am, looking for new things to poke at.

<dannym>janneke: tinycc's "doit" assumes cross compilation. I'm not cross-compiling. Running build.sh via doit thus doesn't work.

<dannym>janneke: Running build.sh manually does--but then I don't have a lot of the environment variables set.

<dannym>janneke: "sh cc.sh tccpp" "worked" the latter way but with a lot of 4 != 8 errors

<dannym>"sh cc.sh tccpp" took 15 min

<dannym>on banana

<dannym>On "sh cc.sh tccgen" I get a lot of 4 != 8 errors, and also get "unexpected size:8"; it takes 19 min

<janneke>dannym: yes, build-32.sh uses a 32bit cross compiler

<dannym>"sh cc.sh tccelf" lots of 4 != 8; takes 10 min and has:

<dannym>unexpected size:8

<dannym>ERROR assign: (assn-expr (p-expr (ident "tmp")) (op "=") (p-expr (ident "addr"))) size[0]:8 != size[0]:4

<dannym>unexpected size:8

<dannym>rank--: not a pointer: #<<type> type: signed size: 1 description: #f>

<dannym>"sh cc.sh tccrun" takes 4 min; no errors

<dannym>"sh cc.sh -gen" errors out: rm: invalid option -- 'g'

<dannym>That's because I didn't set tcc_cpu, because I didn't use doit, because it doesn't work

<janneke>weird, i think it works/worked for me (on banana)

<janneke>i guess part of my recipe is missing

<janneke>these errors are less than great, but x86 has them too

<janneke>at least some of them

<janneke>"less than great" / "terrible"

<janneke>on x86, building tcc takes about 8min, so 15min is not too bad on ARM, i guess

*janneke goes to have a look on banana

<OriansJ`>xentrac: yeah, doing cc_x86 in x86 -> 4626 lines of effort; only x86's half-brained division and lack of registers added any difficulty. AMD64 fixed the second half but unforunately did nothing for division and thus only net saved about 12 lines of effort. but armv7l -> 5995 lines of effort and I understand why people stick to C and above with arm. Comparatively, x86 assembly is so much nicer for general computation. armv7l's everything

<OriansJ`>is conditional and bit-shifting in every instruction is nice for dense bit twiddling but sort of steps upon its own feet by forcing 8bits of immediate as the primary limitation. while in comparision cc_x86 on knight -> 4395 lines of effort; with the only weakness of having only 16bit immediates. but the opcode space is virtually empty and adding support for 8/32/64bit immediates would enable denser binaries. also lack of cmp-set in

<OriansJ`>knight did make for ugly relational expression support in C.

<dannym>janneke: Huh... good to know

<janneke>dannym: so...i'm doing something like (testing that now)

<janneke>in src/mes: guix environment -l guix.scm --ad-hoc guile@2.2

<janneke>CC=arm-unknown-linux-gnueabihf-gcc ./configure; make

<janneke>then ./doit in tinycc

<dannym>janneke: The latter inside the guix environment -l guix.scm of mes ?

<dannym>janneke: That's weird

<dannym>janneke: I wouldn't have thought of trying that :P

<janneke>dannym: yes

<janneke>dannym: look at it this way:

<dannym>janneke: A lot of the later patches for mes-wip from me are to enable usage of mes' pre-inst-env so we don't need to be inside mes environment :)

<janneke>i'm developing mes, in the mes environment...

<janneke>:-) ...

<janneke>then run ./pre-inst-env mescc scaffold/hello.c to compile a c program

<janneke>then, tcc.c is also "just" a c program

<dannym>I don't run ./pre-inst-env mescc scaffold/hello.c inside the mes guix environment either--never did ;)

<dannym>But okay--makes more sense why we see different results then :)

<dannym>(Just like I don't run gcc inside gcc's guix build environment; that one is for building gcc, not for using gcc. It took me a long time to get into the habit of not confusing the two :))

<janneke>well, ... make check => ./check.sh => ./pre-inst-env ... mescc

<dannym>Yes, I can see it now

<janneke>dannym: re gcc: sure!

<janneke>so...

<janneke>we can do mes-based tcc development in the environment for tcc-boot0

<janneke>theoretically, (at least for x86)

<janneke>guix environment (@@ (gnu packages commencement) tcc-boot0)

<janneke>gives you all you need...which is mescc-tools, mes-0.22, nyacc

<dannym>I see

<janneke>BUT we probably want to run our own mes, not /gnu/store/*-mes-0.22/...

<dannym>Yeah

<dannym>The mini-CI I wrote on banana uses this to enter the respective environments (ugh! But it works):

<dannym>dannym@banana:~/src/mes-wip/mes$ cat enter

<dannym>#!/bin/sh

<dannym># guix environment --pure guix --

<dannym>#export GUILE_LOAD_PATH="/gnu/store/hngvis44nkpzfg7x52c9p50hq61f3nik-profile/share/guile/site/2.2"

<dannym>#export GUILE_COMPILE_LOAD_PATH="/gnu/store/hngvis44nkpzfg7x52c9p50hq61f3nik-profile/lib/guile/2.2/site-ccache"

<dannym>export GUILE_LOAD_PATH="/gnu/store/xq8ar9pdks5m56bv7rc7afbpyqbwbxir-profile/share/guile/site/3.0"

<dannym>export GUILE_COMPILE_LOAD_PATH="/gnu/store/xq8ar9pdks5m56bv7rc7afbpyqbwbxir-profile/lib/guile/3.0/site-ccache:/gnu/store/xq8ar9pdks5m56bv7rc7afbpyqbwbxir-profile/share/guile/site/3.0"

<dannym>exec ~/src/guix-master/guix/pre-inst-env guix environment -K --pure mes --ad-hoc nano less gdb git -- "$@"

<dannym>dannym@banana:~/src/mes-tinycc/tinycc$ cat enter

<dannym>#!/bin/sh

<dannym>exec guix environment --pure tcc --ad-hoc git -- "$@"

<dannym>I'm not sure how one is supposed to bootstrap Guix on Debian--what with all those guile libraries Debian doesn't have (guile-readline guile-json guile-zlib guile-lzlib); so I use the hack above O_O

<dannym>bootstrap Guix = compile Guix from git

<dannym>I used to do "guix environment --pure guix ~/src/guix-master/guix/pre-inst-env guix environment --pure mes -- $@" (not a missing newline), but that was not better either

<dannym>Anyway, it works, so whatever :)

<janneke>ah, debian used to have everything, the compression libs are quite new

<dannym>*stops ranting*

<janneke>i need to use the --pure on my x86_64 laptop -- seems not required for banana

<janneke>btw, i configure'd mes like so:

<janneke>GUILE=$(type -p guile) CC=arm-unknown-linux-gnueabihf-gcc ./configure

<dannym>On banana? But how does that CC exist?

<janneke>that's part of the mes environment

<janneke>the mes environment (see guix/git/mes.scm) adds a 32bit cross-compiler

<janneke>to build and test the mes c library using gcc

<dannym>I see. Indeed :)

<dannym>*Edits CI build script*

<dannym>Thanks

<dannym>I'd always use ./configure without any options (Inside ./enter of mes-wip)--works well enough apparently

<dannym>But is maybe the wrong gcc :)

<dannym>*kicks on mes-wip CI job*

<dannym>With CC=arm-unknown-linux-gnueabihf-gcc now :)

<dannym>Btw the reason I planned to use "~/src/guix-master/guix/pre-inst-env guix environment" instead of just plain "guix environment" is in order to use a git clone of Guix for all that stuff because I did plan on editing the mes package definition to use a different mes. Good to know that that maybe isn't necessary (... TODO: test if it is or isn't necessary) :)

<dannym>(Editing the mes package definition in that git working copy)

<dannym>In order to have tinycc use mes-wip

<janneke>yeah, sure

<janneke>once tcc starts to build, you want to switch over and build tcc-boot0, and need a guix git checkout for that

<janneke>dannym: i suppose "guix environment guix" is the best way to bootstrap guix on debian

<janneke>/bootstrap/build/

<janneke>we don't need to bootstrap i guess...

<OriansJ`>janneke: 0,0

<OriansJ`>bootstrapping is the one thing we do here.

<dannym>janneke: Yeah, but it's way too slow and also it adds stuff of the host guix to the environment--which I maybe don't want when building mes, tinycc etc. I'll keep my workaround for the time being :)

<OriansJ`>depending on guix to build guix is a bootstrapping problem.

<dannym>janneke: Slow because even with guix environment guix you'd still need another git clone guix environment pre-inst-env

<dannym>Inside

<dannym>OriansJ`: Yeah :)

<OriansJ`>if nothing else, we can punt leveraging mes-m2 and guile to provide the dependencies for bootstrapping of Mes.c

<dannym>janneke: Status: "sh cc.sh arm-gen" ... 4 != 8 ... unexpected size:8 ... took 8 min

<dannym>to finish

<OriansJ`>eg put a version of MesCC that works with guile in mes-m2 and just depend on guile and mescc-tools from apt then add a script that does the build

<OriansJ`>Then mes can be bootstrapped from mescc as the initial step and we can strip out the other complexity entirely.

<janneke>OriansJ`: hehe, yeah but that's just for development

<dannym>janneke: Status: "sh cc.sh arm-link" ... took 4 min to finish

<dannym>OriansJ`: I agree that that would be much better for actual bootstrapping

<dannym>OriansJ`: I took a lot of shortcuts in order to be able to develop on ARM now as opposed to in a year; among which is using Ubuntu ARMHF with guix installed from a tarball

<OriansJ`>janneke: setting up development should not be harder than bootstrapping, in fact it should be much easier by default.

<dannym>OriansJ`: The rube goldberg contraption I cobbled together annoys me every time I use it ;)

<OriansJ`>dannym: rushing usually ends up forcing us to waste effort and delays the final bootstrap.

<dannym>(In my case, newer Linux kernels have a bug not supporting SATA drives on Banana Pi M2 Ultra, and sure I could spend the next month finding this stupid bug--but I'd rather do mes development. I even bought a new Banana Pi M2U and a new drive; nope, SATA stops working after about 0.5 s)

<dannym>(I brought it up on the linux-sunxi ML, but so far no progress)

<OriansJ`>dannym: well good thing 8GB of eMMC is enough for everything

<dannym>On the other hand, using Banana Pi M2U's supplied images, SATA works at 230 MB/s (!!)

<dannym>So far for several months

<OriansJ`>well in theory mescc should not care about the kernel version and just produce valid ELF binaries

<OriansJ`>(also microSD cards come in 1TB sizes these days)

<dannym>OriansJ`: Hah, yeah, I did development using the SD cards etc before. I really don't like a repeat.

<dannym>OriansJ`: It's way too slow and I don't miss the random minute-long pauses when it finally flushes buffers

<dannym>OriansJ`: Maybe it's fine to use something else than Guix on that; but for Guix: no way

<dannym>I tried it

<dannym>janneke: Status: "sh cc.sh arm-asm" ... took 5 min to finish

<dannym>janneke: Status: "sh cc.sh tccasm" ... 4 != 8; took 6 min to finish

<janneke>nice

<OriansJ`>dannym: I am guessing the 1MB/sec read/write speed probably made for a terriable experience when you had ++GB to write

<dannym>OriansJ`: Yeah, I even got Samsung UHD SD cards later on; those are a little faster when writing; but still... uuugh. What is this? 1990?

<dannym>Meanwhile I can only say good things about SATA drives on R40 ARM CPUs. They are awesome!

<dannym>I boot almost all of Ubuntu and Guix from SATA; only u-boot is still on the SD card

<OriansJ`>dannym: well just spinning rust performance levels; SSDs are a hard drug to give up

<dannym>janneke: Status: "sh cc.sh libtcc" ... only ONE 4 != 8; took 7 min to finish

<janneke>dannym: if you ls -ltrF ~janneke/src/tinycc

<janneke>you see what doit+build.sh did for me

<OriansJ`>although if you are up for gigabit ethernet a sshfs filesystem is also an option if you need better disk performance with the added bonus of system image files

<janneke>and why during development at least we build tcc with ONE_SOURCE, using guile

<janneke>that takes about 6min for mes-tcc

<dannym>janneke: I see. Thanks! I'll try doit again after this run is done :)

<janneke>i also found that: yay, we have a mes-tcc binary that runs --help!

<janneke>that's sooo much better than my very first mescc-built tcc on x86

<janneke>sadly, mes-tcc does not compile a simple .c file...

<dannym>mescc: failed: M1 --LittleEndian --architecture armv7l -f /home/dannym/src/mes-wip/mes/lib/arm-mes/arm.M1 -f tccpp.S -f tccgen.S -f tccelf.S -f tccrun.S -f arm-gen.S -f arm-link.S -f arm-asm.S -f tccasm.S -f libtcc.S -f tcc.S -o tccpp.o

<dannym>When building tcc

<dannym>(So, the non-ONE_SOURCE version doesn't work...)

<dannym>I see

<OriansJ`>dannym: well when debugging M1 errors; first check are the files you told it to read actually there and that you have permission to write the target file

<janneke>dannym: it should! not sure what's going on here

<janneke>hmm...or at least, it could work

<OriansJ`>everything else should produce a useful error message to debug the cause of the failure.

<dannym>I'm sure it's my fault for not defining something in arm.M1

<dannym>We'll see

<janneke>yes, usually M1 problems are pretty trivial to find

<OriansJ`>dannym: it'll say illegal other and show you what wasn't defined

<dannym>It's probably because M1 wasn't found

<dannym>I cannot actually invoke M1 in mes-tinycc's build environment. Because I defined the build environment wrong

<dannym>Invoke it manually with full path and the arguments above works just fine

<OriansJ`> (Correction: Received invalid other;)

<dannym>Invoking

<OriansJ`>MesCC should probably check for the existence of M1 and hex2 in its path and print a meaningful error message if not found.

<OriansJ`>that way such errors are caught instantly and resolution can occur in a faster fashion.

<dannym>Aha, adding mescc-tools and guile@2.2 to my CI's tinycc build environment makes tinycc's "doit" work just fine, too.

<dannym>janneke: Status: "sh cc.sh tcc" ... took 6 min to finish; after that timing information it prints a lot of "ERROR assign: cval.i = s->last_line_num size[0]:8 != size[0]:4"

<dannym>Then 1 min

<dannym>Then + files=tcc.S

<dannym>+ mes-source/pre-inst-env mescc -g -o mes-tcc -L mes-source/mescc-lib -L mes-source/lib tcc.S -l c+tcc

<dannym>+ ./mes-tcc -I mes-source/include -I mes-source/include/linux/arm -static -nostdlib -nostdinc -c crt1.c

<dannym>build.sh: line 131: 8073 Segmentation fault (core dumped) $CC $CPPFLAGS $CFLAGS -static -nostdlib -nostdinc -c crt$i.c

<dannym>Finally I'm where you are :)

<janneke>dannym: \o/

<janneke>and...with a lot of understanding of all the quirks :)

<dannym>Trying to use actual guix master to configure, I get Missing dependencies: guild

<dannym>configure mes-wip

<dannym>CI job just faled

<dannym>failed

<janneke>did you add guile@2?

<janneke>=> guix environment -l guix.scm --ad-hoc guile@2.2

<janneke>we should actually make mes buildable with guile-3.0, esp. for debian

<dannym>The reason was me adding GUILE="(type -p guile)"

<dannym>works just fine when removing it again

<dannym>also with guix master

<janneke>great

<dannym>(no idea why)

<janneke>ah yes, i see

<janneke>well, guix puts guile-3.0 in front, i guess

<dannym>(gdb) r

<dannym>Starting program: /home/dannym/src/mes-tinycc/tinycc/mes-tcc -I mes-source/include -I mes-source/include/linux/arm -static -nostdlib -nostdinc -c crt1.c

<dannym>Program received signal SIGSEGV, Segmentation fault.

<dannym>0x22632e30 in ?? ()

<dannym>(gdb) bt

<dannym>#0 0x22632e30 in ?? ()

<dannym>Backtrace stopped: previous frame identical to this frame (corrupt stack?)

<dannym>argh! No working debug info

<janneke>yeah -- sorry :-)

<dannym>I can't even seem to "display asm", it needs says it debug info for that too

<dannym>needs

<janneke>we have minimal debug info (we could/should have a function call stack only)

<janneke>yes...so that's where the "lotsa debug printing" commit comes in

<janneke>i have been using that to bisect problems in mes-tcc

<janneke>by comparing the trace output produced from arm-unknown-linux-gnueabihf-tcc vs mes-tcc runs

<janneke>another approach would be to actually produce debug info with mescc+M1

<dannym>=> 0x1070bf8 <strstart+56>:

<dannym>0x01070bfc in strstart ()

<dannym>=> 0x1070bfc <strstart+60>: add r9, r9, r11

<dannym>0x01070c00 in strstart ()

<dannym>=> 0x1070c00 <strstart+64>: str r0, [r9]

<dannym>0x01070c04 in strstart ()

<dannym>0x01070c0c in strstart ()

<dannym>=> 0x1070c0c <strstart+76>: cmp r0, #0

<dannym>0x01070c10 in strstart ()

<dannym>0x01070ccc in strstart ()

<dannym>=> 0x1070ccc <strstart+268>: ldr r0, [r11, #-8]

<dannym>0x01070cd0 in strstart ()

<dannym>=> 0x1070cd0 <strstart+272>: ldrsb r0, [r0]

<dannym>0x01070cd4 in strstart ()

<dannym>=> 0x1070cd4 <strstart+276>: and r0, r0, #255 ; 0xff

<dannym>0x01070cd8 in strstart ()

<dannym>=> 0x1070cd8 <strstart+280>: sxtb r0, r0

<dannym>0x01070cdc in strstart ()

<dannym>=> 0x1070cdc <strstart+284>: cmp r0, #0

<dannym>Done by gdb; set logging on; set height 0; while 1; x/i $pc; stepi; end

<dannym>Still isn't crashed

<janneke>nice

<OriansJ`> janneke: if you just use blood-elf; you'll have full debug info for gdb and bt will produce useful information at the crash

<OriansJ`>as blood-elf is in mescc-tools and will allow you to pair down your elf-headers as well

<janneke>OriansJ`: sure, mescc -g uses blood-elf

<janneke>but that's only the call stack, right?

<janneke>it seems that dannym either got a corrupt stack or did not compile with -g

<OriansJ`>janneke: it is everything with a label not prefixed by _

<OriansJ`>so data, variables and every single function label you use.

<OriansJ`>you just need to remember --64 on 64bit targets

<OriansJ`>in fact the exact line is: if(('_' == i->name[0]) && !match(entry, i->name)) file_print("!2\t# st_other (hidden)\n", output);

<OriansJ`>if you were to do: readelf -a bin/M2-Planet | less you'll notice being prefixed by _ isn't enough to be hidden but rather needs to be explicitly hidden with --entry _$NAME

<OriansJ`>hence why the invokation on AMD64 is: blood-elf --64 -f test/test1000/cc.M1 --entry _start -o test/test1000/cc-footer.M1 (to hide _start but show _begin_*)

<dannym>=> 0x1036a64 <expr_const+84>: cmp r0, #0

<dannym>0x01036a68 in expr_const ()

<dannym>=> 0x1036a68 <expr_const+88>: mov r0, #0

<dannym>0x01036a6c in expr_const ()

<dannym>=> 0x1036a6c <expr_const+92>: moveq r0, #1

<dannym>0x01036a70 in expr_const ()

<dannym>0x01036abc in expr_const ()

<dannym>=> 0x1036abc <expr_const+172>: ldr r0, [r11, #-4]

<dannym>0x01036ac0 in expr_const ()

<dannym>=> 0x1036ac0 <expr_const+176>: mov sp, r11

<dannym>0x01036ac4 in expr_const ()

<dannym>=> 0x1036ac4 <expr_const+180>: pop {r11} ; (ldr r11, [sp], #4)

<dannym>0x01036ac8 in expr_const ()

<dannym>=> 0x1036ac8 <expr_const+184>: pop {lr} ; (ldr lr, [sp], #4)

<dannym>0x01036acc in expr_const ()

<dannym>=> 0x1036acc <expr_const+188>: mov pc, lr

<dannym>0x01036acc in expr_const ()

<dannym>Warning:

<dannym>Cannot insert breakpoint 0.

<dannym>Cannot access memory at address 0x22632e30

<dannym>I don't know whether I compiled with "-g"... I just used the tinycc build shell scripts.

<dannym>sp 0x122ccd8 0x122ccd8

<dannym>But as you can see above, some debug info is there, otherwise I wouldn't get the names inside <> either

<dannym>Or it's using the externals as a fallback--who knows

<janneke>dannym: i get this:

<janneke>gdb ./mes-tcc

<janneke>Reading symbols from ./mes-tcc...(no debugging symbols found)...done.

<janneke>(gdb) r -c mes/scaffold/hello.c

<janneke>Program received signal SIGSEGV, Segmentation fault.

<janneke>0x01025074 in gen_cast ()

<janneke>(gdb) bt

<janneke>#0 0x01025074 in gen_cast ()

<janneke>#1 0x01034d08 in expr_land ()

<janneke>#2 0xbefeff0c in ?? ()

<janneke>Backtrace stopped: previous frame identical to this frame (corrupt stack?)

<dannym>Is that the gdb from guix of from Debian?

<janneke>from debian

<janneke>and with the mes-tcc from 'doit', on banana

*janneke goes to apply lotsa debug printing and see

<janneke>hmm, that needs a patch for arm :-)

<janneke>ah, #if !__x86_64__

<janneke>in libtcc.c

<janneke>dannym: i've added two more WIP patches to tcc's mes-0.23

<janneke>dannym: and got first result for a test run diff

<janneke>dannym: diff -y 1 2 > first-result

<janneke>dannym: see banana:~janneke/src/tinycc/first-result

*janneke is afk for a bit

<janneke>dannym: oh, the "lotsa debug printing" commits need to be reverted again, of course

<dannym>Yeah. Thanks :)

<dannym>Which is 1, which is 2 :)

<janneke>dannym: oh oops, see compile.sh, link.sh: 32 bit gcc = 1, mescc = 2

<dannym>Ok, mes-wip on banana CI with the new change CC=arm-unknown-linux-gnueabihf-gcc just finished--it still works fine

<dannym>building

<dannym>and testing

***ChanServ sets mode: +o rekado

IRC channel logs

2020-09-12.log