IRC channel logs

2021-04-23.log

back to list of logs

<OriansJ>xentrac: if bash is doing CD before mkdir finishes and cmp before echo > file completes; a delay might not work as the cd command is just a naked call to sys_chdir and mkdir is just a naked call to sys_mkdir; sync would atleast force a consistent file system state prior to the cmp or cd
<OriansJ>we are talking about 6-8 instructions to do a mkdir syscall here and then another 6-8 instructions for the cd
<xentrac>OriansJ: you're right if the machine reboots in between the two commands
<xentrac>otherwise no
<OriansJ>I don't see how from the perspective of the kernel such an error can occur unless the sys_chdir syscall arrives before the sys_mkdir syscall does
<OriansJ>and on a single cpu system where sys_exit has to be called prior to the shell being informed of a process terminating to execute the next command;
<OriansJ>because wait is being used in bash to ensure it doesn't execute the next command unti lthe first is finished. unless bash special cases builtins and executes them without waiting for the result of a sys_wait to indicate process completion.
<OriansJ>\
<OriansJ>look I might be wrong here but I would like to understand how I am wrong so that we can prove that I am wrong and improve my understanding
<OriansJ>hence why I am explaining my understanding so that it be tested and revised where wrong.
<OriansJ>gforce_d11977: to your question sync is a trivial program, you just call void sync(void); from #include <unistd.h> or in x86 assembly put 36 in eax and do int 0x80
<OriansJ>or I could write a program that does mkdir followed by chdir and if the chdir fails but the mkdir succeeds, we might have a filesystem bug that sync inserted between the two would force correct behavior.
<xentrac>OriansJ: agreed
<xentrac>I think bash does special-case builtins but not in a way that should cause the cd to ever fail
<OriansJ>now there is a second possibility I can imagine for mkdir and chdir failing and that the mkdir program is buggy and received -1 from sys_mkdir and still returned success or the libc always returns zero regardless of what sys_mkdir returns.
<OriansJ>both of those should be very easy to spot in the source code
<xentrac>ooh, I hadn't thought of that
<OriansJ>just a simple grep -R mkdir should be enough to find it
<xentrac>if you can get the error to reproduce under strace -ff it should also be easy to see what went wrong
<OriansJ>but looking at meslibc's lib/linux/mkdir.c that isn't the case and in musl I doubt they would have made such an obvious bug
<OriansJ>xentrac: one would either have to bootstrap strace or include a static version of it to use
<xentrac>you could run the bootstrap build process on a machine that already has working Linux?
<OriansJ>xentrac: well live-bootstrap requires you to provide a working posix kernel
<OriansJ>but outside that the only binaries are those in my bootstrap-seeds
<OriansJ>everything else is built from source
<xentrac>I know
<OriansJ>and checksum'd for sanity checking along the way
<xentrac>I didn't mean you could require people to run it on a machine that already has a working userland
<xentrac>I meant that in order to track down a bug in it, you could run it on a machine that already has a working userland
<OriansJ>xentrac: fair
<xentrac>much as one might run it under qemu, for example
<OriansJ>and showing up 0.5% of the time under a qemu environment that is checksumed and known exactly seems more like a zebra type bug than a horse type bug
<OriansJ>Because the runs should be 100% identical; heck the only thing that might be different is the time
<xentrac>yeah, clearly something is behaving nondeterministically, and concurrency bugs are the most likely zebra there
<OriansJ>but we could eliminate that possibility with: -rtc "2006-06-17T16:01:21" or something consistent
<OriansJ>xentrac: how does one have concurrency bugs in a single core vm?
<xentrac>context switching on interrupts
<OriansJ>The only interrupts would be from the disk writing data and the clock as no network or user input is involved.
<xentrac>yeah
<xentrac>even if you set the clock starting time to something fixed, I think the clock interrupts in qemu are based on real wallclock time (so you can play games) and not VM time
<xentrac>normally Linux arranges for the clock to interrupt it 100 times a second, 4096 in recent kernels
<OriansJ>we could tune that down to 1 per second
<xentrac>that would increase the fraction of such nondeterministic bugs that go undetected for months or years, I suppose
<xentrac>and make them harder to debug
<OriansJ>xentrac: depends if our goal is to find and fix kernel bugs or avoid them to a level that the odds of someone building live-bootstrap hitting them is effectively zero
<OriansJ>because we are probably going to do a bootstrap kernel in the future and that is what live-bootstrap will run on until it is able to properly bootstrap an exact version of Linux.
<xentrac>ultimately we should replace Linux with a kernel small enough to be hand-auditable, but I don't have one in my shirt pocket at the moment
<OriansJ>xentrac: well fortunately we have bauen1 here to help us with that.
<xentrac>yay bauen1
<xentrac>I did learn something vaguely relevant this weekend
<OriansJ>xentrac: care to share with the rest of the class?
<xentrac>JLCPCB has a newish PCB assembly service
<xentrac>traditionally PCB assembly outsourcing takes six weeks and US$500
<xentrac>but lots of people have reported JLCPCB's service takes 1½–2 weeks and they've gotten boards for under US$100
<OriansJ>looks like they have a special off out too https://support.jlcpcb.com/article/146-2-for-1-4-layer-pcbs
<xentrac>I scoped out the limits and the "basic" parts list and it looks like a CPU built out of discrete logic (74HC14s, 74HC08s, CD4051s, 74HC573s, and so on) should be doable for under US$50, conceivably even under US$25
<OriansJ>nice
<xentrac>and maybe be 100 mm × 100 mm
<xentrac>like, a 16-bit CPU
<xentrac>you still have the problem of how to audit your memory hardware, and it would be pretty hard to detect if they replaced one of your 74HC14s with an FPGA programmed to emulate a 74HC14, but that seems like an unlikely threat at the moment
<xentrac>so it'd be a significant step in the direction of full auditability
<OriansJ>agreed
<xentrac>I have a version of Meta5ix recompiling itself to C in about 1.2 million instructions, which is fast enough to be practical on even a sub-MIPS 16-bit processor; I haven't yet gotten it to compile itself to amd64 assembly, let alone an assembly language for a 16-bit machine
<xentrac> http://canonical.org/~kragen/sw/dev3/meta5ix2c.m5
<xentrac>and I think I can probably beat that by a factor of 3 to 5
<xentrac>(speeding up a C program 3× by rewriting it in assembly is unusual, but interpreters are a special case)
<xentrac>I hadn't mentioned any of this stuff here because it's at best tangentially relevant to live-bootstrap and not very finished yet
<OriansJ>well I am interested in the idea
<OriansJ>and I do have some funds for my future hardware work available to help someone else do the work
<OriansJ>xentrac: bootstrapping hardware is the last stage of bootstrapping work anyone will ever have to do.
<OriansJ>So it is fair game to discussion and sharing
<OriansJ>although hardware by itself isn't useful without software and I would wish to encourage CPU ISAs that are human writable/readable; even if it means a few extra chips
<OriansJ>or less instruction density.
<OriansJ>because I refuse to write cc_oisc in oisc assembly
<xentrac>haha
<OriansJ>or cc_brainfuck in brainfukck
<xentrac>yes, and I recall you had a few choice words for the RISC-V instruction formats as well :)
<OriansJ>in a sane architecture doing load R0 0x1 and load R0 0x2 shouldn't require 2 different M1 DEFINE statements unless those statements are DEFINE load ... and DEFINE R0 ..
<OriansJ>heck look at x86 only int 0x80 is the only DEFINE needed for syscalls and the same for AMD64, armv7l and knight-posix
<OriansJ>but AArch64 and RISC-V nope a seperate DEFINE for each and EVERY single syscall supported.
<OriansJ>it delayed the untar.c work in M2libc
<OriansJ>because of instead of doing something sane like LOAD R8 !0x12 or LOAD R8 @0x12 or LOAD R8 %0x12; I had to write a brand new define and validate the hex encoding
<xentrac>I learned last week that the versions of strace and valgrind I was using don't support int $0x80 on amd64
<xentrac>I switched to using the syscall instruction
<xentrac>also all the system call numbers are different
<xentrac>no idea why
<OriansJ>xentrac: because int 0x80 isn't an amd64 syscall but an x86 syscall
<OriansJ>and the AMD64 abi and the x86 abi have different numbers for the syscalls because the AMD engineers profiled the most frequent syscalls and ordered them for AMD64
<OriansJ>hence 60 | sys_exit instead of 1 | sys_exit
<OriansJ>the x86 syscalls are numbered in the order Linus implemented them
<xentrac>oh, is that why they're in that order? why is it beneficial to put more frequent ones first?
<xentrac>amd64 Linux does support the int $0x80 system calls though, and with the i386 system call numbers
<xentrac>but (at least this version of) valgrind and strace don't
<xentrac>(even for 64-bit executables)
<xentrac>actually strace *does* support them for i386 executables
<OriansJ>xentrac: well mov eax !0x7F takes 2 bytes but mov eax @0x80 takes 3 and mov eax %0x8000 takes 5
<OriansJ>xentrac: I believe the int 0x80 behavior is only if LEGACY MODE is enabled; which was designed to run 32bit x86 binaries.
<OriansJ>hence why the register mapping for arguments corresponds with the x86 registers and not the amd64 registers which are different (%rdi | %rsi | %rdx | %r10 vs %ebx | %ecx | %edx | %esi)
<xentrac>yeah, I ran into that too, that was annoying as hell
<OriansJ>xentrac: but it makes perfect sense if the goal is running legacy binaries
<OriansJ>and not for new binaries
<xentrac>legacy binaries can't run in 64-bit mode though
<xentrac>this was a 64-bit binary
<xentrac>like I said strace and valgrind work fine for i386 binaries on amd64
<OriansJ>you really shouldn't ever use int with 64bit binaries
<xentrac>or for amd64 binaries using syscall
<xentrac>yes, strace agrees with you about that
<xentrac>it was really very unhappy with me
<xentrac>IIRC valgrind dumped core
<OriansJ>well normal compilation paths should never produce such monsters
<xentrac>indeed not, yet the kernel supports it
<xentrac>I guess if they're going to reorder the system call numbers anyway, they might as well reorder *all* of them, not just the ones that move across the 127-128 boundary
<OriansJ>even the kernel devs don't consider that a supported
<OriansJ>xentrac: well they did reorder all of them based on the same set they worked with
<xentrac>right
<xentrac>I was thinking "okay fine so we don't want uncommon syscalls to be above 127, but what's the benefit in moving exit from 1 to 60?" and that was my answer
<xentrac>does int $0x80 work in amd64 binaries on current Linux? they did eventually remove extfs and a.out support
<OriansJ>xentrac: well on hardened builds, hasn't worked for years.
<OriansJ>but on vanilla linux, not certain; it isn't a maintained code path
<xentrac>yeah
<xentrac>I think when they removed extfs it hadn't been a maintained code path for 20 years
<xentrac>but probably the timeline for that kind of thing is going to shorten
<OriansJ>depends if there is an in the wild exploit in that code path
<OriansJ>then it can be broken real quick
<OriansJ>with no one noticing
<OriansJ>except maybe you and other people who do crazy things with their compilers
<OriansJ>heck Linux broke ELF segment loading back in the 4.x series; which forced janneke and I to change how we wrote hex2 elf headers
<OriansJ>it wasn't a news story or anyone else really care; because it turned out we were abusing the ELF spec too much and the day came that the code path was simply *FIXED*
<xentrac>what was the problem?
<xentrac>I ran into a thing in stoneknifeforth where I was compiling executables with a load address of 0x1000
<xentrac>and then at some point Linux changed vm.mmap_min_addr to 0x10000 by default
<OriansJ>The binaries would segfault on load
<xentrac>I mean, what was the etiology, not the symptom?
<xentrac>tekknolagi ran into the problem in his fork of SKF and fixed it
<xentrac>rumor has it that 0x8048000 comes from Xenix
<OriansJ>if I remember correctly we loaded the same segement as read-only+execute-only and read/write and it applied both to become read/write/execute and then linux updated and it just became read/write and couldn't execute -> BOOM
<xentrac>oh, that makes some sense, yeah
<OriansJ>well all the bugs when you look at them make perfect sense in the bootstrap
<xentrac>although httpdito gets Linux to map the same page of the executable as RX and RW successfully. it's just that they're two separate mappings
<xentrac>I think StoneKnifeForth may actually do the same thing
<OriansJ>readelf -a foo produced garbage because blood-elf wasn't doing things wrong but when we fixed blood-elf readelf -a foo suddenly zero errors
<OriansJ>^wrong^right^
<OriansJ>So abuses of interfaces and ABIs might work for a while but in the end; it will blow up in our faces.
<OriansJ>hence why I've gotten more strict about doing things the proper way, even if it means more code or work.
<xentrac>oh, yeah, I agree that doing things the proper way — and even the mainstream way — is often safer
<xentrac>I think 0x1000 was a legal load address too
<xentrac>hmm actually SKF is mapping a single RWE segment too. readelf -a has no trouble with it, but objdump silently fails
<xentrac>httpdito, however, loads two segments, one "R E" and one "RW"
<xentrac>they are of course in the same page because the httpdito executable is 2060 bytes; it doesn't have space for two pages
<xentrac>so I bet SKF will fail on current Linux thanks to the same thing you are talking about above. thanks for letting me know!
<OriansJ>we all learn from each other to do what we never could do alone
<xentrac>I keep trying but nothing happens
<OriansJ>xentrac: I wouldn't say the collective results could be classified as "but nothing happens" if anything I'd say it would a fucking spectacular success.
<xentrac>no, I keep trying to learn, but nothing happens
<xentrac>I'm glad your efforts have been more successful --- spectacularly so, I'd say
<xentrac>sorry about the ambiguity
<xentrac>I certainly didn't mean that I keep trying to make you learn! that wouldn't even make sense
<OriansJ>xentrac: one can only learn things they honestly believe are worth learning.
<OriansJ>so the repeated trying to learn and failing is in itself an important lesson. You don't believe commiting that knowledge to memory is worth the effort.
<xentrac>that's one hypothesis
<OriansJ>So if you wish to keep that knowledge commit it to your external memory and cross-reference it in all the ways you can imagine it being needed.
<xentrac>yeah, in Dercuano I was moderately successful at that
<OriansJ>indeed, the cross-reference and linkage needed work but it certainly a good approximation.
<xentrac>Bush's Memex didn't have grep, but Augment/NLS had it by 01969, and I use it a lot on Dercuano (and Derctuo)
<xentrac>yeah, the other thing that really needs work there is executability
<OriansJ>xentrac: grep -iR is one of my favorite code tricks but I prefer a good index more when working on something structured for my external storage.
<OriansJ>emacs org-mode is my bread and butter for notes
<xentrac>doing calculations in a shell buffer and copying and pasting them into text (or in longhand and then photostatting them, as Bush proposed) is a damn poor alternative to a spreadsheet
<xentrac>org-mode is pretty great, yeah
<xentrac>and ^S/^R is often superior to grep for full-text search just because it's faster
<xentrac>^S M-r gives you incremental regexp search, too
<OriansJ>xentrac: I honestly haven't memorized regex search keybindings in emacs; just C-s and C-r
<OriansJ>but that might because my notes are heavily topic clustered
<xentrac>I use emacs for searching a lot of things I didn't write
<xentrac>M-. is pretty useful too
***lukedashjr is now known as luke-jr
<bauen1>OriansJ: the runs aren't quite identical yet the last time i looked, gforce_de1977 made a huge parallel run with make --debug and there where still some differences, mostly due to timing
<bauen1>xentrac: you would probabaly want to build a microkernel, starting with basically a library as os, building a basic hypervisor / message passing microkernel and start adding components until you provide enough POSIX for live-bootstrap
<bauen1>this way you can start from a baremetal monitor running at the highest privilege level, start writing helper functions, e.g. hex0 by hand, then load source code in some way, verify it, use it to compile a hypervisor/message passing microkernel, move yourself to a userspace task and start adding components as you need them, possibly reusing earlier code
<bauen1>and you won't need to reboot, which is quite messy
<bauen1>when you have enough POSIX interface implemented, you run live-bootstrap sysa ; at the end of which you reboot into linux
<gforce_de1977>stikonas: fossy: i dont want to be euphoric, but till now (264 CI-runs) there are zero errors. Seems that my mkdir() overloading changes something....will report more in 4 hours
<gforce_de1977>at the very beginning: http://intercity-vpn.de/bootstrap/helpers.sh.txt
<gforce_de1977>(but now with sync || true )
<bauen1>gforce_de1977: as someone pointed out above, you could also try to run it with strace to see if there are any differences / why the error happens
<gforce_de1977>bauen1: yes, i will do this on next run (so running each mkdir with a statically build mkdir)
<gforce_de1977>(... statically build strace)
<gforce_de1977>ok, in my massrun 311 out of 312 CI-runs are *OK*, there was only 1 failed build: http://intercity-vpn.de/bootstrap/bootstrap.log-multilog-17-1619151233.txt
<gforce_de1977>that means: we have a HEISENBUG: when i insert debug statement, the bug goes away 8-)))
<gforce_de1977>in the next massrun, i will output "/after/automake-1.15.1/build/automake-1.15.1/doc/amhello/amhello-output.tmp" if above error happens again....lets see what is inside
<bauen1>gforce_de1977: maybe you could dump the rootfs of every run ? or at least keep those around that do fail
<gforce_de1977>bauen1: i will think about it. no HDD and no networking. but i can just base64 everything to stdout/logfile.
<Hagfish>for any other project, that would be a really dirty hack
<Hagfish>it reminds me of the guy who got the original ipod to dump its firmware as an audio file in a custom format that he got it to play (with an ACE exploit)
<gforce_de1977>mass-test is already running: it will dump all files in /after to "screen": http://intercity-vpn.de/bootstrap/helpers.sh.txt | see default_src_compile()
<gforce_de1977>(only if there is an error)
<Hagfish># cat "builtins/pipesize.h" || true
<Hagfish>is that defending against the file not existing, or cat not existing?