IRC channel logs
2024-01-03.log
back to list of logs
<pder>QEMU emulator version 8.2.0 (Debian 1:8.2.0+ds-1) <pder>Oddly on the same version of QEMU on the same distro- Debian unstable on a different machine I don't get the hang <stikonas>haven't made any progress with fixing stag0-posix issues in posix-runner, but at least I understood why mes fails to open any files <stikonas>turns out mes checks file descriptors against __FILEDES_MAX which is 4096... <stikonas>so there is no way it will be happy with pure UEFI file descriptors... <stikonas>perhaps I should look at abstracting descriptors to fix mes rather than stage0-posix issues <stikonas>since I'm getting confused with those... <rickmasters>stikonas: darn, I was going to mention that to you when you talked about whether you needed per-process file descriptors. <rickmasters>stikonas: sorry, it took me a long time to debug mes. it corrupts memory when you exceed __FILEDES_MAX... as you probably figured out <rickmasters>stikonas: I remembered I increased that but ran into another problem later and decided there was no option but to implement per-process fds <rickmasters>stikonas: I can't remember the issue off hand though, sorry <stikonas>rickmasters: it's impressive though how you managed to debug it all <rickmasters>stikonas: I'm curious how you figured it out. I had to run detailed comparisons of linux vs builder-hex0 with detailed debug output. <stikonas>rickmasters: well, I was just stepping in gdb dissassembly <stikonas>and gdb was returning larget file descriptor as is normal for UEFI <rickmasters>stikonas: I think it took me like a week or two to find it manually - no debugger in builder-hex0 :( <stikonas>rickmasters: there is no debugger in posix-uefi either <stikonas>so with something like qemu-system-x86_64 -cpu qemu64 -net none -m 4G -enable-kvm -drive if=pflash,format=raw,unit=0,file=/usr/share/edk2-ovmf/OVMF_CODE.fd,readonly=on -drive if=ide,format=raw,file=build/disk.img -monitor stdio -s <stikonas>then I inserted the following two lines in sys_call_3 function <stikonas>so qemu ran it until it got stuck in syscall <stikonas>then I started gdb, typed "target :1234" <stikonas>and jump out of the forever loop with break *address_of_next_line and jump *address_of_next_line <rickmasters>hmm, well tracking it back to memory corruption is what took me so long... <rickmasters>anyway, I think maybe what led me to per-process file descriptors was that a process was reading the same file multiple times concurrently <rickmasters>and there were two read offsets that needed to be maintained separately <stikonas>cause right now I'm just offloading it to UEFI... <rickmasters>but I was using a global table and returning an index to that table and the table only had one current offset being maintained <stikonas>I don't think stage0-posix opens multiple files at the same time <stikonas>I can skip stage0-posix with my kernel... <rickmasters>it was an issue with reading headers that read other headers <rickmasters>it would circle back to the same header and screw up the previous offset <stikonas>hmm perhaps I'll have to keep track all offsets <stikonas>and then reset then when restoring process... <rickmasters>does uefi allow opening the same file twice and maintaining different read offsets separately? <rickmasters>well, I'm assuming that's what you'd do - every open gets its own independent file descriptor from UEFI <rickmasters>so builder-hex0 has all the file descriptor abstraction/indirection you'd probably need if you want to use that for reference... <rickmasters>stikonas: you just need a table with the UEFI handle and current_offset <rickmasters>stikonas: but you've got to maintain the current offset on all I/O <stikonas>yeah, I'll take a look, though it might take some time <stikonas>probably makes sense to build it step by step <stikonas>first I'll start with per-process map of UEFI fds to small fds that are smaller than FILEDES_MAX <rickmasters>stikonas: maybe you can leverage UEFI to handle offsets. it has SetPosition, GetPosition apparently <stikonas>well, that's what I was doing up till now <stikonas>which is why I was saying it might make my job much easier <stikonas>if UEFI can open same file multiple times <stikonas>posix-runner has a lot of complexity hidden in M2libc... <rickmasters>I see you've got a good start on fork/exec/waitpid. That's black magic. <stikonas>it's a good start but I think it's still buggy... <stikonas>well, you probably spent weeks debugging those things... <stikonas>though C makes some lower level stuff a bit more complex... <stikonas>especially when you need to deal with things like stack <stikonas>anyway, right now I am able to run way more stuff directly than via kaem <rickmasters>yeah it was extremely difficult and I ran into a lot of problems getting it working. <stikonas>so there must be something still wrong in fork/execve/waitpid... <stikonas>(well, right now it is zeroied for the first process but not on execve or brk syscalls....) <rickmasters>there are some dark corners around copying and restoring chunks of the stck. it was somewhat miraculous that it works as well as it does. <stikonas>we are not using UEFI stack actually at all <stikonas>so UEFI stack is quite small, I think only 128 KiB <stikonas>so even in stage0-uefi I had to request another memory area that I'm using as stack <stikonas>rickmasters: at the very least after all this I'll better understand how builder-hex0 works... <stikonas>before that I didn't have enough time to dive into its internals <rickmasters>So, the one thing I'm not clear on... the memory models of processes in builder-hex0 assumes a specific memory range, but <rickmasters>how do you make the same assurances with UEFI and how do you know UEFI is not using the memory? <stikonas>rickmasters: I have to request memory from UEFI using allocatepages <stikonas>(which is abstracted into malloc in M2libc) <stikonas>so all the calls to malloc/calloc in posix-runner.c actually request memory from UEFI <stikonas>luckily though on AMD64 it is much easier to have position independent binaries <stikonas>i.e. M2-Planet uses stuff like lea rax, [rip+label] <stikonas>rather than loading absolute address of label <stikonas>so basically binaries are loaded into whereever UEFI allows <stikonas>(before that I also have to set model specific registers with the address of syscall handler) <stikonas>rickmasters: there were some annoying hangs on UEFI after returning from syscall (due to syscall chaning SS and CS registers) <stikonas>I guess this syscall stuff is a bit different on x86_64... <rickmasters>well, I wish I had circled back to clean up or document that code better. I bet its been a challenge to decipher. <stikonas>well, I haven't looked at the whole thing <stikonas>but the things I looked at where alright <stikonas>anyway, posix-runner.c would at least act as some kind of high level prototype to builder-hex0 if somebody wants to first understand it at higher level before diving in <stikonas>it's the wrong order chronologically but oh well <rickmasters>stikonas: so the way I was thinking of doing the UEFI posix OS was to read all files into memory, exit UEFI entirely, then take over interrupts, etc <stikonas>but Googulator suggested the current approach <stikonas>otherwise we need to implement more drivers <rickmasters>stikonas: I don't know enough about UEFI to know which makes the most sense; just thinking out loud <Googulator>an important caveat is that ExitBootServices also kills access to GOP beyond a simple framebuffer <Googulator>& I'm not sure if you can set up a text-mode "frame"-buffer using GOP that can be used afterwards <Googulator>so if we go for early ExitBootServices, that might require implementing an fbconsole in the kernel <Googulator>pder: try setting qemu machine type to "pc-i440fx-6.2" <Googulator>8.2 might have changed the default to pc-q35 instead <oriansj>I agree with rickmasters; you'll probably have an easier time with your POSIX skipping UEFI as much as possible. Use UEFI only to load the files you need into memory and ExitBootServices. VGA direct access isn't complicated (I even already have code for that) just use address 0xB8000 and Entries in the VGA buffer take the binary form CCCCCCCCBBBBFFFF (B is the background color, F is the foreground color and C is the ASCII character) <oriansj>just remember 80 rows and 25 columns by default <oriansj>then int index = ((VGA_COLS * row) + col) << 1; buffer[index + 1] = color; buffer[index] = character_you_wish_to_write; <Googulator>oriansj: that will only work if the GOP driver leaves the GPU in a VGA emulation mode <oriansj>(I susggest 0x0C for the color of errors and 0x0A for regular messages) <oriansj>well that isn't much more complicated <Googulator>& of course you can't just call VBIOS to set up a text mode again... because *there's no VBIOS* <Googulator>so to render text, you need code to convert that text into pixels <Googulator>UEFI itself has code for doing that - but it's part of boot services, not runtime <rickmasters>orianjs: well stikonas has done a lot so far so I'd probably see how the UEFI path plays out at this point. <Googulator>right now, it writes text data into VGA RAM using the previously set up text mode <Googulator>on top of that, on modern systems, there's no guarantee that the display output IP even supports a native text mode in hardware <Googulator>yesterday, I was running bare metal builds on 2 separate boards connected to the same display, switching between inputs to see both - I kept forgetting which board I was seeing... <rickmasters>Well, I assume we've got to move to a real kernel at some point, I can't imaging Fiwix being just be a UEFI proxy <Googulator>until I realized I can tell because one board (Intel GPU) used an angular font, the other (Nvidia) a lot more rounded <Googulator>because Fiwix was relying on the GPU's VGA compatibility hardware & VBIOS to render glyphs <Googulator>but it's good news that if you set up a graphics mode & tell Fiwix to use it, it can render glyphs on its own <Googulator>doing that pre-Fiwix without UEFI boot services is the challenge <Googulator>you have to encode a font in hex0/hex2 somehow, without blowing up file sizes too much <rickmasters>well, we've already got UEFI way past hex0/hex2, so any transition could leverage the highest compiler. <Googulator>are we at the point where we can directly load binary data from a file (i.e. a font)? <Googulator>because just having C doesn't help that much, it would just mean encoding fonts in C rather than hex2 <Googulator>unless you can split font data out of source files <Googulator>once fonts are solved, I agree it's fairly easy to implement an fbconsole using blt <Googulator>blt can take care of scrolling, as well as drawing new glyphs <oriansj>Googulator: we could build a font source to C data like convert cat.png -define h:format=bgra -depth 8 cat.h does <oriansj>or add the ability to embed binary files in M2-Planet output <oriansj>(which strangely would be quite simple to do) <oriansj>I believe it would be the new #embed keyword in C <rickmasters>I think trying to use UEFI up to compiling an launching Fiwix seems like the straightforward plan... <rickmasters>I was skeptical that would be possible but what stikonas has looks promising and avoids reimplementing video services before Fiwix. <Googulator>Fiwix itself will need changes to support scaling fonts, as you can't assume 8x8 fonts will be readable on a modern machine with UEFI <pder>Googulator: I am trying your suggestion of -machine pc-i440fx-6.2 now <pder>Is it ok to have machine kernel-irqchip=split -machine pc-i440fx-6.2 <Googulator>IIRC it's -machine pc-i440fx-6.2,kernel-irqchip=split <Googulator>we ignore the same memory blocks in Fiwix as before, they're just ignored later <Googulator>& none of that comes into play unless kexec-fiwix is also modified to pass additional high memory blocks to Fiwix <Googulator>(because right now, the e820 map provided by the BIOS is ignored, and the one hardcoded in kexec-fiwix is used instead) <pder>Interesting, it doesnt make sense to me why that would cause fiwix to hang. I'll let you know what the last text I see printed from Fiwix <pder>Would the amount of ram in the host machine have anything to do with this? <Googulator>one more thing you could try if this doesn't help is disabling KVM - if that fixes it, then it's probably your host kernel <pder>I actually have two machines both running Debian unstable and I only get the Fiwix hang on one machine. They should be identical kernels and qemu version, but I'll double check <pder>Maybe Fiwix can print more debug info <pder>This is the last thing fiwix prints <Googulator>memory: total=1048576KB, user=705304KB, kernel=342752KB, reserved=520KB <Googulator>kernel: text=325KB, data=67KB, bss=165KB, i/o buffers=262144, inodes=72537 <Googulator>it's mostly bare-metal related, but it also makes Fiwix's memory map a bit more conservative <pder>Unfortunately, the bare-metal-linux hung in the same spot <rickmasters>pder: you said bare-metal but you mentioned qemu before ? <Googulator>pder: it looks as if it's hanging when trying to init the secondary IDE slave device (which doesn't exist) <pder>hmm I would like to compare the output when Fiwix hangs and when it boots properly. BTW I just tried checking out master and cherry picking the PAE passthrough support commit and Fiwix booted and carried on <Googulator>Is there a CPU difference between the machine where it works, and where it doesn't? <pder>working cpu: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz. non working: Intel(R) Core(TM) i7-14700K <Googulator>hmm... removal of some legacy MMU behavior in Raptor Lake? <pder>I thought I read someone had else had trouble with live-bootstrap on a machine with 64GB of RAM, and thats what this one has <pder>it doesnt make sense though that qemu would be affected <Googulator>so BIOS (or rather, UEFI) will remap some physical RAM above 2^36 <Googulator>I'm pretty sure I did successfully bootstrap on a machine with 192GB RAM though <pder>are you talking about running on bare metal or through qemu? <Googulator>running a test now with the original Fiwix on the 192GB machine <pder>I did not try disabling kvm, will probably not have results until the morning <pder>interesting, well I am glad there is at least a workaround for me right now even if it doesnt seem like it should fix this issue <Googulator>I was also looking through the memory init code of our patched Fiwix - there are quite a few places where things could be going wrong <Googulator>e.g. where it's not clear whether an address is meant to be interpreted as physical or virtual <Googulator>but it's not clear why storing and ignoring (as opposed to discarding) e820 blocks above 2^32 would make a difference <Googulator>booted the exact same commit on the Zen 4 machine - no issue with Fiwix <pder>I should maybe try testing my RAM <Googulator>or maybe Intel dropped some legacy hack from their recent MMU implementations, which AMD still kept in Zen 4 <Mikaku>Googulator: Fiwix supports the QEMU/Bochs Graphics Adapter as a framebuffer console <Mikaku>regarding the pder's pastebin on the Fiwix hang, you need to know that the current serial console driver is buggy because it might not show the last kernel messages if it hangs <Mikaku>I plan to fix this bug in the serial console during the 1.5.0 version <Googulator>that will need to change for UEFI support, where all we get is a straight preconfigured linear framebuffer <Mikaku>Googulator: no, the kernel also supports VESA VBE 2.0+ compliant graphic cards <Googulator>as long as the bootloader configures a graphical mode first, right? <Mikaku>in the VESA VBE case though, the kernel expects the framebuffer address and setup in the Multiboot1 structure from the bootloader <Googulator>My main concern was that modern GPUs might not offer any native text mode at all <Googulator>With the current setup of BIOS -> builder-hex0 -> kexec-fiwix -> Fiwix, we're guaranteed to be in text mode when Fiwix starts <Googulator>And in stikonas' UEFI approach, posix-runner can get away with not dealing with a graphical framebuffer by retaining UEFI boot services until posix-runner hands off to a real kernel, and using boot services to render text <Googulator>But once that real kernel starts, it has nothing to rely on for drawing text <Googulator>GPUs on x86 possibly still have text mode & VGA emulation support in hardware, but on ARM or RISCV, it's common to have a pure graphical output block with no notion of text mode <Mikaku>for the hardware that only supports fb, we'll need a new kernel parameter to pass the fb memory address, resolution, bpp, font size, etc. more or less the same parameters that kernel expects from the bootloader when using VESA <Googulator>Can Fiwix currently do font scaling? (FB-only hardware often comes with high-DPI displays.) <Mikaku>Googulator: Fiwix only supports 8x8, 8x14 and 8x16 fonts which are the most common sizes with normal resolutions (800x600, 1024x768, 1280x748) <Googulator>8x14 is also what one would use for HD-style resolutions, right? <Mikaku>Googulator: hmm, I don't remember exactly why I included that font size, but for sure was not for such big resolutions because all my testing PCs are really old :-) <Mikaku>checking the console resolution in my Linux workstation at 1920x1080, it says 'frame buffer device 240x67' which looks like it uses a 8x16 font <matrix_bridge><Andrius Štikonas> Well, small font on 4k system is better than nothing <pder>Googulator: I tried a bootstrap from the master branch but removing the -enable-kvm option and that worked all the way through gcc 13 <janneke>...but at least guile didn't build reproducibly back then, and quite possible none of those binaries did? <janneke>it could also be that the static-binaries weren't upgraded since 20130105; i haven't looked into that <pder>Another update to my Fiwix hang issue. I tried adding some printk debugging to Fiwix so I added a couple of printk calls to ata_init(). Strangely when I ran the bootstrap it did not hang <pder>Something appears to hang between ata_init() and mem_stats()- it is also possible it is locking up somewhere else and I just never see the output from mem_stats() <rickmasters>pder: I've gathered from Mikaku (Fiwix author) that he uses this option for hard drive: <rickmasters>-drive file=${IMG},format=raw,if=ide,cache=writeback,index=0 <rickmasters>pder: perhaps you could modify rootfs.py to use the same and see if that helps <pder>rickmasters: I will try that. Those options are probably in kexec-fiwix somewhere? <pder>or are these qemu options? <pder>got it, thanks. have you encountered this issue before where Fiwix hangs somewhere after detecting drives? <rickmasters>I'm not sure about that index=0 - rootfs.py can use two drives so that might need to be adapted... <rickmasters>pder: yes, but never got stuck on that excepting the case where I found I needed -machine kernel-irqchip=split to make it work <pder>ok, i am definitely using kernel-irqchip=split <rickmasters>I wasn't able to get to the root cause of why that was needed. I just chalked it up to a likely obscure kernel bug but I'm not sure about that. <rickmasters>BTW, the problem was serial port not working - irqs not firing <pder>so I think that PAE passthrough patch is not the fix for my hang- I just have to add a printk call and its fixed :) <pder>I will check- should know in a couple minutes <rickmasters>pder: if not, you can stick an infinite loop into the code, and if it hits the loop you know it got that far. <rickmasters>pder: you can do that all the way into assembly by sticking in a .here jmp here infinite loop <rickmasters>pder: ... because an infinite loop WILL peg qemu and that's a signal you can use <pder>I just tried with those qemu drive options and got a KVM internal error during the mes build <pder>I havent ruled out possibly a RAM problem in this machine <pder>I went back to the master branch, ran a bootstrap until Fiwix hung and my cpu utilization for qemu is around 1-2% <pder>is a race condition possible at this point in the boot process? <rickmasters>pder: i'm not sure exactly where you are hanging but Fiwix runs linear on startup until it starts the first context switch <pder>Based on the output I see I never see the output of mem_stats() <pder>mem_stats is called from mm/swapper.c <pder>I believe Mikaku suggested that I might not be seeing all output if there is a hang <rickmasters>add this line of code to peg qemu: __asm(".byte 0xEB, 0xFE"); <rickmasters>it can narrow down exactly how far you are getting in the code - or you can use gdb as stikonas suggested <pder>I'm not familiar with the technique of using gdb with qemu, but I shall look into it <pder>I will try first your suggestion to see if I can narrow down where the hang is occuring <Mikaku>pder: if you are using serial console and you cannot see all messages on a hang, try incrementing the value NR_CB_QUEUE in include/tty.h <pder>Mikaku: thanks, I will try that <pder>Mikaku: Basically the last output I see is from ata_init(). I don't see output from mem_stats(0 <pder>I'm adding __asm(".byte 0xEB, 0xFE"); after mem_stats() and running again to see if mem_stats() completes <matrix_bridge><Andrius Štikonas> pder: basically if you run qemu with -s, you can attach to process inside qemu using target localhost:1234 from inside gdb <pder>got it, will definitely try <matrix_bridge><Andrius Štikonas> And forever jmp loop is good way to jump to specific location <stikonas>and mes now fails with a different error <stikonas>the not nice thing is that even simple stuff like mes --version doesn't yet work...