IRC channel logs

<azert>solid_black: I figured out pl011 is just a 8502 compatible uart

<azert>same as mine

<azert>I fixed a few issues in the driver but still hardcoding everything overall

<azert>I get this output:

<azert>GNU Mach 1.8

<azert>Kernel command line: foo=bar

<azert>Booting in EL1

<azert>vm_page: page table size: 262144 entries (20480k)

<azert>vm_page: DMA: pages: 262144 (1024M), free: 213940 (835M)

<azert>vm_page: DMA: min:13107 low:15728 high:26214

<azert>Model name: Olimex A64-Olinuxino

<azert>module 0: module-hello ${host-port} ${device-port} $(task-create) $(task-resume)

<azert>1 bootstrap modules

<azert>panic ../kern/bootstrap.c:685: user_bootstrap: Cannot load user executable module (error code 6000): module-hello

<azert>sneek: later tell solid_black that we just need to figure out the last error! any clue

<azert>sneek: later tell solid_black we just need to figure out the last error! any clue?

<solid_black>azert: woohooo!! \o/

<solid_black>aarch64 mach booting on something that isn't qemu "virt" *and* is real hardware!

<solid_black>how awesome is that!

<solid_black>have you fixed the module's reg property?

<solid_black>other than that, it's likely that the boot module just gets overwritten

<solid_black>I've been loading modules from ROM so far, so there's no logic to mark the pages used by modules as used

<solid_black>do you have a branch where I can pull your uart additions?

<solid_black>does your board have a GICv2?

<solid_black>also is the latest commit (the one that enables E2H), or without that one?

<azert>With the latest

<solid_black>'cause you're booting in EL1 (which is actually better for now, until I figure out more about how virtualization works)

<solid_black>that would mean that your board doesn't support VHE/E2H

<azert>Yes it means that but it’s fine for me, I can start with this now

<solid_black>sure, we don't have virtualization implemented anyway

<azert>I’ll send you a link with the code, the modifications are minimal

<azert>Yes, everything is exactly the same as yours in hw terms

<azert>The pl011 is not named that way

<azert>I was wondering, should we go for a device_server in userspace for aarch64?

<solid_black>erm, no

<solid_black>there are devices inside Mach, and there are devices in userland

<solid_black>"time" is an example device inside Mach

<solid_black>you need a server to implement that

<azert>I think we should just support the basics in kernel space

<solid_black>well yes, that's the plan

<solid_black>Mach drives timers, interrupt controllers, and UARTs

<solid_black>everything else is up to userland

<solid_black>weeeell, some more little things go into Mach too

<solid_black>like psci that we already have

<azert>What is an interrupt controller in « device » terms?

<solid_black>well, it is a hardware device that you have to interact with

<solid_black>not much to it in "device" as in device_ RPCs

<solid_black>though there'll be some API for userland to register interrupt handlers, and to ack/eoi interrupts

<solid_black>which might be the same as the existing one on i386, or different, we'll see

<solid_black>so, did you fix the reg property?

<azert>What do you mean?

<azert>For the module?

<azert>To me the device_server looks just like another name directory I think I really need to understand it..

<azert>I’m not sure the issue is the reg property, for the module..

<azert>I fixed it I think. What are the 0x0 you added?

<solid_black>I mean the reg prop of the module, yes

<solid_black>I added 0x0's 'cause I saw you got confused by this

<solid_black>do you understand how reg and address-cells/size-cells work?

<azert>What should it be? Let’s say I have a module of a certain size loaded to a certain address

<solid_black>address-cells being 2 means you have to supply to 32-bit "cells" for the address

<solid_black>same for size

<solid_black>it's big endian, so if your address is 0x1234, you go 0x0 0x1234 for the address

<solid_black>of course if your address is > 4GB, the first cell wouldn't be 0x0

<solid_black>same for the size

<solid_black>does that make sense?

<azert>Ok that’s easy

<solid_black>yeah, otherwise Mach is just going to look in a wrong place

<azert>I think there is still an error

<azert>I’ll try to fix it

<azert>address-cells should be 2

<solid_black>yes, both should be 1

<solid_black>err, 2

<azert>what about the other property?

<solid_black>please see the first example in aarch64/BOOTING

<solid_black>the one that has /* More modules... */

<azert>Btw putting the module at 0x4000 is not an option in my board

<solid_black>try putting it at some high address

<solid_black>like high end of the RAM

<solid_black>it doesn't have to be 0x4000, that's just a ROM address in "virt"

<solid_black>so, does the example make sense?

<azert>Im afraid the device tree will be moved at a fixed address no matter what

<solid_black>that's also fine

<azert>Yes the example makes sense

<solid_black>so you say address-cells = 2, size-cells = 2, and then supply two cells for each

<azert>It’s not fine if it gets overwritten and currently we don’t pass this info I think

<solid_black>it's the module itself that gets overwritten, by Mach

<solid_black>simply because I haven't implemented not overwriting it

<azert>What about the device tree? Did you implement the reservation logic?

<solid_black>kind of, with a hack

<solid_black>see the "If the kernel itself or the DTB is loaded in this region of memory, exclude them form the heap." comment in pmap.c

<solid_black>but yeah, that's my point, you're stumbling on exactly those code paths that are in "proof of concept" status, and need to be improved

<azert>There is an error in your uart driver: on arm it should be iomem32

<solid_black>what's "iomem32"?

<azert>And the statut register is offset 5 (meaning 20) and not 4

<azert>Iomem32 means that you have to shift all offsets by two and do endianness magic if on big endian

<solid_black>but also yes, the UART driver is not in any serious state too, issues are very likely

<solid_black>uhh

<azert>This info is written in the device tree

<solid_black>certainly not in mine?

<solid_black>can you share your device tree?

<azert>You don’t have a reg-shift property?

<solid_black>no

<azert>reg-shift = <2>

<solid_black>I mean, your model is different, it's not PL011

<solid_black>and UARTRSR is at offset 0x4 for PL011, I just checked in the manual

<azert>Also reg-io-width

<solid_black>pl011@9000000 { clock-names = "uartclk\0apb_pclk"; clocks = <0x8000 0x8000>; interrupts = <0x00 0x01 0x04>; reg = <0x00 0x9000000 0x00 0x1000>; compatible = "arm,pl011\0arm,primecell"; };

<solid_black>this is mine ("virt"'s) pl011

<azert>Ok that’s weird

<solid_black>what's weird? it's just a different model

<azert>We need a very generic uart driver

<solid_black>we're going to have to write a separate driver for it

<solid_black>no, that's not possible

<azert>Nope

<azert>It’s the same interface

<solid_black>evidently it's not

<solid_black>that's the part I was trying to understand, and y'all explaned to me

<solid_black>UART is a generic term

<solid_black>they may be compatible in what format/protocol they speak over a wire, but the way they interact with the CPU differs

<azert>It’s true it’s generic but there is a standard: the 8502

<azert>And I think lp011 comply to it

<azert>This https://en.m.wikipedia.org/wiki/16550_UART

<solid_black>"The 16550 UART (universal asynchronous receiver-transmitter) is an integrated circuit designed for implementing the interface for serial communications. The corrected -A version was released in 1987 by National Semiconductor.[1] It is frequently used to implement the serial port for IBM PC compatible personal computers"

<solid_black>that's... obviously not the same as PL011 by Arm?

<solid_black>no really, I'm not going to pretend that I know anything about this, 'cause I don't

<solid_black>but your understanding doesn't hold up either

<solid_black>Arm PL011 is a very specific (but common?) UART model that is different from whatever was used on the IBM PC, and different from what your board has

<solid_black>and this is ok

<solid_black>raspberri pi has yet another uart model

<solid_black>Luca's board has yet another one

<solid_black>we're just going to have drivers for them all

<solid_black>and learn which driver to use from the device tree

<azert>My theory is that a driver like this one https://github.com/torvalds/linux/blob/master/drivers/tty/serial/8250/8250_early.c will be compatible with most

<azert>Check OF_EARLYCON_DECLARE

<solid_black>I'm not seeing "arm,pl011" among those OF_EARLYCON_DECLARE stanzas?

<solid_black>whereas I am seeing it in https://github.com/torvalds/linux/blob/fec50db7033ea478773b159e0e2efb135270e3b7/drivers/tty/serial/amba-pl011.c#L2602

<azert>Yes there is not

<solid_black>(no idea what "amba" is)

<solid_black>"This is a generic driver for ARM AMBA-type serial ports. They have a lot of 16550-like features, but are not register compatible."

<azert> https://github.com/raspberrypi/documentation/blob/develop/documentation/asciidoc/computers/configuration/uart.adoc

<azert>The PL011 is a capable, broadly 16550-compatible UART

<azert>But maybe this is wrong

<azert>If registers are incompatible then they will have different drivers

<solid_black>we've already seen that in the 0x4 vs 0x5 offset thing

<azert>Ok

<solid_black>any luck trying to put the boot module at the high end of ram?

<azert>I cannot do it now, I’ll tell you tonight

<solid_black>ok

<azert>solid_black: this is the dummy driver I implemented for my board https://paste.debian.net/1313459/

<azert>For things like IO barriers and raw writes, do we have them implemented already in some headers?

<azert>Doesn’t make any sense to keep them there

<azert>Also I stripped the endianness safety from them since it was too complex to implement for me

<azert>Of course, it’s stdout only for now

<solid_black>another KVM demo, from LWN: https://lwn.net/Articles/658511/

<solid_black>they implement PSCI in the kernel

<azert>Check the netbsd virtualization api: afaik it is based on a different model

<solid_black>any links / pointers?

<solid_black>I've only heard of bhyve

<solid_black>NVMM?

<solid_black>NVMM sounds very similar to KVN

<solid_black>let's ask a different question too, I guess: what would you want to run on top of QEMU/Hurd/Mach, and why?

<solid_black>to reiterate, "my" plan (it's not mine, that's how things have been working on L4s since forever IIUC) is not only about reusing the task/thread/vm APIs that we have for configuring virtual machines, it's also about reusing 90% of the implementation in the kernel too

<solid_black>a thread that's running in EL0 is not that different from a thread that's running in EL0 or EL1

<solid_black>there'd be some additional thread_state, and... that's it?

<solid_black>well ok, not that simple, you also have to virtualize interrupts (vGIC) and timers & stuff

<solid_black>the way FEAT_VHE works, the vm_map/pmap translation we're applying is already the 2nd stage translation, and 1st stage is disabled IIUC

<solid_black>so we'd turn off TGE, and the guest will be able to enable it's MMU, aka stack 1st stage translation before ours 2nd stage

<solid_black>s/it's/its/

<azert>solid_black: your plan is the good one

<azert>People (including me) will just run linux

<azert>I think it is important that you plan for the possibility of assigning real devices such as an Ethernet card to single vm instances

<solid_black>I don't think I have to do anything special for that at the Mach level

<azert>Not for now

<azert>Support for smmu is eventually due

<solid_black>the real userspace hypervisor (QEMU) would mmap the ethernet card into the VM task's "physical" memory

<solid_black>and cook up the appropriate nodes in the device tree

<solid_black>one trouble with this all btw is SErrors

<solid_black>qemu never raises them IIUC

<solid_black>but real hardware might, especially if you tickle some peripheral a wrong way

<solid_black>and IIUC there's not even a good way to correlate a received SError to what task caused it, since it's asynchronous

<solid_black>this is going to be an issue with userland device drivers, and even more of an issue with hardware passthorugh to VMs

<azert>I don’t know if this helps you https://www.linaro.org/blog/kvm-pciemsi-passthrough-armarm64/

<azert>It focuses on pci, I’m not sure all peripheral can be passed through since like you said the bus might lack support

<azert>solid_black: tried loading the module in high memory

<azert>this is the output:

<azert>GNU Mach 1.8

<azert>Kernel command line: foo=bar

<azert>Booting in EL1

<azert>vm_page: page table size: 262144 entries (20480k)

<azert>vm_page: DMA: pages: 262144 (1024M), free: 213940 (835M)

<azert>vm_page: DMA: min:13107 low:15728 high:26214

<azert>Model name: Olimex A64-Olinuxino

<azert>module 0: module-hello ${host-port} ${device-port} $(task-create) $(task-resume)

<azert>1 bootstrap modules

<azert>task loaded: module-hello 1 2

<azert>start module-hello:

<azert>then it hangs

<azert>looks like something is still broken? Any ideas?

<azert>Maybe the console for user programs is not wired up?

<solid_black_tmp>azert: try unifdef0'ing the debugging printfs in trap.c ("Sync exc from EL0!\n", "Sync exc from EL1!\n"), I've left them in for this very purpose

<solid_black_tmp>The console ("console" device) is not wired up for user programs indeed, but the tests use mach_print(), which just forwards to in-kernel printf(), so that should work

<azert>solid_black: I suspect it never makes it to el0

<azert>A bug in thread_bootstrap_return ? Just a suspect

<azert>I don’t see how else it would get stuck and not raise any exception from el0

<azert>Wait, where does it print this “start module-hello”? I cannot find this in the source code

<azert>Ok, boot_script_task_resume

<azert>That does nothing special. This bootstrap protocol is highly convoluted. I’m pretty sure it never lands in EL0

<azert>I’m quite confident it was made by a PhD student, or a master studenr

<azert>funny old stuff i just found: https://www.gnu.msn.by/prep/tasks.hurd.html

IRC channel logs

2024-04-08.log