<youpi>damo22: I dropped the vm_deallocate from the device_write RPC, and it goes much farther <youpi>e2fsck is however really not happy <youpi>the group descriptor somehow got corrupt <Pellescours>damo22: youpi: I already got thit issue in november (before my ext2fs changes) <Pellescours>you are were I was when I tested rump last time, fsck is not able to run and writes makes some timeout when doing poweroff <Pellescours>if you try to add a second disk (handled by rump) and create an ext2fs on it, it will crash <Pellescours>youpi: I tried extended attributes on ext2fs and it worked fine, for what I did no problem <Pellescours>yeah, I mounted a file-system using the --extended-attribute option. And actually I mounted at the same time the same fs with another translator not using --extended-attribute and it did not said that fs was mounted <Pellescours>But mounting twice the same fs without --extended-attribute is not possible because the second one says the fs is already mounted (or it mount it as read only I don’t remember) <Pellescours>I did not tried to mount with --extended-attributes t <youpi>I mean: was it really recording translators in xattrs? <youpi>e.g. dpkg-reconfigure hurd to re-create /dev entries etc. <youpi>to make sure tha tthe xattr code is raelly getting used <Pellescours>I didn’t checked really, but I mounted the fs by starting the translator manually, and I did it by mounting another fs stored in a 100M file, I did not tested with the root partition. I will test it <Pellescours>And I monunted the fs with a ext2fs that I built from source also <youpi>if the filesystem doesn't contain xattrs, the xattr code won't actually be used <youpi>so that's not actually testing anything <Pellescours>how do I know that the fs contains xattrs it’s an option to pass to fsck (or mkfs) right? <youpi>no it's an option to pass to ext2fs when mounting it *before* setting translators in it <youpi>did you actually read about what xattr were supposed to be used for in the hurd? <Pellescours>But I settrans with the xattr option, so it should use the xattr. I read the xattr, it’s to store the translator info inside the xattr <youpi>what do you mean by "I read the xattr" ? <youpi>what do you mean by "store the translator info inside the xattr"' <youpi>to be sure that you understand what we are talking about <youpi>xattr are used to store passive translator entries <youpi>if you are not doing anything about passive translator entries *inside* the mounted FS, you are not actually exercising xattr <Pellescours>I read the code, and as you said I understand that xattr is used to store the passive translator entries <youpi>ok, then did you set passive translator entries after mounting with xattr enabled? <Pellescours>So to test it correctly I need to start the translator as passive and check if it behave correctly after a reboot <youpi>and yes, testing with a root filesystem is mandatory before I'd be crazy enough with enabling this blindly for all hurd debian systems... <youpi>you need to set passive entries *INSIDE* the FS <youpi>the question is not about recording the passive entry *for* the ext2fs getting run <youpi>but telling the ext2fs with xattr to record passive entries *inside* the FS <youpi>and thus with xattr instead of the currently gnu-specific extension <Pellescours>I understand that you don’t want to enable it blindly, I want to help to test it correctly <Pellescours>but the --extended-attribute option of ext2fs does not set the passive entries inside the FS? <youpi>it just tells ext2fs to use xattr to record passive entries <youpi>it doesn't immediately converts all passive entries like that <youpi>it's only new entries that will use xattr <youpi>ext2fs will be able to read both passive entries using the old gnu extension and the new xattr <Pellescours>but I created a new fs, so it was empty. Wasn’t that enough? <youpi>if you don't create passive entries in it there's no xattr involved <Pellescours>Because I’m not sure to know how to creat passive entries <youpi>as I mentioned above, dpkg-reconfigure hurd <youpi>do you actually know what a passive entry is? <youpi>damo22: it looks to me like there are memory mapping issues <youpi>I tried to create a program that just reads 1MB from wd0 <youpi>that crashed the storeio, but also pfinet and the shell in which I started the program <youpi>there must be fishy things with physical memory <youpi>something like rump not taking care that the ahci dma might still be writing to the buffer <youpi>even if it was unmapped after the device_read <Pellescours>youpi: yeah it’s a node with a passive translator attached to it. I forgot to mension but I attached my translator to a file in /root. I will do the dpkg-reconfigure ;) <youpi>what do you mean by "my translator" ? <youpi>again, the passive entry thing is about being created *inside* the fs ran with xattr enabled <youpi>otherwise you're just setting a passive entry on your root filesystem <youpi>really, please make sure that you precisely understand each and every word that I'm writing <Pellescours>I will said all what I said. I created 1 file called test (to be the translator node), another called fs (of 100M with a mkfs.ext2 to it). I compiled hurd, and then I did a settrans test ext2fs --enable-xattrs fs <youpi>so you created a passive entry in your root filesystem to start an ext2fs with xattr enabled <youpi>what I'm saying is creating a passive entry *INSIDE* the ext2fs with xattr enabled <damo22>when you "settrans -p" you are telling the filesystem to record a passive entry against the file in the filesystem, is this always stored in the xattr? <youpi>currently it's store in a gnu extension of the ext2 format <youpi>the whole point is about storing it in an xattr instead <youpi>but to actually exercise it, settrans has to be done *inside* the ext2fs with xattr enabled <damo22>why are we talking about xattrs if we have bigger problems <damo22>lets not confuse the issue we are having by trying to use xattr new feature <Pellescours>your right I forgot that the translator node is not stored inside my fs file but in my test file which is in my / fs (so without xattr) <Pellescours>damo22: for nothing I was just telling that I tested xattr and I was actually not doing it correctly <damo22>so yeah, if you are testing xattrs you need to settrans -p on a file IN the filesystem you want to test <damo22>maybe we need to use async io so the buffer can be unmapped right at the end <youpi>I don't think async i/o will simplify things <youpi>a synchronous i/o is supposed to be over when it returns <youpi>it'd be useful to write down somewhere on the wiki the walkthrough of the function calls behind rump_sys_pread, to make sure what layers are involved and what they do, before entering the actual driver <damo22>ok i can mount it in userspace and do a gdb backtrace <Pellescours>I just tried the xattr properly and when trying to start a passive translator with extended attributes, It says "No space left on device". When I start it as active translator, it works. <Pellescours>Actually I did something wrong in my test, forget what I said <Pellescours>It seems to work actually (when I assign enough space to the fs that stores the xattr) <Pellescours>gtg, I don’t tested correctly yet so I don’t know if it works <damo22>how can i not have ext2fs running? <damo22>hmm how do i launch gdb on rumpdisk if the process has not started yet <damo22>ive attached a real disk to an ahci controller and rebuilt gnumach to ignore ahci <damo22>ive booted off ide, with a separate rootfs <damo22>ahh Kernel is already driving an IDE device, skipping probing disks <damo22>youpi: why are there two patterns of vm_deallocate in rumpdisk_device_{read,write}? if the write fails, don't you just deallocate then? <damo22>it seems you are deallocating the data before it has a chance to succeed and return <damo22>oh once you write the data to the disk you dont need the data buffer anymore <damo22>where is the buffer deallocated after rump_sys_pread? <youpi>the kernel does during the message transfer <youpi>the vm_deallocate in the error case surprised me as well <damo22>io-read.c in libdiskfs is confusing: it uses different buffers depending on the length it wants to read <damo22>so if *datalen contains a length already, it assumes the *data buffer is already allocated? <youpi>but you're allowed to reallocate it if you want <youpi>so the way rumpdisk does it should be fine as well <youpi>I'd say don't try too hard on that side, since what is happening is corruption in *other* processes <youpi>that's not supposed to happen with IPCs <youpi>so there's low probability that it's a misuse of the IPC <youpi>and rather a misconception in rumpdisk about pending dmas etc. <youpi>really, that's not supposed to happen that you do an IPC wrongly and then *another* process gets corrupted <youpi>it'd be extremely surprising that there'd be such bug in gnumach that'd have been unnoticed so far <damo22>so the bug is in use of rump or rump internals? <youpi>as I wrote already several time, my best guess is that a dma is actually still pending even when rump_disk_pread has returned <youpi>but since I have no idea how rump is actually using dmas, buffers, etc. <youpi>thus my question about having a walkthrough <youpi>to be sure how things are actually supposed to work <damo22>a walkthrough from where to where? <youpi>from the rump_sys_pread down to the driver that starts the dma <damo22>or put a breakpoint on the dma request <youpi>since there are probably request queues in between <damo22>it did many of these ahci_bio_start and ahci_bio_complete cycles, and then failed <damo22>but finding the last one is difficult <youpi>it's be useful to have the debugging symbols to see the parameters <damo22>is it supposed to create debug packages? <youpi>it just creates them automatically <damo22>oh the static binaries dont have debug symbols <youpi>that's probably not unfixable <damo22>should all the .a files keep the debugging symbols? <youpi>gnu_srs: procps 2:3.3.17-6+hurd.1 is in <Pellescours>maybe that's not related but default when I use an ahci disk in hurd it uses the same irq number than my ethernet card. And it's the pfniet buffer which is corrupted. Maybe it's just a coincidence <youpi>it's not a buffer which is corrupted, but pfinet itself <youpi>also the shell in which I had started the command <youpi>a shared irq is just about notification <youpi>that doesn't affect dmas at all <damo22>"Riastradh: once it returns (rump_sys_pread) its done (the dma transfer)." <damo22>Riastradh: (Each driver's interrupt handler _does_ return a bit saying whether the interrupt was for that driver or not, so the OS can use that to notice interrupt storms, although NetBSD just records it in evcnt(9) on some ports and ignores it on others.) <youpi>could it be that netdde and rump step on each other concerning the dma engine? <Pellescours>ok, that’s what I thought too, that’s just a coincidence <youpi>e.g. the write supposed to be for rump happens actually in a netdde buffer, thus overwriting completely unrelated stuff <damo22>is there a single dma engine that is shared in the system? <youpi>what I'd really like to know also is how exactly is used the physical memory allocated with the contiguous allocator <youpi>I have only old memorie of times when that could have been so :) <damo22>maybe we are using the same dma buffer in netdde and rump <damo22>apparently it's just memory, not a special set of four pages that they have to share for DMA <damo22>but vm_allocate_contiguous() is what we use right? <youpi>so far there was just one user <damo22>hmm dont we need to assert that pmin + size < pmax ? <damo22>otherwise the allocation may cross the pmax boundary even though it is supposed to be contained in [pmin, pmax] <youpi>not an assertion but making it fail, yes <youpi>ah no it's supposed to be already enforced by the selector configuratio <youpi>so an assert should be fine yes <youpi>I mean possibly the compiler makes it an it and thus truncated to 0 <youpi>but you can as well just print it inside mach <youpi>to make sure that it's received properly <damo22>the pmax gets passed in from netdde and pci-userspace as 0x100000000 <Pellescours>youpi: I added x-xattr-translator-records to my fstab, did a dpkg-reconfigure hurd, reboot, did again a dpkg-reconfigure hurd to ensure xattrs were enabled, reboot again and no issue :D <Pellescours>mount shows that the fs is mounted with x-xattr-translator-records <Pellescours>I also tested with a passive tranlator in a fs that I mounted with xattrs (a fs in a file) and the passive translator (hello) was started correctly when I accessed to the node after reboot <youpi>could you also check by mounting from linux that there are xattrs recorded ? <youpi>there are probably linux tools for that <Pellescours>yeah I removed th -m "./" but I still don’t see anything <damo22>what about getfattr -d /path/to/mnt/dev/sd0 <damo22>youre in linux so you have to mount it to a different folder you cant just use /dev right? <Pellescours>nothing, I also installed attr in my hurd VM and did the command from the running hurd and nothing <youpi>I don't think we have rpcs for xattrs yet, so getfattr wouldn't see them <Pellescours>ok, so I gonna recheck from my linux. I did a settrans -g /dev/zero && dpkg-reconfigure hurd to ensure that it was reset <Pellescours>I confirm on linux, no fattr seen to my /mnt/dev/zero <damo22>maybe you can inspect the inodes with debugfs <gnu_srs>youpi: Tks for the local build of procps. Shouldn't the actual bug, #987557 (and procps maintainer), be noticed about this too? <youpi>you already noticed the bug didn't you? <gnu_srs>Yes, but the maintainer does not seem to create updated versions frequently. Now when there is a kFreeBSD bug filed, #1005285, something might happen RSN. ***roptat is now known as Guest2562
<youpi>damo22: more precisely it's the backtrace to rumpcomp_pci_virt_to_mach that can be interesting <youpi>as of now it looks a bit bizarre <youpi>(inside rumpdisk_device_read) <youpi>but inside rumpcomp_pci_virt_to_mach <youpi>(gdb) p/x ((unsigned long* )$esp) [1] <youpi>Mmm, in that rumpcomp_pci_virt_to_mach, vpi_phys_addr <youpi>we'd need to make sure it didn't overflow <youpi>that alone can very well explain the craziness I had observed (the completely-unrelated pfinet crashing) <damo22>the vm_allocate_contiguous is returning a physical address above 4GB <youpi>the read function doesn't use allocate_contiguous <damo22>i put a print in vm_allocate_contiguous and when rump_dma thing called it i got an address above 0x100000000 <youpi>was the vm offset really that one ? <youpi>yes but I mean the vm offset <youpi>afaict the allocate_contiguous is used for something else <youpi>and the dma is done directly into the buffer to be passed to the IPC <youpi>which is a good thing actually <damo22>is pmax supposed to restrict the paddr or the vaddr? <youpi>but what I mean is to check that the vmaddress is what you expect <youpi>it seems you are expecting that rump_dma is getting allocations from vm_allocate_contiguous <youpi>I'm saying that it seems it's just getting what was vm_allocated from the rump_read <damo22>no i dont think so, i think it caches the dma address when dma is set up <youpi>"thinking" is never enough with code <youpi>I've seen so many times when my thinking was wrong <youpi>just because there's always something out of my radar <damo22>i will compile static binary with debug symbols, i cant seem to load the dynamic ones