IRC channel logs

<youpi>damo22: the hang looks like out of memory indeed

<youpi>it seems like there are missing cleaning in mach calls or RPCs

<youpi>the thing is: some of those have to go to lowmem (<1G) for the kernel to be able to access to it directly

<youpi>and thus even if there are gigabytes free, if that lowmem is full mach is stuck

<damo22>youpi: ok thats good news i guess

<youpi>yes

<youpi>actually I had thought about something else: pci-arbiter and rumpdisk have to use mlockall()

<youpi>otherwise if they get swapped-out we're hosed

<youpi>and it made the problem worse

<youpi>looking at rumpdisk it indeed grows

<damo22>i need to print off_t from librump

<youpi>and locking its memory in makes it keep its memory busy :)

<damo22>i think its 4

<damo22>but setting it to 8 did not fix my strange file issue

<damo22>where else could off_t be misformed?

<youpi>all the path from device_read/write down to the driver

<damo22>i bought a w530 thinkpad for $60 that is stuck in a boot loop

<damo22>i can solder a socket onto one of the extra headers to flash it with coreboot externally

<damo22> https://doc.coreboot.org/_images/w530-2.jpg and schematics for CN100 http://zamaudio.com/mbox2/w530-jst.png

<youpi>really, as I said, make yourself a setup where you issue just a few reads at a very large offset, and put prints in the device_read function and the eventual ahci driver, to check that they match. if they don't, you know the issue is in between and you can step down in between

<damo22>well, its a bit tricky because i cant boot off rumpdisk and only issue a few reads

<damo22>i guess i need a copy of /

<damo22>81MB/s over usb3

<damo22>is that possible??

<damo22>11737760256 bytes (12 GB, 11 GiB) copied, 143.541 s, 81.8 MB/s

<damo22>youpi: how do i attach two disks? qemu-system-i386: -device ide-hd,drive=udisk,bootindex=2: Can't create IDE unit 1, bus supports only 1 units

<youpi>the qemu details are tricky

<youpi>it completely depends on how you set them up exactly

<damo22>-drive id=rootfs,file=rootfs10G.dd,format=raw,bus=0,unit=0 -drive id=udisk,file=/dev/sdd,format=raw,bus=1,unit=0

<youpi>it seems that my fixup was effective, rumpdisk remains at a few dozen MB

<damo22>that seems to work but gnumach detects sd0 sd1 sd2 hd0

<damo22>and hangs

<youpi>I'm lost, is that with q35?

<damo22>/extra/qemu/bin/qemu-system-i386 -M q35,accel=kvm -smp 2 -m 4096 -net user,hostfwd=tcp::8888-:22 -net nic -curses -drive id=rootfs,file=rootfs10G.dd,format=raw,bus=0,unit=0 -drive id=udisk,file=/dev/sdd,format=raw,bus=1,unit=0

<youpi>ok, I've seen gnumach find out some hd0, I don't know why, possibly the qemu q35 tries to emulate some ide as well

<youpi>but can't you just run the system on ide, and only one test disk with ahci?

<damo22>probably, i need to read the options again

<damo22>it shouldnt be this hard

<damo22>how do i refer to the ahci built in bus?

<damo22>bus=0 does not work

<damo22>ok i attached a second ahci

<damo22>controller

<damo22>nice, i have a second disk with a / partition

<damo22> http://paste.debian.net/plain/1208398/

<damo22>these two reads dont work properly because i can see the ls is invalid

<damo22>gah i cant get librump to print to console in the disk driver

<damo22>how do i disable stripping of libs in dh?

<damo22>i have no debug symbols in librump

<damo22>hey youpi i got an interesting crash http://paste.debian.net/plain/1208413/

<damo22>when the filesystem was mounted ro

<damo22>why is it trying to "drop a node" when the filesystem is readonly

<damo22>how do i load symbols in gdb for a shared lib that is not loaded yet?

<damo22>it does not load the debug symbols for librump

<damo22>(gdb) sharedlibrary /usr/lib/librump.so

<damo22>No loaded shared libraries match the pattern `/usr/lib/librump.so'.

<damo22>Thread 31 hit Breakpoint 1, rumpdisk_device_read (d=0x1000ff80, reply_port=187, reply_port_type=18,

<damo22> mode=0, bn=291622944, count=4096, data=0x1b832f0c, bytes_read=0x1b832e18)

<damo22>is count in units of bytes or sectors?

<youpi>damo22: dropping a node is just removing it from the page cache

<youpi>DEB_BUILD_OPTIONS=nostrip

<youpi>count is bytes

<damo22>ok my problem is that the dynamic libs dont go in the symbol table as entries visible with ldd

<damo22>i think librump is loaded with libdl

<damo22>so gdb does not know they are being used, until they get called, and then its too late to load debug symbols

<youpi>librump is linked in statically, see the makefiles

<damo22>even in rumpdisk? not rumpdisk.static?

<youpi>see the rumpdisk/Makefile

<damo22>wdstrategy (wd0) sizeof(daddr_t)=8 blkno=291622944 bp->b_blkno=1252511007243239424

<damo22>Thread 31 hit Breakpoint 1, sys_pread (l=0x10030900, uap=0x1b832d14, retval=0x1b832d0c)

<damo22>(gdb) p *uap

<damo22>... offset = {pad = -1012908032, le = {datum = 149310947328}...

<youpi>(gdb) p/x 291622944

<youpi>$1 = 0x1161d020

<youpi>(gdb) p/x 1252511007243239424

<youpi>$2 = 0x1161d02000000000

<youpi>that looks like a 32bit/64bit bug in the call

<youpi>something like one side things 32bit and the other side things 64bits for a parameter

<damo22>it could be i used %lld

<damo22>to print the value

<youpi>the compiler would have complained

<youpi>did you check the warnings?

<damo22>maybe i can break on wdstrategy

<damo22>yeah its my mistake with the printf

<damo22>wdstrategy (wd0) sizeof(daddr_t)=8 blkno=291360776 bp->b_blkno=1251385004257181696

<damo22>(gdb) p *bp

<damo22>$1 = {b_u = {u_actq = {tqe_next = 0x0, tqe_prev = 0x0}, u_rbnode = {rb_nodes = {0x0, 0x0}, rb_info = 0}, u_work = {wk_dummy = 0x0}}, b_iodone = 0x0, b_error = 0, b_resid = 0, b_flags = 1048576, b_prio = 2, b_bufsize = 2048, b_bcount = 2048, b_dev = 3, b_data = 0xfa16800, b_blkno = 291360776 ...

<damo22>it looks like with my patch, the offset in sys_pread is correct

<damo22>but it still doesnt explain my broken file

<gnu_srs>youpi: FYI, I've not shut down mahler, the host is down now due to a possible HW error.

<youpi>ok, thanks, I was wondering

<damo22>what happens if the bp->b_bcount is 2048 but the request was for 4096 bytes?

<damo22>could it be truncating every block in half?

<youpi>depends at what point you are looking at the data

<youpi>possibly the transfer is just not finished

<youpi>I wouldn't expect a bug that grave here, however, since rumpdisk is used in production already, and all that rumpdisk is doing is calling rump_sys_pread

<damo22>where does it check if the bytes_read == count in the request?

<damo22>i dont see that in block_rump.c

<youpi>that's missing indeed

<damo22>what is it supposed to do

<damo22>read again?

<youpi>no, that would fail again

<youpi>just return that value

<youpi>and it's up to the caller to cope with it

<youpi>I would mach_print("short read\n"); because taht's really not supposed to happen in practice

<damo22>okay

<youpi>possibly the file is simply at the very end of the read limit

<damo22>no short reads i can see

<damo22>its a bit weird that it calls diskfs_free_node() when doing ls

<youpi>do you have a backtrace?

<damo22>not sure how to get one

<damo22>gdb on ext2fs?

<youpi>for instance yes

<damo22> http://paste.debian.net/plain/1208434

<damo22>that was triggered in the ext2fs on my rumpdisk by typing ls -la on the directory with my strange file

<damo22>the file works normally without rumpdisk

<damo22>?--------- 0 root root 0 Oct 20 2019 hooks

<damo22>ext2fs: /dev/wdd0s3: warning: bit already cleared for inode 5767226

<youpi>it has 0 links indeed

<youpi>does that file show some links outside rmupdisk?

<damo22>how do i tell

<youpi>the first 0

<youpi>taht's the number of links

<youpi>diskfs_drop_node is normal, it's just getting out of the cache

<youpi>but it finding np->dn_stat.st_nlink == 0 is what is surprising

<youpi>and that's what ls says indeed

<damo22>well it must have links because it appears as a directory on non-rumpdisk and has files inside

<youpi>so I guess it's just reading the inode that fails

<damo22>i can get the same listing when i boot off a different controller

<youpi>did you get the block number of that inode with debugfs?

<damo22>not yet

<damo22>drwxr-xr-x 2 demo demo 4096 Mar 9 2020 hooks

<damo22>debugfs: blocks /demo/git/rumpkernel-0~20191130/.git/hooks

<damo22>23069329

<damo22>/dev/sd0s3 106811392 316526592 209715201 100G 83 Linux

<damo22>(106811392+23069329)*512

<damo22>66498929152

<youpi>note that block numbers are relative to the beginning of the partition

<youpi>also, here the problem is with the block is the block of the inode structure itself, not the inode content

<damo22>yes, i just added the start offset of the partition

<youpi>since that's what contains the numref, type of inode, etc.

<youpi>i.e. the imap command in debugfs, probably

<damo22>debugfs: imap /demo/git/rumpkernel-0~20191130/.git/hooks

<damo22>Inode 5767226 is part of block group 704 located at block 23068675, offset 0x0c80

<damo22>the first 0x134 bytes are populated of the inode, the rest are zero

<youpi>at offset 0x0c80 ?

<damo22>i just dumped the inode with dump_inode

<youpi>with which disk driver?

<damo22>the one that works

<youpi>ok

<damo22>its believable, since its just a directory with a few files

<youpi>my point is getting a block number and making sure you know you have the right block number

<youpi>the inode structure itself doesn't care about the number of files in the directory

<youpi>the bytes along the inode structure are just other inode structures that are part of the same inode block

<damo22>okay

<youpi>once you know where the inode structure is, you can try to read it with rumpdisk, and debug that

<damo22>so do i run debugfs with rumpdisk driver?

<youpi>for instance

<damo22>dump_inode on that inode returned a zero byte file

<damo22>with rumpdisk

<youpi>dump_inode dumps the content of the inode, not the inode structure itself

<youpi>if the inode structure is hosed, then dump_inode will read a random block

<youpi>it's the inod structure that you want to try to read

<youpi>since that's what contains the numref, type of inode etc.

<youpi>if *that* is hosed, there is no point in looking at dump_inode, it'll fail of course, because the block number will be bogus

<damo22>3200 a481 0000 6c0b 0000 7a05 d55d 54b3 ab5d ....l...z..]T..]

<damo22>3220 6c29 0e5c 0000 0000 0000 0100 0800 0000 l).\............

<damo22>3240 0000 0000 0000 0000 df01 2800 0000 0000 ..........(.....

<damo22>oops hex

<damo22>wrong offset

<youpi>at least that looked like an ext2_inode :)

<damo22>0760 0000 0000 0000 0000 0000 0000 0000 0000 ................

<damo22>1000 a481 0000 7904 0000 c705 d55d 49b3 ab5d ....y......]I..]

<youpi>(modulo the byte inversion)

<damo22>why is 1000 the next value after 0760?

<damo22>i dont understand block_dump

<youpi>it means it's all 0 in between

<youpi>but why 0760 ?

<damo22>it said my offset was 0xc80

<youpi>which block_dump command did you use exactly?

<youpi>then it's 0c80

<damo22>debugfs: bd 23068675

<youpi>hexdumps always use jex

<youpi>hex*

<damo22>or do i add the offset 0xc80 to the block

<youpi>it's an offset *inside* the block

<damo22>before dumping?

<damo22>ok

<damo22>so its all zero then

<damo22>but it looks like a short read

<youpi>while doing exactly the same with non-rump doesn't show zeroes?

<damo22>because there is no *

<youpi>the ponit is trying to read exactly the same block both ways

<damo22>ok will do

<youpi>and check whether it's different

<youpi>and then debug exactly that

<youpi>since taht's where things go wrong

<youpi>what do you mean by "there is no *" ?

<damo22>usually there is a line by itself with

<damo22>*

<damo22>when there are all zeros below

<damo22>but i got two sequential lines with missing offsets between

<youpi>I thought you said there was no short read in rumpdisk_device_read?

<youpi>I mean, debugfs cannot know about a short read if rumpdisk_device_read is not reporting it

<damo22>what do i return in the case of a short read

<damo22>EIO?

<youpi>as I said, just the amount of bytes

<youpi>but *ALSO* put a mach_print

<youpi>since taht's not suppose to happen

<damo22>ok yeah i did

<youpi>and with a mach_print you're sure to know about it

<youpi>if you don't get a mach_print then it means it doesn't happen

<damo22>wait, D_SUCCESS

<youpi>it's as simple as that

<damo22>or number of bytes

<youpi>yes, success, and put the size where it shall be

<damo22>ok

<damo22>errr its dumping different data

<damo22>completely

<damo22>its like the offset is somewhere else in the disk?

<youpi>so that's a reproducer

<youpi>and you can try to debug exactly that with e.g. a mere dd command

<damo22>hmm cant find the actual offset

<damo22>does ext2fs count blocks from the start of the partition?

<youpi>yes

<damo22>sudo dd if=/dev/sd0s3 of=testfrom3 skip=23068675 bs=4096 count=1

<damo22>got it

<damo22>XXXX libstore: dev_read recnum=291360792

<damo22>(291360792-106811392)/8

<damo22>23068675

<damo22>but the contents are different

<damo22>ok:

<damo22>testfrom3withoutrump-recnum-184549400

<damo22>testfrom3withrump-recnum-291360792

<damo22>so rump is not subtracting the start of the partition

<damo22>?

<youpi>why would it be? it doesn't even know about partitions

<damo22>storeio is not working correctly then

<damo22>root@zamhurd:~# showtrans /dev/wdd0s3

<damo22>/hurd/storeio -T typed part:3:device:@/dev/rumpdisk:wd0

<youpi>?

<youpi>why would storeio not work?

<damo22>the part:3 thing

<youpi>well, what it does is precisely add the offset of the partition

<youpi>but I have no idea what you are on actually

<youpi>what exactly the numbers you quoted above mean

<youpi>what is at which layer

<youpi>does rumpdisk_device_read() get the proper offset?

<damo22>recnum in libstore is the wrong offset

<youpi>where in libstore?

<damo22>i have to reboot to get the right line

<youpi>I don't really need the line, I just need to know at which layer

<youpi>you mean dev_read?

<damo22>yes

<damo22>dev_read

<damo22>these numbers above recnum-* are the values from dev_read

<damo22>when i dd the same block on different drives

<damo22>drivers

<damo22>ah, im using a different translator setup for the two different modes:

<damo22>/hurd/storeio sd0s3 without rump and

<damo22>the other one is -T typed part:3:device:@/dev/rumpdisk:wd0

<youpi>sd0s3, it's the kernel that will do the translation

<damo22>i guess storeio adds the partition offset to the recnum for rumps mode?

<youpi>it does so when the part layer is involved, simply

<damo22>is there a way to search a block device for a sequence of 4096 bytes?

<damo22>then i can find the offset of this strangely mapped block

<youpi>there are probably tools for that, but I don't think it's so easy to find out such tool

<youpi>while you just need to check along the device_read path that the offset remains correct

<youpi>I mean

<youpi>if you see that it's offset by some amount, great, but that won't tell you *where* the bug is

<youpi>finding *where* the bug is is basically most often the first thing to find out

<youpi>the *why* can come after that

<youpi>(and is quite often obvious when the "where" is determined very precisely)

<damo22>goodnight

<youpi>gn

<youpi>damo22: just like I said, the difficult part is finding out where the bug lies, not why :)

<youpi>so the good news is that I could let the debian installer run a long time

<youpi>the bad news is that console-setup is broken due to the tempfile removal :)

<youpi>but I'll build packages, that'll be a good test as well

IRC channel logs

2021-08-21.log