IRC channel logs


back to list of logs

<youpi>damo22: the hang looks like out of memory indeed
<youpi>it seems like there are missing cleaning in mach calls or RPCs
<youpi>the thing is: some of those have to go to lowmem (<1G) for the kernel to be able to access to it directly
<youpi>and thus even if there are gigabytes free, if that lowmem is full mach is stuck
<damo22>youpi: ok thats good news i guess
<youpi>actually I had thought about something else: pci-arbiter and rumpdisk have to use mlockall()
<youpi>otherwise if they get swapped-out we're hosed
<youpi>and it made the problem worse
<youpi>looking at rumpdisk it indeed grows
<damo22>i need to print off_t from librump
<youpi>and locking its memory in makes it keep its memory busy :)
<damo22>i think its 4
<damo22>but setting it to 8 did not fix my strange file issue
<damo22>where else could off_t be misformed?
<youpi>all the path from device_read/write down to the driver
<damo22>i bought a w530 thinkpad for $60 that is stuck in a boot loop
<damo22>i can solder a socket onto one of the extra headers to flash it with coreboot externally
<damo22> and schematics for CN100
<youpi>really, as I said, make yourself a setup where you issue just a few reads at a very large offset, and put prints in the device_read function and the eventual ahci driver, to check that they match. if they don't, you know the issue is in between and you can step down in between
<damo22>well, its a bit tricky because i cant boot off rumpdisk and only issue a few reads
<damo22>i guess i need a copy of /
<damo22>81MB/s over usb3
<damo22>is that possible??
<damo22>11737760256 bytes (12 GB, 11 GiB) copied, 143.541 s, 81.8 MB/s
<damo22>youpi: how do i attach two disks? qemu-system-i386: -device ide-hd,drive=udisk,bootindex=2: Can't create IDE unit 1, bus supports only 1 units
<youpi>the qemu details are tricky
<youpi>it completely depends on how you set them up exactly
<damo22>-drive id=rootfs,file=rootfs10G.dd,format=raw,bus=0,unit=0 -drive id=udisk,file=/dev/sdd,format=raw,bus=1,unit=0
<youpi>it seems that my fixup was effective, rumpdisk remains at a few dozen MB
<damo22>that seems to work but gnumach detects sd0 sd1 sd2 hd0
<damo22>and hangs
<youpi>I'm lost, is that with q35?
<damo22>/extra/qemu/bin/qemu-system-i386 -M q35,accel=kvm -smp 2 -m 4096 -net user,hostfwd=tcp::8888-:22 -net nic -curses -drive id=rootfs,file=rootfs10G.dd,format=raw,bus=0,unit=0 -drive id=udisk,file=/dev/sdd,format=raw,bus=1,unit=0
<youpi>ok, I've seen gnumach find out some hd0, I don't know why, possibly the qemu q35 tries to emulate some ide as well
<youpi>but can't you just run the system on ide, and only one test disk with ahci?
<damo22>probably, i need to read the options again
<damo22>it shouldnt be this hard
<damo22>how do i refer to the ahci built in bus?
<damo22>bus=0 does not work
<damo22>ok i attached a second ahci
<damo22>nice, i have a second disk with a / partition
<damo22>these two reads dont work properly because i can see the ls is invalid
<damo22>gah i cant get librump to print to console in the disk driver
<damo22>how do i disable stripping of libs in dh?
<damo22>i have no debug symbols in librump
<damo22>hey youpi i got an interesting crash
<damo22>when the filesystem was mounted ro
<damo22>why is it trying to "drop a node" when the filesystem is readonly
<damo22>how do i load symbols in gdb for a shared lib that is not loaded yet?
<damo22>it does not load the debug symbols for librump
<damo22>(gdb) sharedlibrary /usr/lib/
<damo22>No loaded shared libraries match the pattern `/usr/lib/'.
<damo22>Thread 31 hit Breakpoint 1, rumpdisk_device_read (d=0x1000ff80, reply_port=187, reply_port_type=18,
<damo22> mode=0, bn=291622944, count=4096, data=0x1b832f0c, bytes_read=0x1b832e18)
<damo22>is count in units of bytes or sectors?
<youpi>damo22: dropping a node is just removing it from the page cache
<youpi>count is bytes
<damo22>ok my problem is that the dynamic libs dont go in the symbol table as entries visible with ldd
<damo22>i think librump is loaded with libdl
<damo22>so gdb does not know they are being used, until they get called, and then its too late to load debug symbols
<youpi>librump is linked in statically, see the makefiles
<damo22>even in rumpdisk? not rumpdisk.static?
<youpi>see the rumpdisk/Makefile
<damo22>wdstrategy (wd0) sizeof(daddr_t)=8 blkno=291622944 bp->b_blkno=1252511007243239424
<damo22>Thread 31 hit Breakpoint 1, sys_pread (l=0x10030900, uap=0x1b832d14, retval=0x1b832d0c)
<damo22>(gdb) p *uap
<damo22>... offset = {pad = -1012908032, le = {datum = 149310947328}...
<youpi>(gdb) p/x 291622944
<youpi>$1 = 0x1161d020
<youpi>(gdb) p/x 1252511007243239424
<youpi>$2 = 0x1161d02000000000
<youpi>that looks like a 32bit/64bit bug in the call
<youpi>something like one side things 32bit and the other side things 64bits for a parameter
<damo22>it could be i used %lld
<damo22>to print the value
<youpi>the compiler would have complained
<youpi>did you check the warnings?
<damo22>maybe i can break on wdstrategy
<damo22>yeah its my mistake with the printf
<damo22>wdstrategy (wd0) sizeof(daddr_t)=8 blkno=291360776 bp->b_blkno=1251385004257181696
<damo22>(gdb) p *bp
<damo22>$1 = {b_u = {u_actq = {tqe_next = 0x0, tqe_prev = 0x0}, u_rbnode = {rb_nodes = {0x0, 0x0}, rb_info = 0}, u_work = {wk_dummy = 0x0}}, b_iodone = 0x0, b_error = 0, b_resid = 0, b_flags = 1048576, b_prio = 2, b_bufsize = 2048, b_bcount = 2048, b_dev = 3, b_data = 0xfa16800, b_blkno = 291360776 ...
<damo22>it looks like with my patch, the offset in sys_pread is correct
<damo22>but it still doesnt explain my broken file
<gnu_srs>youpi: FYI, I've not shut down mahler, the host is down now due to a possible HW error.
<youpi>ok, thanks, I was wondering
<damo22>what happens if the bp->b_bcount is 2048 but the request was for 4096 bytes?
<damo22>could it be truncating every block in half?
<youpi>depends at what point you are looking at the data
<youpi>possibly the transfer is just not finished
<youpi>I wouldn't expect a bug that grave here, however, since rumpdisk is used in production already, and all that rumpdisk is doing is calling rump_sys_pread
<damo22>where does it check if the bytes_read == count in the request?
<damo22>i dont see that in block_rump.c
<youpi>that's missing indeed
<damo22>what is it supposed to do
<damo22>read again?
<youpi>no, that would fail again
<youpi>just return that value
<youpi>and it's up to the caller to cope with it
<youpi>I would mach_print("short read\n"); because taht's really not supposed to happen in practice
<youpi>possibly the file is simply at the very end of the read limit
<damo22>no short reads i can see
<damo22>its a bit weird that it calls diskfs_free_node() when doing ls
<youpi>do you have a backtrace?
<damo22>not sure how to get one
<damo22>gdb on ext2fs?
<youpi>for instance yes
<damo22>that was triggered in the ext2fs on my rumpdisk by typing ls -la on the directory with my strange file
<damo22>the file works normally without rumpdisk
<damo22>?--------- 0 root root 0 Oct 20 2019 hooks
<damo22>ext2fs: /dev/wdd0s3: warning: bit already cleared for inode 5767226
<youpi>it has 0 links indeed
<youpi>does that file show some links outside rmupdisk?
<damo22>how do i tell
<youpi>the first 0
<youpi>taht's the number of links
<youpi>diskfs_drop_node is normal, it's just getting out of the cache
<youpi>but it finding np->dn_stat.st_nlink == 0 is what is surprising
<youpi>and that's what ls says indeed
<damo22>well it must have links because it appears as a directory on non-rumpdisk and has files inside
<youpi>so I guess it's just reading the inode that fails
<damo22>i can get the same listing when i boot off a different controller
<youpi>did you get the block number of that inode with debugfs?
<damo22>not yet
<damo22>drwxr-xr-x 2 demo demo 4096 Mar 9 2020 hooks
<damo22>debugfs: blocks /demo/git/rumpkernel-0~20191130/.git/hooks
<damo22>/dev/sd0s3 106811392 316526592 209715201 100G 83 Linux
<youpi>note that block numbers are relative to the beginning of the partition
<youpi>also, here the problem is with the block is the block of the inode structure itself, not the inode content
<damo22>yes, i just added the start offset of the partition
<youpi>since that's what contains the numref, type of inode, etc.
<youpi>i.e. the imap command in debugfs, probably
<damo22>debugfs: imap /demo/git/rumpkernel-0~20191130/.git/hooks
<damo22>Inode 5767226 is part of block group 704 located at block 23068675, offset 0x0c80
<damo22>the first 0x134 bytes are populated of the inode, the rest are zero
<youpi>at offset 0x0c80 ?
<damo22>i just dumped the inode with dump_inode
<youpi>with which disk driver?
<damo22>the one that works
<damo22>its believable, since its just a directory with a few files
<youpi>my point is getting a block number and making sure you know you have the right block number
<youpi>the inode structure itself doesn't care about the number of files in the directory
<youpi>the bytes along the inode structure are just other inode structures that are part of the same inode block
<youpi>once you know where the inode structure is, you can try to read it with rumpdisk, and debug that
<damo22>so do i run debugfs with rumpdisk driver?
<youpi>for instance
<damo22>dump_inode on that inode returned a zero byte file
<damo22>with rumpdisk
<youpi>dump_inode dumps the content of the inode, not the inode structure itself
<youpi>if the inode structure is hosed, then dump_inode will read a random block
<youpi>it's the inod structure that you want to try to read
<youpi>since that's what contains the numref, type of inode etc.
<youpi>if *that* is hosed, there is no point in looking at dump_inode, it'll fail of course, because the block number will be bogus
<damo22>3200 a481 0000 6c0b 0000 7a05 d55d 54b3 ab5d ....l...z..]T..]
<damo22>3220 6c29 0e5c 0000 0000 0000 0100 0800 0000 l).\............
<damo22>3240 0000 0000 0000 0000 df01 2800 0000 0000 ..........(.....
<damo22>oops hex
<damo22>wrong offset
<youpi>at least that looked like an ext2_inode :)
<damo22>0760 0000 0000 0000 0000 0000 0000 0000 0000 ................
<damo22>1000 a481 0000 7904 0000 c705 d55d 49b3 ab5d ....y......]I..]
<youpi>(modulo the byte inversion)
<damo22>why is 1000 the next value after 0760?
<damo22>i dont understand block_dump
<youpi>it means it's all 0 in between
<youpi>but why 0760 ?
<damo22>it said my offset was 0xc80
<youpi>which block_dump command did you use exactly?
<youpi>then it's 0c80
<damo22>debugfs: bd 23068675
<youpi>hexdumps always use jex
<damo22>or do i add the offset 0xc80 to the block
<youpi>it's an offset *inside* the block
<damo22>before dumping?
<damo22>so its all zero then
<damo22>but it looks like a short read
<youpi>while doing exactly the same with non-rump doesn't show zeroes?
<damo22>because there is no *
<youpi>the ponit is trying to read exactly the same block both ways
<damo22>ok will do
<youpi>and check whether it's different
<youpi>and then debug exactly that
<youpi>since taht's where things go wrong
<youpi>what do you mean by "there is no *" ?
<damo22>usually there is a line by itself with
<damo22>when there are all zeros below
<damo22>but i got two sequential lines with missing offsets between
<youpi>I thought you said there was no short read in rumpdisk_device_read?
<youpi>I mean, debugfs cannot know about a short read if rumpdisk_device_read is not reporting it
<damo22>what do i return in the case of a short read
<youpi>as I said, just the amount of bytes
<youpi>but *ALSO* put a mach_print
<youpi>since taht's not suppose to happen
<damo22>ok yeah i did
<youpi>and with a mach_print you're sure to know about it
<youpi>if you don't get a mach_print then it means it doesn't happen
<damo22>wait, D_SUCCESS
<youpi>it's as simple as that
<damo22>or number of bytes
<youpi>yes, success, and put the size where it shall be
<damo22>errr its dumping different data
<damo22>its like the offset is somewhere else in the disk?
<youpi>so that's a reproducer
<youpi>and you can try to debug exactly that with e.g. a mere dd command
<damo22>hmm cant find the actual offset
<damo22>does ext2fs count blocks from the start of the partition?
<damo22>sudo dd if=/dev/sd0s3 of=testfrom3 skip=23068675 bs=4096 count=1
<damo22>got it
<damo22>XXXX libstore: dev_read recnum=291360792
<damo22>but the contents are different
<damo22>so rump is not subtracting the start of the partition
<youpi>why would it be? it doesn't even know about partitions
<damo22>storeio is not working correctly then
<damo22>root@zamhurd:~# showtrans /dev/wdd0s3
<damo22>/hurd/storeio -T typed part:3:device:@/dev/rumpdisk:wd0
<youpi>why would storeio not work?
<damo22>the part:3 thing
<youpi>well, what it does is precisely add the offset of the partition
<youpi>but I have no idea what you are on actually
<youpi>what exactly the numbers you quoted above mean
<youpi>what is at which layer
<youpi>does rumpdisk_device_read() get the proper offset?
<damo22>recnum in libstore is the wrong offset
<youpi>where in libstore?
<damo22>i have to reboot to get the right line
<youpi>I don't really need the line, I just need to know at which layer
<youpi>you mean dev_read?
<damo22>these numbers above recnum-* are the values from dev_read
<damo22>when i dd the same block on different drives
<damo22>ah, im using a different translator setup for the two different modes:
<damo22>/hurd/storeio sd0s3 without rump and
<damo22>the other one is -T typed part:3:device:@/dev/rumpdisk:wd0
<youpi>sd0s3, it's the kernel that will do the translation
<damo22>i guess storeio adds the partition offset to the recnum for rumps mode?
<youpi>it does so when the part layer is involved, simply
<damo22>is there a way to search a block device for a sequence of 4096 bytes?
<damo22>then i can find the offset of this strangely mapped block
<youpi>there are probably tools for that, but I don't think it's so easy to find out such tool
<youpi>while you just need to check along the device_read path that the offset remains correct
<youpi>I mean
<youpi>if you see that it's offset by some amount, great, but that won't tell you *where* the bug is
<youpi>finding *where* the bug is is basically most often the first thing to find out
<youpi>the *why* can come after that
<youpi>(and is quite often obvious when the "where" is determined very precisely)
<youpi>damo22: just like I said, the difficult part is finding out where the bug lies, not why :)
<youpi>so the good news is that I could let the debian installer run a long time
<youpi>the bad news is that console-setup is broken due to the tempfile removal :)
<youpi>but I'll build packages, that'll be a good test as well