IRC channel logs

<damo22>hi

<azert>damo22: hi

<gnu_srs2>WDYM: (remember the xy problem)?

<damo22>gnu_srs2: if you assume x is the problem but it doesnt cause y ? im guessing something like that

<etno>I suppose this one: https://en.wikipedia.org/wiki/XY_problem

<solid_black>hi!

<damo22>hello

<damo22>i am running a slightly modified gnumach to detect timeouts in locks, but it does not seem to time out

<damo22>something about having two processor sets with no overlapping cpus is causing a deadlock

<damo22>i wrote a simple program to execute a task on slave_pset

<damo22>and another simple program to hit a gnumach rpc repeatedly in parallel

<damo22>when i run ./smp ./test 7 it runs for a little bit and then deadlocks, but no locks are timing out

<damo22>when i check the runqs, the slave_pset is empty

<damo22>maybe the test program has called an rpc, waiting for a lock but the thread doing the ipc has gone to sleep and never woke up

<damo22>because the lock is locked by another thread calling the same rpc

<damo22>now both threads have gone to sleep and never rescheduled back onto the runq?

<youpi>did you check their backtraces?

<damo22>how?

<youpi>that will tell you what they are stuck on

<youpi>well, trace ?

<youpi>didn't you ever use trace in the kdb ?

<youpi>trace/tu

<youpi>to also have the userland part

<damo22>hmm no

<youpi>you *

<youpi>really

<youpi>*

<youpi>should read documentation :)

<youpi>you'll discover a lot of things :)

<damo22>oh i didnt know i can get a backtrace during runtime even if no exception

<damo22>trap*

<youpi>really, people, read the doc...

<youpi>the rtfm mantra is forever

<youpi>(and "it's long" is not a good excuse: yes there is a lot to learn, that's for your goodness)

<damo22>but how do i get a backtrace on a different cpu?

<youpi>I don't know, probably the doc knows

<damo22>i dont think kdb lets you switch to a different cpu yet

<youpi>but possibly kdb doesn't actually stop the other cpus

<youpi>so it can't actually print a backtrace

<rekado>Gooberpatrol66: laminar might be better for the purpose of automatically applying patches from a mailing list and spawning processes in response. Cuirass is fine for when you’ve got code in git already and you’ve got a stable manifest of things to build.

<damo22>that is true

<youpi>(and that's something that could possibly be fixed with an IPI)

<rekado>Gooberpatrol66: Cuirass fits the specific Guix use-case very well, but it might not be the right tool for a generic evaluation of arbitrary patches.

<damo22>actually kdb runs trace on entering it

<damo22>so that doesnt help

<damo22>because its only on cpu0

<youpi>but other cpus are idle, aren't they?

<damo22>yes runqs are empty

<youpi>then you can trace any thread

<youpi>since they're not running

<damo22>ohh

<damo22>db{0}> show thread $task68.2

<damo22>task 68(f6754ea0): thread 2(f664c640) R.....

<damo22>db{0}> trace/tu $task68.2

<damo22>switch_context(f8891000,c10cc0ec,f64c1e50,c1007076,f8890000)+0x20c

<damo22>0xf6763bb0(c10cc0e0,f8890000,f8891000,3,1,1,1000,1)

<damo22>mach_port_names(f5bd1038,fa0a203c,f64c1f10,fa0a204c,f64c1f14)+0xc9

<damo22>_Xmach_port_names(f5405010,fa0a2010,c1033c6d,f5bd1040,f6708270)+0x43

<damo22>ipc_kobject_server(f5405000,27,0,1000)+0x93

<damo22>mach_msg_trap(2804e8c,3,18,40,21)+0x985

<damo22>theres 200% load on the host

<damo22>but its deadlocked

<damo22>100 is from kdb spinning

<damo22>and another one from ipc_mqueue_send

<damo22>there must be a problem in ipc_mqueue_send ?

<damo22>db{0}> trace/tu $task68.6

<damo22>....

<damo22>_Xgsync_wake(f662c010,fa0a4010,0,c1050012,f6754ea0)+0x4b

<damo22>ipc_kobject_server(f662c000,1000,f662c000,c102c99e)+0x93

<damo22>ipc_mqueue_send(f662c000,0,0,0)+0x247

<damo22>mach_msg_trap(4808b0c,1,30,0,0)+0x7cf

<damo22>or mach_msg_trap ?

<damo22>ipc_mqueue_send() has some strange locking

<damo22>it does ip_lock(port);

<damo22>then branches in an if to unlock it

<damo22>i think the problem is, ipc_mqueue_send() does ip_unlock(port) then blocks the thread, and does ip_lock(port) if/when it wakes up... what if some other thread tries to use the port? wont it be unlocked so the other thread can lock it and then it will try to lock it twice when it wakes up?

<damo22>maybe it should hold the lock throughout its sleep?

<youpi>?

<youpi>I don't understand what you are saying

<youpi>usually you *don't* want to hold a lock while sleeping

<youpi>since that'll prevent any other thread from being able to acquire the lock

<youpi>of, course, threads accessing the same port need to do this coherently

<youpi>e.g. not steal messages from another thread or such

<youpi>or it can

<youpi>depending on the semantic

<youpi>e.g. several readers on a pipe

<youpi>normally with ipc ports there is just one thread doing a receive

<youpi>but there can be several senders

<youpi>so they do need to be able to get the lock to put their message

<damo22>ok

<youpi>(again, please read book chapters about producer/consumer principles)

<damo22>i use activemq at work, i kind of know this

<damo22>just not the low level locking part

<damo22>its difficult to follow because the kernel can send messages to itself?

<solid_black>you almost never want to hold any locks while sleeping, yes

<solid_black>it makes perfect sense that if it foes ip_unlock() before sleeping and ip_lock() upon waking up

<youpi>(that's part of the usual principe of hoare monitors, also known as condition variables, i.e. pthread_cond_wait)

<damo22>vm_map_lock, vm_map_verify vm_map_lookup are all called at the same time on the same lock i think

<damo22>one write lock and two read locks

<damo22>c10b40e0

<damo22>map lock

<damo22>solid_black: you might know more on this than me

<damo22>the kernel map lock is being highly contended

<solid_black>do you mean it's just slowing things down, or are there bugs / deadlocks / races because of this?

<damo22>how do we ensure no deadlocks on that

<damo22>i see a deadlock

<solid_black>generally, deadlocks are solved by establishing a locking hierarchy

<solid_black>do you have more details?

<solid_black>like backtraces

<damo22>(09:48:54 PM) damo22: vm_map_lock, vm_map_verify vm_map_lookup are all called at the same time on the same lock

<solid_black>and what threads are waiting on

<damo22>i can get backtraces for all the threads

<solid_black>but what is the issue? trying to lock the same lock is fine; only one thread gets it, and the others wait

<solid_black>a deadlock happens when e.g. thread 1 is holding lock A and is trying to lock the lock B, which is held by thread 2 who's trying to lock the lock A

<damo22>i think the issue we have is that the threads are separated by cpu sets

<damo22>so kernel runs on cpu0

<solid_black>why would that matter?

<damo22>my process calloing rpcs runs on cpu1-7

<damo22>i am sending tons of the same rpc to kernel

<damo22>during that, the task sending the rpcs locks up

<damo22>i have backtraces where the same map lock is being written and read by multiple threads simultaneously

<damo22>the threads are sitting there stopped

<solid_black>does lock_done() / thread_wakeup_prim() wake up the other CPU core? (like with an IPI or something?)

<damo22>not sure

<solid_black>because that could surely lead to a deadlock

<damo22>how

<solid_black>two threads running on different CPU cores contend for the same lock, one thread gets it, the other one goes to sleep waiting for it, there is nothing else on the second core to run, so it enters low power mode (machine_idle), the first thread releases the lock once it's done, but never pokes the second core to wake up

<damo22>:D

<damo22>perhaps when this occurs, cpus in a different processor set must send an IPI?

<damo22>to the other ones

<damo22>one

<solid_black>I wish I understood how the Mach scheduler actually works so I could give better advice :|

<damo22>you must be right, we are missing a cpu wakeup

<damo22> * Sleep locks. These use the same data structure and algorithm

<damo22> * as the spin locks, but the process sleeps while it is waiting

<damo22> * for the lock. These work on uniprocessor systems.

<damo22>maybe this is broken on smp

<damo22>hmm we dont have sleep locks for SMP

<azert>Hi damo22. Just an idea feel free to ignore it: is the kernel making the assumption that it runs in cpu0, as you mentioned?

<azert>I doubt that is always the case, when a thread enters the kernel from another cpu it will stay there

<azert>But maybe I m not correct

<azert>Unless there is a complicated way to force the kernel in cpu0, I don’t see why that would happen

<Pellescours>azert: for now with the current setup of damo, the kernel is forced to run on cpu0. Because the 1st cpu set (on which kernel run) contains only cpu0

<azert>How can this be?

<azert>Userspace threads enter the kernel all the time. There is nothing forcing the kernel in cpu0

<azert>Unless the scheduler is broken

<azert>And it doesn’t wake up tasks on other CPUs waiting on a lock in kernel spave

<azert>Why would that be the case?

<azert>I don’t know if I’m clear

<damo22>the default_pset contains one processor just 0

<azert>Ok but how would the kernel be tied to this?

<damo22>the kernel task is set up with default_pset as far as i know, just like all tasks

<damo22>i created a second pset called slave_pset containing all the other cpus

<azert>Ok, but threads enter the kernel all the time, do you agree with that?

<azert>Let’s say a thread from a slave processor enters the kernel, that will be in a slave provesssor

<damo22>hmm

<Pellescours>not sure but I don't think so. to ask something to the kernel you send a message (syscall/rpc) then the thread is put in pause. But this don't mean the kernel will start executing on the cpu

<azert>The kernel cannot make the assumption to be in the default_pset, it could only for some specific threads like kernel side services maybe. But I’m not sure gnumach have such things

<damo22>no i think the scheduler only executes tasks on particular pset

<azert>Pellescours when you do a syscall you don’t do a full context switch, normally

<Pellescours>azert: gnumach does continuation and not preemption

<azert>How does this matter at all? De facto kernel code will run on slave processors

<azert>That’s how x86 is designed

<damo22>i think the context is switched on a syscall

<damo22>to the kernel task

<azert>On syscall there is no magic involved : no other cpu is signalled

<azert>The thread keeps executing in kernel space whatever processor it is

<azert>You could imagine a stupid design where syscalls are just shims to wake up the kernel in another cpu, but I doubt that’s what is happening

<damo22>ok youre right i think

<azert>Good I hope that’s useful.. I’m not sure

<damo22>the kernel executes on all cpus?

<damo22>even with psets?

<azert>At least part of it, yes, absolutely

<damo22>ok but in the case where i am sending RPCs to the gnumach task

<damo22>for IPC

<damo22>i dont think the kernel executes on the AP

<azert>I don’t know what happens.. in a sane design that rpc would just run a function

<damo22>to send its message

<azert>It doesn’t ?

<damo22>i dont know actually

<azert>Maybe gnumach doesn’t shortcut rpcs.

<azert>Me neither

<azert>At least the scheduling of the rpc is kernel code on slave processors. That’s for sure

<damo22>the task i run containing 8 threads all run on APs only, and they execute rpcs in the gnumach task that runs on cpu0, i think the sleep locks are broken for SMP

<damo22>i get slightly different behaviour when i change the sleep locks to be interruptable

<damo22>msleep(1000*3600*8)

<damo22><---

<azert>damo22: but are you sure that vm_map (a syscall) for a task running on processor 5 goes to execute on processor 0? A part the bug you’re hunting, that would be totally insane

<youpi>I very much doubt that it's done so

<youpi>the syscall trap will just enter the kernel on which the user thread was running

<youpi>and execute the kernel code there

<youpi>it's just the kernel *threads* which are bound to cpu0

<solid_black>azert damo22: kernel task vs other tasks is not the same thing as running in kernel mode or user mode

<solid_black>kernel threads only ever run in kernel mode, but user threads run both

<solid_black>a syscall for a user thread gets taken from user mode to kernel mode, on the same CPU

<solid_black>and the way kboject RPCs work, an RPC server routine is alays executed immediately

<solid_black>so you're not actually messaging the kernel task, it runs in your thread

<solid_black>so "the task i run containing 8 threads all run on APs only, and they execute rpcs in the gnumach task that runs on cpu0" <-- that's not how this works

<gnu_srs2>Hi, how to define ifreq_int, needed for e.g. #define SIOCGIFMETRIC _IOWR('i',23, struct ifreq_int) /* get IF metric */

<etno>gnu_srs2: this looks to me like a routing notion, and I have difficulty to see why this could be useful at low level. Moreover, the linux man page states that this is not implemented by linux. Does the code you are porting rely heavily on this information ?

<gnu_srs2>etno: Thanks for your comment. My question is how struct ifreq_int, struct ifreq_short is defined. No header file defines it, see /usr/include/i386-gnu/bits/ioctls.h and

<gnu_srs2>/usr/include/net/if.h: define _IOT_ifreq_short and define _IOT_ifreq_int

<etno>gnu_srs2: I don't see how this struct has an interest if the syscall using it does not exist. But I just started my journey with Hurd, so I might be missing something :-)

<youpi>I agree with etno :)

<youpi>it's not implemented in the pfinet and lwip translators, so it will essentially fail anyway

<gnu_srs2>youpi: Maybe you should go through header files and comment out definitions not being implemented :)

<gnu_srs2>Finding not-supported stuff only creates frustration and waste of time :(

IRC channel logs

2024-02-16.log