IRC channel logs

<mark_weaver>but even if that's fixed, there's this thorny problem that if two threads try to load the same module concurrently, one of them is likely to think that it's already loaded, even though it's only partially loaded.

<mark_weaver>really, there should be some kind of locking to make sure that only one thread tries to load a given module.

<mark_weaver>however, code needs to be able to access a partially-loaded module, if it's the thread that's currently loading it.

<mark_weaver>we can't simply put a "currently-loading" lock on each module, because that would lead to deadlocks in the case of circular dependencies.

<mark_weaver>anyway, I'm thinking on it now.

<civodul>oh, right

<civodul>trickier than one would think...

<dje42>Can you put a lock on the entry point to loading a module?

<mark_weaver>while I'm at it, I'm going to try to fix that "compiling multiple modules" bug that's causing problems for guix.

<civodul>mark_weaver: the name space thing?

<civodul>that'd be great

<mark_weaver>I don't think we can do it the way you suggested though.

<mark_weaver>or at least I don't see a sane way to do it.

<mark_weaver>what I'd like to do, however, is to keep a record of when a module has been fully loaded.

<mark_weaver>my thought is that the thread that is currently loading a module will effectively have access to that module before it's fully loaded, but if any other thread tries to access it, it will have to wait for the load to finish.

<mark_weaver>but there also has to be deadlock detection.

<mark_weaver>suppose there are two modules A and B that use each other.

<mark_weaver>now suppose that thread A tries to load module A, at the same time as thread B tries to load module B.

<mark_weaver>they cannot both wait for each other to finish loading, or else there will be deadlock.

<dje42>Single-threading module loading, plus a lock on each module seems like it could work.

<mark_weaver>so I think we need to keep a record, for each module, of which module it's waiting to load.

<mark_weaver>and if a cycle is detected, it will simply not load the module (as is done now)

<mark_weaver>dje42: anything involving a lock on each module is likely to lead to deadlocks, because you'd need to have an ordering on the locks, no?

<mark_weaver>(or at least you'd have an uphill battle to convince me that it wouldn't lead to deadlocks)

<dje42>What I mean is a thread-safe test on whether the module has been loaded.

<mark_weaver>well, I'm reluctant to make module loading completely single-threaded. some modules might take a long time to load.

<mark_weaver>in fact, I think some modules might never finish loading. they might have the main program within their top-level expressions.

<dje42>If it's an easy first step to fix the problem (and something more elaborate could take awhile), I'd be ok with it.

<dje42>(take awhile to properly implement)

<mark_weaver>I'm going to make it a priority to fix this properly and soon.

<dje42>Can one determine in advance if a module will never finish loading? [I'm guessing not, or at least not easily enough.] Nasty.

<dje42>Hmmm, can loading a module be aborted and retried?

<mark_weaver>a module could lead to an uncaught exception while it's loading, yes. we need to take that into account as well.

<mark_weaver>I think I have the basic outlines of a workable plan...

<mark_weaver>I just need to think it through.

<dje42>Then if(!) one can detect a deadlock one could throw something in one thread (undoing all its locks) and starting over.

IRC channel logs

2014-01-04.log