IRC channel logs

<rekado_>for real-world workflows GWL has a terribly slow step before anything starts

<rekado_>it’s the moment when it claims to generate “all scripts and their dependencies”.

<rekado_>I don’t remember what exactly it’s doing there, but it uses Guix over an inferior.

<rekado_>this is a bit sad, because it does a lot of work here even when it doesn’t need to compute anything new.

<rekado_>the scripts already exist, but GWL doesn’t know it.

<civodul>yes, i remember you reported a performance issue with inferiors

<civodul>but i never got around to investigating

<civodul>could GWL do cache things similar to 'guix shell'?

<civodul>that's a game changer IMO

<rekado_>yes, that was what I planned

<rekado_>the first run will still be very slow (several minutes just for generating scripts), but when nothing has changed it should just load from the cache.

<civodul>several minutes is terrible

<civodul>hey zimoun!

<zimoun>civodul: hey!¬

<civodul>zimoun: any thoughts on the ISC tutorial?

<zimoun>I just answered back to you right now. :-)

<zimoun>Sorry, I have been for some days off (working as “farmer“ close to Marseille ;-))

<civodul>ah excellent, thanks!

<zimoun>Then, I have been somehow busy at work, damned! :-)

<civodul>oh you must have had a good time

<civodul>(in Marseille, not at work :-))

<zimoun>yeah, a friend is starting “vegetable business” and I helped them to build stuff as hothouse.

<zimoun>civodul: do you have a draff for this tuto proposal?

<civodul>zimoun: not yet, but that's where you come in! :-)

<civodul>we have an empty Git repo now

<civodul>that's a start

<zimoun>for this Friday, it will be hard for me. My backlog is already HUGE. And Friday, another day off (Aveyron this time ;-))

<zimoun>A template somewhere?

<civodul> https://gitlab.inria.fr/guix-hpc/isc-tutorial-2022

<civodul>worry not, it's for Thursday, not Friday :-)

<zimoun>ahah! Right. So even more difficult. ;-)

<rekado_>somewhat surprisingly job submission via Guile DRMAA still works

<rekado_>I’m rearranging a few things to use the process-engine abstraction (kinda annoying to have all these similar concepts here: runner, script, process, blah)

<civodul>yay, having proper GWL/scheduler integration sounds pretty exciting

<rekado_>and this will unlock some much needed cleanup

<rekado_>the old way of generating special scripts for Grid Engine required wrapping the generated process scripts, and this made caching much more complicated as more information had to be threaded through the machinery

<rekado_>it all becomes simpler

<rekado_>I like that

<civodul>very cool

<civodul>so Grid Engine talks DRMAA as well?

<rekado_>yes

<rekado_>the old way *only* worked for Grid Engine, because it has a command line feature to declare that a job depends on another submitted job (by name)

<rekado_>the new way should work for slurm, the various grid engine clones, and whatever else uses DRMAA v1

<civodul>so now you gained support for Slurm and probably other schedulers, right?

<rekado_>correct

<civodul>really nice

<rekado_>and with the whole restructuring that’s going on now I should be able to add support for guile-aws as well.

<civodul>woow, this is starting to look like serious changes! :-)

<rekado_>a bit wasteful at first (start one VM per job and kill it afterward), but I have some ideas to make it behave more … economically

<civodul>Bezos will thank you for not optimizing too much on his next space trip

<rekado_>yeah, but right now I’m stymied by this very helpful error message: “error: ice-9/boot-9.scm:1752:10: In procedure with-exception-handler: error: #{.}#: unbound variable” :)

<civodul>that's a dot interpreted as a symbol

<rekado_>yeah

<rekado_>no backtrace, though, so I don’t know where…

<rekado_>wisp is full of dots!

<civodul>ah

<rekado_>bizarre: adding “#:use-module (drmaa v1 high)” to (gwl workflows) fixes the problem.

*rekado_ is confused

<rekado_>it’s enough to add (system ffi-help-rt). I wonder why.

<civodul>weird

IRC channel logs

2022-01-25.log