IRC channel logs

<rekado_>my talk proposal for GWL in the HPC devroom was accepted

<rekado_>but I just realized that mid-January (the deadline for recording the talk) is very soon

<rekado_>I’d really like to have a few more features by then

<civodul>rekado_: yay!

<civodul>well, a bit of extra pressure

<civodul>but i guess there's already enough to talk about

<rekado_>that’s true, but the GWL is embarrassingly lacking when it comes to execution.

<rekado_>we generate launcher scripts that can only be used with Grid Engine

<civodul>ah yeah

<civodul>an "implementation detail"? :-)

<rekado_>it is, but it also reveals how little thought went into all this

<rekado_>well, that’s not true

<rekado_>some thought went into this, but I haven’t come up with anything better yet

<civodul>heh

<rekado_>what’s great about this “launcher script” method is that it’s all self-contained. There are no weird run-time dependencies. These are dumb Guile scripts.

<rekado_>I wanted to add DRMAA support, raw SSH cluster support, and perhaps even support for dumb ad-hoc AWS clusters.

<rekado_>none of these solutions would really be self-contained

<rekado_>but perhaps this doesn’t matter

<rekado_>after all that’s why we have Guix

<rekado_>the workflows themselves have non-trivial dependencies; why not the whole execution engine itself?

<rekado_>here’s why not: Guile DRMAA and Guile AWS are barely working

<rekado_>“baking” the workflow is fine, but “baking” these execution things seems wrong

<rekado_>AWS is a (slowly) moving target; you want the latest Guile AWS at execution time.

<rekado_>you can’t necessarily use the old Guile AWS that existed when you baked the workflow.

<rekado_>so… should the GWL be a runtime dependency then…?

<rekado_>is that acceptable? All the other workflow language folks seem to think so, but it kinda feels inelegant to me.

<rekado_>and *that’s* why I haven’t really come up with anything new in that area.

<civodul>hmm

<civodul>instead of producing a script, it could produce a description (say an sexp)

<civodul>and then there'd be a run-time gwl command that would consume that description and use the appropriate AWS/DRMAA thing

<civodul>that way run-time support would be disconnected from the workflow

<rekado_>yes, that’s what I meant with GWL being a runtime dependency

<rekado_>but … does it need to generate a description at all? The workflow itself is the description.

<civodul>perhaps it's more convenient to "lower" the workflow to a low-level intermediate representation

<rekado_>yeah

<civodul>just like we have "packages" and "derivations"

IRC channel logs

2020-12-27.log