IRC channel logs


back to list of logs

<rekado_>my talk proposal for GWL in the HPC devroom was accepted
<rekado_>but I just realized that mid-January (the deadline for recording the talk) is very soon
<rekado_>I’d really like to have a few more features by then
<civodul>rekado_: yay!
<civodul>well, a bit of extra pressure
<civodul>but i guess there's already enough to talk about
<rekado_>that’s true, but the GWL is embarrassingly lacking when it comes to execution.
<rekado_>we generate launcher scripts that can only be used with Grid Engine
<civodul>ah yeah
<civodul>an "implementation detail"? :-)
<rekado_>it is, but it also reveals how little thought went into all this
<rekado_>well, that’s not true
<rekado_>some thought went into this, but I haven’t come up with anything better yet
<rekado_>what’s great about this “launcher script” method is that it’s all self-contained. There are no weird run-time dependencies. These are dumb Guile scripts.
<rekado_>I wanted to add DRMAA support, raw SSH cluster support, and perhaps even support for dumb ad-hoc AWS clusters.
<rekado_>none of these solutions would really be self-contained
<rekado_>but perhaps this doesn’t matter
<rekado_>after all that’s why we have Guix
<rekado_>the workflows themselves have non-trivial dependencies; why not the whole execution engine itself?
<rekado_>here’s why not: Guile DRMAA and Guile AWS are barely working
<rekado_>“baking” the workflow is fine, but “baking” these execution things seems wrong
<rekado_>AWS is a (slowly) moving target; you want the latest Guile AWS at execution time.
<rekado_>you can’t necessarily use the old Guile AWS that existed when you baked the workflow.
<rekado_>so… should the GWL be a runtime dependency then…?
<rekado_>is that acceptable? All the other workflow language folks seem to think so, but it kinda feels inelegant to me.
<rekado_>and *that’s* why I haven’t really come up with anything new in that area.
<civodul>instead of producing a script, it could produce a description (say an sexp)
<civodul>and then there'd be a run-time gwl command that would consume that description and use the appropriate AWS/DRMAA thing
<civodul>that way run-time support would be disconnected from the workflow
<rekado_>yes, that’s what I meant with GWL being a runtime dependency
<rekado_>but … does it need to generate a description at all? The workflow itself is the description.
<civodul>perhaps it's more convenient to "lower" the workflow to a low-level intermediate representation
<civodul>just like we have "packages" and "derivations"