IRC channel logs


back to list of logs

<civodul>hey zimoun!
<zimoun>procrastination :-)
<rekado_>zimoun: I’m currently attempting to translate pigx-rnaseq from Snakemake to GWL
<zimoun>rekado_: Awesome!
<rekado_>it’s a little difficult to turn a workflow with file-based targets into processes
<rekado_>Snakemake (like GNU Make) encourages this kind of backward connection of the graph
<rekado_>when we wrote PiGx we realized that we would actually like to have named tasks that don’t correspond to file names.
<rekado_>the result was a hacky association list from ‘target’ names to files that we expect to be produced
<rekado_>much of the work in translating this to the GWL is to turn the workflow inside out
<zimoun>yeah, I had hard time to explain to my colleagues this :-) That file-based is limitation… And because Snakemake is Python, it is possible to turn into function but it becomes a mess
<rekado_>one thing that has become very clear is that Snakemake falls apart as soon as you feel the need for procedures
<zimoun>let me know if it is possible to help by sharing the workload or anything
<rekado_>Python restricts lambdas to one line, so even simple inline things quickly turn ugly
<rekado_>zimoun: I may need to pick your brain later
<rekado_>I don’t know if any of this looks easy to people who are not me.
<zimoun>exactly! Which appears naturally for complex workflow, for example doing different alignment at different step: which naturally fits as a function with parameters.
<zimoun>well, the recent updates you did appear easy to me, at least. ;-)
<rekado_>I already noticed a few … oddities
<rekado_>Snakemake has a ‘log’ field for its rules, but I don’t know if they are treated specially
<rekado_>I moved that as a named output to ‘outputs’
<rekado_>but I wonder if there should instead be generalized handling of I/O streams
<rekado_>another thing that bothers me a little is local definitions
<rekado_>the syntax for ‘let’ in Wisp is awkward, so I made the ‘procedure’ field introduce a definition context
<rekado_>it’s possible to define values at the top of the ‘procedure’ field
<rekado_>but this also means that people have to write ‘procedure’ all the time just to be able to define local values.
<rekado_>this makes the definition of the process ‘payload’ a little uglier
<zimoun>ah, I see.
<rekado_>here’s what I have so far:
<rekado_>it’s not fully translated yet
<rekado_>I added a definition of ‘get’, which is useful for safely accessing nested alists
<rekado_>(e.g. for parsed configuration files)
<rekado_>so ‘get config "counting" "counting_mode"’ is like config['counting']['counting_mode'] but with error checking
<zimoun>about ’procedure’, reading your file, it does not appear to me akward
<zimoun>I find clear that ’log’ is in ’outputs’
<zimoun>the ’get’ is really a good idea
<rekado_>near the bottom we have report1, report2, and report3 — all of these can be merged with just one extra argument to the process
<rekado_>I wanted to translate first and simplify later
<rekado_>another thing that’s up for debate is the handlebars style of variable interpolation in code snippets
<rekado_> (
<rekado_>these {{foo}} things annoy me.
<rekado_>it’s ironic how people can put up with this while also complaining about parentheses in S-expr syntax :)
***rekado_ is now known as rekado
<zimoun>the file looks clear to me. Except the usual WISP akwardness. For example, genemeCoverage, the dots and string-appends or the outputs where the dots and indentation: subtile :-)
<rekado>yeah, I’m not entirely happy with wisp.
<zimoun>instead of {{foo}}, what do you suggest?
<rekado>I don’t know yet.
<rekado>but I was wondering if it would be a good idea to implement some sort of unquoting
<rekado>the {{placeholders}} can only take on a previously defined value
<rekado>with : and :: they can take on one or muliple named values from lists
<rekado>e.g. {{inputs:gtf}} or {{inputs::reads}}
<rekado>BTW, this idea also came about because I wanted to avoid the awkward wisp let
<rekado>so maybe *that* is something that should be addressed instead
<rekado>maybe by offering a clojure-like let
<civodul>a Lokke front-end!
<civodul>(not quite what you had in mind, but now every time i read "Clojure" i shout "Lokke")
<zimoun>personally, reading {{foo:bar}} is fine with fine. The dot and indentation wisp is a bit less clear. I mean, I am mentally translating to Scheme with parenthensis to understand. Maybe not enough habits, I do not know.
<rekado>in wisp land you need to start a line with a dot when the first thing on the line is not a procedure (or syntax)
<zimoun>civodul: I miss the Clojure / Lokke reference )-:
<rekado>zimoun: lokke is Clojure for Guile
<zimoun>not always, string-append NEWLINE “foo” “bar” should work in WISP
<zimoun>civodul: I have not be selected for the training on next Monday. I will be in Group 2. I hope you are prepared because I will have questions about Guix :-p
<rekado>zimoun: can you point me to a line where this is not the case?
<civodul>zimoun: heh c'mon
<rekado>in wisp it must be: string-append\n . "foo" "bar"
<rekado>(or all on one line)
<rekado>because otherwise "foo" would be applied to "bar" and the result is fed to “string-append”
<zimoun>in you file, process translate_sample_sheet_for_report with cwd, trim_galore_pe drop-right\n string-split\n 2, just below without-exension read2\n “_val…”
<zimoun>process salmon_quant in procedure define arguments, string-append\n “-r”
<rekado>oh, I think that’s a mistake
<rekado>you can think of the dot as “identity”
<rekado>looks like I’m missing a few dots …
<zimoun>well, that’s the problem with WISP, IMHO.
<rekado>I think it’s unnecessary, too.
<rekado>isn’t it known ahead of time that we cannot possibly apply a string literal to anything?
<rekado>or a keyword
<rekado>maybe there’s something we can do about this…
<rekado>either by making strings, numbers, and keywords applicable, or by being smart somewhere earlier.
<zimoun>is wisp infix for arithmetic?
<rekado>it doesn’t collide with srfi 105
<rekado>but it’s not enabled by default
<rekado>I’ll talk to Arne about the leading dot. I’m sure he’s thought about this for a while.
<rekado>zimoun: there may be a way around this similarly to how commit 7e099dd2428dbe21bcb1530e9bc49e3c4514cb99 made code snippets self-applicable
<zimoun>civodul: is guix-hpc still an alive alias? Is this the right email to coordinate the annual report?
<civodul>zimoun: yes, that should work
<civodul>hmm i think efraim, bonz060, and PurpleSym are not on that alias, but they may have something to share
<civodul>maybe mbakke too?
<rekado>(call-with-input-string "string-append\n \"hello\" \"world\"\n \"yes\"\n\n" (@@ (language gwl spec) gwl-scheme-read-chunk)) ==> ((string-append "hello" "world" "yes"))
<rekado>this is only for strings that follow line indentation, so I think this is not so bad
<rekado>now the question is whether Arne can be convinced to accept this change
<rekado>unfortunately, it doesn’t help with keywords. I can’t recognize srfi-88 keywords. Strings and numbers are easy, though.
<zimoun>civodul: thanks, I will drop an email later this evening to guix-hpc + CC the 4 other people.
<zimoun>rekado: awesome the extension. If the extension has the same name as a script, then the script is first. Maybe this precedence could be reverse? WDYT?
<zimoun>rekado: about string and WISP, in my mind . means “no new (“, so I am confused by your example.
<rekado>zimoun: I thought that extensions might be broken, so you’d always want the core Guix features untouched. But then it’s just a matter of unsetting GUIX_EXTENSIONS_PATH, so maybe it’s better to let extensions override core commands.
<rekado>that would allow for more flexibility
<rekado>for example, I could offer a new version of ‘guix deploy’ that adds AWS support, falling back to the existing ‘guix deploy’ script for non-AWS stuff.
<rekado>re wisp: there’s no dot in my example. With this change you can start a new line with a string without having that string be applied to anything else.
<rekado>so it does the right thing even without a leading dot.
<zimoun>about extension, I agree. What I have in mind is override ‘guix search’ for instance.
<rekado>yeah, I think it makes sense to invert the order.
<zimoun>re wisp, I see. Well, I need to read myself :-)
<rekado>I think I’ll do without srfi-88 and use (read-set! keywords 'prefix) instead
<rekado>I’ll lose keywords like ‘this:’, but I can still use ‘:this’
<rekado>I just didn’t want to have the noise of ‘#:this’
<rekado>with keywords that start with a colon I can tell the wisp reader to avoid application of keywords
<zimoun>sounds reasonable to me