IRC channel logs


back to list of logs

<rekado>not me
<civodul>i love those testimonials because i know they come from folks who know how they ended up here
<civodul>to someone new, it might feel like they landed on :-)
<civodul>yet we have nothing to sell, only derivations and backtraces!
<zimoun>Yeah, cool! :-)
<jonsger>ACTION is especially interested in those "experience reports" on deploying Guix to HPC, clients etc...
<rekado>I’m preparing my talk for tomorrow; it’s a 1 hour introduction to reproducibility, containers, and Guix to an audience of sysadmins.
<rekado>the least pleasant part of all this is setting up the problem; hard to do without coming across as unfair
<rekado>I’ll have to mention Conda and why it’s solving a *different* kind of problem
<rekado>and later I’ll have to do the same to application container software
<efraim>can you pin a specific conda commit or is it more like using docker and getting whatever version of libraries/binaries are included in that 'base' version (of debian or another distro)?
<efraim>ACTION has never used conda
<rekado>it’s mechanism is “name=version”, passed to a SAT solver.
<rekado>you pin versions, not commits or hashes.
<efraim>so similar to a classical distribution, but without the strict 'one version of a library in the archive'
<rekado>in the past this involved a lot of LD_LIBRARY_PATH
<rekado>more recent versions make fewer assumptions about the target system and include more “base libraries” in the dependencies to be installed
<rekado>but it still happens with regularity that you can’t install packages on RHEL because the global glibc or gcc is too old.
<flypaper-ultimat>ooh, will a video or the slides be available of it somewhere? I'm trying to convince our sysadmins. (though their main objections are that the daemon requires root)
<rekado>I try to make a point of conda being a good thing for users (because it sure beats having sysadmins weigh the pros and cons of setting new defaults for all users of a shared system)
<rekado>flypaper-ultimat: I hear it will be recorded, but I don’t know any details.
<rekado>the root thing keeps coming up here as well
<civodul>yeah, i’ve heard it so many times
<rekado>(does singularity still use a setuid helper?)
<civodul>it has to
<rekado>(singularity = apptainer)
<civodul>unless unprivileged user namespaces are enabled
<civodul>but that’s still rarely the case on clusters
<rekado>“Apptainer runs "rootless" by default. It allows unprivileged users to leverage containers and prohibits privilege escalation within the container; users are the same inside and outside the container.”
<civodul>“by default” is important i guess :-)
<rekado>I guess that a lot of admins would accept Guix if we just installed a setuid thing instead of a root daemon
<civodul>there used to be one in the nix daemon
<civodul>that is, the daemon could invoke a setuid helper
<flypaper-ultimat>(our cluster apparently does have unpriviliged user namespaces enabled, to add a datapoint)
<zimoun>rekado: a good reference is
<zimoun>“managing software complexity“
<zimoun>it was a large project by Roberto Di Cosmo, Stefano Zacchiroli, etc.
<zimoun>For tomorrow, maybe it is a bit too much. :-)
<zimoun>In this old project, they study many topics that Guix now fixes, by design.
<zimoun>The model of Conda with “name=version” leads to solve a NP-Complete problem for installing.
<rekado>is this only a problem for *upgrading* or also for reinstalling an environment from an unchanged set of constraints?
<zimoun>The problem to solve is: install this package given this repository.
<zimoun>The question is thus the state (stability?) of the repository.
<zimoun>Somehow, the “theoretical” question is: the SAT solvers (or similar) are they stable/efficient/robust/etc. enough to be practical considering the time-frame of a scientific projetc?
<rekado>do we know anything about whether conda repositories are in practice immutable?
<rekado>from Konrad’s message in the guix-science list I gather that the archive does seem to change over time
<zimoun>I do not know. I remember discussions in
<civodul>there was a good thread on guix-science:
<rekado>> We published the environment file, but a few months later conda could not reconstruct it any more. They had updated the compiler infrastructure, which requires a rebuild of all packages. But they didn't rebuild all the versions from the past, so most older environment files becameunusable.
<rekado>I find myself stumbling over Konrad’s messages, blog posts, and papers repeatedly, nodding along enthusiastically
<zimoun>Yeah, I think that’s expected and documented by Conda
<zimoun>Because packages are added to the Conda index repository, the given repository R becomes R’. And the SAT solver finds a solution. However, because it is not using the same repository R vs R’, there is no guarantee that’s the same solution (although that’s the same set of constraints “name=version”)
<zimoun>(I do not speak about un-tracked system-wide dependencies ;-))
<zimoun>From my point of view, the reproducibility of Conda environment relies only on the “quality” of the SAT solver.
<civodul>yeah, i found ‘solvers.html’ above to be insightful
<civodul>the diagram clearly shows that it’s stateful
<zimoun>what would be really interesting is the example that Thibault started: rebuild a complicated computational environment once per month using a set of constraints.
<rekado>zimoun: untracked system dependencies have been a recurrent problem in my limited experience
<rekado>e.g. binaries that assume a more recent libc
<zimoun>Yeah! I think Conda is doomed on two sides: system-wide dependency and solver.
<rekado>IIUC some repositories are stricter than others when it comes to system dependencies
<zimoun>For instance, are you able to regenerate this Conda environment ?
<rekado>I think so, yes
<rekado>I did “guix shell conda” and then ran “conda env create --name=test --file=$HOME/Downloads/environment.yml”
<rekado>took a while, but it installed a whole bunch of things
<rekado>zimoun: I notice that the environment file does contain some hashes
<zimoun>Well, after conda asks stuff about sudo… so I do not know if the script works.
<zimoun>“guix shell -CN shell -- conda env create --name=test --file=$HOME/Downloads/environment.yml” works for me too.
<civodul>ACTION loves that dependency on ‘sudo’
<civodul>rekado, PurpleSym: you can configure the guix-science etc. repos to send a POST request to (where NAME is the jobset name, like ‘guix-science’)
<civodul>i’ve done that for the repos hosted at
<civodul>hi tech
<rekado>is there any payload to the request?
<rekado>does the endpoint require validation / authentication? Or can anyone POST to /evaluate to trigger an evaluation?
<civodul>anyone can POST to /evaluate, no token, nothing: party time!
<civodul>there’s a threshold on trigger frequency, so it’s safe
<civodul>pretty nice to get immediate feedback
<civodul>well, “immediate”: evaluation still takes a bit of time