IRC channel logs
2023-09-18.log
back to list of logs
<civodul>i love those testimonials because i know they come from folks who know how they ended up here <civodul>to someone new, it might feel like they landed on runguix.com :-) <civodul>yet we have nothing to sell, only derivations and backtraces! <jonsger>ACTION is especially interested in those "experience reports" on deploying Guix to HPC, clients etc... <rekado>I’m preparing my talk for tomorrow; it’s a 1 hour introduction to reproducibility, containers, and Guix to an audience of sysadmins. <rekado>the least pleasant part of all this is setting up the problem; hard to do without coming across as unfair <rekado>I’ll have to mention Conda and why it’s solving a *different* kind of problem <rekado>and later I’ll have to do the same to application container software <efraim>can you pin a specific conda commit or is it more like using docker and getting whatever version of libraries/binaries are included in that 'base' version (of debian or another distro)? <rekado>it’s mechanism is “name=version”, passed to a SAT solver. <rekado>you pin versions, not commits or hashes. <efraim>so similar to a classical distribution, but without the strict 'one version of a library in the archive' <rekado>in the past this involved a lot of LD_LIBRARY_PATH <rekado>more recent versions make fewer assumptions about the target system and include more “base libraries” in the dependencies to be installed <rekado>but it still happens with regularity that you can’t install packages on RHEL because the global glibc or gcc is too old. <flypaper-ultimat>ooh, will a video or the slides be available of it somewhere? I'm trying to convince our sysadmins. (though their main objections are that the daemon requires root) <rekado>I try to make a point of conda being a good thing for users (because it sure beats having sysadmins weigh the pros and cons of setting new defaults for all users of a shared system) <rekado>flypaper-ultimat: I hear it will be recorded, but I don’t know any details. <rekado>the root thing keeps coming up here as well <rekado>(does singularity still use a setuid helper?) <civodul>unless unprivileged user namespaces are enabled <civodul>but that’s still rarely the case on clusters <rekado>“Apptainer runs "rootless" by default. It allows unprivileged users to leverage containers and prohibits privilege escalation within the container; users are the same inside and outside the container.” <civodul>“by default” is important i guess :-) <rekado>I guess that a lot of admins would accept Guix if we just installed a setuid thing instead of a root daemon <civodul>there used to be one in the nix daemon <civodul>that is, the daemon could invoke a setuid helper <flypaper-ultimat>(our cluster apparently does have unpriviliged user namespaces enabled, to add a datapoint) <zimoun>it was a large project by Roberto Di Cosmo, Stefano Zacchiroli, etc. <zimoun>For tomorrow, maybe it is a bit too much. :-) <zimoun>In this old project, they study many topics that Guix now fixes, by design. <rekado>is this only a problem for *upgrading* or also for reinstalling an environment from an unchanged set of constraints? <zimoun>The problem to solve is: install this package given this repository. <zimoun>The question is thus the state (stability?) of the repository. <zimoun>Somehow, the “theoretical” question is: the SAT solvers (or similar) are they stable/efficient/robust/etc. enough to be practical considering the time-frame of a scientific projetc? <rekado>do we know anything about whether conda repositories are in practice immutable? <rekado>from Konrad’s message in the guix-science list I gather that the archive does seem to change over time <zimoun>I do not know. I remember discussions in id:86y1rt5xoz.fsf@gmail.com <rekado>> We published the environment file, but a few months later conda could not reconstruct it any more. They had updated the compiler infrastructure, which requires a rebuild of all packages. But they didn't rebuild all the versions from the past, so most older environment files becameunusable. <rekado>I find myself stumbling over Konrad’s messages, blog posts, and papers repeatedly, nodding along enthusiastically <zimoun>Because packages are added to the Conda index repository, the given repository R becomes R’. And the SAT solver finds a solution. However, because it is not using the same repository R vs R’, there is no guarantee that’s the same solution (although that’s the same set of constraints “name=version”) <zimoun>(I do not speak about un-tracked system-wide dependencies ;-)) <zimoun>From my point of view, the reproducibility of Conda environment relies only on the “quality” of the SAT solver. <civodul>yeah, i found ‘solvers.html’ above to be insightful <civodul>the diagram clearly shows that it’s stateful <zimoun>what would be really interesting is the example that Thibault started: rebuild a complicated computational environment once per month using a set of constraints. <rekado>zimoun: untracked system dependencies have been a recurrent problem in my limited experience <rekado>e.g. binaries that assume a more recent libc <zimoun>Yeah! I think Conda is doomed on two sides: system-wide dependency and solver. <rekado>IIUC some repositories are stricter than others when it comes to system dependencies <rekado>I did “guix shell conda” and then ran “conda env create --name=test --file=$HOME/Downloads/environment.yml” <rekado>took a while, but it installed a whole bunch of things <rekado>zimoun: I notice that the environment file does contain some hashes <zimoun>Well, after conda asks stuff about sudo… so I do not know if the script works. <zimoun>“guix shell -CN shell -- conda env create --name=test --file=$HOME/Downloads/environment.yml” works for me too. <civodul>ACTION loves that dependency on ‘sudo’ <civodul>i’ve done that for the repos hosted at gitlab.inria.fr <rekado>is there any payload to the request? <rekado>does the endpoint require validation / authentication? Or can anyone POST to /evaluate to trigger an evaluation? <civodul>anyone can POST to /evaluate, no token, nothing: party time! <civodul>there’s a threshold on trigger frequency, so it’s safe <civodul>pretty nice to get immediate feedback <civodul>well, “immediate”: evaluation still takes a bit of time