IRC channel logs

<rekado>zimoun: hello!

<rekado>I’ve been thinking about R and propagated inputs

<rekado>I’ve been stumbling upon a problem repeatedly: guix install r-something; time passes; guix pull; guix install r-whatever —> conflict

<rekado>this cannot be worked around with “guix pull && guix upgrade r-”

<rekado>because for some reason “guix upgrade” won’t upgrade package *variants*

<rekado>so colleagues who follow my advice of upgrading all R packages before installing new ones are confused

<rekado>and so am I

<rekado>the latest workaround is to use a manifest

<rekado>guix package --export-manifest > /tmp/m; guix package -m /tmp/m; guix install r-new

<rekado>this is clumsy, though

<civodul>rekado: i and others came to the conclusion that we should recommend 'guix shell' as the main tool for researchers

<rekado>yeah, guix shell is the best

<civodul>this is what Konrad suggested in https://10years.guix.gnu.org/video/guix-as-a-tool-for-computational-science/

<rekado>it’s predictably, declarative, fast, etc

<rekado>predictable*

<civodul>(last slide)

<civodul>yeah

<rekado>I don’t know why it’s not used more often here, though

<civodul>not that it helps you with your immediate problem though :-)

<rekado>it might be a mental block

<rekado>the name “guix install” suggests permanence and stability

<civodul>right

<civodul>until recently, i/we would advertise "guix install" as the way to get started

<civodul>because it looks familiar

<rekado>for things like guix.install() (in R) I don’t think I can use “guix shell”

<rekado>I’ll probably switch to building and installing manifests under the hood, though

<rekado>I wonder if we can ever do without propagation in R

<zimoun>rekado: here my advice is to only use manifest.scm (and channels.scm).

<zimoun>I just ban “guix install”, well it is just for a quick test but the only way to control the environment is manifest.scm + channels.scm.

<zimoun>Moreover, for newcomers I simply mention “guix shell”.

<rekado>right

<rekado>still, I feel icky about propagated inputs

<zimoun>Then, depending on the skill of the users, I explain profiles.

<rekado>I would like us to avoid the problem in the first place

<rekado>the cause of the conflict error is that we refuse to install variants of the same package into the profile directory

<rekado>that’s fair

<rekado>but we only attempt to install variants of the same package because of propagation

<rekado>and we use propagation because simply adding /gnu/store directories onto the load path wouldn’t help as the order of discovery is linear and not scoped per package

<rekado>the load path is a global variable, and the namespace registry mapping files to names is global, too

<rekado>I wonder if we can scope it

<zimoun>Yeah, I agree about propagated inputs. I remmber reading something on guix-devel or help-guix about a similar issue.

<rekado>(and I wonder the same for Guile)

<civodul>in pretty much all languages but Node.js, there can be only one module with a given name

<civodul>i'm not sure scoping would help if the language is not prepared for it

<rekado>yes

<civodul>what would help is a mechanism à la RUNPATH

<rekado>in R you can load namespaces but not attach them

<civodul>interesting

<zimoun>IMHO, we have a kind of dilema between “imperative” which requires a solver to avoid conflict and “declarative” which implies the manifest dance (no guix upgrade)

<rekado>and some packages then use the equivalent of Guile’s (@ (other namespace) foo)

<civodul>ok

<rekado>so there’s the assumption that namespaces have unique (and predictable) names

<zimoun>Well, I do not find back the message.

<zimoun>Hum, I am missing how propagated inputs is avoidable for R. But it is possible, for sure it will be better.

<zimoun>However, for the general case, the only way to avoid some conflicts of propagated inputs when using “guix upgrade” would be to add a version resolver… which would defeat the main message about reproducible computational environment.

<rekado>I don’t think it’s possible to avoid with R.

<rekado>I just read the code in src/library/base/R/library.R

<rekado>there’s a clear assumption that there’s a global mapping from plain names to objects; when a namespace has been attached it will be loaded

<rekado>when a namespace has been loaded it won’t be loaded again

<rekado>we would have to patch R significantly to work around this, and I’d rather not attempt it

<rekado>R packages would have to have their own namespace mapping (falling back to a parent environment); while the mechanism for nested environments exists it is not used for namespace discovery

<rekado>the machinery is also distributed between different procedures, each of which would have to be aware of the modified discovery mechanism

<zimoun>well, heavily patching R would lead to more work for maintenance

<rekado>taking a step back: could we just create a really long load path with absolute /gnu/store directory names instead of propagating things?

<rekado>let R deal with conflicts

<rekado>from R’s perspective it is totally fine to have multiple conflicting entries on the load path

<rekado>as long as nothing from conflicting locations is attached and loaded R won’t mind

<rekado>instead of having a package propagate its inputs it would merely record the locations of its inputs

<rekado>a profile hook would then assemble the R_LIBS_SITE value by concatenating and uniquifying the recorded locations for all leaf packages

<rekado>we would shift the problems (due to bad user habits) from Guix to R

<rekado>in other words: let R be the bad guy

<zimoun>is there no conflict when uniquifying?

<civodul>ACTION just had a tech discussion on the maintenance of Guix on https://www.grid5000.fr

<zimoun>civodul: oh cool! And so, conclusion? Guix is not enough staffed? ;-)

<civodul>heh :-)

<civodul>it was mostly a recap about what needs to be done when updating the daemon, whether/where to run 'guix publish', etc.

<civodul>currently there are ~8 sites, each of which runs 'guix publish'

<civodul>and so there's like 10 URLs behind '--substitute-urls'

<civodul>which in hindsight is a bit overkill :-)

<civodul>that's an interesting setup because that's a "cluster of clusters"

<zimoun>wait, they are serving substitutes. Is it not public?

<civodul>they're serving them on the "local" network, within Grid5000

<civodul>i don't think it's very useful though

<zimoun>well, it is useful for communicating, at least. :-)

<zimoun>I mean, somehow a kind of call. ;-)

<civodul>sure it can serve as an example

<rekado>zimoun: when each package installs its own search path, and the search path is made up of absolute file names then there should not be a conflict in generating the effective search path

<rekado>zimoun: my only worry is that the search path could get very long, and I don’t know if R is okay with that

<zimoun>rekado: hum, I do not know. Well, do you think it is worth to have a workaround for bad user habits?

<rekado>it’s not Guix’s job to police users

<rekado>there are two benefits: we get rid of propagation and we don’t fail until the user actually does something bad

<rekado>propagation clutters manifests, so getting rid of it is a benefit in and of itself.

<rekado>bah, this is disgusting: https://github.com/xnnba1984/DoubletCollection/blob/c0d62f1853942ee6a087eaf7b000d9e4261e2dfd/R/FindDoublets.R#L99

<rekado>r-doubletcollection from Guix ends up trying to install some python thing via conda

<rekado>I find code like this to be incredibly disrespectful

<rekado>this screwed up a user’s conda installation, and since conda doesn’t have roll-backs it’s messy to restore the environment

<zimoun>It is not about police users. It is to set “something bad” with more stringency. ;-)

<zimoun>Yeah… « reticulate::py_install("scrublet",pip = T) » is… ouch!

<zimoun>ACTION has to go, see you.

<rekado>sorry, “police” is too strong

<rekado>to me it’s an unnecessary restriction brought about by the fact that we use propagation

IRC channel logs

2023-01-27.log