<zimoun>rekado_: piouf! I have almost clean all the mean I did with the wip-r branch and aside a couples of packages, all the updates build now. I will push it tomorrow. <zimoun>Well, I have learnt a lot about the workflow with this. :-) <PurpleSym>rekado_: I’ve been running into this “we just install unreleased -dev packages from GitHub that are totally incompatible with stable versions” paradigm as well. Is it possible to point the CRAN importer to a GitHub repository right now? <rekado_>it’s in the manual, but “guix import cran --help” is awfully terse <PurpleSym>Oh, you’re right. I must have overlooked that. <PurpleSym>I also thought about overriding install.packages, but for now I just disabled it via RStudio’s `allow-package-installation=0` <rekado_>I’m not an actual R hacker, so I don’t know if it’s possible to override the implementation of install.packages in the base library without patching it. <rekado_>I’d like to rename the existing implementation and fall back to it in case installation via Guix has been disabled <rekado_>using the same name would unlock installation of packages through Guix via RStudio and all other tools that happen to use install.packages. <PurpleSym>(The same is true for `pip install` as well) <PurpleSym>As for implementation, RStudio registers a hook using .rs.registerReplaceHook(). The implementation is in src/cpp/r/R/Tools.R (registerHook). Does not look RStudio specific right now, but I’m also not a R hacker. <rekado_>I’d like to make this change in R itself. I find RStudio a little confusing, because some of the features it provides are only possible because it implements its own shell around libR.so, so a bunch of features are not backed by the R language itself but by custom extensions in their own shell. <zimoun>rekado_: There is are still 9 packages that do not build. I am going to push to my branch on GitHub. I have rewritten the history and the each commit builds, aside corner cases. <zimoun>civodul, PurpleSym: about yesterday “workflow“ conversation, well, on one hand the answer seems .guix.scm or GWL, on the other hand, I do not think the old bad habbits will change. Recently, discussing about LaTeX and co, <https://infomath.pages.math.cnrs.fr/talk/2020-2021/latex/> (French), it is clear that a lot of bad practises are still used… and it si *so* slow to change. Anyway. <civodul>slow to change should not be an excuse for doing nothing ;-) <zimoun>civodul, rekado_: about R, colleagues and popularity, I am seeing more or less the same thing as rekado_ in my lab. It is easier to do quick&dirt and it works than fix the hard way. Especially when a lot of researchers do not really undertand what the tool is doing (not to say: they do not know what they are doing). Again, bad habbits are so hard and slow to change… out of scope of Guix, maybe. <zimoun>civodul: yeah, we should be able to produce PoC as you did with 10years later. And maybe one action could be to try that Guix should be part with the ReScience review process. <zimoun>but somehow, we are lacking manpower :-) Not to say bootstrap issue. ;-) <zimoun>it could be better to do older that 2019, e.g., 2016 or 2015, but the inferiors are not going so far, therefore, 2016 is more a guix-past exercise. <civodul>we achieved quite a few things during that hackathon a few months ago <zimoun>I put on my TODO list: try to reproduce a 2019 one and see how Guix fits the picture. But the TODO list is already too long to be efficient. ;-) <zimoun>well, it could be an interesting feedback for the future (hypothetical) day we have been discussed. :-) <rekado_>zimoun: you should use a vector instead of a list ;) <rekado_>zimoun: my take is that we can co-opt existing habits <rekado_>if people are used to installing stuff directly from github using devtools: let them! <rekado_>with a little bit of planning I think we can turn this habit into a valid use of Guix <rekado_>the R importer is one of the best, in my totally biased opinion; it produces R packages for Guix in a very reliable fashion that lends itself to automation <rekado_>but whenever PYTHONPATH is discussed my brain leaves me <rekado_>I’ve been an active participant in and instigator of PYTHONPATH discussions in the past, but nothing ever came of them <rekado_>the last I read was Hartmut’s analysis of our options and their downsides <rekado_>I do think we need to stop setting PYTHONPATH, but I lack the patience to re-read all past discussions to see a path forward <rekado_>I’ll gladly leave this to people who aren’t yet fed up with Python :) <zimoun>I agree about habbits. But similarly I am still seeing more than often \def instead of \newcommand with LaTeX. You can have the best luatex engine, if user use old bad habbits and shoot them in the foot, what is doable? <zimoun>one point is the person wants to mix the ‘pip -e’ packages with Guix packages. <rekado_>see, my experience has been that LaTeX *and* lualatex are considered low-level stuff by a new generation of R users <rekado_>so eventually these habits have a way of disappearing for new poor habits :) <rekado_>frankly, I think using ’pip’ is a valid use case <rekado_>but it’s one of those things that we shouldn’t break needlessly <rekado_>there can be fundamental obstacles, but anything beyond that should not be due to sub-optimal design on our side <rekado_>it is fine, for example, to install gcc-toolchain and build your own software without packaging it for Guix <rekado_>that’s often far from the best way to do things, but we don’t make it impossible <rekado_>there are quite a few things wrong with our use of PYTHONPATH, and they lead to actual problems; so while we’re motivated to look for alternatives it’s good to see if they collide with well-established habits <civodul>it's problematic that PYTHONPATH discussions haven't led to a concrete set of actions we could take <rekado_>there’s a danger in having a person take the lead just as much as there is in apathy <rekado_>I appreciate Hartmut’s emails, for example, but I fear that the result was that it all seemed much more complicated, so that *fewer* people were interested in moving forward. <rekado_>it’s hard to collectively circle in on a solution when the problem appears to spiral out of control and becomes less accessible to interested parties. <civodul>i remember being overwhelmed by these discussions <zimoun>rekado_: I agree with your points, metaphorically my point is The Right Thing is learn music (instrument theory, music thoery, etc.) and today people (scientifist) just grad an out-of-tune guitar, watch a video on well-known platform and practise enough just to play alone around the camp fire. Then they say: hey! I know music and I am not interested by trying your crazy instrument in this band even if I can do better with both. <zimoun>That’s your example with your or my colleagues, for instance. :-) <rekado_>zimoun: I’m going to lean far out the window now, but… I think it’s absolutely fine for people *not* to know music theory and have fun playing an instrument, even when they could do better and perhaps enjoy more. <rekado_>my lab colleagues certainly know their stuff <rekado_>but they use software as a means to an end <rekado_>if at some point in their journey they realise that they probably should have been a little more principled at the beginning — that’s fine <rekado_>often enough, though, it’s totally fine not to do the right thing from the start <rekado_>sometimes quick exploration is a goal in itself <rekado_>there are a few systems out there that claim to describe the final environment you ended up with, so that you can report on your setup for publications <civodul>one takeaway here is that guix should support quick exploration better <rekado_>this shows that people often start their explorations haphazardly <zimoun>exactly, I am perfectly on the same wavelength. And that’s why it is hard to fight and change the bad habbits. It is the downside of this journey. <rekado_>Guix has the principled side pretty well covered <rekado_>but we still have ‘guix install foo’ <rekado_>we have a lot of the pieces we need to extend a branch to those who often find themselves on the less principled side of things <rekado_>‘install via import’ is an idea that has popped up a few years back <rekado_>I think the R importer is very close to being ready for a feature like that <civodul>at the Guix Day, we discussed it again someone said "yeah Nix does it" <rekado_>I think we tried a little too hard to make it nice back then. <rekado_>we wanted to build not S-expressions but actual package values <rekado_>and that turned out to be harder than anticipated <civodul>but that was for one specific importer no? <rekado_>the print thing was used with the JSON importer, I think <zimoun>Not related: ‘./pre-inst-env guix build -m ~/tmp/Guix/list-bioconductor.scm -k -v0” only shows one fail when I know there are more. What do I miss? <rekado_>civodul: it’s a pity I don’t recall the details :-/ <rekado_>I just remember that I hit a serious roadblock <rekado_>zimoun: does it actually build them all or does it stop when it hits the first? <zimoun>I thought -k is to --keep-going so continue even with failure <rekado_>zimoun: oh, sorry, I thought that’s “-K” <rekado_>civodul: I think we can keep producing S-expressions for the time being, because that’s what we have. <rekado_>we would want to produce package expressions *anyway* <rekado_>doesn’t really matter if the package is generated from the expression or the expression from the package value <rekado_>for reproducibility and to enable exploration we should generate package expressions and save them somewhere <zimoun>rekado_: is it not what “guix import” already does? <rekado_>“guix import” generates expressions, but it neither saves them nor does it build them <civodul>rekado_: yes, but since we have print.scm, it may be simpler to generate a <package> and then convert it to an sexp <zimoun>guix import > foo.scm and I am not convinced that import (impure) should go with build (pure). <civodul>well, i guess one difference would be recursion <rekado_>zimoun: yes, redirection works, but foo.scm is not a module yet <civodul>anyway, we need to look at what it takes to appeal to our target audience, so to speak <civodul>and then plan to do work in that direction <rekado_>zimoun: so, you’d have to add a module header, then place the file in a directory, then set GUIX_PACKAGE_PATH and *then* you can finally build the thing <rekado_>but more realistically you’d then fix some mistakes in the foo.scm module first :) <zimoun>my point is just that I am not convinced that mixing impure (import whatever the output is) and build (pure) in only one go (guix install https:///foo.com/bar) is a right thing to do. <rekado_>it’s not “right”, but is it convenient? <zimoun>however, having “guix import“ independent of the ‘importer’ could be nice <rekado_>it also stops being impure when the installation operates on the generated (and thus now ‘static’) package expressions :) <rekado_>we’re talking about streamlining common workflows <rekado_>right now I still can’t tell people at work to use the CRAN importer because they need to know just a little too much to make it a useful tool for them. <rekado_>it only depends on the mutable outside at the time of invocation <rekado_>by that same token so is all in gnu/packages/bioconductor.scm <zimoun>and you get is not necessary what I get. <rekado_>and people can still share the generated ‘user module’ <zimoun>to me, it is not fine. Because it is the root of evil. <rekado_>they could even feel generous and contribute it as patches to Guix itself. <rekado_>after all it’s just a streamlined invocation of ‘guix import -r’ <zimoun>from my point of view, this “convience“ against “right“ leads to Dockerfiles hard to reproduce. But that’s only my opinion. :-) <zimoun>rekado_: there is still something that I am not understanding with ’build -k’,. Anyway it appears that somehow I have fixed the broken packages by fixing depend ones. <civodul>zimoun: thing is, there are plenty of options to do quick and dirty hacks <civodul>so if Guix doesn't help with the quick-exploration approach, people resort to Docker, pip, etc. <civodul>so i think it's fine to have tools that support that, as long as caveats are documented <civodul>we could also make that reproducible, actually, if the profile contains enough metadata <zimoun>I will think about it. But I almost sure that I disagree with this statement. :-) <zimoun>Using this “kind of argument” leads to run non-free because it is convenient, and it is not a problem as long as caveats are documented. :-) Well, purism vs pragmatism, somehow. <civodul>also note that you're the one using the term "purism" here :-) <civodul>it's a fact that we'll never have all the existing free software packaged in Guix <zimoun>purism in the meaning of pure vs impure, as functional approach. :-) <civodul>the question becomes: what do we offer to deal with software unavailable in Guix? <civodul>zimoun: yes, i don't like that terminology, seems a bit frightening to me, judgmental and all <zimoun>I agree with you about scientific software unavailable in Guix, and I remember asking exposing <license> for this very same reason and your arguments here were mine at the time, mine were yours. ;-) <zimoun>my question is simply: does it make sense to mix “import” (impure) with “build” (pure)? <zimoun>use the state of outside, not stateless, not: what you get is what I get (WYGIWIG :-)) <zimoun>Well, I will think about it. ;-) The streamlining and reproducibilit is a good question, IMHO. <zimoun>rekado_: I have 356 packages that build. Missing only 2. <rekado_>zimoun: excellent! Do you need help with the missing two? <zimoun>rekado_: yes maybe. Thank you for let me the time to learn. The journey was very fruitful. :-) I have tried to unknot the mess I created, well, it should be ok, even if maybe some propagated inputs still remain and they are not necessary. And I will try to summarize what I have learnt and then open a discussion with the Cuirass’s guy to see how to should be possible to avoid rebuilding all locally and avoid blocking with <zimoun>CI. Maybe use Bayfront, I do not know. <zimoun>I have to go now, see you later here. :-) <rekado_>zimoun: if you’re worried about extra inputs you could reset the version strings of all modified packages and re-run “guix refresh -t bioconductor” <rekado_>it would print all suggested deletions <rekado_>(and then you can throw away the reset version strings) <civodul>perhaps you could list some of the available packages beside RStudio (done or planned) <civodul>one goal here would be to get others to join <civodul>that means showing there's already people committed to growing the channel, and giving an idea of what packages would be welcome, how to contribute, etc <civodul>i'd remove the reference to the non-free counterpart <civodul>maybe clarify that "more relaxed" means "free software not built from source, such as npm stuff" (and not "non-free software") <civodul>maybe get rekado_ + zimoun + roelj to actually contribute ;-) <zimoun>I did nothing with guix-science, if I remember correctly. ;-) <civodul>it should be presented as an announcement of several research institutes joining forces and calling for contributions, IMO <civodul>in the short term, they'll probably stay separate <civodul>because colleagues have their habit, and it's very niche software anyway <civodul>but then, if we succeed at announcing guix-science loudly, it could change eventually <civodul>rekado_: did you hear from birdsite BTW? :-) <zimoun>civodul, PurpleSym: about the draft, maybe one line about the Substitutes and how to get them. From my PoV, the avaibility of subsitutes is one key of success. :-) <civodul>rekado_: is there any recourse? other ways to contact them, "lost my password" boxes or something? <rekado_>I wrote them several emails, responded to all (automated) emails they sent me, went through their help pages multiple times <rekado_>I don’t think there’s any more I can do <civodul>rekado_: thanks for taking the time to go through all this! <civodul>we're fortunate email, IRC, etc. don't work like that :-) <rekado_>it’s very unsatisfactory. Our “suspicious activity” that apparently deserves of termination of our account is to use Tor. <rekado_>but maybe they are just too busy fact checking realDonaldTrump, so their support staff can’t fix this for us. <zimoun>rekado_: I am rebuilding everything (detect missing HDF5 annoying empty dir) and I need to fix a couple of commit messages. Well, I have rewritten the history, so then nothing should be broken. The draft result is here https://github.com/zimoun/guix <zimoun>civodul: bug#42371 is annoying about massive rebuild «guix build: error: all build users are currently in use;…». So I end up with a shell for-loop :-(. And I am not sure to have the right tools to debug that. <civodul>zimoun: i don't experience it so you'll have to fix yourself muahaha <zimoun>it is because you are not building enough ;-) <zimoun>somehow, the scheduling strategy of the dameon is not clear for me <civodul>"scheduling strategy" is a strong word <PurpleSym>civodul: I’ll look at it on Monday again. There are three packages listed in the fourth paragraph. Getting people to contribute is the hard part and I don’t know how to do that. <PurpleSym>I just feel the three of us (rekado, roptat and I) are doing alot of duplicated work and it would be a good start to have one package for RStudio instead of three. Not a PR person. <civodul>agreed, and i understand you may not be enthusiastic about PR <civodul>that's probably what it takes if we don't want it to be yet another scientific channel tho <zimoun>PurpleSym: by roptat you mean Roel not Julien, right? <rekado_>would be nice to have a migration strategy <rekado_>I would, for example, add guix-science as a dependency to guix-bimsb and then re-export packages in guix-bimsb <PurpleSym>↑ That’s what I’ve been doing for a while when moving packages to Guix proper. <zimoun>rekado_: for now I have rebuild 235 packages and only 3 have non-dterministic issue (which I will not fix but only report :-)) <rekado_>zimoun: I might try fixing them if you tell me their names <zimoun>rekado_: for now r-methylkit-1.16.0 -r-biocparallel-1.24.1 r-rhdf5filters-1.2.0 <zimoun>on the branch, I still need to clean commit message since packages are marked as BROKEN but they are not anymore <rekado_>methylkit is developed by the lab here, so if there’s something that causes the build to be irreproducible I’m sure they’ll accept a patch to fix that <rekado_>let me know when the commit messages are fine; I’ll then fetch. <zimoun>ok, the propagated-inputs are not all clean yet <zimoun>rekado_: one more r-affycompatible-1.50.0 :-) ***rekado_ is now known as rekado
<zimoun>rekado: 2 packages are BROKEN as the commit message says. The rest builds. All the history, package per package, should build; even if I have not double-checked that. In the last state, all the packages from Bioconductor build; double-checked. All the propagated-inputs are not set at the minimum. I could finish to unknot the mess I did with that but before, I would prefer that you give a look and let Cuirass builds and check.