IRC channel logs
2021-08-04.log
back to list of logs
<PurpleSym>Phew, clinfo works with a Guix-built libamdocl64.so now. Do we have anything in place to tell the ICD loader where to find it? It seems it is looking at my system’s /etc/OpenCL directory. <civodul>PurpleSym: i think it's needed supposed to check /etc/OpenCL, no? <PurpleSym>Yeah, but thankfully there’s a search path if you install ocl-icd :) <civodul>BTW, i got my first successful pytorch (CPU) build yesterday <PurpleSym>Any chance of getting ROCm into Guix proper? It’s entirely open source, but – well – it needs firmware blobs in the kernel. So probably nonguix territory? <civodul>i suppose the userland bit (the part that's free software) could go in Guix proper <civodul>proprietary firmware is needed for AMD GPUs? <PurpleSym>Yes, their GPU’s don’t do anything useful without firmware nowadays. <civodul>so "pip install torch" downloads an 830MB wheel that contains things like libgomp.so and libcudart.so (!) <civodul>i think we should take a step back and raise awareness of these issues <civodul>cuz i/we keep preaching to the choir but the vast majority of users probably doesn't understand why we'd bother with Guix packaging when "pip install" works "so well", even for non-Python packages like this one <rekado>it’s a real challenge to communicate. What could be wrong about this when it evidently works? <rekado>it’s the same barrier that needs to be broken when communicating what functional package management brings to the table <rekado>the first brick falls when people admit that conda/docker/whatever may not be as “elegant” as Guix, but it “does get the job done” <rekado>but I feel that to many users this reframing as “elegance” makes Guix seem “academical” in the worst sense. <civodul>rekado: yeah, it's not really helping <civodul>it's like "you folks are doing nice work, thumbs up; in the meantime, i'll get the job done with that other tool" <rekado>and this is the exact response I got from some users at the MDC <civodul>perhaps it's a mistake of ours to communicate on improvements to what we already have, when the basic motivation is not well understood outside of our circles <efraim>perhaps a more in-depth example of using transformations and manifests? I was thinking of writing something up. <efraim>currently I'm replacing sdl2 with a later version and grafting openssl compiled with -march=native for the marginal boost <rekado>“you are doing nice work, thumbs up; in the meantime, I’ll get the job done with LD_LIBRARY_PATH.” <civodul>efraim: that'd make for a great blog post! i feel we need more hands-on posts, like "how i did X" <civodul>is "pip install" actually supposed to work in Guix? <civodul>it downloads stuff and eventually fails with "Read-only file system", apparently while trying to write to /gnu/store/...-python <efraim>so... `guix environment pytorch -- pip install --user pytorch --no-binary all` or something? <PurpleSym>Or in a virtual environment. I do that quite often with editable projects. <civodul>tested in: guix environment -CNP --ad-hoc python-wrapper nss-certs openssl coreutils grep sed findutils <civodul>so, isn't pip stepping on CONDA's toes? <wehlutyk[m]>I was following the conversation from afar and thought I can add something to the communication challenge mentioned earlier <wehlutyk[m]>you're probably aware of this, but the incentive system in academia is also working against people being able to see this problem. A PhD is 3 years, with evaluations every year. Most postdocs are 1 or 2 years, sometimes down to 6 or 9 months. <wehlutyk[m]>The pressure for stable positions after postdocs is huge and growing. So it's very difficult for people to spend time making sure their code will reproduce 10 years down the line. <wehlutyk[m]>This is probably even worse in deep learning where the shelf life of a new network architecture is less than 2 years <civodul>i think pushing scientific reproducibility, and thus reproducible deployment, also means inevitably bringing such issues to the table <wehlutyk[m]>On the other hand, my experience is that Guix is an easy sell for HPC users outside computer science and physics, as larger teams need to install a lot field-specific software, and coordinate people's environments (at least this is what I'm seeing now in neuroscience -- the only challenge still being that I haven't found the time to get a shared daemon to run as non root) <civodul>first time i hear it's an "easy sell" :-), but yes, it should be possible to make a point that Guix helps with these things <wehlutyk[m]>maybe some publishers or funders could change that pressure, although I don't see anybody forcing people to use a specific package manager :) <civodul>i think we need to raise awareness of the issues before we can discuss specific tools <civodul>rekado: we have xnnpack now, which i think tensorflow-lite could depend on <civodul>(not sure what it brings exactly, but surely something) <drakonis>the pitch could use some mild touching up, convincing people to try it after getting too comfortable with nix has been an uphill battle <drakonis>explaining to them why they should use it over nix <drakonis>most of the time the appeal to them has been nix's package availability <rekado>civodul: woo, excellent job packaging pytorch! <rekado>civodul: tensorflow-lite can use xnnpack, but it insists on building it from source. I wanted to patch the build system later to include xnnpack. <rekado>I don’t think it’s that important to convert nix users to Guix. <rekado>drakonis: I’m pretty sure that the R packages in Nix are not actually all usable. <rekado>their quantity over quality approach is frustrating to me, because it means that I need to do all the hard work instead of being able to just copy :) <rekado>also: their tensorflow-lite package only builds the .a lib, which is not great. <rekado>as soon as you need a system library it gets bad <rekado>and you know… in that case you might as well use install.packages, devtools, ratpack, or the many other pure R package managers. <drakonis>what frustrates me about nix is that besides the obvious quantity over quality problem, is that it is really inconsistent <drakonis>it has a dozen ways to do the same thing, each one with specific edge cases and focus <drakonis>also i was pulled into a conversion with a haskeller interested in guix because he got burnt by nixpkgs maintainers being really irresponsible with changes <drakonis>one of the things he pointed out was that there's a combinatorial explosion in nixpkgs <drakonis>there is a set of packages duplicated for each different ghc version <civodul>drakonis: i guess some would argue that having one package set per ghc is a feature <civodul>back when i contributed there wasn't much of an effort to keep things consistent (there was no peer review actually) <civodul>but things have improved since then, it seems <drakonis>nixpkgs is still being blocked for days though <civodul>i mean, scaling up peer review is hard <drakonis>they have bots for checking if a patch builds <drakonis>necessary when they get hundreds of patches a week <drakonis>it's impressive how they haven't dropped below 4 digits of issues and PRs <dstolfa>civodul: does guix have a way to say "please build all of the packages that depend on this package"? if so, it might be possible to scale up review in a way that a buildbot can just build everything in CI and give automated feedback, at least for some things. <drakonis>it would be interesting to have a branch that has automatically bumped packages <dstolfa>drakonis: these are also many bumps that look trivial, but aren't <dstolfa>2 line change, it builds, great! except like thousands of packages fell over as a result <drakonis>a substantial amount of PRs in nixpkgs are that <civodul>here you can see patches from guix-patches and red/green buttons telling whether they apply, build, etc. <drakonis>regarding sets of ghc packages, it should be trivial to write a function to switch the ghc version for the packages, right? <dstolfa>civodul: cool, so this is basically building all dependents of the package too? <dstolfa>drakonis: guix already has that for a number of languages <civodul>dstolfa: ah that i'm not sure actually, but it could <dstolfa>yep, but the build farm is very big... :) <dstolfa>and would prevent fallouts of untested things <drakonis>dstolfa: i mean invoking map to apply the package to a set <drakonis>i see a certain appeal in mentioning that <drakonis>afaik, all build systems let you pick which package you want to use for the build, but there isn't any written example on how to get a group of packages to build using a different version <drakonis>that works for me, but they might not understand how that correlates to this <drakonis>by the way, why isn't isnt the manual pointing to the latest commit by default? <drakonis>people tend to read the manual for 1.3.0 and not the current one <drakonis>i've had an occurrence of someone checking the manual and looking for a service that's already included in the latest revision <civodul>it'd be nice to have version links in a corner of the manual web pages <drakonis>it's more in the interest of not having to tell people that they're looking at the manual for the stable version and that it is out of sync with the latest one <drakonis>since search engines like google point to the stable manual by default <drakonis>civodul: thanks for your work with kicking off guix btw <drakonis>dstolfa: that is indeed what i want, but applying that to a group with packages, map would probably do the trick? <dstolfa>drakonis: is there a reason it can't be explicit in the packages themselves? it would be good for it to be explicit, if only for people looking at it later to be able to see that it depends on a particular compiler version <dstolfa>huh. why do you need to duplicate the packages? <dstolfa>i'm not sure i fully understand your problem <drakonis>okay so, nixpkgs has a series of sets of packages with different compiler versions <drakonis>its not really needed, but there's people that don't really understand how to set the compiler version for each package for their own local consumption <dstolfa>drakonis: i'm not sure i understand why this is necessary in the first place, what's wrong with using the default? <drakonis>how do i put this, rather concerned with how they don't get to pick which compiler version they want to use for the software <dstolfa>and is there a reason they are rather concerned, or is this just complaining for the sake of complaining? i'm asking because there might be a way to achieve what they want without any of this <drakonis>unless they want to test behavior against different compiler versions <drakonis>there is always a way to achieve what they want without making it overly complex <dstolfa>drakonis: uh, the simplest way to do this i guess is to just change the default compiler version <dstolfa>see gnu/packages/rust.scm, the very last line <drakonis>i mean doing things like getting a group of packages and defining them with a different compiler i guess <drakonis>programatically so they dont need to write a shitload of code i guess? <dstolfa>if they want to do it for everything, just set the default compiler to something else build everything locally i guess <dstolfa>use a custom guix channel and that's it <dstolfa>it's doable, it's just not really sensible IMO <dstolfa>i understand some people need to change to compiler for one reason or another, but if the reason is "i wanna use a different compiler", then it's probably best to just change the default and build everything locally *shrug* <dstolfa>of course you can map over it since it's just guile records but i feel like it's not the prettiest code <drakonis>technically this is the same kind of hackery that happens in nix afaik <dstolfa>they'd probably need to check that the builds they're doing are reproducible and compose well with the rest of guix... there's no saying what changing a default compiler might do (or building some with 1 and others with another without testing) <drakonis>i don't think that'd be much of a consideration right now <drakonis>i was checking how it defines the packagesets and woah this stuff is just ugly