<civodul>i think that although those views are more popular now, they're losing ground
<civodul>both from a reproducible science viewpoint and from a security viewpoint
<civodul>it's becoming a harder sell to say "look, we have higher priorities"
<zimoun>civodul, rekado: They marks a point about “guix build guix --no-grafts --check”. Guix is not reproducible.
<zimoun>civodul, rekado: it is hard to rationally debate because there is a fundamental philisophical disagreement about what Scientific Knowledge means. Their tyhread is saying: let do engineering work. For sure but that’s not a scientific method. Their view is for HPC what cooking is for chemistry. That’s fine. It does not mean there is a good cooking against bad chemistry; or vice-versa. It is fundamentally
<zimoun>different. This is the question: if scientific method is not based on total transparency, how can we collectively verify that there is no mistake and thus the resulting knowledge is indeed grounded on universal facts.
<zimoun>Last, as I said when reviewing the blog entry, concerns about security is another space and I do not fully buy the Guix argument because 1. the bootstrap is run on binary linux kernel, so still chicken-egg problem and 2. hardware is easier to compromise by vendor than software, IMHO.
<zimoun>rekado: perhaps, a philosophical disagreement is maybe “religious”. ;-)
<civodul>zimoun, if you ever read this: i agree, this is about verifiability, not reproducibility
<civodul>if i were not atheist, i'd be religious about verifiability rather than reproducibility :-)
<drakonis>although i shouldnt be taking on more tasks than i can handle
<zimoun>civodul: about security, I will rehash it. Somehow, my point is against what do you protect? What is the surface? All this dance about bootstrap and reproductibility is against various attacks but this surface is not closed yet because kernel and hardware. For sure, bootstrap and repro helps. For one hand. :-)
<zimoun>For the other hand, what is the surface to introduce a backdoor to free software? CVE or mistake by packager speaks by themselves. Well, I am simply pragmatic here, I guess. ;-)
<zimoun>I agree that security and scientific stuff are the 2 faces of the same coin.
<zimoun>However, in the scientific case, I have something (e.g. a different plot) that triggers the audit.
<zimoun>In the security case, the attack hiddes as much as possible the something that could trigger the audit.
<Noisytoot>you missed: <civodul> zimoun, if you ever read this: i agree, this is about verifiability, not reproducibility
<Noisytoot><civodul> if i were not atheist, i'd be religious about verifiability rather than reproducibility :-)
<rekado>drakonis: I’m not sure of the details of what “student job” really means at the MDC, but you’re welcome to send me a quick email (email@example.com) and I’ll send you some info about the project and whatever I manage to figure out about the employment conditions.
<rekado>zimoun: “religious” is often used to mean “unreasonable” or “extreme”. Especially when the context is probably one of the most secular endeavors in human history. Nah, this is intentionally disparaging. Not cool.
<civodul>0. Is is fine to for tools and projects to have different goals, as long as it is clear to users. Calling the other views “religious” is uncalled for.
<civodul>1. We can’t build a secure software supply chain without reproducible builds—they give us verifiability.
<civodul>2. We can’t have reproducible builds unless we control the whole software stack.
<civodul>3. Does reproducible science really need bit-for-bit reproducibility, or is a laxer form or reproducibility sufficient? Bit-for-bit reproducibility is the only form that enables practical verification.
<civodul>4. Slightly off-topic because the post is not so much about Guix: Guix *is* used in #HPC and achieves performance portability (for MPI, for maths) through engineering means described at https://hpc.guix.info/blog.
<zimoun>civodul: mouais :-) For sure, fixing all the holes is the thing to do. But I miss how fixing the tiny mouse hole would really help to protect my house when in the same time some windows is still open. Anyway!
<civodul>i think the strategy here was to depict Guix as idealist and impractical, and conversely to depict Spack as pragmatic, practical, and the only way to achieve performance
<zimoun>«2. We can’t have reproducible builds unless we control the whole software stack.» To me, here “reproducible builds” is too strong. We can’t verify build unless we control the whole software stack. IMHO.
<rekado>oh, but this binary was downloaded from the nvidia website, and it has a version number. I control the whole software stack.
<rekado>it’s easy to disagree with the argument, but it does succeed at presenting spack as the more “reasoned”/“middle of the way” approach
<rekado>just by looking at the number of likes the intro tweet got I think this kind of reasoning is appealing to a lot of people
<zimoun>civodul: HPC cluster has to be treated as any other instrument to measure data. Do you ask that the microscope or genomic sequencer produce bit-to-bit data? And they are both a full stack of closed stuff.
<rekado>with computer-based experiments one of the most important inputs is the bits we use.
<zimoun>Cooking is reproducible and provide a knowledge. Why is not considered “scientific”?
<dstolfa>i wonder how anyone using proprietary software to reproduce results of science can be so sure that the black box doesn't just get very hot, and then printf some numbers that it fetched from some database that it initially submitted the results to :)
<zimoun>civodul: yes, point #3 will be attacked. Even by me I think. :-)
<civodul>zimoun: yes, i withdraw point #3 and refine it to software deployment :-)
<civodul>todd_: perhaps you wanted to comment so we can reduce the number of round trips? ;-)
<todd_>So hey I think I agree with nearly all of your points. Not advocating for some sort of enlightened centrism here. We should do all we can to improve reproducibility. Which is why we're trying to make proprietary stuff as reproducible as possible *too*.
<rekado>todd_: we gotta comb our hair before making our case in front of the twitter crowd :)
<drakonis>the twitter crowd will judge you with the eyes of a god
<todd_>How do you propose we use GPUs, at least before there is an open runtime for them? You'll find it rather hard to do well w/o libcudart.
<rekado>ah, that’s where the “religious” comes from ;)
<todd_>also note that of the forthcoming US exascale machines, we're using the GPUs with a more open software stack. It was a big deal for selection. Even AMD's stack comes with both completely open and proprietary pieces though.
<civodul>todd_: there's OpenCL, there's RoCM, but really, that's not what the article is about
<drakonis>there's a ongoing development regarding gpu runtimes
<drakonis>besides rocm and opencl, there's intel's thing as well
<civodul>though i'd certainly argue that building atop CUDA is a problem
<drakonis>and there's a cuda compiler being written in llvm that isn't proprietary
<rekado>drakonis: the point is though: practically speaking there is nothing but CUDA.
<todd_>If we didn't build atop CUDA, turnaround time for our physicsts would be a lot slower, and the AMD software stack is currently not pretty.
<todd_>I would claim it gives you a lot more than verification right now. Since I don't see anyone actually scaling repro build verification.
<zimoun>civodul, rekado: no the microscope analogy is *not* fallacious. Eveything looks like a nail when you have a hammer. :-) Microscope is exactly the same as HPC cluster: you can control very *well* everything. Applying your arguments to HPC cluster, you are requiring that full transparent microscope.
<todd_>but that's not to say you couldn't use it as a verification mechanism one day.
<rekado>zimoun: FWIW here at the MDC microscopes are like sequencers; the real fun begins with software (which in the case of our microscopy lab is all written by the phd students).
<drakonis>todd_: you should look outside of the hpc crowd for this
<civodul>todd_: we can tell about cases where non-verifiability has proved very problematic though
<drakonis>the nixos folks are the other side of the coin here and they are likely to have some examples
<civodul>the bitcoin-mining flatpaks, the Zoom spyware, etc. etc.
<todd_>I know there are repro build efforts outside HPC. If the claim is that verification provides security, my question is how. Verifying that a binary maps to source only gives you security if the source is trustworthy.
<civodul>of course, but then there's collective and individual control: i as an indivual can't tell whether this particular source is doing what it claims to do
<zimoun>civodul: this is the net of the issue when one thinks about producing scientific knowledge. How to deal with opaque tools in the chain? Do we consider these opaque tools as variability? As we do for almost any experiment?
<todd_>but in the meantime we do trust some vendors. It's the best we can do. We are not, as you say, "doomed". I think we do ok.
<civodul>also, i didn't write that the tools are "doomed", period
<civodul>i wrote: "these tools are doomed to be not only unsafe but also opaque"
<todd_>bitwise reproducible builds are great. If we can get to a point where we can build down to libc for spack, we will have them. IMO there are going to be places where you have to trust a binary for quite a while.
<civodul>perhaps "bound to" would be more appropriate
<todd_>and we should be able to repro those as best we can.
<civodul>not being a native speaker means lots of surprises :-)
<civodul>(also "these tools" was about the binary distros, pip & CONDA here)
<todd_>sure -- though we also rely on binaries in Spack and let people do "impure" builds.
<civodul>but i gather that it's the one sentence that triggered the reaction
<civodul>Spack is still primarily "source-based" though, so to me that's quite different
<todd_>My argument is that there is a lot you can do for reproducibility even if you cannot adopt a system that requires you to build down to libc.
<todd_>(and also that there are things below that -- and at some level, you trust something)
<civodul>yes, but software builds is what we work on--not CPUs, not hard disks, etc. :-)
<todd_>and yeah it sucks that CUDA is proprietary -- but we and a lot of sites use it, and at least for now we're not going to stop and sacrifice a ton of perforamance. But we would like our builds to be reproducible too.
<civodul>mind you, we have colleagues using it too, and with Guix
<civodul>and that's ok, but that doesn't mean we should not collectively question that
<civodul>from many different angles actually, not just "reproducible science"
<todd_>you could argue that though we use CUDA, we put several hundred million toward questioning that by supporting the competition
<drakonis>there's also intel's arc gpus, its interesting.
<civodul>todd_: anyway, i hope you'll concede that we're not religious but just holding different views
<civodul>and that the condescending tone was uncalled for
<civodul>the upside is that i now have a better understanding of our divergence, different priorities, and perhaps things that weren't clear in the post
<civodul>i guess we can have an argument on twitter and still be friends :-)
<todd_>I think we agree we should do as much as we can. Not faulting you guys for doing bitwise, but faulting other ecosystems for not doing it and ignoring the decisions they had to make also comes off as a bit condescending.
<todd_>I could argue that many of the Guix posts come across as rather sanctimonious -- if you read them, it comes off as "if it's not bitwise it's not worth doing"
<civodul>for software builds, there's a rather large consensus that "if it's not bitwise, it's not worth it"
<civodul>again, this goes way beyond our tiny community
<civodul>but also: this is just for software builds
<civodul>and perhaps that was a source of misunderstandings
<civodul>i do take note that the posts come off as condescending and sanctimonious
<dstolfa>bitwise reproducible builds might not matter as much for reproducing results in say, biotech, but they do matter for performance evaluation of systems (quite a bit...)
<dstolfa>it ultimately depends on what you're doing, but if i pass in the same flags on the same system and same hardware, i better get the same result...
<rekado>todd_: it doesn’t appear in advocacy posts obviously, but I’m dealing with conda users on a daily basis.
<dstolfa>at the very least it should be reproducible within that one system, at least IMO
<rekado>and from my perspective anything that makes users’ lives easier is worth doing
<todd_>I mean you guys aren't the only ones trying to get around conda :)
<todd_>it would be very awesome to see a system set up to do bitwise verification of binaries... regularly. And to do it in a distributed way. I am curious if you all have ideas for how to do anonymous verification effectively. Is there a scenario where you do not have to trust the verifier?
<zimoun>todd_, civodul: from my understanding, Guix introspects the tools themselves and try hard to apply the scientific method (i.e., transparency) to this introspection. Spack deals with cluster as any other experimental instrument, then treat the variability as any other scientific experiment deals with variability. Does it make sense?
<todd_>if you could do that, I think you could really crowdsource the verification.
<todd_>@zimoun: Spack has recipes just like guix. They just do not go all the way down to libc right now; we tag the builds with the OS used and the compilers. If you build everything with Spack, you can introspect all the builds above those things -- if you want.
<todd_>you can also say that, e.g., mvapich is "external" and you can use the system one.
<civodul>zimoun: it makes sense to me; now i understand the analogy