<civodul>Steap: make sure to send me your slides! <Steap>civodul: they should be available on Pycon.fr soon! <ArneBab_>davexunit: that’s peoplel who don’t understand that autotools have their features for good reasons… <davexunit>ACTION sent another message to the rubygems folks <davexunit>I don't think they understand where I'm coming from, and I'm struggling to make it clear to them. <davexunit>this situation is so depressing. I've put sooo much effort into Ruby on Guix and it's probably all for nothing. <mark_weaver>davexunit: I don't know much about this, so maybe it's not feasible, but what about going back to the plan of doing "git init" to create a fresh repo from the pristine source checkout? <cehteh>haha .. i should grab some popcorn and wait for them to reimplement guix/nix in ruby <cehteh>i sometimes feel ruby has some rather strong sworn in commmunity <mark_weaver>I confess I'm only vaguely aware of the issues here, so perhaps I'm just spouting nonsense :-/ <davexunit>mark_weaver: that gets around one major problem, but it will really suck to go back to that. <davexunit>there's other, smaller issues that the 'git init' trick doesn't solve like auto-generated ChangeLog or perhaps more important files that need the actual git history. <davexunit>in practice, a good number of gems on rubygems.org do have test suites and things <davexunit>and the ones that don't are still source code releases <davexunit>so my hope was that I could convince the rubygems maintainers to care about users being able to run tests on gems going forward, but they seem to only care about making the smallest possible gems that only include files needed at runtime. <davexunit>I have pointed them at PyPI as an example to emulate, but they don't seem interested in learning from them, either. <davexunit>cehteh: the ruby community is not very friendly. <cehteh>i didnt wanted to say that so directly :) <davexunit>it seems that every time I interact with another ruby programmer I receive poor treatment. <davexunit>I got thoroughly flamed for daring to ask a question in #ruby without having a test case already written to show the problem and using 'foo' and 'bar' as example variable names. <davexunit>and apparently no one sympathizes my desire for independent verification of gem tests in any sort of automated fashion. <davexunit>they claim that gems are binaries (yet in practice they are source code) but have no concept of a source package. <davexunit>and Bundler, the ubiquitous dependency management tool built on top of RubyGems, has a script that generates the boilerplate for new gems. that script creates a very important file, the gem specification file, that *shells out to git* to determine the list of files to include in releases. <davexunit>I think we can all agree that coupling the build system to the version control system isn't great design, but nearly every single gem is inflicted with this. <davexunit>the whole system appears broken, yet it feels like no one but me sees any problems. <mark_weaver>davexunit: if their position is that the gems don't have full source code with tests, then maybe we can't use gems after all. <mark_weaver>maybe we need to go back to using the source code repo directly <davexunit>mark_weaver: their position seems to be that they don't encourage shipping tests, however a good number of developers do this. <davexunit>but the future doesn't look bright, and that worries me. <davexunit>they say that a gem should contain, among other things, "Code (including tests and supporting utilities)" <davexunit>so there's some serious confusion even amongst the rubygems maintainers about what a gem even is. <efraim>40 minutes on build phase for the mplayer-1.2 update <efraim>"Starting from this release, MPlayer no longer ships with an internal copy of libdvdnav and libdvdcss. You can use the libraries provided with your distribution, or compile and install them yourself. They will be autodetect at configuration time." time to fix the inputs and build again <civodul>that's a move in the right direction <efraim>it looks like libdvdnav and libdvdcss aren't included because they're used by ffmpeg <efraim>also from their webpage: "The tarball already includes a copy of FFmpeg, so you don't need to fetch it separately." <civodul>davexunit: bah, you're sending us bad news re gems :-/ <civodul>well that's fine, we'll let it rest for a while <cehteh>davexunit: still understandable with the upstream attitude .. but about the argument that a .git is not deterministic, cant it be hacked somehow to consider it deterministic when the git branches/tags etc have the correct hash. from such a viewpoint git is deterministic and its storage internals are just implementation details <cehteh>i think that would cover a lot other things as well and i like the idea not to roll release tarballs or whatever but give some git hash and done <cehteh>(completely ignore .git, what matters is the checkout) <davexunit>cehteh: but you can't import something downloaded from the net into the guix store without specifying it's hash <davexunit>so, until git clones can be made deterministic, this will continue to be a problem. <cehteh>i think that needs to be reconsiderd .. a git checkout is all what needs to be deterministic <davexunit>cehteh: it can't be reconsidered, how can you possibly trust something whose hash changes with the wind? <cehteh>the hash of the checkout needs to be trusted and stable of course <davexunit>source code downloaded through git is not exempt from strict hashing requirements. <davexunit>the way we handle git repos right now is by downloading them and deleting the .git directory. <cehteh>but i can trust an arbitary .git even if someone stashed his porn archive mixed into a linux source tree <cehteh>as long i have a hash to checkout i wont see these <cehteh>except for the .git deletion it could be just ignored and/or cleaned to some extent <davexunit>but deleting the .git directory breaks the ruby build system. almost every gem relies on running 'git ls-files' in order to build. <davexunit>what would be great is a variant of git clone that is deterministic, importing only the objects that reflect the history of a desired commit. <cehteh>even if .git contains malicious content, as long you can checkout a trusted hash you get what you asked for <davexunit>except the .git directory is different every time someone pushes a commit <davexunit>and thus a machine cannot verify that the source it got was the same as what the packager got. <cehteh>what really doesnt matter to your checkout <cehteh>thats why i say just ignore .git <cehteh>one could reset and filter branch to some deterministic state <davexunit>along with every other source of nondeterminism. <cehteh>throwing away everything not needed <cehteh>but still packing etc may yield non deterministic packfiles <davexunit>if the git cloning process can be made deterministic, I would be very excited. <cehteh>i think its more irrelevant than you think <cehteh>what matters is that the checkout and the history of some commit is deterministic <cehteh>everything beyond is just a implementation detail <davexunit>you're missing a key detail: the directory that is considered the source for a package build *must* match a SHA256 hash in Guix. <cehteh>see it like your hdd or ssd storing data in some bitpatterns which might be not deterministic as well <davexunit>and it's not an irrelevant one, nor something that should be relaxed. <cehteh>i tihnk thats rather something guix should work around <davexunit>to relax that requirement compromises the entire system. <davexunit>and we already work around it... by removing .git entirely. <davexunit>preserving the git database is currently impossible. <cehteh>a simple (but not complete) fix would be to ignore .git instead deleting it (besides you can do out-of-tree checkouts with git, that would be saner) <cehteh>next sanity check would be to confirm that the current checkout really matches the intended git sha and is reachable/included in the .git <davexunit>but the .git directory cannot enter the build container because it's a known source of non-determinism. <cehteh>when i discovered nix years ago i was rather thinking "why dont they build the store on top of git" :D <cehteh>i mean there are always ways to shoot yourself into the foot .. if your build is not deterministic then it is not deterministic, having .git there or not might just be some way to generate some non determinism *if* you abuse is in some ill way <cehteh>note that (IIRC) some time ago there where some probabilistic optimization strategies which led to non deterministic compilation in gcc .. dunno what happend to that if it got removed or disable <cehteh>such would completely defeat guix/nix .. but still be some implementation detail <davexunit>that's more than an implementation detail to us. <davexunit>our goal is bit-identical binaries. we're not there yet, but we're working on it. <cehteh>but the non deterministic build is the culprit, not the existence of .git <davexunit>the existence of .git *is* a problem because it's downloaded from the net and we have to make damn sure that what we fetch from the net is bit-identical to what we expected to receive. <mark_weaver>cehteh: ignoring the .git file, but leaving it there, would amount to putting ones faith in a rather dubious assumption: that the contents of the .git do not matter <davexunit>there's simply no room for ignoring anything. <mark_weaver>if we need the .git directory because the build system will be performing git operations, then it would entail taking on faith that the results of those operations somehow always end up the same <cehteh>mark_weaver: not faith, your *checkout* aka source should have a determinstic hash .. and the binary generated then should be deterministic as well <mark_weaver>cehteh: but ruby build systems run git commands as part of the build. that's why we need to keep the .git directory. <cehteh>iff the build system does some magic git stuff which makes the build non deterministic then its not git's fault <cehteh>even running git commands should be fine, as long it stays deterministic <mark_weaver>so we'd have to be convinced that the differences in the contents of the .git directory cannot result in any change in the git operations. <cehteh>if not, its broken, i completely agree <mark_weaver>cehteh: that's a very slippery slope, to start making assumptions like that <davexunit>fixed-output derivations need a hash. they need to be deterministic. the real solution here is for git to provide a way to do a deterministic checkout <cehteh>a git checkout *is* deterministic <mark_weaver>next thing you know, the security of our entire system depends on a huge code base, when what we're trying to do is very simple: verify that the downloaded source matches what we expct. <davexunit>cehteh: if I run git clone on a repo, compute the hash, then commit to that repo, re-clone it, and compute another hash. those two hashes will differ. <mark_weaver>I've tried to explain, and it's not getting through, and I don't have any more time to spend on this now. <cehteh>seems some misunderstanding on both sides <mark_weaver>for most of our packages that take sources from git repos, the contents of the .git directory are not used at all during the build, so we can simply remove it, and then there's nothing non-deterministic left. <cehteh>yes .. but killing a .git is like killing a kitten :D <cehteh>would be nice to have the .git stores externally and do checkout into your build dir <mark_weaver>but if we need to keep the .git directory around, then that's a potential source of non-determinism, and who knows how it could be used for malicious purposes. if we don't verify it, we are opening ourselves up to someone exploiting security holes in git, for example. <mark_weaver>not to mention that git uses SHA-1 as the hash, and SHA-1's security is not very good anymore. <cehteh>you need to trust git that it cant be used that way true <mark_weaver>davexunit: I don't understand why "git init" as an early build phase is so bad. <mark_weaver>is there a practical problem with that approach? or maybe it doesn't solve all of the problems? <cehteh>anyway .. keeping a .git could save a lot bandwidth instead cloning again, you may consider to have some mutable .git store somewhere and do the checkouts elsewhere so that no .git taints them but it stays somewhere safe <mark_weaver>on GuixSD, if you add that to the list of 'packages' in your OS config, it might automatically show up as an option from the 'slim' display manager. <mark_weaver>if not, you can certainly make a ~/.xsession file that runs it manually <serhart>mark_weaver: yeah, it showed up for me in slim when I added it to my os config <davexunit>mark_weaver: it solves a major problem, which is good, but I've encountered other problems where more things required git, but actually needed the real history to generate the right file. <davexunit>those are lesser problems, so maybe we can add an optional phase to 'git init && git add .' when necessary. <davexunit>and be selective about which gem archives are OK and which need to be bootstrapped. <paroneayea>davexunit: I think joeyh is mostly right about git checkouts being a perfectly fine, even great approach <davexunit>paroneayea: *if* they can be made deterministic. <paroneayea>davexunit: in some ways, if done appropriately, maye even better than a tarball: it may be that a tarball has some pre-release steps that you can't reproduce from the official source <davexunit>yes, mark_weaver has also expressed interest in doing the bootstrapping ourselves. <paroneayea>it might be wayyyy too much history to check out <paroneayea>I wonder if our git checkout command currently does that... <davexunit>but I think we can only do shallow clones in certain cases <paroneayea>happy to see all this haskell love hit the list, thanks Paul, if you're on IRC :) <zhong>mark_weaver: I add i3-wm to (packages (cons* i3-wm xfce...)), I run "guix system reconfigure /etc/config.scm" and get error "In procedure module-lookup:Unbound variable: i3-wm" <davexunit>zhong: you must import the module that contains that variable <efraim>zhong: you have to declare it at the top also <davexunit>i3-wm is a variable in the (gnu packages wm) moduel <paroneayea>davexunit: I almost considered posting your email thread to /r/linux but decided maybe that would stir up unnecessary trouble :) <davexunit>paroneayea: yeah, I was considering similar things, but decided against it for the same reasons. ***Xe is now known as shadowh511
***shadowh511 is now known as Xe
***aeva` is now known as aeva
<civodul>iyzsong: it seems Inkscape fails to build with the newer GLibmm <civodul>in turns that prevents us from building a system <civodul>oh i think i have a fix for that one <davexunit>ACTION posted one last response to the rubygems thread <davexunit>just wanted to reiterate that they still do not understand my concerns. <davexunit>the person whose username is e2 clearly does not understand what I've asked for. <davexunit>he doesn't see why a package maintainer would need source code! <davexunit>or why a package maintainer would want to run a project's test suite! <civodul>"they don't really need the sources" <davexunit>but yeah, I guess we'll have to evaluate on a case-by-case basis whether what's hosted on rubygems.org is OK to use. <davexunit>and when it's not, fall back to something like the old ruby build system did. <civodul>"So ironically, as a packager, you don't need access to sources at all" <civodul>that person lives in a different world <davexunit>yes, exactly. it's clear that it's a waste of time to discuss further. <davexunit>all I can do is hope that I've planted a seed and they'll change their mind some day. <civodul>between the "GitHub hipsters" and "ourselves" <civodul>where ourselves means the GNU, Debian, etc. crowds <davexunit>get with the times, no provides the source to their free software anyway! <civodul>the free software world is being dockerized or githubized <davexunit>this is the state of "modern" software development <davexunit>a bunch of people who haven't learned from the last 3 decades or so of software development deciding what is best. <civodul>maybe we are all uneducated to someone else ;-) <civodul>but yeah, i've become a bit scared of the gems, cabals, pips, and whatnot <civodul>i hadn't realized the depth of that gap <civodul>seriously, pip looks better than cabal, yes <davexunit>I haven't done enough research on that one yet, though. <davexunit>I know the host binaries, but do they host source. <civodul>it typically downloads jar files from the net <civodul>grr, inkscape wants more than 700MiB to store its .o with debugging info <xentrac>I'm less of a fan of gem, cabal, and pip, but I have to admit that they seem to be opening up software distribution to a lot of people <xentrac>and npm now has more free software than Debian, by lines of code if not by usefulness <xentrac>I didn't know that pip actually hosted source code <keverets>xentrac: is that really true? What do you use to measure lines of code in npm vs Debian? <lfam>Well, I think a lot of people are sick of "the gems, cabals, pips, and whatnot". To me that is one of the many great reasons to adopt guix or nix. <lfam>How many different packaging systems are we supposed to learn? <civodul>xentrac: actually yes, things like npm or cabal really create "parallel worlds" in a way <civodul>lfam: to their defense, they are all quite easy to use and similar ***francis7-awol is now known as francis7
<davexunit>xentrac: npm only sort-of hosts free software, though. they host a lot of minified stuff, which is not source code. <davexunit>and they, like rubygems, make no effort to give users the source code that was used to build that binary. <davexunit>so I question that they have more free software than Debian. <davexunit>also, node packages are notorious for being tiny, and thus there are tons of them, so counting the number of packages is a poor measure for the amount of free software available.