<roelj>sneek: later tell rekado: It seems that the import from a Python shell inside the eolie environment is just fine.. http://paste.lisp.org/+7IZ7 There seems to be a problem with finding dependencies, as LD_LIBRARY_PATH is needed to find libwebkit2gtk-4.0.so.37.
<reepca>It's going... eh. I see the C++ scanner that's all "let's hash and compare 32 characters for every character we see!" and shudder, but I suppose it's probably I/O bound anyway. And of course it's not that bad, because it first checks that all 32 characters are base32-valid. But I still really feel like I should improve that somehow.
<Apteryx>Sorry, I've been away for too long it seems :)
<Apteryx>OK! That's the code that looks for references to graft?
<reepca>Nah, it's the code that looks for references to detect in general after a derivation is built.
<Apteryx>Interesting. I didn't know that such happened at that time.
<Apteryx>Are these kept in the database or something?
<reepca>Apteryx: I ended up using two 8400 GS'es. It's actually sort of lucky I ended up with two of them by accident (problem with the first one turned out to be an adapter issue), because each one only has 2 crtcs. So I need two of them to drive 3 monitors.
<lfam>reepca: Yeah, that's a serious issue in practice. AFAIK, we've found and fixed two very serious bugs of that sort in Python and GCC
<lfam>reepca: Those references are used for grafting, too, right?
<reepca>I haven't looked at the grafting code, but I'd expect so.
<Apteryx>So right now our garbage collector is "leaking" whatever reference doesn't appear in clear in the binaries/built product?
<lfam>Python eggs used to be zip-compressed, obscuring references. Then, there was a string-chunking optimization in GCC that caused store references to be obscured.
<lfam>Apteryx: I wouldn't call it a leak since it leads to store items being GC-ed too soon rather than accumulating ;)
<Apteryx>I guess they are used for grafts; otherwise why would there even be the requirement that they *must* be unencoded to be saved in the DB?
<lfam>How would you find them if they weren't in the clear?
<reepca>Apteryx: the only reason they must be unencoded is because we don't see them otherwise.
<lfam>When mark_weaver found the GCC issue, he suggested that it could be possible to adjust the reference scanner to be able to find the chunked references. I don't think we implemented this, however.
<reepca>but conceivably the same issue could apply to grafts - patching a reference that is split up or unrecognizable doesn't really work.
<lfam>It helps illustrate the severity and complexity of the issue of references in Nix / Guix systems
<Apteryx>lfam: No it was deemed to complicated and prone to fail if the compiler changes; so GCC was patched to not do that instead.
<Apteryx>*too, *changed. I need to reread myself before hitting that enter key...
<Apteryx>reepca: Granted, we don't see them if they're not plain text, but it seems at that point in time in the daemon you have the knowledge of opaque references as well. If those references are added and also used by grafting I would think they would cause problems there.
<lfam>Often I find unecessary or miscategorized inputs in package patch review. Sometimes dependencies are removed upstream when updating and we don't notice it right away. If we treated the list of inputs as the list of references, we would end up with spurious references. So, there are trade-offs either way
<Apteryx>reepca: 8400 GS... With nouveau driver? I just recovered my 2006 desktop, it's equipped with a single 8800 GTS. From what I've read the power management is not working yet so it's going to be slow for 3D, but otherwise seems like it'll be fine with no binary blob.
<lfam>Also, the current grafting implementation wouldn't work with encoded or obscured references, since the software wouldn't know how to execute the bug-fixed dependency.
<lfam>I mean, it wouldn't work whether or not we have a way to register references when they would otherwise be obscured.
<lfam>But, I wish we had a more powerful build farm so that we could rely on grafting less often and for shorter periods of time.
<reepca>I'm a bit unfamiliar with most build systems - how feasible would it be to do a scan after configuring but before compiling? What would the risk of spurious references be like?
<lfam>There are often references in the built objects themselves, so they need to be scanned, too. That GCC issue I linked above was a case like that.
<lfam>As for the risk of spurious references, I don't know because I've never tried it :)
<reepca>But in order to get into the built objects they must at some point be in the source, right?
<lfam>I haven't read Eelco Dolstra's thesis on Nix, but it seems like the idea behind scanning references assumes they will never be obscured. Then, the reference scanner is a perfect idea, because it registers only things are actually referred to by the store item.
<lfam>I'm not sure I understand. The source code won't always include store references after configuring. In the gnu-build-system, there will be some store references in the build directory after configuring, but I don't think it's the case for all the build systems.
<lfam>And, the references that would be found at that point may only be needed for building, and so we don't need to register them.
<reepca>Hmm. What if we only scanned *.c and *.h files? Could references be introduced through other means like compiler flags or environment variables?
<lfam>I realize that my use of the verb 'register' may be different. We definitely want to protect build-time things from the garbage collector during building. But in this conversation I've mostly been thinking about what happens after the build is finished.
<Apteryx>I somehow thought it just validated the hash were the same.
<reepca>Apteryx: there are two hashes for an output, the "store path" hash, which just depends on the inputs and derivation and which you see in the path, and an internal hash stored in the database of the actual contents of the output.
<sneek>rekado, roelj says: It seems that the import from a Python shell inside the eolie environment is just fine.. http://paste.lisp.org/+7IZ7 There seems to be a problem with finding dependencies, as LD_LIBRARY_PATH is needed to find libwebkit2gtk-4.0.so.37.
<sneek>fturco, ng0 says: with Gentoo experience it helps to compare the Gentoo Handbook and the GuixSD documentation. I found it relatively easy at first, compared to the first Gentoo install I did (Gentoo nowadays is very easy for me aswell, too many installs past)
<sneek>fturco, ng0 says: I wanted to do an Gentoo / GuixSD comparison after a while… I might still do it in a comparison of systems and then ultimately from a Gentoo-person perspective which parts of Guix are the parts in Gentoo you are used to, etc..
<fturco>i created two subvolumes on my btrfs partition
<fturco>now i'm trying to run: guix system init /mnt/etc/config.scm /mnt
<jsierles>civodul: so since guix gc doesn't seem to show all my built packages, is there any other way to see them or query the database/daemon?
<reepca>jsierles: is it important that you only see built *packages*, or will all stuff in the store do? Built packages are a subset of store items. "All store items" would just be the union of guix gc --list-dead and guix gc --list-live. At the daemon level it doesn't know about packages, only about derivations, but built packages are also built derivations. You can query derivation outputs from the daemon, but you'll also get stuff like the
<reepca>output of a derivation that downloads source. But the derivations usually fall into patterns that you should be able to filter relatively easily.
<jsierles>reepca: yeah, i can see they are filterable. just wondered if there was a distinction internally
<jsierles>i only want to see packages, since the goal is to provide a simple package installation tool on top of guix, only using what's already in the store
<jsierles>didn't want to do this since i have to parse out the version numbers, etc. and I'm sure that info is available somewhere I don't have to parse it out
<jsierles>perhaps adding a --json option to 'guix gc --list-x' would be useful
<jsierles>i suppose another concern is that since guix definitions are a moving target, some software could be in the store which no longer can be installed using 'guix package'
<jsierles>for example, i have a profile with python 2.3.1, and guix updates it to 2.3.2. my old profile would still work, but i couldn't create a new profile with python 2.3.1, since there is no longer a definition for it. correct?
<rekado>fturco: are you purposefully compiling things from source?
<fturco>rekado, no, but i had to add the --fallback option in order to go on
<fturco>by the way it built gcc-5.4.x, but 7.1.0 is available too
<reepca>fturco: aye, the version of gcc used in gnu-build-system doesn't get changed very much both for reasons of stability and because it requires rebuilding the world. Build glibc with a different gcc and now everything that depends on glibc has to be rebuilt to use that new version of glibc, and so on. One of the consequences of functional package management.
<fturco>reepca, i like rebuilding the world, but not on my first attempt at installing guixsd