<Apteryx>sneek: later tell slim404: that guild error is normal, or at least innoffensive. It is because guild compiler front script is trying to auto-compile the Guile module it interprets. It usually saves this to $HOME but here fails to do so because in the build environment there is no $HOME. <Apteryx>sneek: later tell slim404 the real problems seems to do with a segfault in the tests: line 5: 1116 Segmentation fault ... ***fr33domlover1 is now known as fr33domlover
<IpswichTriptych>So, I'm unsure of how to proceed. I cannot complete a guix pull because it invariably times out. <IpswichTriptych>My thought is to guix build all of the largest packages... but for example "guix build java" is not possible... <IpswichTriptych>Any advice? For clarification, I'm trying to install a simple GuixSD configuration on a Lenovo X60 with onl y 1GB of RAM and 4 GB swap. The installation was a success, but I have so far been unable to complete guix pull because once the swap is utilized, the compiling step takes so long that the process times out after 3600 seconds <IpswichTriptych>I have more RAM on the way, but the X60 motgerboard only allows a maximum of 3GB, so this hardware configuration may simply not be the right one <lfam>IpswichTriptych: Did you try setting the --timeout and --max-silent-time options to guix pull? <IpswichTriptych>i realize now that those options are listed in guix pull --help, but that wasn't apparent to me before <sneek>Welcome back civodul, you have 4 messages. <sneek>civodul, wingo says: idle thought, the slot allocators uses integers as bitmasks <sneek>civodul, wingo says: with functions with many slots (more likely the case with long functions; heuristics can be complicated) that means more bignums <sneek>civodul, wingo says: i wonder if those finalizers are perhaps cause of some of our gc woes? <sneek>civodul, wingo says: i.e. bignum finalizers <civodul>i've spent literally days on this, but i haven't been able to pinpoint the issue yet :-/ <sneek>wingo, stis says: in case you stall a thread for later execution in fibers, you will unwind the stack and the problem is that a finally clause which is a dynwind will execute <sneek>wingo, stis says: if there was a per thread variable that indicated if the unwinding was of this case one can direct the finally clause to ignore the unwinding <wingo>civodul: do the disappearing links in the weak table not disappear? <wingo>well. many questions, hard to organize them <civodul>wingo: it looks like a possibility; for instance, move_disappearing_links did not check return value, and upon GC_NOT_FOUND, i think the new entries would lack a disappearing link <civodul>i fixed that, but that didn't help :-/ <wingo>this situation is weird tho. is the weak table causing algorithmic badness or just incrementally worse mem usage? <wingo>this is the source properties table, yes? <civodul>wingo: in the bug above, it's any weak table <civodul>for the compiler, it's probably the source property table, yes <wingo>well sure, just trying to think about our compilation problem <civodul>when we're compiling python.scm, the source property table is really big <civodul>so it definitely plays a role into the memory/CPU badness <civodul>maybe not the only explanation, but certainly one of them <wingo>have you also been tracking the finalizers queue length? <wingo>depending on how libgc was compiled <wingo>does the finalization queue length stay low or does it explode? <wingo>wondering if the weak tables never get vacuumed b/c that happens via a finalizer <civodul>roughly the number of dis. links is proportional to the number of entries <civodul>and the number of finalizers is constant <wingo>cool, that rules out finalizers <civodul>vacuum_weak_table does get called, a bit too infrequently, but it does its job <wingo>it should get called once per GC <civodul>uh, i can reproduce the problem on a pure C example <civodul>i'll email the libgc folks for advice <efraim>looks like I might've been hasty pushing binutils@2.29.1 to core-updates, more CVEs coming in, and gcc-boot0 isn't linking to libstdc++-boot0 <civodul>the bug has moved yet another level up ;-) <civodul>i find myself much less efficient than a wizard ;-) <civodul>i had been meaning to write it down for some time already :-) <efraim>hmm, with binutils-2.29.1, gcc-boot0's libcc1.so can't find libstdc++.so.6 in the runpath <htgoebel1>roptat; Now that maven is finished we need something like a maven-central importer :-) <Apteryx>Was there a change recently where compilation messages are now hidden by default? <Apteryx>I have some guix commands takeing forever and I have no idea what is going on. <Apteryx>I think it's because I changed git branch and lots of Guile modules need to be interpreted (I should run make). <Apteryx>efraim: this happens when I try to test the addition of "perl" as a native-input to gcc-4.7.4... Apparently it tries to rebuild the world but I have no idea what is going on, there's no output on the CLI and top says guix is using all of my CPU (apparently no subprocess are doing compilation, so it's purely a Guile/Guix thing). <Apteryx>Eventually it dies with a stack overflow exception... <efraim>Apteryx: when you add it to 4.7 it gets propagated until it is overwritten <efraim>The stack overflow is probably from a circular dependency somewhere with the perl <efraim>I think GCC5 is the next time native inputs get explicitly declared <efraim>We use GCC 4.9 and 5 for building up from the bootstrap binaries <Apteryx>yes, that's why I had to declare perl as a native-input in gcc@5 as well. <Apteryx>(I'm really trying to fix all of our gcc-*). Except the gcc-boot0 apparently which I shouldn't touch? <Apteryx>Circular dependency of perl... this happens when A <--requires-- B anb B --requires--> A ? <efraim>And GCC-final, but with all the inherits the safest option is to work with 4.7 4.8 6 and 7, and to 'undo' the changes that would happen to 4.9 and 5 <efraim>Yeah, although I'm trying to think of the circle that adding Perl would make <Apteryx>hmm, is it really a concern that perl is part of our bootstrap closure (is it not already?) <Apteryx>I would think that perl is needed to build the kernel Linux, no? <efraim>I'm pretty sure we're using Perl-final for GCC-final <bavier>civodul: another issue with spack is having a *compatible* python installed <Apteryx>efraim: for gcc-final we are recycling perl-boot0. <Apteryx>it's used indirectly through the closure of texinfo-boot0 that is used for gcc-final. <Apteryx>yep, and the texinfo is already used for gcc@5, which has "perl" has an input. So my patch should be OK as long as I remove perl for the gcc-boot0 package (with alist-delete as is done for texinfo) and add perl-boot0 for gcc-final... and fix the circular dependency. WDYT? <efraim>Is Perl-boot0 the only Perl in commencement.scm? <efraim>If there's still a curxlular dependency guix should let you know somehow <Apteryx>efraim: yes, perl-boot0 appears to be the only perl dependency in the commencement.scm module. <Apteryx>Is guix really able to warn for circular dependencies? <efraim>Its normally stack overflows or unending 'thinking' before building <Apteryx>I think after I switch to perl-boot0 in commencement.scm it should be fixed. <mekeor>is there a reason why there's no guix-package for pdftk apart from "nobody did it yet"? <civodul>mekeor: i suppose there's no other reason :-) <Apteryx>efraim: circular dependencies got solved after filtering out perl from gcc-boot0! Thanks for the hints. <Apteryx>I'm building it finally, will check the result, and then I'll resend a v3 patch. <Apteryx>civodul: good job at digging the memory issues. Do you have an idea of the share of memory leakage attributable to the bdwgc bug you found? <Apteryx>My system is too slow... I have to keep trying at the btrfs raid stuff, that would unlock an install on a much more capable desktop. <Apteryx>efraim: any idea when the next core-updates merge is set to happen? <efraim>Staging is up next, core-updates could be soon after <efraim>Depends if we need a separate python cycle <efraim>good news, I think I fixed core-updates <efraim>time to build out to hello again <Apteryx>random question: are info manuals meant to be read in a mono-spaced font? It seems some formatting would break otherwise, but at the same time, a roman font would be more readable for long texts. <janneke>Apteryx: info manuals, well texinfo can be printed through latex <janneke>the parts that need to be mono-spaced should be inside some @verb[atim] environment <janneke>err, that may be TeX instead of LaTeX <Apteryx>right! I was just wondering if my configuration was standard for the Emacs info reader; it looks like it is. <Apteryx>I've only mapped the 'Fixed' font familiy to a monospace font (Hack) in my fonts.conf file... I think 'Fixed' was some strange looking console like font otherwise. <IpswichTriptych>i'm currently at 68.0% of compiling 635 files /gnu/packages/music.scm . . . <slim404>is python 3 available as a guix package ? <sneek>Welcome back slim404, you have 2 messages. <sneek>slim404, Apteryx says: that guild error is normal, or at least innoffensive. It is because guild compiler front script is trying to auto-compile the Guile module it interprets. It usually saves this to $HOME but here fails to do so because in the build environment there is no $HOME. <sneek>slim404, Apteryx says: the real problems seems to do with a segfault in the tests: line 5: 1116 Segmentation fault ... <lfam>slim404: Yes, it's called 'python' <lfam>slim404: Python 2 is packaged as python-2 <janneke>slim404: do you know about guix package -A? <janneke>slim404: do you know about guix package --help? <slim404>janneke: it lists all the packages it seems, thanks <janneke>that might come in handy for such questions ;-) <slim404>janneke: guix package -A | grep python gives too many results <IpswichTriptych>BTW, I noticed last night that when running "guix pull --verbose --verbosity=" level 4 or 5 failed, but 3 or below did not fail <lfam>slim404: The output of `guix package --search` is in recutils format, so you do sophisticated filtering and searching. <lfam>guix package --search=. | recsel -e "name ~ 'python'" -p name,version,synopsis <lfam>The argument to --search is a regular expression, so --search=. means "print all the packages" <lfam>The output of that command is still sort of verbose, but it's better than looking through the entire list of packages <lfam>Of course, you need recutils installed to have the `recsel` command <lfam>Alternatively, `guix package --search=^python$` <lfam>I know, it's not really that accessible. It would be great to have a GUI "app store" interface for Guix packages <lfam>Does anyone else have intermittent connectivity issues to git.savannah.gnu.org? <lfam>It seems like 50% of the time, `guix pull` fails because the connection to the Git repo fails :( <IpswichTriptych>lfam: I have not personally experienced any connectivity issues to git.savannah.gnu.org, but I will be cognizant of them in the future <lfam>Thanks IpswichTriptych! It could just be me, but I don't notice similar issues with other sites <Apteryx>lfam: arg... my gcc build failed after 12000 sec because of lack of space. <lfam>How much space is not enough? <Apteryx>hehe, I meant to tell this to efraim though, my bad :) <Apteryx>lfam: hmm... I'm not sure. I was pretty annoyed about it so I ran "guix gc" which freed 14 GB. <lfam>Thanks for updating dnsmasq, efraim <lfam>Some of these bugs are just terrible, and never going to patched in "the wild" <lfam>Never going to be patched, that is <lfam>You can trigger CVE-2017-14491 with a CNAME response to an A/AAAA query <Apteryx>IpswichTriptych: yes! It's commonly in those waters. <Apteryx>I'm running off an older, small SSD which gives me 55 GiB of usable storage. It's not much! <bavier>though I think the number reported freed by 'guix gc' doesn't consider deduplication <bavier>e.g. 10GiB reported freed, but disk usage of /gnu/store only drops by 2GiB because the other 8GiB was previously being deduplicated away anyhow <bavier>I may be wrong, just something I noticed recently <lfam>My version of Guix is ~4 days old but there is still no substitute available for libreoffice :/ <lfam>Hm, looks like there is a more recent substitute for libreoffice <lfam>efraim, I've just sent what I hope the is the final revision of the go-build-system / Syncthing patches <civodul>lfam: re substitutes, i've noticed that too <civodul>actually there's a substitute for /gnu/store/cnjmgq88sqzdzxhidz81i4jmvp7m48nz-libreoffice-5.3.6.1 <lfam>I just took the "latest" libreoffice substitute by using my Git checkout <civodul>probably there's been a time window during which there was no evaluation <civodul>and so we'll never get substitutes for things that correspond to that period <lfam>This issue about location-addressed resources (tarballs) changing their contents is frustrating! <janneke>lfam: thanks for your insight, not just a github thing <lfam>But, ultimately, a pervasive issue that I bet will affect every single package source in time <lfam>I noticed today that the upstream source for our net-tools package is 404 :/ I guess that SourceForge's Git snapshots are cached for a short period and only generated when they can tell a human is requesting it through the web browser, rather than through the URL we are using <lfam>I should email SourceForge and ask if they can change this. I really wanted to avoid fetching net-tools with Git <janneke>thinking about the content addressed-url, laertus (or someone using --no-substitutes) could use guix download on that url and thehn continue? <lfam>Or maybe ask net-tools to make a new release <lfam>Yeah, in this case I think the user should handle it ad-hoc, janneke <janneke>lfam: for a moment i thought it was impossible for a user to get around and we needed to consider making a release -- if there's a workaround then we're good, sorta ;-) <lfam>Well, in my opinion, if they are committed to using --no-substitutes, and the old source is not available anywhere, they could always just use the new source. This will invalidate substitution of all referring packages, but it shouldn't matter if they are using --no-substitutes <lfam>But, we should try to find a better solution for core dependencies like libgit2 <lfam>Also, I hope the libgit2 team will start releasing their own tarballs <janneke>lfam: how would they use the new source? they would need to copy the .scm and change the hash...that's much to ask as part of a `guix pull' install? <lfam>Hm, true, I didn't consider `guix pull`. I'm not sure how it would work for that case <janneke>lfam: yeah, that was the case at hand. you could argue the need to support --no-substitutes but otoh having that is kinda essential. <janneke>yeah, libgit2 is somewhat special for us...i hope that they take some responsibility somehow <lfam>Well, immediately, I guess the best option is to recommend they take our substitute / content-addressed copy of the relevant tarball <laertus>i see the need to get a tarball of libgit2 if you don't already have git.. but... i already have git <laertus>could guix fall back on just using git? <lfam>There's a difference between Git and libgit2 <laertus>no, i mean use git to get the source <lfam>One issue with Git is that it has a huge dependency graph, and so we'd rather avoid making it a core dependency of Guix <laertus>i understand.. but that's broken now <laertus>so in the case of major breakage like this, could there be a fallback to using git to get the source of libgit2 rather than just breaking completely? <lfam>That will fetch the relevant file and put it in /gnu/store so that you can continue with `guix pull` <laertus>will guix check the hash of that before trying to build with it? <lfam>The Git program itself is not really something we can rely on, IMO. The dependency graph is simply too large and unreliable <lfam>I worry it would only exacerbate the problem of a fragile dependency graph <laertus>i understand.. but for people who already do have it, it could be used, no? <lfam>We'd need to make changes to how Guix works, but I understand what you mean <laertus>yeah, you'd have to check out a specific tag, and do some kind of verification on that <laertus>it's not as clean or simple as a tarball <laertus>but the tarball method seems borked for the foreseable future <lfam>I wonder if the result of a Git checkout of libgit2 is bit-reproducible to the unpacked libgit2 tarball? <lfam>One of the suggestions in the upstream libgit2 discussion was to use the hash of the uncompressed (or maybe unpacked) tarball. This is interesting, and some projects sign that instead of the tarball. <happy_gnu[m]>I was told you are working in packaging Go and Syncthing <lfam>It would mean we have to trust more code (the decompressor and maybe the tar implementation), which would be something to think about seriously before doing <lfam>laertus: Whatever code runs before we can check the hash needs to be considered as attack surface, and we should minimize it <laertus>lfam: yeah, i understand the concern <laertus>another thing i was thinking is if they are having problems with tars, maybe they could use another format, like zips or something <lfam>laertus: I've seen HTTP interception garbage come in response to `guix pull` or other Guix downloads. We have to take a defensive stance. And since most of the compression and tar implementations are written in C, I'd probably argue against exposing them in this way <laertus>i'm not sure if there is some better bit-reproduceable archive format than tar.gz <laertus>though i guess it might be possible to do an untar and uncompression in guile.. at least in principle <lfam>Yeah, it's definitely possible <laertus>but it's still a larger attack surface, no matter how you slice it <lfam>laertus: Are you familiar with the concept of fixed-output derivations? In Guix, they are derivations that we know the result of in advance, like package sources. <happy_gnu[m]>lfam: why the last ones? Are you not packaging anymore :( ? <laertus>lfam: not really.. i've only heard of derivations (in the guix sense) for the first time yesterday.. but i think i understand what you mean <lfam>happy_gnu[m]: No! :) I mean, I hope they are the latest Go build system and Syncthing patches before I push them :) <lfam>laertus: There is a chapter on derivations in the manual. Basically, they are an intermediate level of build scripts, between the packages you see in 'gnu/packages' and what the guix-dameon reads and executes. When building a package derivation, we don't know what the hash of the result will be until we build it. Fixed-output derivations are simply derivations where we *do* know the hash in advance <lfam>happy_gnu[m]: I hope so! It works for me anyways. <lfam>laertus: I'm interested in removing fixed-output derivations from the category of "substitutes", so that people who have security concerns with substitutes can get package sources in a more reliable way <lfam>It should be something you can turn off, however, in case you have privacy concerns with fetching sources from sites beside the upstream source <laertus>lfam: i think i'm going to have to read up on how that works exactly before i feel i can trust it <lfam>The general problem of old versions of Guix being unbuildable without substitutes is more general, however. A while ago, we had a similar issue when a GnuTLS test accidentally failed to "wrap" the expiration date of a test TLS certificate. You can't build that version of our GnuTLS package anymore unless you change the date of your system (dangerous!) <lfam>The only safe option was to use our substitute of GnuTLS, or invalidate your entire package graph by skipping the test in your local copy of the GnuTLS packaging <laertus>is it possible that even though the output of a fixed derivation is known, there could be some unexpected side-effect of buliding it that could compromise your system? <lfam>And I've seen other instances of "code with an expiration date" too <lfam>As far as I can tell, only from the code that runs on the source tarball before the hash is verified. After the hash is verified, you can be sure the source code is exactly what is expected <laertus>so what's the difference between that an just having a hash of the source? <lfam>This assumes the hash function is cryptographically secure <lfam>Can you clarify your question? <laertus>well, you said "you can be sure the source code is exactly what is expected"... that kind of sounds like what a hash of the source gives you <laertus>so what is the difference between the fixed derivation (which give you knowledge that the source is exactly what you expected) and just using a hash of the source? <lfam>Exactly. Perhaps I misunderstood your question about unexpected side-effects <lfam>I recommend reading the chaper on derivations and reading a few derivations (they are the .drv files in /gnu/store). You will see that a fixed-output derivation is called "fixed-output" because it includes the hash of the result of running it. <lfam>If Guix runs the derivation and the result does not match the hash, the result is rejected. <civodul>lfam: you can't exactly guix download the /file URL because it'd have the wrong name <civodul>so you have to "wget -O libgit2-0.25.1.tar.gz ..." <civodul>and then "guix download file://$PWD/libgit2*.tar.gz" <civodul>we should probably support URLs in the other order <janneke>civodul: what about guix download --file-name=NAME <url> <laertus>lfam: i'll read the guix manual on derivations, but from your description it sounds like there's only a hash of the output of running it, which means a hash of the files generated, but i'm concerened that there could be a kernel exploit in the code which does not generate any file fingerprints to hash in the first place <laertus>could a fixed-derivation be crafted to generate the same exact files (and thus the same hash) as another fixed-derivation, but also contain an exploit that the latter did not have? <lfam>Yeah, that sort of issue is always a concern <lfam>Wait, I said "Yeah" in reply to your earlier message. <lfam>I don't have an answer for your second message. <laertus>a hash of the original source tarball seems to be a solution for this problem that a fixed derivation does not address <lfam>You are miscategorizing "original source tarball" and "fixed output derivation" as replacements for each other <laertus>yeah, i think i need to read the docs :) <lfam>A derivation is a build script that is used to create a thing in /gnu/store (apologies to civodul for surely butchering the terminology and finer points) <lfam>Any time Guix downloads a package source, it runs a derivation that downloads the source <lfam>Any time Guix builds a package, it runs a derivation that builds the package <lfam>Some derivations are "fixed-output", because the include the expected hash of the result of running the derivation <lfam>About your previous question, "could a fixed-derivation be crafted to generate the same exact files (and thus the same hash) as another fixed-derivation, but also contain an exploit that the latter did not have?" <lfam>It's an interesting and real problem, but not limited to the case of downloading things. <lfam>The software we rely on is super-buggy and that will be the case for a long, long time <lfam>I'm talking about all of it, not just Guix <lfam>A sufficiently motivated person could probably make your computer do anything they wanted if they are able to intercept your internet connection and mutate HTTP traffic, or convince you to download a file somehow <laertus>lfam: you said earlier "I'm interested in removing fixed-output derivations from the category of "substitutes", so that people who have security concerns with substitutes can get package sources in a more reliable way" <laertus>i guess i don't fully understand why they're in substitutes in the first place <laertus>i only have a very tenuous grasp of how guix works, and definitely need to read more before i can comment intelligently on this <lfam>I don't know the original motivation. I suspect a combination of privacy concerns and it being an implementation detail that is tricky to change <lfam>Like, perhaps somebody doesn't want the Guix or GNU projects to know what software they use. They could achieve that with --no-substitutes. <lfam>But other people with privacy concerns may be comfortable with GNU knowing what software they use, but don't want all the upstream projects to know. They have a different response to the question of "privacy" <laertus>i meant more of why derivations were in substitutes <lfam>My original statement was misleading. <laertus>my own reason for using no-substitutes is that i've heard that building from source is more secure in guix than using substitutes.. nothing to do with privacy considerations <lfam>Do you have an idea of what substitutes are, in general? <lfam>Yeah, that's the primary use case <laertus>and not run by the guix project itself necessarily <lfam>By default, Guix will only download substitutes from our own servers, but you can add other substitute servers if you want <lfam>So, these pre-built binary substitutes are one type of substitutes. But, we also offer substitutes of source tarballs, Git source checkouts, etc. <laertus>so would using the guix project's own substitutes be any less secure than building from source? <lfam>Derivations are not the things that are substituted. They are instructions for building a package, downloading a source tarball, and other things like that. The results of the derivations are what can be substituted. <lfam>I was thinking we could split the idea of "substitutes" into pre-built binaries and source tarballs, basically. It's just an idea, and mabye not the right approach, IDK. <lfam>As for "so would using the guix project's own substitutes be any less secure than building from source?" <lfam>If you build from source, you are relying on the security of your own computer to build the package properly. If you use our substitutes, you are also relying on the security of our servers to build and serve the right binary substitute. So, the number of things you trust has increased. <laertus>i guess i'll stick with source, then, for now <laertus>though my computer is so damn slow at doing that from guix for some reason <laertus>my last "guix pull" took over 9 hours, with about 4 hours of that being just automake testing itself <laertus>and that guix pull didn't even complete (because of that libgit2 issue) <laertus>i'll definitely try to see if i can modify the guix program to not do tests <lfam>Currently, there is a bug in Guile that causes the Guile compiler to be slower and use more memory than it should. We aim to fix this but it will take time. <lfam>What kind of CPU are you using? <laertus>a 4- or 5-year-old 2-core AMD E1-1500 <laertus>an old, super cheap laptop with 8 gigs of ram <laertus>otoh, i've used it to build gentoo packages virtually non-stop, and they're usually not that bad, as at least they don't do these long tests <laertus>i do kind of suspect there's something wrong goign on with guix <laertus>i've run perf on my system while guix was building this stuff and running its tests, and it was using only something like 3% of the cpu <laertus>i have a feeling that there's some kind of off-cpu performance issue going on <laertus>but it's not present when i build natively on gentoo... so maybe it has something to do with the containers guix uses <laertus>anyway.. i'm still investigating it... or i was when guix died because of the libgit2 issue <laertus>so, back to what you said about "we could split the idea of "substitutes" into pre-built binaries and source tarballs" <mekeor>ACTION needs a guix-package of xine-lib <laertus>xine.. now that's a name i haven't heard in a while <mekeor>it can EITHER use gstreamer OR xine :D <mekeor>ACTION why packages are called gst-… instead of gstreamer-… <laertus>there's dia-libs/gst-plugins-base and media-libs/gstreamer <laertus>err i mean media-libs/gst-plugins-base <lfam>mekeor: That's how they are named upstream