IRC channel logs

<rekado>looks like I’ll have to figure out a way to speed up ‘guix upgrade’ when users have accumulated changes to their profile.

<rekado>I have another user here who kept doing ‘guix install this; guix pull; guix install that’, and now ‘guix upgrade’ takes >5 mins to do nothing.

<rekado>‘guix upgrade’ also doesn’t seem to upgrade all packages, so the user cannot run ‘guix install’ due to conflicts.

<rekado>~/.guix-profile/manifest weighs in at 24M

<rekado>I’ll use --export-manifest and then install packages from the generated manifest.

<rekado>the new manifest file after the upgrade is also 24M big

<rekado>with R packages there’s a *lot* of duplication in that file

<rekado>perhaps it would be good to support references

<rekado>or maybe it would be enough to cache more

<rekado>read-manifest needs about 27 seconds on that 24M file

<civodul>rekado: could you share one of these manifests so we can profile things?

<civodul>or we can have a manifest that just picks say 50 r-* packages to reproduce that?

<civodul>it's important to have that fixed

<rekado>I see that READ alone takes a little over 5 seconds. On this NFS thing it’s 27s.

<rekado>so there isn’t much we can do on the Guix side to improve it, other than to generate smaller manifest files.

<civodul>the Scheme 'read' or the syscall?

<rekado>(on my laptop with SSD a little caching by store item name reduced the time to process the manifest file from ~8 secs to ~4.5 secs)

<rekado>Scheme READ

<civodul>okay

<rekado>scheme@(guix-user)> ,time (define m (call-with-input-file "/tmp/bad-manifest" read))

<rekado>;; 27.116401s real time, 5.478444s run time. 2.620153s spent in GC.

<civodul>terrible

<civodul>can you try with (read-disable 'positions)?

<rekado>;; 18.041167s real time, 3.645781s run time. 1.860371s spent in GC.

<civodul>less bad

<civodul>we could also compress things by supporting references to shared substructures

<rekado>right, that’s what I meant above.

<rekado>we could use the store name as a unique key

<civodul>i was thinking of SRFI-38

<civodul>if that works, it comes for free

<civodul>problem is that using write-with-shared-structure would introduce a syntactic incompatibility

<civodul>so instead we'd probably need to roll our own

<civodul>but first it'd be good to prototype with SRFI-38 to see if it helps at all

<rekado>is compatibility a concern at all? I thought we could just bump the manifest format version number and do whatever.

*rekado didn’t know about srfi-38

<civodul>rekado: bumping is fine (mostly), but here would fail to parse #1#=xyz and such

<civodul>so we'd get an ugly crash rather than a "format not supported" error

<civodul>s/here would fail to parse/'read' would fail to parse/

<civodul>rekado: the manifest at https://web.fdn.fr/~lcourtes/pastebin/big-manifest.scm.html gives me a 632K manifest file

<civodul>i'd really like to know how to get a 24M file :-)

<civodul>i mean that's two orders of magnitude higher

<rekado>I opened a bug report with four packages that get us to 7.1M

<rekado>(I Cc’d you)

<rekado>I can prepare a manifest with some trial and error that pushes the file to 24M

<civodul>thanks rekado

<civodul>7.1M is already a good start :-)

IRC channel logs

2022-05-18.log