IRC channel logs
2017-06-14.log
back to list of logs
<lfam>fps: I recently spent a day reading through Petter's old work-in-progress code for a go-build-system. Now I have a good idea of how to finish it, just need to find the time. <lfam>IPFS is one of the things I really want to get packaged <fps>it's basically just ipfs add -r /gnu/store/....-bash-static-4.4.12, updating the list, adding it and republishing it <fps>added module-imports [one version] <fps>not sure how it handles timestamps or hard links, or permissions, yet <lfam>I think it could be a great way to decentralize substitutes <fps>yep, and doing away with the need for dedicated build servers, though those can still be useful to build the most needed packages <fps>i a user were to build a package there would still have to be a way to communicate back the ipfs hash to the rest of the guix world <lfam>I think we'd still want the build farm, because we want people to only accept substitutes signed with a key they trust, and that key is centralized <lfam>Fully decentralized building will require some good ideas about "trust" <reepca>does --with-source disable sha256 checking of the download? <fps>i guess reproducibility and a majority vote would have to come into it somewhere.. <lfam>reepca: I haven't looked in the code, but I think it must do that <lfam>fps: Some sort of consensus system? <fps>i.e. once a number of people have built a package and independently published it there's a certain degree of trust in the derivation <fps>if their outputs agree <lfam>Redistributing Hydra's substitutes over IPFS would be a good start, before we tackle that harder problem <fps>problem is to prove that they independently derived the package. needs some sort of POW <lfam>Sweet dreams of decentralization! <lfam>See, it starts to get complicated quickly ;) <rain1>you can add -buildmode=pie when compiling golang projects <lfam>BTW, if anyone is interested in picking up Petter's work, I'm happy to discuss it with you <lfam>I worry I won't be able to touch it for a couple weeks at least <fps>lfam: indeed, it gets complicated.. but we'll see what happens if we let the problem linger in the back of our heads for a while ;) <lfam>The WIP go-build-system implements the basic idea, but it needs to be rewritten in a more idiomatic style. Also, some things that were removed when Petter was prototyping should be put back in <lfam>Basically, it symlinks all the dependencies into a Go-idiomatic union directory. <lfam>Anyone reading the logs, ping me if you're interested :) <fps>lfam: sadly i have not the slightest clue about go or about implementing build systems in guix, so i am not qualified to help :( <lfam>fps: That's where I started, too :) <rain1>hm how do you approach the import problem? golang just points to a git repo, not to a specific commit <lfam>rain1: If I end up doing the work, I'll use Syncthing as the test package for the build system. It has a few dozen dependencies, some of which have their own dependencies, and the Syncthing project specifies which Git commit they build each dependency form <lfam>build each dependency from <lfam>So, the packages will all be built from that Git commit <lfam>If some of those libraries end up being required for other packages, and need other Git commits... I'll cross that bridge when I come to it <lfam>We won't let Go download the dependencies at build-time; there's not networking in the build environment. We'll provide them all locally, similar to how we approach Python builds. <lfam>Honestly, I still don't fully understand how Go programs are supposed to be built. I keep reading things like "golang just points to a git repo, not to a specific commit" and ask myself, "Really? That doesn't seem right" <lfam>But then I talk to Go programmers about this and they all get this dark look and start sighing <lfam>It doesn't make sense to me that the idiomatic build system for a new language just pulls the latest commit of a development repo <reepca>lfam: perhaps they expect a non-development repo? <lfam>I guess saying "development repo" was needlessly derogatory ;) They must expect rock-solid master branches <lfam>Well, lots of things are written in Go, so it must work out somehow <OriansJ>janneke: thank you for the patch, I've also made some interesting developments with META II <reepca>... Never mind. Of course the most flustering of errors are the typos. <emyles>Trying to build a new python package, I get "phase `unpack' failed after 0.0 seconds" <reepca>Looking through gnu/packages/python.scm, I notice that most of the sources are in .tar.somecompressionformat form, and yours is in .zip form. A quick search reveals that python-pytest-mock, for example, also downloads something in .zip form, and it uses "unzip" as a native-input. <reepca>Builds are done in an isolated environment, so you have to explicitly tell it to make unzip available <reepca>(apparently that's not an implicit input from python-build-system I guess) <emyles>okay, thanks a lot, I'll have another go <ng0>ACTION gives sneek a coffee <ng0>reepca: I've seen the message <ng0>I "fixed" some graphic lags I had with one GPU by using E21, which to my surprise is faster than XFCE4 with rendering of websites or videos <ng0>yeah.. I just read the saved log sometimes, sneek doesn't deliver everything <reepca>Hm, trying to get an old linux-libre to work and linux-modules-builder is failing, unable to find a bunch of .ko files <janneke>OriansJ: oh, good! I have now removed all jump/address/offset calculations from mescc and fully leverage the hex2/hex3 label resolving <nalaginrut>hi folks, is there a document to teach me how to build all packages in my local server. I have reason to do it anyway <sneek>Welcome back nalaginrut, you have 1 message. <sneek>nalaginrut, ArneBab_ says: the scoping, the data formats, and the type conversion. <nalaginrut>well, I never know sneek can monitor outside of #guile ;-) <efraim>i ad-hoc it with 'guix package -A | cut -f1 | sort -u | shuf | xargs ./pre-inst-env guix build --fallback --keep-going' <nalaginrut>efraim: are you answering my question? Let me elaborate it, I want to build a repo for it <efraim>there's a way to do it with cuirass, but I haven't been following its development, sorry <sneek>m-o, civodul says: you could do (dynamic-func "scm_run_finalizers" (dynamic-link)) and use the FFI <fps>nalaginrut: you can just enumerate all packages, and then write a for loop that calls guix package -i <package> --fallback or something <fps>takes on the order of a couple of days possibly <efraim>With grafts you have to build the packages with --no-grafts for --keep-going to work <efraim>I also run a guix-publish service on one of my aarch64 boards <jsierles>is there a way to use a manifest, or something like 'guix package -p' with 'guix pack'? <jsierles>what does this mean? warning: collision encountered: /gnu/store/s45c3nf4swhkgwrsc7bbnp3abhik2wwd-gfortran-7.1.0/libexec/gcc/x86_64-unknown-linux-gnu/7.1.0/install-tools/mkheaders /gnu/store/yr9rk90jfpq7xcbh24aryrnsd2i3qvwa-gcc-toolchain-7.1.0/libexec/gcc/x86_64-unknown-linux-gnu/7.1.0/install-tools/mkheaders <jsierles>not clear if this conflict is something i need to worry about. <rekado_>jsierles: It might not be a problem. <rekado_>jsierles: I installed both gfortran and gcc-toolchain into the same profile (both at version 5, though), and I haven’t had problems. <rekado_>this conflict is due to the fact that gfortran is a subset of gcc and installs some files that have the same name. <rekado_>what are the differences between these files? <roelj>Is there a way to use a package in a G-Expression, that needs its propagated-inputs to function properly? I see that the propagated inputs are not in the package's output path, so using a G-Expression, it's missing crucial bits to run a program in the package. <roelj>This will not work: (system (string-append #$r "/bin/Rscript my-script-that-needs-r-genomicranges.R")) <rekado_>roelj: that’s because you don’t have a profile at this point, right? <roelj>Possibly, one way to work around it is to build a profile of a bunch of dependencies first. Is there already something to run a G-Expression in an environment? <roelj>Oh, and I had to use #$r-minimal instead of #$r, because that is an empty directory (everything is propagated there). <civodul>roelj: yeah you could build a profile <roelj>civodul: Can you point me to code that does this without doing all too much other stuff? :) <roelj>That's the last thing I need to have the Guix Workflow Language ready for production. <civodul>roelj: that's 'profile-derivation' in (guix profiles) <civodul>you give it a <manifest> and it returns the derivation of the profile <roelj>That sounds a lot easier than what I think I had to do <civodul>the simplest form is: (profile-derivation (packages->manifest (list r ...))) <roelj>Then, how can I run a G-Expression inside that environment? <civodul>what kind of execution environment would you like? :-) <civodul>container, vm, derivation, "here and now", etc. <roelj>Hehe, "here and now" for now (and here), <civodul>you might have to do 'gexp->script' and run the resulting program <civodul>there's currently no 'gexp-eval', but we should add it <roelj>I have been using gexp->derivation up until now <roelj>And then derivation->script. <roelj>I use a custom derivation builder.. <civodul>i think you should avoid opening connections to the build daemon in the middle of the program <civodul>with %store-monad there's a connection that's "threaded" through all function calls <roelj>So that a connection is only made when it actually needs to make a connection, and that a connection will be reused instead of duplicated. <civodul>roelj: in guix/scripts/*.scm, the approach is to open the connection once at the beginning and then to reuse it all along <civodul>that avoids bad surprises, notably related to GC <civodul>so there's usually one 'with-store' form (which calls open-connection) and then the 'store' argument is passed to all the functions <civodul>either explicitly or via %store-monad <civodul>i hope that makes sense and doesn't sound too much like lecturing :-) <civodul>plus, that's not what you were asking for ;-) <roelj>Oh no, it makes sense.. I didn't understand how those guix/scripts/*.scm handled the store connection. <roelj>So, it's definitely worth looking into again.. Even though I believe that my current solution may be APInsane, but very effectively ;) <roelj>civodul: Anyway, thanks! Much to process again <efraim>I'm still hunting down the aarch64 test failure for ffmpeg, I think Arch arm gets around it by not running the tests <rekado_>IT moved Guix to a new NFS share with different settings, and it seems to be quite a bit faster than before. <rekado_>7 seconds for doing nothing, compared to 7 minutes. <rekado_>still slower than on my workstation, but I’m already happy <jsierles>still debating here whether to use something like GlusterFS with NFS, or stick with our GCSfuse setup <jsierles>nfs is going to be faster of course. but gcsfuse works quite well and gives us free backups <rekado_>AFAIK NFS is sequential, so performance can be pretty bad <roelj>rekado_: I'm also interested in the details <roelj>Good to read that it's faster now :) <civodul>did commit 015f17e8b9eff97f656852180ac51c75438d7f9d and the one before help on NFS? <civodul>i wonder if there's a trick to simulate slow disk i/o on Linux, like 'tc' <roelj>civodul: Put your USB ports to 1.1-only mode and mount it as /gnu ;) <roelj>Or simply mount it over NFS.. <rekado_>I’m still waiting for IT to perform a few more changes <rekado_>once that’s done I’m free to test the performance impact of that commit <roelj>I have that commit applied. For 'time guix environment --ad-hoc coreutils --pure --no-substitutes --no-grafts -- true', the timings are like this: 55.702s, 28.124s, 23.728s, 18.530s, 17.294s, 16.819s, 16.374s, 21.436s. <roelj>So, the first time always takes much longer (due to NFS caching I guess). And the other timings look like an improvement <jsierles>rekado_: in your R article, you suggest a technique for still supporting install.packages. however since R_LIBS is set to the store, doesn't that mean R will try to write there? <ryanwatkins>Hey guys, is there any reason why I might have trouble running binary files on GuixSD? <ennoausberlin>Hello. I played around with configurations and after bare bone install I decided to add X support. Most packages are installed but the X binary is not there. How can I find which package is related to that file? guix package -s X gives me too many hits <ennoausberlin>I dont need the full gnome desktop. So I just wanted a X server and clfswm or stumpwm as window manager <jsierles>is there a way to list available packages, including those available in GUIX_PACKAGE_PATH? <jsierles>found it - guix package --list-available. <OriansJ>janneke: today there should be another few hundred more M0 definitions for x86 opcodes, I hope to have all the ones you need before the end of the day <rekado_>jsierles: R will offer users to install to a directory that they can write to. <rekado_>jsierles: I’m using a slightly more complicated profile for users in my research group <janneke>OriansJ: ow...what an amount of work! do i need that much? <OriansJ>janneke: probably not, but I am included encoding for instruction combinations that would be used by optimizing compilers <rekado_>sneek: later tell ennoausberlin You only need to create an executable script ~/.xsession which ends on “exec stumpwm”. It will be executed automatically. <rekado_>sneek: oh, this botsnack is dirty. Next time. <jsierles>is there a guide for installing modules from guix git? i see you reference a specific tag in your profile <rekado_>jsierles: sadly, this doesn’t exist yet <jsierles>rekado_: ok. i don't mind doing the work. just wondering really how to use the git repo for modules source <janneke>OriansJ: better to be complete while you're at it? <rekado_>you’d have to either use “guix pull” with a specially crafted URL, or use plain git. <jsierles>if i use git, how can I inform to use the git repo for modules source? <jsierles>alright. it's just 'guix pull' now takes like 20 minutes <janneke>OriansJ: mescc can do with a lot of cleanups still, but i figure that the hex2 (and/or hex3) output as ugly as implemented right now will help greatly with that <rekado_>jsierles: if you use “./pre-inst-env guix” it will set the load path such that the git repo’s modules are used. <rekado_>pre-inst-env is created after “./bootstrap && ./configure --localstatedir=/var && make” <rekado_>note that when switching to an older version of Guix you will have to rebuild some files <rekado_>sometimes you even have to delete certain .go files manually because of ABI breakage. <jsierles>solution should be to never use older versions of guix <rekado_>that doesn’t sound like a solution to me :) <OriansJ>janneke: quite likely. Also, why did you select 0x8048000 as your base address for your elf binaries? <rekado_>older versions of Guix have an older “configuration” of the graph, so it may be useful <jsierles>reminds me - regardless of guix upgrades, profiles and packages are always immutable forever? <rekado_>everything in the store is append-only <jsierles>meaning if i have a store mounted and am using a profile, is that profile guaranteed to work forever? <rekado_>you only ever lose things from the store by garbage collection, i.e. invoking “guix gc” <rekado_>“guix gc” will only delete things that are no longer pointed at <jsierles>but it shouldn't be necessary to use an older guix to, for example, use an older version of a specific package <rekado_>if you already have that specific package in the store then you’re right <rekado_>but if your store never had that package or no longer has it (after “guix gc”) you may need to check out an older version of Guix to realise that part of the older graph you’re interested in. <jsierles>ok. the checkout's job would just be to get the package into the store, right? <janneke>OriansJ: i tried 0x0 but that did not work. gcc toolchain uses 0x8048000 and that worked. <jsierles>ideally checking out an old guix would just be a temporary batch job for creating a profile. <janneke>OriansJ: i know nothing about elf... <rekado_>checking out a version of Guix gives you that particular configuration of the graph. At that point you can then build part of the graph, which adds things to the store. <OriansJ>janneke: that is because the bottom 1MB is reserved if I remember correctly <jsierles>my doubt is about the /var/guix/db - is that going to be affected by moving around across versions? <OriansJ>janneke: p_vaddr and p_physaddr set to 0x10000 with e_entry set to 0x10054 will work <rekado_>the db only keeps track of things that are in the store. <jsierles>rekado_: right. in reality there would not be a single guix install anywhere. just batch jobs which manage the store. clients would issue the jobs, but not have access directly to guix <rekado_>the db format hasn’t changed so far. <jsierles>alright. that sounds reasonable. just need to look into the git business. <janneke>OriansJ: ah, ok :-) I wondered but did not worry too much about it <jsierles>the job should not have ABI issues since it will do a fresh 'preinstall' each time <OriansJ>janneke: I wonder where they got that number from <rekado_>jsierles: that would probably be just as slow as “guix pull”, though. <rekado_>jsierles: you’d also have to make sure to use the right Guile for the version of Guix you’re dealing with. <rekado_>(only recent versions can use Guile 2.2.2, for example.) <janneke>OriansJ: prolly a secret hint about their boy-or girlfriend <rain1>ddid you all see the nice tutorial <OriansJ>janneke: quite possibly, kinda like how some disassemblers behave badly when given add eax,[esp+0x1234567] <jsierles>maybe the answer then is to just keep copies of each 'guix pull' around <rain1>just a nice clearly written guide of the basics <rekado_>jsierles: or to install the “guix” package via Guix but overriding the source tarball. <rekado_>no idea if that would be better, haven’t thought this through. <jsierles>will have to see in practice. really we just need to work out how to meet demands of users, and don't know what those are yet :) <rekado_>jsierles: maybe your experiments can help us identify some of the open questions in implementing channels. <OriansJ>rain1: well it answers the question of how gcc got that number, 128MB of stack space with the stack below the code growing down. <OriansJ>rain1: I honestly don't get those two design decisions: stack growing towards zero and defining a standard stack base address which explicitly prohibits the largest and fastest growing part to only 1/32 of the address space <sneek>ennoausberlin, you have 1 message. <sneek>ennoausberlin, rekado_ says: You only need to create an executable script ~/.xsession which ends on “exec stumpwm”. It will be executed automatically. <OriansJ>janneke: the good news is you'll have probably the most complete x86 opcode encoding document in the x86_defs file <OriansJ>janneke: but of the 192,384 possible x86 opcode+register encodings, I'm stopping after 4K <OriansJ>(that also includes all of the prefixes, etc) <janneke>OriansJ: i'm amazed...doing a quick grep in mes <janneke>grep "['\\`](" module/mes/as-i386.mes | wc -l and grep '^ *#x' module/mes/as-i386.mes | wc -l <janneke>reveals that mescc currently uses ~140 opcodes <OriansJ>janneke: probably a smaller, separate definition file just for MES, with only 140 lines <OriansJ>janneke: speaking of which, I should have that done shortly <janneke>OriansJ: i'm wondering about our next step, there are so many things to work on? <OriansJ>janneke: go for the wins you think are fastest and easiest <OriansJ>janneke: I'm currently focused on giving you as much power to play with for bootstrapping as possible. <janneke>OriansJ: yes, i see and appreciate that, hex2 and mescc_tools have been amazing steps -- that's why i ask if you see an obvious next target <janneke>i'd very much like to work towards replacing guix's bootstrappping [binaries], we need full connection from stage0->mes and also mes->gcc... <OriansJ>janneke: once hex2 is fully in MES, we have enough for a baseline connection. <janneke>OriansJ: i think hex2 is now (since yesterday) fully in MES <OriansJ>janneke: the connection from mes->gcc is probably going to go like this: MESCC_minimal->guile->MESCC->TCC->GCC <janneke>OriansJ: that's right, i should have a look at eval.c <OriansJ>janneke: I probably need to get back to work, cleaning up your make process, etc to ensure getting new people started hacking on MES because a smooth process <janneke>OriansJ: yes, you should be cleaning up my make process :-) <OriansJ>and with that 1,400 Definitions for x86 <janneke>OriansJ: another thing i was wondering: what about the .c tools in stage0 and MESCC_Tools, would it help if mescc could produce (annotated) hex2 for them? <OriansJ>janneke: sure, they could serve as valid tests for mescc <janneke>OriansJ: but we need hex2 versions of all of them, right? i guess for stage0 you have hand-coded hex2 versions for them already? <OriansJ>janneke: if you notice, all of them were hand converted to assembly and depending on the bootstrap step were converted to hex0, hex1, hex2 or left as is <OriansJ>janneke: but yes, converting them to hex2 and adding them as tests for MESCC_Tools would be useful <janneke>OriansJ: i was wondering if mescc could help with that -- possibly no much development is necessary? <janneke>mes is already include in guix, if we have Mes depend on MESCC_Tools, that needs to be packaged too <OriansJ>janneke: well I was planning on adding MESCC_Tools as just an addition to mes