<davexunit>I want to change the syntax of 'guix environment' to be more like 'sudo', or 'strace'.
<davexunit>so that the specified command can be run via 'execlp' instead of 'system'
<davexunit>but I'm not sure how to make that happen given the current way of doing things where one can specify many package names like 'guix environment guile emacs gcc'
<codemac>davexunit: +1. does the guix cli have the concept of '--' ? it coud be 'guix environment --pure <pkg> -- <cmd>'. though, to me it's really separate command like 'guix shell -e '(expr)' / -p 'pkg' <cmd>'
<davexunit>it wouldn't be a separate command. it's just an alternate way of saying the same things we're saying now.
<davexunit>I think I can kill 2 birds with 1 stone here by also making the various options compose better.
<davexunit>for example, I've found myself running something like 'guix environment guile' to get all of guile's inputs
<davexunit>but then I want to also add another package in an ad-hoc manner
<davexunit>so I spawn *another* shell: 'guix environment --ad-hoc strace'
<davexunit>I'd like to be able to compose all of these options together in a single command
<daviid>davexunit: really nice indeed! and this will really help guile to grow, which for us is really cool!
<sir123>Afternoon fellow Guix :) So I've had an idea for a while, in doing some study of the Guix system, and comparing with other libre distributions, along with some thoughts on how a good distro is put together, I have a few questions about the Guix system.
<sir123>First of all, as someone who cares about free software, I know I value the ability to control my own computing. A natural extension to this is compiling your own packages. GNU Guix offers a chance to understand the makeup of the packages I use, and have greater control over my system.
<sir123>So, I wanted to ask this: to what extent does Guix isolate the building of packages, allowing reproducible builds?
<sir123>i.e. when I build a package, what does the system do to isolate the build from the rest of the functional system?
<mechanical>Build processes are carried out by the guix-daemon program under one of many potential users, as specified by the --build-users-group in a chroot environment, run the program is able to be run as root.
<amz3>there is bootstrap procedure, I'm not familiar with it
<amz3>that said, building you won OS is massively more easy with guix
<amz3>the bootstrap procedure includes guile the language in which guix is written
<sir123>If the build system could be controlled, that is, I could set Guix to use my own trusted build system, I could have far better control over my own computing.
<sir123>I believe Guix could potentially provide that granular control.
<amz3>e.g. building a gentoo from scratch is not documented at all. Guix has a way to build everything from scratch provided you have guix on a machine
<amz3>(gentoo is always built via livecd. Bootrasping is only done for new architecture is only done for new architecture so it's not documented...)
<amz3>I think this area is better documented that in other distributions
<sir123>This is my concern. If a user did not trust the set packages, Guix provides a way to build programs and the system to still be aware of it than forcing the users to accept binaries, as is done in other distributions.
<sir123>The user can have complete control over his/her computing.
<amz3>(regarding build systems, yes you can change the build system, remove the upstream build system and write you own. Or create custom bootstrap binaries and build you own OS with it)
<amz3>sir123: yes, maintainer are aware of those. You never have to trust upstream binaries and can build things on your own
<amz3>sir123: around 10:00 AM GMT at least davexunit will be here, he is more knowledgeable than me
<yenda>that's wierd I was trying to update python2.7.9, it built, then I tried building ansible, a test from passlib failed, I tried building again 2 times and both worked
<mark_weaver>yenda: many packages have test suites that non-deterministically fail and will succeed if retried. it is quite unfortunate, and we should work to fix those problems when we have time.
<mark_weaver>this happens quite a lot on our build farm, hydra.gnu.org.
<mark_weaver>amz3, mechanical: it's good that you recommended that sir123 come back when someone more knowledgeable is here. even if you avoid binary substitutes, guix always starts from a set of "bootstrap binaries" that we provide, avoiding the host system entirely except for its kernel. so far, we have always cross-built those bootstrap binaries using guix itself (although the initial x86_64 bootstrap was originally done from NixOS), but it
<mark_weaver>could in principle be done from an arbitrary system using its libraries.
<mark_weaver>gnu/packages/bootstrap/*/* contains a few statically-linked executables for each supported architecture, which are used within the isolated build container to download and unpack the rest of the bootstrap binaries from alpha.gnu.org, and then the process proceeds from there.
<mark_weaver>we have an easy, automated way to build all of those bootstrap binaries from within guix. see the "porting" section of the manual.
<mark_weaver>e.g., I created the initial bootstrap binaries for mips64el-linux and armhf-linux by cross-compiling them, using guix on x86_64-linux
<mark_weaver>and then I used the resulting guix system to *natively* compile a better set of bootstrap binaries which are the ones we actually use on mips and armhf now.
<mark_weaver>(cross-compilation is not as reliable, mainly because the autoconf scripts must make guesses about the target system)
<amz3>is it possible to use distcc to compile the bootstrap binaries compiled on the native system?
<mark_weaver>we don't currently support distcc, and I don't see how it could be done without sacrificing the reliability of the isolated build container.
<mark_weaver>our build containers deliberately isolate the build processes accessing the network.
<mark_weaver>(except for the so-called "fixed output" derivations where the sha256 hash of the build outputs are known in advance)
<mark_weaver>fixed output derivations are used for downloading the source code of packages, e.g. for our 'origin' forms where a sha256 is included
<mark_weaver>the system is designed to ensure that the only things build processes have access to are the declared build inputs (and the things they depend on).
<mark_weaver>unable to access or communicate with anything that is not a controlled environment
<mark_weaver>however, it is already not quite as isolated as I'd like. for example, it still uses the kernel from the host environment, which in practice introduces "impurities" in the build, and can affect the results.
<mark_weaver>this is not just theoretical. some of the test suites for our core packages pass on some kernels and fail on others, because they end up testing detailed behavior of the kernel which changes from version to version.
<mark_weaver>and some packages have build systems that try running bits of assembly code to test for features in the processor itself, and then create binaries optimized for that processor.
<amz3>(by secure I understand that the compilation output has no backdoors introduced by the host which might be compromised)
<mark_weaver>and then of course there are things like the real-time clock. many test suites include very tight timeouts that fail on slower machines (like MIPS) or even on powerful machines that are heavily loaded.
<mark_weaver>amz3: understood. we try our best to avoid that. the only thing we actually use from the host system is the kernel itself.
<mark_weaver>(and everything below it, e.g. the "Management Engine" that is in all newer Intel-based systems)
<mark_weaver>also, if you build guix from source code starting from an arbitrary distro, then the guix-daemon (as well as guix itself) is built with the toolchain and libraries from the host distro, and guix uses the guile from the host distro.
<mark_weaver>however, when actually building packages, the guix-daemon ensures that nothing from the host environment ends up in the isolated build environment.
<mark_weaver>I worry about computer security a lot, and would like to work toward better solutions, but it's a hard problem. there is huge amount of complexity, proprietary systems we cannot yet avoid, etc, that could compromise the security of the system.
<mark_weaver>for example, we don't have any bootstrap method that doesn't involve starting from megabytes of existing binaries, whose ultimate source goes back in an unbroken line to the earliest computers.
<mark_weaver>to compile anything from source code, you must start with a working computer system, which at least must include a kernel, compiler, linker, and some other basic tools, and that's quite a lot of stuff.
<mark_weaver>not to mention the processor with all of its microcode that we can't even see the source code for.
<mark_weaver>and all of the code within the dozens of other processors all over modern computers, e.g. in the hard drive, network devices, etc.
<mark_weaver>so there's a completely independent processor, running a complete OS, in modern intel systems that lives within the northbridge.
<mark_weaver>it has complete dominion over the "main" CPU and memory, and direct access to the network.
<mark_weaver>specially formatted incoming network packets are sent to this independent processor before they are even seen by the main processor at all.
<mark_weaver>and it's accessible even when the machine is turned off, as long as there is power to the system (e.g. the battery is present)
<mark_weaver>the newest machine we've managed to get working without this Management Engine is the Libreboot X200, which is what I use.
<mark_weaver>in newer machines, it seems to be more integral to the working of the system. there are some newer systems that we've *almost* managed to get working, but 30 minutes after boot the system shuts down, for lack of an intel-signed binary running in the management engine.
<mark_weaver>the folks on #libreboot can tell you more about these things.
<mark_weaver>the Purism Librem machines will include this Management Engine, because no one knows how to get such machines working without the Management
<mark_weaver>sir123: it's a crazy project of mine, but at some point I'd like to create a bootstrap that starts with a tiny interpreter supporting a subset of scheme, as a coreboot payload, and from there build to a compiler capable of bootstrapping
<sir123>i.e. if we could build a secure environment (what that means is a problem in and of itself) on the host distro, then get the user to trust a set of initial tools, then build out from inside that
<mark_weaver>well, I don't know why you would trust some other random distro more than this one.
<mark_weaver>but for purposes of implementing David A. Wheeler's answer to the "trusting trust" problem, I would like to support bootstrapping from a diverse set of starting points and comparing the results.
<sir123>mark_weaver: it's a crazy dream of mine to be able to build a trusted gcc buildchain on one system, then move it into another distro to form a 'system' toolchain that can be used for compilation, having it isolated and controlled, 'trusted'
<sir123>i'm reading through your link now, interesting
<mark_weaver>I'm not sure I understand your "crazy dream", but in any case I would like to provide a documented way to create a set of bootstrap binaries capable of bootstrapping guix from a diverse set of starting systems.
<mark_weaver>see gnu/packages/make-bootstrap.scm contains the code that cross-builds the bootstrap binaries for an arbitrary platform from an existing guix system.
<sir123>So the basic concept is to have a compiler, that may or may not be trusted, then build the untrusted compiler out of a trusted compiler (GCC, for example), and compare bit-for-bit with the untrusted compiler?
<mark_weaver>note that there are some patches that need to be applied to some of the bootstrap programs, e.g. guile.
<mark_weaver>you should be able to follow the same approach from an arbitrary system, manually.
<sir123>I recognise that Guix could be the first system to liberate users from the developers, if you get what I mean.
<mark_weaver>sir123: the idea is to build the same version of GCC from a diverse set of starting environments, and then use those GCCs to compile itself one more time.
<mark_weaver>and then from there, you can look at the more detailed test logs, re-run the tests manually in strace or gdb, etc.
<mark_weaver>inside the /tmp/nix-build-* directory there is an 'environment-variables' file that contains the environment variables present during the build, which is important because e.g. the PATH contains the /gnu/store directories for the tools used in the build.
<sir123>i like the HOME variable: "/homeless-shelter"
<mark_weaver>so, to re-run some things in that environment (albeit not isolated now), I will typically do "env -i $(which bash)" to spawn a new shell with an empty environment, and then source the environment-variables file to load those settings, and then work from there.
<mark_weaver>when you first mentioned helping with the documentation, it was in the context of your statement "<sir123> The documentation, at least for me, didn't really help me to understand that.", so I assumed you had an idea of what you wanted to work on.
<davexunit>mark_weaver: quick guile question. I have a file descriptor for the master side of a pty, when I call (fdopen master "r+"), it returns a port object for the slave side. do you know how to open the master fd such that it is the master side of the pty?
<sir123>PaulePanter: hope you get your bug fixed. happy hacking :)
<mark_weaver>PaulePanter: I'm not sure why he needs to build it within Guix to fix that bug. Looking at our current workaround, I guess the issue is that unlike most configure scripts, with freetype it's not sufficient to merely pass CONFIG_SHELL=... to the configure script. instead, we set it as an environment variable before running ./configure
<mark_weaver>Guix automatically passes CONFIG_SHELL=... to configure scripts by default, but that didn't work for freetype
<mark_weaver>PaulePanter: but if the freetype developer has a question, maybe he should post his question as a comment in the bug tracker, and we can take it from there?
<PaulePanter>mark_weaver: Sounds good. I just skimmed the bug tracker for a different bug and saw that he asked for help and that there was no reply.
<mark_weaver>to be clear, setting the CONFIG_SHELL environment variable is a workaround that we use for the freetype package
<PaulePanter>mark_weaver: I guess, that not all developers are that fluent in autoconf(?) or writing good build scripts. So he asked if you somebody else could provide a patch.
<davexunit>and still need to wrap other low-level interfaces to turn off echoing for the pty so input isn't printed twice
<davexunit>well, the 'make' segfault wasn't exactly related to needing to create a new pty, but rather needing to bind mount the host's current controlling terminal as /dev/console within the container
<davexunit>now this works!: ./pre-inst-env guix environment --container guix -E make
<davexunit>(with a hack that bind-mounts the host /bin/sh into the container)
<davexunit>now building a clean checkout of guix in a container. going well so far.
<davexunit>ah, running 'make' downloads bootstrap binaries. never realized that.
<davexunit>so, a container to build guix need network access. I should add a flag to 'guix environment' that doesn't unshare the network namespace.
<mark_weaver>davexunit: hmm, I wonder how that's dealt with in our existing 'guix' recipe.
<mark_weaver>certainly, the environment to build our 'guix' package does not have network access, because it is not a fixed-output derivation.
<yenda>amz3: this^ but I disabled all plugins and it didn't work either, I always get a 401 unauthorized error, but with conkeror for isntance no problem
<mark_weaver>yenda: python-libffi-mips-n32-fix.patch is a fix to the bundled copy of libffi in python. that patch was accepted in upstream libffi and is in the latest libffi release, so there's a good chance that the newest version of python-2 has picked it up.
<mark_weaver>but please verify that the change made in the patch is already in the python sources before removing that patch.
<mark_weaver>the sqlite-3.8.4-test-fix patch might have been picked up also. iirc, it came from upstream python anyway.
<yenda>mark_weaver: yes, both patches have been picked up, at least in 2.7.10
<davexunit>mark_weaver: yes, that's true. it seems to be a problem with my container only.
<davexunit>since what I do is obviously not the same as build.cc.
<davexunit>yeah, build.cc sets TEMPDIR to the temporary build dir
<davexunit>I think I'll create a /tmp in 'guix environment --container' and set TMPDIR accordingly.