IRC channel logs

<PotentialUser17>Hello! Is anyone here familiar with using guix offload and guix publish together (possibly with guix deploy as well)? I currently have two instances on AWS running guix, one of which is offloading builds to the other. I would like to have that "master" instance also make its store available via publish. The issue that I'm having, assuming I'm

<PotentialUser17>reading the docs correctly, is that publish will only serve substitutes. It will not initiate a build on that machine if a substitute does not exist. Is that correct? What I would like to have happen is a CI pipeline (in something like a github or gitlab repo) creates an environment using a channels and manifest file and those substitutes are

<PotentialUser17>pulled from that "master" instance. In the event that the substitutes do not exist on that "master" instance I would like "master" to offload the build instead of trying to do the build on the lower power machine running the CI pipeline.

<PotentialUser17>Does that make sense? Has anyone here done something like that before?

<civodul>PotentialUser17: hi! it does; looks like you need to set up offloading, which is not related to ‘guix publish’

<PotentialUser17>Offload is already set up and working

<civodul>ah good

<civodul>so i guess you’re all set?

<PotentialUser17>I'm just confused about how to trigger builds from the CI machine if the substitute does not exist on the "master" (or publish) machine

<civodul>if the substitute does not exist, then Guix will try to build the thing

<civodul>if offloading is set up, then it will offload it

<PotentialUser17>So CI machine asks publish machine for a substitute, if the substitute does not exist I would like the publish machine to offload to a more powerful build machine and convey that substitute back to the CI machine

<PotentialUser17>Will the publish machine, if offload is set up, automatically do this?

<civodul>ah no, that’s not how it works

<PotentialUser17>Or will the build happen on the CI machine?

<civodul>exactly

<civodul>‘guix publish’ really does nothing but publish binaries

<civodul>it does not spawn builds

<PotentialUser17>Ok so the CI machine needs to be able to access the build machines

<civodul>yes

<civodul>(or the CI machine can be a build machine itself)

<PotentialUser17>So if the CI machine does not receive a substitute from the publish machine it will then automatically offload that build (assuming offload is set up)

<civodul>yes

<PotentialUser17>I'd like to use low power machines for CI, so having stronger build machine to offload to is preferred

<PotentialUser17>Ok that all makes sense

<civodul>great :-)

<PotentialUser17>Last question: is there a way with guix deploy to spin up an instance on AWS?

<PotentialUser17>So if I need to offload a build can I have guix turn on that machine on AWS?

<civodul>no, not yet at least, but ‘guix deploy’ is modular and has backends for different kinds of deployments

<civodul>it has one for DigitalOcean for instance

<civodul>so i guess it could have one for AWS (not familiar with it though)

<PotentialUser17>Yeah I had tried to google around for one but didn't see a whole lot

<civodul>but then, deployment is far from instantaneous

<PotentialUser17>I saw some old logs of rekado and others discussing it, but it didn't look like it really went anywhere

<civodul>so the machine would need to be ready beforehand, somehow

<civodul>ah yes, they know better than me :-)

<PotentialUser17>Right, I would pretty much just need a way to turn the machine on. I'm fine with having the instance already created on AWS.

<civodul>i guess this whole area needs work, but it’s not an unusual need

<PotentialUser17>CI machine requests a substitute from publish machine, if it does not receive one CI machine will offload the build, if a build happened CI machine will run guix copy to transfer the store to the publish machine so future CI runs will have the necessary substitutes and no builds need to happen

<PotentialUser17>Does that sounds like a reasonable workflow? Leaving aside the whole deploy stuff

<civodul>offloading essentially copies things inputs to the build machine, and then copies the result back to the original machine

<civodul>so one way to set it up is to have ‘guix publish’ running on the head node

<civodul>that node will provide substitutes

<PotentialUser17>Maybe I'm confused

<PotentialUser17>so I want to have an always on publish machine that can serve substitutes to many other machines (CI or otherwise)

<PotentialUser17>but that machine won't run builds if a requested substitute does not exist, right?

<civodul>correct

<PotentialUser17>Ok I think I'm understanding then

<civodul>‘guix publish’ won’t spawn builds

<PotentialUser17>So running 'guix publish' on the head node won't help me, since in this case the head node is just some low power CI machine that's turned on as needed

IRC channel logs

2024-04-01.log