IRC channel logs

<vasi>Googulator: sent you a PR for the coredns issue: https://github.com/Googulator/azurelinux/pull/1

<vasi>Googulator: Based on the log files and my system stats, it looks like the Azure build is downloading massive ccache tarballs and potentially using them as inputs while bootstrapping! See some evidence here: https://gist.github.com/vasi/d2a29ea58ac18f28ce7905effa993cc2 and the ccache config file here: https://github.com/Googulator/azurelinux/blob/3.0-bootstrap/toolkit/resources/manifests/package/ccache-configuration.json

<vasi>It looks like we actually do get hits on the ccache cache, I've added it to the gist above. We better either turn off USE_CCACHE, or modify the config somehow so it only uses local ccache. This is all horribly under-documented, I don't see anywhere in the docs that remote ccache exists, but it's definitely being used on my box.

<fossy>that's quite unfortunate

<fossy>thankfully, even if annoying to fix, shouldn't change the bootstrap at all (well, apart from disabling it)

<vasi>Yeah, it's just really concerning that it's so hidden! I'm not sure how to ensure that nothing _else_ weird is going on. Maybe run the build with networking off?

<vasi>Anyhow, restarting my bootstrap from after the toolchain build, this time with USE_CCACHE=n. Hopefully the cache wasn't hiding any build failures

<fossy>yeah, networking off is usually a good bet

<Googulator>That's... bad.

<Googulator>I initially built with ccache disabled, but had to enable it because one build failed with a weird issue related to ccache without it

<Googulator>probably a missing build-time dependency declaration for ccache in one of the SPECs that need it

<Googulator>All of this strongly suggests MS never builds Azure Linux with ccache disabled

<Googulator>Not the first time something documented as supported actually doesn't work - Ubuntu is listed as a supported build host (apparently MS bootstrapped on Ubuntu originally), but some of the SPECs have directives introduced in a newer RPM than the one available in Ubuntu repos

<nikolar>as if microsoft has ever cared about reproducibility or bootstrap paths

<Googulator>nikolar: surprisingly, Azure Linux has the cleanest bootstrap of all RPM- or DPKG-based distros I've seen so far

<Googulator>Microsoft's choice of Ubuntu (a DPKG-based distro) as their bootstrap host is likely what enables us now to try to rebootstrap in the first place - had they chosen an RPM-based host (e.g RHEL), we would probably be facing the same dependency hell as Fedora or openSUSE

<Googulator>Fedora's build system just assumes it can pull every dependency needed from upstream

<nikolar>that's probably just an accident rather than a well thought out decision

<Googulator>well, they had to do it, due to starting from a DPKG-based host and building an RPM-based target

<Googulator>even here, there's some "cheating" involved - the dependency graph of the upstream repository is consulted to fill in missing Provides: information

<Googulator>also, there's one known piece of binary taint: OpenJDK is pulled in as a prebuilt binary package, due to Java being needed quite early in the build system

<Googulator>luckily we do know how to bootstrap OpenJDK already :)

<vasi>Agreed, I haven't seen any other rpm/dpkg based distros that are even vaguely bootstrappable

<vasi>Googulator: also there's a few packages with binary-based bootstraps, like rust

<Googulator>Azure's is the only build system I know that produces real binary packages, and yet has the provisions for building and following a dependency graph

<Googulator>yeah, Rust is another one that pulls in binaries

<vasi>(we do know how to bootstrap rust of course, but trying to get the whole mrustc-chain somehow usable by the Azure bootstrap doesn't sound fun)

<Googulator>also, OCaml, which actually _has x86 binaries committed into its upstream Git repository_

<nikolar>that's a decision of all time

<Googulator>For Rust, I was thinking of precompiling bootstrap compilers outside the Azure Linux build env, and then injecting them into Azure Linux in place of the official ones

<Googulator>same for OpenJDK

<fossy>^ this is the way to go i think

<Googulator>another idea I had was, since Azure Linux's build system is largely separate from its package definitions, we could try building other RPM/DNF-based distros' SPEC files using the AzL Toolkit

<Googulator>e.g. Fedora is meant to be built using Koji one package at a time with full upstream repos available, but assuming its SPECs declare their dependencies properly, Azure Linux Toolkit should have no problem making a dependency graph for them

<vasi>I think guix and stagex may be the only existing distros that bootstrap rust all the way from source

<Googulator>there's also an overlay for Gentoo to do it

<vasi>Oh fascinating, I can't seem to find that overlay via google. Got a link?

<vasi>this one? https://gitlab.com/stikonas/gentoo-bootstrap

<vasi>whooa it's actually been integrated upstream? ok, i take it back, guix + stagex + gentoo as well

<Googulator>Yes, it seems so...

<Googulator>I wonder if any distro has the new mrustc

<Googulator>which can start from 1.74

<vasi>isn't it likely that the Fedora SPEC files have circular dependencies? that's why you can't build eg: Debian from source easily

<vasi>The Azure Linux folks had to explicitly decide an order for their bootstrap toolchain

<Googulator>anyway, AzL has Rust 1.75 and 1.86 currently, with 1.84 and 1.85 available in past git commits - that means, we could even do the bootstrap within the build system by adding mrustc and 1.74, and then daisy-chaining them with dependencies

<Googulator>interestingly, the Azure dependency graph isn't cycle-free

<Googulator>instead, there's actually a cycle resolution stage early in the build, which I don't fully understand atm

<Googulator>that cycle resolution stage is also what appears to use upstream dependency information

<vasi>you mean in the toolchain construction, eg here? https://github.com/microsoft/azurelinux/blob/3.0/toolkit/scripts/toolchain/container/toolchain_initial_chroot_setup.sh

<vasi>or in the actual package builder somehow?

<Googulator>It runs right before package build

<Googulator>After the toolchain is done

<Googulator>BTW, looks like AzL's upstream is openSUSE: https://github.com/microsoft/azurelinux/pull/11295#discussion_r2044799554

<Googulator>makes sense given the historic cozy relationship between MS and Novell

<vasi>this thing? https://github.com/microsoft/azurelinux/blob/3.0/toolkit/tools/internal/pkggraph/pkggraph.go#L1406C97-L1406C100

<vasi>it says it resolves cycles using packages from "PMC" (ie: "packages.microsoft.com", binaries)

<vasi>i hope that doesn't mean yet another source of random binaries

<Googulator>It only appears to use the package metadata from the upstream repo there

<vasi>`fixCyclesWithExistingRPMS` notes that it only looks for upstream prebuilts if RESOLVE_CYCLES_FROM_UPSTREAM is set, ok

<Googulator>yes, and that's off by default, and cannot be enabled if DISABLED_UPSTREAM_REPOS=y

<Googulator>IGNORE_VERSION_TO_RESOLVE_SELFDEP is also off by default

<Googulator>it still does use upstream Provides: information to work around undeclared Provides: entries

<Googulator>RPM has support for autodetecting provided libraries, pkg-config entries, etc. - this has the unfortunate effect that if you rely on it, the binary package will "know" what it can provide, but its source package won't

<Googulator>a misfeature if you ask me :)

<Googulator>at the very least, undeclared provides should be a warning

<matrix_bridge><Andrius Štikonas> Googulator: Gentoo indeed starts from 1.74

<Googulator>looks like I figured out what MS did to resolve dependency cycles

<Googulator>it seems they have a rule that "every dependency cycle must have at least one node that's part of the toolchain"

<Googulator>and then the Grapher golang tool transforms the cycle A->B->C->A to A::toolchain->B->C->A::final

<Googulator>and if there's a new cycle, they bootstrap one package in it and add it to the toolchain

<vasi>They also seem to break mutual-runtime-dependencies, see fixIntraSpecCycle

IRC channel logs

2025-09-09.log