Skip to content

Fedora Go Unbundling is Broken

Intro

For some time now, the Fedora Go Special Interest Group (hereinafter “Go SIG”) has been building Go packages based on individually Go modules installed into $GOPATH/src and packaged into RPMs instead of using Go modules. This post explains why I think the current system is broken and explores another option: vendoring.

Numbers

Note

This data was generated by rough fedrq queries. There may be a couple packages missing that depend on golang in irregular ways.

Currently, the Fedora Go SIG maintains 18581 library packages from 17162 source packages. We maintain 556 application packages3 (i.e., packages that provide binaries / applications that end users can run) from 459 source packages4. Considering that some source packages provide library applications and library packages, that amounts to a total 18515 source packages.

Evidently, there is a large imbalance between the amount of library packages that we maintain versus the amount of applications we maintain. With quite few active members of the Go SIG, it proves quite difficult to keep such a large number of package in proper shape.

Compatibility and Maintainability

Maintaining these library packages is not exactly straightforward. The Go packaging ecosystem is nonideal for distributions that wish to maintain each library as a separate package—it was never designed that way. A fair number of libraries do not publish tagged releases, so each project depends on an arbitrary commit of those libraries. Other widely used libraries, such as the golang-x-* projects, only have 0.Y.Z beta releases, despite being depended on by a large number (494) of source packages6. (This is a recent development; for a while, these packages did not have tagged releases either.) Furthermore, the Go module ecosystem lacks a way for packages to set compatible version ranges like, for example, the Python and Rust crate ecosystems. Each project just sets a single version in the go.mod file. We cannot rely on upstream metadata or stable versions to help us determine whether an update is incompatible with other packages in the distribution like other Fedora language ecosystems do.

These inherent issues with the Go package ecosystem design makes it quite easy to break other packages when updating libraries. See the Fedora Release Engineering tracker for how many times we have had to revert library updates that ended up breaking many other packages. As of late, we have struggled to keep packages functional and up to date. However, I’d be remiss if I did not acknowledge the heroic work of Go SIG members who have worked on cutting the amount of the packages that are out of date or that fail to build from source.

One solution is to run impact checks in Copr and rebuild reverse dependencies before updating any Go library, but this is quite time consuming with such a large number of packages maintained by so few people. Impact checks are not foolproof either, as when libraries completely remove import paths, Copr will just pull old library versions.

Go modules support

Currently, our packaging tooling—namely, go-rpm-macros—does not support Go modules, as the module system is not exactly amendable to using a local registry of bundled dependencies. The effort to modify the tooling in https://pagure.io/GoSIG/go-sig/issue/35 is stalled. We still rely on the deprecated GO111MODULE=off mode which does not work with some newer packages that use sub-Go modules without a lot of finagling.

Vendoring

One solution is to vendor libraries within each package. This involves running go vendor, creating an archive containing the vendor directory, and uploading that to the lookaside cache as a secondary source, in addition to the project’s primary source archive. Note that we cannot download Go modules inside the main build process, as Fedora’s buildsystem builds packages with network access disabled.

Vendoring removes the need to maintain so many library packages and ensure that they are compatible with each other. If we retired all the library packages, we would be able to significantly reduce our maintenance burden. Vendoring also allows us to enable Go modules and stop using the deprecated GO111MODULE=off mode.

However, vendoring is not without its issues. For one thing, following Fedora’s Bundled Software Policy and Licensing Guidelines is more difficult with vendored packages. Packagers need to account for the license identifiers of all the vendored code and construct a cumulative SPDX identifier to add to the License: tag. Packagers also must include all license files from each vendored library contained within the package. Re-performing a license audit after each update is tedious. The Bundled Software Policy also requires including bundled(golang(PATH)) = VERSION Provides for each vendoring library. Handling library vulnerabilities can also be difficult: each package needs to be manually updated and patched to use the new library version.

I have worked on tooling to mitigate these issues. We have had an RPM generator to automatically generate the aforementioned bundled() Provides for some time now. I started developing the go-vendor-tools project. See my report from this week for more information about the tool. It aims to make creating vendor archives and handling licensing a relatively frictionless process. The tooling will also allow each packager to regenerate vendor tarballs with new library versions to mitigate security vulnerabilities.

Using the builtin tooling (i.e., go mod vendor) with some helper scripts to vendor Go modules is more practical than attempting to work around the intractable issues with unbundling a software ecosystem that is not at all designed for that purpose.

  • Updated 2024-03-25: Add section about Go modules

  1. fedrq pkgs -P 'golang(*)' | wc -l ↩

  2. fedrq pkgs -F source -P 'golang(*)' | wc -l ↩

  3. fedrq wr go-rpm-macros $(fedrq subpkgs golang) -s | fedrq subpkgs -i --arch x86_64 | wc -l ↩

  4. fedrq wr go-rpm-macros $(fedrq subpkgs golang) -s | fedrq subpkgs -i --arch x86_64 -F source | wc -l ↩

  5. fedrq wr -s $(fedrq subpkgs golang) go-rpm-macros | wc -l ↩

  6. fedrq wr golang-x-\*-devel -s | wc -l ↩