Skip to content

Blog

Each week, I publish a summary of the open source work I completed, feature a musical, and discuss general events.

Week of 2023-03-03

Open source

I have devoted most of my open source dev time this week to go-vendor-tools. See last week’s report and “Fedora Go Unbundling is Broken” for more information. There will likely be more discussion on this during Monday’s Go SIG meeting.

Next week, I plan to work on finishing up the antsibull-core v3.0.0 release and some of the other Ansible-related tasks I kind of shirked this week. There are a few active Steering Committee discussions on the Ansible Forum. I also need to finish the next steps of my Ansible package–wide testing effort. Mainly, that consists of checking back on each issue I filed against broken collections.

Musical of the week

The musical of the week is Avenue Q. Think about it as a somewhat dirty (“adult”) staged musical version of Sesame Street. The show consists mostly of puppets, sans the three human characters. Avenue Q’s eclectic cast of characters (attempt to) get about their lives in an “outer-outer borough” of New York City. The musical dives head first into racism, dating, financial instability, sexuality (the show addresses the gay Bert & Ernie theory), sex work, and other topics. Apparently, someone at MTI thought it would be a good idea to develop Avenue Q School Edition. The school edition replaces “The Internet Is For Porn” with “My Social Life Is Online”; I guess the puppet sex scene is also removed. Avenue Q inspired Trey Parker and Matt Stone to author The Book of Mormon musical.

Suggested reading

The Atlantic April 2024 Issue’s cover story is “The Golden Age of American Jews Is Ending” (gift link). The article discusses the rise of antisemitism on the left and right of the political spectrum and how Jews have been ostracized from the liberal movements we helped establish.

Maxwell

Reply on the Fediverse

Fedora Go Unbundling is Broken

Intro

For some time now, the Fedora Go Special Interest Group (hereinafter “Go SIG”) has been building Go packages based on individually Go modules installed into $GOPATH/src and packaged into RPMs instead of using Go modules. This post explains why I think the current system is broken and explores another option: vendoring.

Numbers

Note

This data was generated by rough fedrq queries. There may be a couple packages missing that depend on golang in irregular ways.

Currently, the Fedora Go SIG maintains 18581 library packages from 17162 source packages. We maintain 556 application packages3 (i.e., packages that provide binaries / applications that end users can run) from 459 source packages4. Considering that some source packages provide library applications and library packages, that amounts to a total 18515 source packages.

Evidently, there is a large imbalance between the amount of library packages that we maintain versus the amount of applications we maintain. With quite few active members of the Go SIG, it proves quite difficult to keep such a large number of package in proper shape.

Compatibility and Maintainability

Maintaining these library packages is not exactly straightforward. The Go packaging ecosystem is nonideal for distributions that wish to maintain each library as a separate package—it was never designed that way. A fair number of libraries do not publish tagged releases, so each project depends on an arbitrary commit of those libraries. Other widely used libraries, such as the golang-x-* projects, only have 0.Y.Z beta releases, despite being depended on by a large number (494) of source packages6. (This is a recent development; for a while, these packages did not have tagged releases either.) Furthermore, the Go module ecosystem lacks a way for packages to set compatible version ranges like, for example, the Python and Rust crate ecosystems. Each project just sets a single version in the go.mod file. We cannot rely on upstream metadata or stable versions to help us determine whether an update is incompatible with other packages in the distribution like other Fedora language ecosystems do.

These inherent issues with the Go package ecosystem design makes it quite easy to break other packages when updating libraries. See the Fedora Release Engineering tracker for how many times we have had to revert library updates that ended up breaking many other packages. As of late, we have struggled to keep packages functional and up to date. However, I’d be remiss if I did not acknowledge the heroic work of Go SIG members who have worked on cutting the amount of the packages that are out of date or that fail to build from source.

One solution is to run impact checks in Copr and rebuild reverse dependencies before updating any Go library, but this is quite time consuming with such a large number of packages maintained by so few people. Impact checks are not foolproof either, as when libraries completely remove import paths, Copr will just pull old library versions.

Go modules support

Currently, our packaging tooling—namely, go-rpm-macros—does not support Go modules, as the module system is not exactly amendable to using a local registry of bundled dependencies. The effort to modify the tooling in https://pagure.io/GoSIG/go-sig/issue/35 is stalled. We still rely on the deprecated GO111MODULE=off mode which does not work with some newer packages that use sub-Go modules without a lot of finagling.

Vendoring

One solution is to vendor libraries within each package. This involves running go vendor, creating an archive containing the vendor directory, and uploading that to the lookaside cache as a secondary source, in addition to the project’s primary source archive. Note that we cannot download Go modules inside the main build process, as Fedora’s buildsystem builds packages with network access disabled.

Vendoring removes the need to maintain so many library packages and ensure that they are compatible with each other. If we retired all the library packages, we would be able to significantly reduce our maintenance burden. Vendoring also allows us to enable Go modules and stop using the deprecated GO111MODULE=off mode.

However, vendoring is not without its issues. For one thing, following Fedora’s Bundled Software Policy and Licensing Guidelines is more difficult with vendored packages. Packagers need to account for the license identifiers of all the vendored code and construct a cumulative SPDX identifier to add to the License: tag. Packagers also must include all license files from each vendored library contained within the package. Re-performing a license audit after each update is tedious. The Bundled Software Policy also requires including bundled(golang(PATH)) = VERSION Provides for each vendoring library. Handling library vulnerabilities can also be difficult: each package needs to be manually updated and patched to use the new library version.

I have worked on tooling to mitigate these issues. We have had an RPM generator to automatically generate the aforementioned bundled() Provides for some time now. I started developing the go-vendor-tools project. See my report from this week for more information about the tool. It aims to make creating vendor archives and handling licensing a relatively frictionless process. The tooling will also allow each packager to regenerate vendor tarballs with new library versions to mitigate security vulnerabilities.

Using the builtin tooling (i.e., go mod vendor) with some helper scripts to vendor Go modules is more practical than attempting to work around the intractable issues with unbundling a software ecosystem that is not at all designed for that purpose.

  • Updated 2024-03-25: Add section about Go modules

  1. fedrq pkgs -P 'golang(*)' | wc -l ↩

  2. fedrq pkgs -F source -P 'golang(*)' | wc -l ↩

  3. fedrq wr go-rpm-macros $(fedrq subpkgs golang) -s | fedrq subpkgs -i --arch x86_64 | wc -l ↩

  4. fedrq wr go-rpm-macros $(fedrq subpkgs golang) -s | fedrq subpkgs -i --arch x86_64 -F source | wc -l ↩

  5. fedrq wr -s $(fedrq subpkgs golang) go-rpm-macros | wc -l ↩

  6. fedrq wr golang-x-\*-devel -s | wc -l ↩

Week of 2024-02-25

Open source

I released ansible 9.3.0 this week. I used the new release automation developed by Anwesha Das, as well as the new release announcement automation I implemented in antsibull.

I cut a new release of Fedora’s Golang RPM Macros. It addresses some longstanding issues, including the ability to pass our compiler flags to upstream Makefiles and other build scripts and removing clashes between C LDFLAGS and Go LDFLAGS. The release also include a patch from Zbigniew Jędrzejewski-Szmek to enable build reproducibility for Go packages. Package build IDs were randomly generated previously and are now generated in a reproducible way.

I attended the bi-monthly Fedora Go Special Interest Group meeting. We discussed migrating our repositories from Pagure to Gitlab to allow us to take advantage of Gitlab CI and Packit. The CI situation with Pagure is not great, and the upstream project’s maintenance is in question. Fedora has its own namespace on gitlab.com, and it integrates with the Fedora Account System so we can give everyone in the Go SIG group access without having to manage permissions in multiple places. Gitlab’s interface is a bit… busy, but it works well enough. I guess its ‘open core’ model is better than Github’s completely closed source model.

I started a new go-vendor-tools project to enable creating vendored Go packages in Fedora while following the bundling and licensing guidelines. That project will be moved to the new fedora/sigs/go Gitlab namespace once Infra creates it for us. The tool definitely needs more work, but so far, it can:

  • Create Go vendor tarballs in a reproducible way
  • Crawl the vendor directory for license files using askalono-cli and generate a cumulative SPDX license identifier for the project
  • Implement a TOML configuration file to mark license files that askalono is unable to detect
  • Implement RPM macros to install all the detected license files into the package’s %{_licensedir} and verify that the package’s SPDX identifier matches the package’s RPM License: tag

The package uses a—dare I say—innovative Packit setup to run integration tests for the code and RPM macro side, in particular. If it ends up working out well, I may adopt a similar setup for the main Go macros and Ansible macros projects that I maintain. That probably warrants a separate bog post. In any case, I am a big proponent of automated testing, especially for RPM macros where one small mistake has the propensity to break all 23 thousand source packages in the Fedora package collection. I have explored various approaches for RPM macro testing, including pure pytest setups, a YAML file containing test cases, and the aforementioned Packit-based integration testing.

Musical of the week

The musical of the week is Little Shop of Horrors. The musical combines humor with deeper themes: morality; death; and the consequences of fame. I saw the show Presidents Day weekend off Broadway with Darren Criss (of “Glee” fame) playing the lead, Seymour Krelborn.

Until next time,

Stay sane and stay sure

Maxwell

Reply on the Fediverse

Week of 2023-12-10 (and the other weeks I missed)

Open source

I got back to improving our developer tooling for ansible-documentation. Amongst other tasks, I integrated the docs build process into the nox configuration.

I performed Fedora distgit and ansible-documentation code reviews as usual.

As discussed in My First Ansible Release, I have been working on improving the testing for the collections in the ansible package. My PR antsibull to enable this is now ready for review! The PR allows cloning all the collections in the ansible package, running sanity tests, and collecting the results. I also created https://github.com/ansible-community/package-test-results to store the test results and coordinate filing issues against broken collections. Feedback is welcome in the forum post!

I helped create a hot fix antsibull-docs release to address a new ansible-core change that broke the Ansible documentation.

I started working on plans for the next major release of antsibull-core, the library used by the other antsibull tools.

I started working on improving sourcehutx’s documentation.

I reported an issue against dnf5 for an issue that broke fedrq’s CI.

Speaking of fedrq, I have done some work on cleaning up the API and preparing to add support for recursively resolving package queries. The Repoquery.resolve_pkg_specs interface has gained some additional arguments to enable this. I spent some time going through dnf and libdnf5’s package Goal APIs and planning out a compatibility layer between them in the same way I’ve done for the Package and Query APIs.

I have also worked on tomcli, my CLI tool for querying and modifying TOML files and cut 0.4.0 and 0.5.0 releases (changelog). I improved the documentation a bit. I added new subcommands and more flexible output formatting. I reduced the dependencies by switching from typer to click for CLI parsing and from hatchling to flit_core for the Python build backend. The goal is to make tomcli easier to bootstrap so it can be adopted more easily by baseline Python packages in Fedora. I would like to release 1.0.0 and finish stabilizing the CLI API.

I submitted a PR to Fedora’s python-click package to make it easier to bootstrap.

Musical of the week

The musical of the week is In the Heights. Lin Manuel Miranda wrote In the Heights in college long before he wrote Hamilton. The musical centers around a group of Dominican American families living in the gentrified Washington Heights neighborhood of New York City. I have managed to memorize the entire opening number. I will be seeing a local production of the musical in February. There is a movie version of the musical on (HBO) Max1.

Maxwell

Reply on the Fediverse

  1. HBO must have payed a ton of money to purchase max.com… ↩

Week of 2023-11-19

Open source

I released Ansible 9.0.0 and subsequent Ansible 9.0.1. It turns out, there was an error in our new setuptools configuration, so we needed to release a hot fix.

I’ve continued my work on improving QA for the ansible community package. See Testing collections within the ansible package on the Ansible Forum for more details!

I updated ansible-core to 2.16.0 and ansible to 9.0.0 in Fedora. We have had packaged pre-releases up until now to enable support for Python 3.12.

I submitted PRs to the ansible-documentation repository to fix our broken 404 page template. Thanks to @felixfontein for helping diagnose the issue!

I also updated ansible in EPEL 8 to correspond to RHEL 9.3’s ansible-core 2.15.x update.

I worked locally on some changes to tomcli to make output formatting more flexible.

Per usual, I helped with ansible-documentation and Fedora distgit PR reviews.

Musical of the week

The Musical of the Week is Spring Awakening. Spring Awakening, based on the 1891 German play of the same name, explores adolescent sexuality through the lens of repressive late 19th century Germany. The musical deals with complex subjects, including religion, suicide, sexual abuse, abortion, and coming to terms with one’s sexual identity. The “taboo” nature of these topics makes them all the more important to discuss. Spring Awakening’s leads, Melchior Gabor and Wendela Bergmann, were Jonathan Groff and Lea Michelle’s breakout roles, respectively.

Maxwell

Reply on the Fediverse

Week of 2023-11-05

I released Ansible 8.6.0 and Ansible 8.6.1. See My First Ansible Release for more details!

I submitted a couple PRs to ansible-build-data to improve the release process documentation and to antsibull to improve parts of the release playbook and fix a regression I introduced in another change 😬.

FESCo approved my Pydantic V2 change. @music and I are now working on the package reviews and other steps to integrate the new major version.

I packaged bindep and ansible-builder for Fedora. They are currently waiting in the updates-testing repository.

I also packaged flit-core for the alternative python3.11 stack in EPEL 9.

RHEL 9.3 was released this week, so I updated the ansible package in EPEL 9 accordingly. In RHEL 9.2, ansible-core was built against python3.11, but it’s now back to the default python3 (python3.9) stack. I updated ansible-core to 2.16.0 in Fedora as well.

The musical of the week is Beetlejuice.

Maxwell

Reply on the Fediverse

My First Ansible Release

My first ansible package release and reflections on the process.

Intro

This week, the Ansible community package release process was opened up to the community (i.e., non-Hatters) for the first time! This is a pretty big milestone for our community. The Ansible Community Steering Committee made decisions and policies for the Ansible package, but we were never able to handle the releases ourselves — until now. I was the release manager for this week’s Ansible 8.6.0 release and @felixfontein, another community member, took care of Ansible 9.0.0b1. As the maintainer of the downstream Fedora package and co-maintainer of our build tooling, it was great to see the process in action!

Release process overview

Note

The ansible community package contains a set of independent Ansible collections that are curated by the community, and it pulls in ansible-core. Each major version of ansible depends on a specific major version of ansible-core and contains specific major versions of the collections it includes.

The release process involves putting the appropriate collection versions together, downloading the collection artifacts, and combining them into a single Python package with the proper metadata. Along with the package, a combined changelog and porting guide are generated for all of the collections in the package. The porting guide, changelog, and other static data about the specific collection versions included in each ansible release is kept in the ansible-build-data repository.

Hiccups

I also have a newfound appreciation for the work of the Ansible release manager and am already working on process improvement for future releases. The Ansible 8.6.0 release, in particular, was not without hiccups.

We had one upstream collection that had a semantic versioning violation and another collection that published a release with syntactically invalid Python files. The latter issue broke downstream packages even for users who do not utilize the broken plugins. This is unfortunately not the first time collection maintainers have published releases with Python syntax errors. In the end, I pinned the semver-violating collection to an earlier version and had to publish an 8.6.1 hottfix release to fix the other issue.

Testing and QA issues

We generally rely on upstream collections to follow our guidelines and run CI tests before publishing their releases. However, some collections, especially the commercial ones with convoluted internal <-> external repo sync pipelines, do not seem to consistently test releases before publishing them to Ansible Galaxy 😥.

The Steering Committee and Release Management WG are discussing running some basic smoke tests of our own on top of the ansible package. To this end, I submitted a PR to add a basic Python byte-compilation test to the ansible release playbook. This should at least catch the most egregious issues.

We are also discussing running at least a subset of ansible-test sanity’s tests on top of all collections, but that presents its own can of worms. Mainly, running all of the tests for every single collection is very time-consuming and resource intensive. Also, we would need to figure out how to retrieve the collection sources. The Galaxy artifacts are not guaranteed to include all of the files needed to run sanity tests while the content tagged in the upstream Git repository may not match the Galaxy artifact’s content.

Week of 2023-10-22 and Israel statement

Open Source

The Ansible Community Steering Committee is working on revamping our voting process to use the new community forum. We previously used Github Issues for discussions and created Github Discussions with +1/-1 comments for voting that we counted manually. The plan is to switch over to the forum and use the nice poll feature to count votes for us. I participated in discussions bikeshedding about specific process details and reviewed the policy change PR.

The Go macros in Fedora have a lua function/RPM macro to generate a Go package name based on an import path by replacing / with - along with other normalization. It turns out the way that the macro generates names for compat packages is contrary to the Packaging Guidelines and has been for the past five years. @eclipseo and I discussed the situation and he submitted PRs to go-rpm-macros and go2rpm to change the naming algorithm that I reviewed.

I worked on backporting these changes to the go-rpm-macros-epel EPEL 9 backport. We have had an entirely separate package that selectively overrides macros, as RHEL ignored our bugs and PRs to fix its broken patches. I decided to switch to a source git-based structure instead of keeping code in distgit. There is now a separate epel9 distgit branch in the upstream go-rpm-macros repository.

I have continued working on the dev tooling in the ansible-docuemntation repository and fixed an issue with our pipx installation instructions.

As usual, I reviewed various distgit PRs and Ansible documentation changes.

Musical of the Week

The musical of the week is Hadestown. Hadestown recounts the Greek tragedy of Orpheus, a wide-eyed songwriter and son of a muse, and Eurydice, a tortured young girl simply searching for something to eat and a warm place to sleep. Hadestown’s jazz-infused score and terrifying industrial re-imagination of the Underworld bring the chilling tale to life. I saw the musical in Chicago last summer and quite enjoyed it other than the overly pitchy Orpheus understudy.

Israel

I’m also featuring The Decolonization Narrative Is Dangerous and False from The Atlantic that counteracts some problematic rhetoric about Israel and covers the wider context of the conflict.

I condemn the Hamas terrorists who killed, mutilated, raped, and kidnapped innocent civilians, including grandparents, pregnant women, and children. The Kibbutzniks who Hamas attacked were peaceful, left-leaning Israelis, many of whom advocated for peace, opposed the hard right government and West Bank settlement expansion, and even spoke Arabic. I also mourn the loss of innocent Palestinians in Gaza, while recognizing Israel’s right to neutralize the Hamas terrorists in a manner that minimizes civilian deaths. Hamas seeks to sabotage the peace process and annihilate the Jewish people, all while deliberately endangering its own people.

Maxwell

Reply on the Fediverse

Week of 2023-10-15 (and more!)

My migraines and depression have been quite bad the past couple weeks, so I took a little break from these posts.

After much ado, I finally submitted my Fedora Pydantic v2 Change for review. After more ado and some poking from @decathrope, the Change was finally announced. Thanks as well to @music for his help with the Change.

I have done my usual work on the ansible-documentation repository to review PRs and improve dev tooling. I started a discussion on squashing vs. merging vs. rebasing when applying Github pull requests. I worked on an improvement to the way our issue/PR triager and pip-compile Github Actions authenticate with the Github API. I reviewed a couple PRs from the community. I updated our PR triager to warn contributors when their changes modify automatically generated files.

I also reviewed a change to enable codespell in the ansible-documentation repo. One of the issues is the naive way in which the tool parses its configuration. I plan to work on a PR to improve it next week.

I worked on packaging hatch-requirements-txt for Fedora. It’s a hatchling plugin that allows sourcing package requirements from a requirements.txt external to pyproject.toml. Of course, it had a new unpackaged test dependency that in turn required an entirely new build system called whey that in turned required even more dependencies. I decided to patch the new test dependency to use setuptools…

I preformed my usual ansible, ansible-core, and ansible-collection-* package updates and assisted with a couple distgit PR reviews.

I helped resolve a situation where duplicate packages were introduced into Fedora.

I’ve contributed some minor changes to the new maubot-fedora Matrix-native Zodbot replacement.

I helped with efforts to move Ansible Community Steering Committee votes to the new Ansible Community Forum.

The musical of the week is Waitress. Waitress, based on the 2007 movie by late director Adrienne Shelly, focusses on a masterful pie baker and waitress. Jenna grapples with life in a small Southern town and an abusive husband. The musical sensitively chronicles Jenna’s journey towards self reliance and realization, as well as her dream of ending her toxic marriage and establishing her own pie shop.

Maxwell

Reply on the Fediverse

Week of 2023-09-17

I wrote a draft Update_To_Pydantic_Version_2 Change to update Pydantic to version 2 in Fedora. I need to clean it up and submit it for community feedback.

As part of this work, I submitted a PR to Matrix synapse to add compatibility for the new version. For now, it uses the pydantic.v1 compat module until pydantic v1 support can be dropped completely.

I attended the FESCo meeting to discuss an issue relating to Fedora ELN. It seems we have reached some common ground.

I attended the Ansible Docs WG meeting and helped with the ansible-documentation repository branching process.

In case you haven’t heard, the Ansible community has a new forum at https://forum.ansible.com. I started a forum group for folks who maintain Ansible-releated packages in Linux distributions. So far, we have folks from Fedora, Arch, and the Ubuntu PPA!

I helped with some changes to the rust-packaging macros to fix an issue affecting written-in-Rust Python extension modules and automate vendoring for RHEL/ELN.

I participated in miscellaneous Ansible Community Steering Committee discussions and Fedora distgit PR reviews, as usual.

The musical of the week is Les Misérables, a classic and one of my favorites. I saw it twice, once in Chicago and again in London while I was in the vicinity for Flock to Fedora. It is a sung-through musical, which I particularly like. There is a reason it is one of the longest running musicals.

Maxwell

Reply on the Fediverse