Streamlining AWX Releases

Project Discussions
, , , , ,
1

Hi folks,

For those of you who don’t know me, I’m John “Gundalow” Barker and have been part of the Ansible team at Red Hat for over 8 years. I am an engineering manager and currently responsible for the Ansible engineering team that focuses on the upstream community. You can find me on this Forum or Matrix as gundalow.

In a recent blog post, we talked about some upcoming changes we’ll be making to the AWX project. This will sound familiar if you attended Red Hat Summit/Ansible Fest 2024. As we acknowledged at both AnsibleFest and in the blog, we know we didn’t go into all the details as they are still forming.

The issues we face have been building for many years, we’ve either ignored them, pushed them to another day or attempted adjustments, and while we are just beginning to talk about how we might evolve AWX, the thoughts we have are many, but unstructured. We have lots to overcome but also need to focus on how we address them as both an open source project and as one of the upstream components of Red Hat’s Ansible Automation Platform.

While usage of AWX has grown beyond what we ever anticipated, this did not happen without encountering a fair share of challenges along the way. Unsurprisingly, over time, it has become more and more challenging to make changes, both upstream and downstream. While we have done our best to support our community by enabling and collaborating with contributors, triaging and debugging issues, and providing regular releases, we have struggled to support the project and product as usage continues to increase.

The changing needs of the project and the broader ecosystem have led to the realization that we must rethink AWX. To meet these increasing demands, we need to transform it into a more modern and scalable solution.

Reaffirming the intent of AWX

As AWX is now a fairly well-known and established open source project, we realize that not everyone may be familiar with its history. Originally, AWX was the internal name for the closed-source product Ansible Tower, written and sold by Ansible and then Red Hat after the former was acquired. In the FAQ that was published when AWX was very first open sourced by Red Hat in 2017, it stated:

Q: WHAT’S THE DIFFERENCE BETWEEN AWX AND ANSIBLE TOWER?

AWX is designed to be a frequently released, fast-moving project where all new development happens.

Ansible Tower is produced by taking selected releases of AWX, hardening them for long-term supportability, and making them available to customers as the Ansible Tower offering.

This is a tested and trusted method of software development for Red Hat, which follows a similar model to Fedora and Red Hat Enterprise Linux.

With the introduction of AWX, we spent months going through the process of getting it ready for the world to see and use. From the very beginning, we have always wanted to stay true to Ansible and Red Hat’s dedication to collaboration and open source, as we believe it results in higher quality and more secure software.

The years that followed the initial release have resulted in a wide set of contributions, ranging from new features to bug fixes, documentation, and a community of users supporting each other and the project.

Through our experience in supporting AWX, one of the main challenges of accepting active contributions was the initial design of the Ansible Tower project all those years ago, and the law of unintended consequences. Since the codebase was not originally designed as a community-minded project, when we do get contributions, a seemingly sensible change in one location causes knock-on impacts elsewhere, making it difficult to have confidence we are not breaking something down the delivery pipeline.

While we want to make changes that enable more collaboration with our community, at the same time we also must reduce the overhead we as the Ansible Engineering team within Red Hat spend supporting AWX so that we can more effectively serve both our upstream community and paying customers.

Our goal here isn’t to offer an exact replica of our commercial product for free. Instead, we aim to foster an innovative and collaborative environment for upstream development within the Ansible Ecosystem that ultimately results in higher quality code and a better experience for everyone.

Changes to how we version AWX

You may have seen our recent AWX Office Hours meeting where we proposed a shift to Calendar Versioning (CalVer) for AWX. This is the first change we’re making and is something that will continue to roll out across all of the Ansible projects, like we have already done in our Ansible Development Tools projects such as ansible-lint.

Why CalVer over SemVer?

Traditionally, AWX has used Semantic Versioning (SemVer). This approach implies a certain level of stability and support, and has led to confusion and mismatched expectations over the years. AWX has always been a “one-way street” - no stable/release branches, and tags have always been created from the most recent commit on the devel branch at a certain point in time. CalVer, on the other hand, better aligns with our intent of a single, rapidly evolving development stream.

As we’ve started socializing this idea, one FAQ has been how we will be denoting and communicating breaking changes. This and some other things we don’t know yet are captured a little further down. What we do know, is that release notes will continue to be auto-generated from Pull Request titles. We will also be engaging the community as we begin to make changes to our tooling to look into possible ways of programmatically detecting or automating this.

Additional changes: How we build and distribute AWX

As we begin to make changes to AWX, we need to balance the effort we put into maintaining it with making progress towards its future. The alternative is we fork the project and start afresh in parallel, but we are reluctant to do that. As those that have maintained large projects might understand, it’s nearly impossible to merge diverging branches back together later without incurring a lot of work. Now let’s take a look at some steps we’ll be taking in order to achieve what we’ve outlined above.

Pausing releases

Until we’ve completed some of the changes we’ve described in this post, we would like to focus our limited time and energy on what comes next for the project. Therefore, AWX 24.6.1 will be the last release until we implement CalVer and other changes around our build and release processes.

During this time, we will still be making and accepting new contributions. If they happen to be against an area of the codebase that’s undergoing large changes, we may put them on-hold, though we don’t anticipate many, if any, in these areas.

For those that would like to stay on the bleeding edge, you are free to build the images yourself using the tooling and documentation in the repo.

Additional changes: common services

Part of our strategy for Ansible Automation Platform is to build standard, common services for its various components. This reduces duplicated code, makes it easier to maintain, and makes it easier for people to contribute.

As we continue to move towards a more services-based architecture, it will also be critical that we think about how all of the components within AAP communicate with each other. An early example of this will be in the next version of AAP, where we will introduce a new unified UI and provide a single place to log in and manage the platform. We hope to discuss these changes more in the coming weeks to give a more comprehensive overview of more forward-looking ideas in the Ansible Forum.

What we don’t know yet

We don’t have all the answers, and that’s part of why we are using the forum to gather feedback and allow dialogue. Some current unknowns are:

  • Should we use YY.MM.DD or YYYY.MM.DD
  • How breaking changes will be communicated as we move from SemVer to CalVer.
  • At what cadence AWX will be released. We are currently considering nightly builds, or perhaps weekly snapshots.
  • The first CalVer release. This is dependent on how timely we can make the changes to our tooling.

Looking ahead

We understand we are talking about some large structural changes to how AWX is developed today, though they are needed to allow continued development. The initial response to these changes during AWX Office Hours was positive and attendees understood how this would allow faster development and easier contribution.

We look forward to the discussion and collaboration with our contributors and upstream users to help shape the future structure, and see how we can work together to ensure we can continue to scale your automation needs. All ideas are welcome, we hope you will find time and reason to participate.

Join the discussion

We value your input on these changes. You can share your thoughts asynchronously by replying to this Forum Post

I’ll also add a link later to allow you to provide anonymous feedback form if you prefer.

Stay updated

To stay in the loop with all things AWX, join The Forum, follow the News & Announcements and follow the awx.

Links in this AWX update series

Useful links

Updates

  1. Corrected proposal to be YY.MM.DD or YYYY.MM.DD
  2. Added link to CalVer PR
14 Likes
2

Someone just pointed out that Python are considering moving to CalVer.

1 Like
3

I would suggest to have a look at Conventional Commits. In my experience with my own project and with customers, it becomes the responsibility of each author and reviewer to make sure that each merge request/commit has the necessary information, but then it becomes easy to programmatically extract the necessary information from the git log.

2 Likes
4

Thanks for those links.

In AWX we currently use the PR description to define what type of change this is, the following is from the AWX release process.

PR descriptions contain: Bug, Docs Fix or other nominal change or New or Enhanced Feature

Getting current versions
    awx: 24.3.1
    awx-operator: 2.16.1
PR https://github.com/ansible/awx/pull/15196 votes z
...
PR https://github.com/ansible/awx/pull/15150 votes y
PR https://github.com/ansible/awx/pull/15191 votes z
https://github.com/ansible/awx/compare/24.3.1...devel
awx devel is 13 commit(s) ahead of release 24.3.1

All commits voted, the release type suggestion is y

What I’m not sure is how that gets represented in the CalVer schema.

1 Like
6

Mea culpa, I had not read the official CalVer site so my previous comment was a bit ignorant.

I would suggest YYYY.0M.0D because it makes the date unambiguous, and it’s easier to read and parse a long list of versions when all are the same character length.

5 Likes
7

There’s a lot to digest here, and I have a lot of ideas to share. I’m going to try and stay on topic here while I share my feedback.

While I don’t expect AWX to support “stable” release branches, I’ve always been annoyed that there’s no reference point for what AWX version/commit matches what Ansible Controller version downstream. Even with the understanding that AAP will apply changes to its AWX fork downstream, it would still be nice to know from which commit AWX was forked from. Creating an “aap-x.y” branch in AWX for subsequent AAP releases would be welcome. It might even be useful to your internal team if you needed to back-port fixes applied upstream instead of cherry picking them after the fact.

So there’s a few different things around “breaking changes” and release cycles I want to talk about.

Code deprecation/breaking design changes

These things should fit somehow into your normal release and announcement workflows. Whether or not deprecations should be given a specific support period before fully removing them is another topic.

Releases with unintended breaking changes

Things like the awx-operator deleting the wrong PVC or in the wrong order of operations for things like postgresql upgrades, or changes that otherwise prevent parts or all of AWX from working as expected. Particularly borked releases need a process for purging all related tagged images (not just the main culprit images, but things like operator bundle/catalogs/etc). As well as integrating with the regular announcement workflow to notify users of known issues, removal of these images, and whether or not to expect a hotifx or wait for the next release cadence.

CICD is a must

Having some level of CICD built around addressing breaking changes is definitely a must after you switch to CalVer. With SemVer, if x.y.0 introduces breaking changes, they may get fixed in x.y.1, so as a general rule, it’s safe for consumers (users) to assume that a given z version should be skipped if there’s any subsequent z version. Hotfixes could be released even the same day. I’m not sure how same-day-hotfixes would look in CalVer, but regardless of what that tag looks like, it might make sense to remove images with breaking changes altogether (whether that’s the main image, the operator, or the operator’s catalog/bundle/manfests/etc). If “bad” images are removed, there’s less concern (for us users) about using the ones that remain.

PS about awx-operator-catalog

Additionally, the awx-operator-catalog could be improved with more detailed entries, replaces, and skips/skipsRange that exclude the broken images, but only if that can easily be automated (I do this easily enough for SemVer, but CalVer might be a headache).

The operator catalog release cycle itself is something else to think about. As it is right now if you switch to weekly/nightly releases, every release would throw PR’s at the community catalogs, upstream and downstream. I don’t know how automated or noisy that might be.

YYYY.MM.DD for sure. Including leading 0’s as @eburgueno suggested.
Edit: I’m actually thinking maybe without leading 0’s to keep the CalVer consistent with SemVer enough to allow “version type” tooling to continue working as expected. Maybe even doing weekly release versions with incremental z version for out-of-cycle patches, i.e: YYYY.WW.0 YYYY.WW.1

Make posts to all the usual release announcement mediums notifying the breaking changes, presenting any disaster-recovery processes if applicable (like that PVC deletion one a few months ago), and purging those particularly broken images from Quay.

Maybe at most a weekly release cadence (Mondays? So hotfixes can be applied quickly during the work week if needed), and a separate nightly tag for nightly builds if they are desired.

2 Likes
8

Sorry about the wal-o-text and if it reads a little funny. Been writing that off and on all day.

2 Likes
9

Thanks for the post @gundalow. It’s great to have incremental updates on what’s being discussed and what’s been decided as the project goes through these changes.

I see two topics that seek immediate feedback - here’s mine:

How often to release AWX?
I’d vote for a weekly release cadence. Weekly releases allow features and bugfixes to get into the codebase quickly, and most weekly releases would contain some meaningful change. Daily builds generate a lot of “releases” but many of them would not differ much from each other.

How to name the CalVer releases?
I agree with everyone I’ve seen chime in so far: YYYY-MM-DD, with leading 0s when appropriate - for example, 2024-09-07 for the 7th day of September, 2024.

2 Likes
10

I specifically prefer YYYY.MM.DD with period delimination. This allows for “version type” tooling to still work as expected, as it’s similar in notation and incrementation to SemVer.

2 Likes
11

Another option to consider with the CalVer.

As I mentioned in my post earlier, it would be useful to have some way to designate hotfixes, especially if they are released same day. If you were to do weekly releases, then the x.y version could simply be YYYY.WW. Then the z version could be incremental releases of the same week if there are any out-of-cycle patches.

So, for e.g, this week’s release would be: 2024.27.0, and a subsequent patch released today would be 2024.27.1.

1 Like
12

What are we talking about here:

  • More supervisord processes/services
  • More split-up supervisord processes (eg: task web …) and containers?
  • More stuff like EDA (separate “optional” container that works with AWX)
1 Like
13

I don’t really like CalVer. It’s hard to see what the new version implies. CalVer looks like a marketing instead of a technical versioning to me.

For example:

  1. 1.0.0 → 1.0.1 bugfix release, no breaking changes just one ore more bugs fixed
  2. 1.0.1 → 1.1.0 new features, no breaking changes just new features (and maybe bugifxes)
  3. 1.1.0 → 2.0.0 There are some breaking changes.

CalVer doesn’t tell you anything about this.

4 Likes
14

I kind of agree but it is not happening now?

Reading the release notes of 24.0.0 → What are the breaking changes?
I’m all for semantic versioning but then I would like to have the following chapters in the release notes

  • Breaking
  • New Features
  • Bug fixes

If this is not present in the release notes semantic versioning is kinda irrelevant.

So I am kind of ambivalent to semantic versioning vs cal ver. I like to have better release notes.

In an Ideal world:

One document that list all breaking changes. When you update you just take that document scroll to the version you have and read till the version you are upgrading to.

3 Likes
15

I understand you have to have some uniquifying string to identify a particular release, whether SemVer, CalVer, CodeName, or something else. I probably prefer CalVer if only because it gives a feel for how old different components of a software stack are relative to one another. Comparing projects with different release cadences is apples and oranges, but at least with CalVer you’re comparing all your apples to the same orange. Otherwise – to me – they are just strings.

But the point I want to make is this: Just because you switch to CalVer for naming releases doesn’t mean you have to throw out SemVer and its benefits (see @mariolenz’s post #13 above). A line in the release notes saying “This YYYY.MM.DD release corresponds to SemVer X.Y.Z” would be enough. Or a line in a meta.yml file like

   semver: x.y.z

would suffice, wouldn’t break anything, and still provides the kind of information people use to make SemVer-based decisions.

1 Like
16

Hi
Lots of great discussion here, I’ll make some time to reply.

@shanemcd has just created Initial move to CalVer ansible/awx#15358 PR, please reply here with your feedback in the forum

2 Likes
17

I don’t see any docs changes in the CalVer PR. Can we have at least a mention in the README file to say what sort of CalVer the team has gone with?

18

Hi folks,
We’ve posted a follow up in Refactoring AWX into a Pluggable, Service-Oriented Architecture please take a look

1 Like
20

I wanted to contribute my longer form idea on the CalVer vs SemVer conversation, which I did here. The TL;DR is that I do not think that the majority of users will have a sensibly different experience in either case and boils down to individuals points of view

2 Likes
21

Hi all, a new post in the AWX refactoring series has been posted in the forum, you can check the post here: Upcoming changes to AWX Operator installation methods

1 Like
22

Just FYI that since we are doing so much work in the AWX repo, we wanted to ensure the AWX documentation remains useful, so we are freezing the AWX documentation publishing. This ensures all the heavy work happening in the awx devel branch will not impact docs for now.

This also means we won’t have any documentation updates for some time.

3 Likes