Upcoming changes to AWX Operator installation methods

Hi everyone,

In a couple of recent posts, we’ve talked about Streamlining AWX Releases as well as Refactoring AWX into a Pluggable Services-Oriented Architecture. Now we’d like to announce plans from Red Hat engineering to make changes to how AWX Operator gets released.

First, let’s reiterate something that we said in a previous post because the changes I’m going to outline fit into that message:

“While we want to make changes that enable more collaboration with our community, at the same time we also must reduce the overhead we as the Ansible Engineering team within Red Hat spend supporting AWX so that we can more effectively serve both our upstream community and paying customers.”

Let’s talk about AWX Operator

AWX Operator is a convenient way to deploy AWX in Kubernetes environments. Work on the operator started back in 2019 when Red Hat open-sourced the codebase for Ansible Tower. The operator was originally created by the community, with a vision to grow it into an installation method with capabilities to manage the full lifecycle of an AWX deployment.

In 2022, Red Hat made the AWX Operator available from OperatorHub.io. The intention was to take away some of the pain with installing AWX on Kubernetes. However, this has lead to some unintended consequences:

  • Red Hat customer confusion, after seeing the AWX Operator entry, about whether or not Red Hat recommended its use for production environments, and if Red Hat provided support for the AWX Operator. (Red Hat has never provided support for the AWX Operator. Red Hat Ansible Automation Platform is the supported product.)
  • Increased engineering burden on Red Hat to maintain the release process.

Also in 2022, the Helm chart was added to the AWX Operator code repository from a community contribution. For folks who are not familiar with Helm charts, they are a packaging format for installing AWX on Kubernetes. Similarly to the situation with OperatorHub.io, we’ve observed some unintended consequences of having the Helm chart in the AWX Operator repository:

  • Expertise for maintaining the Helm chart is outside the Red Hat engineering team.
  • Community owns the Helm chart, but Red Hat engineering plans to refactor AWX could lead to conflicts or breakage. For instance, the Helm chart now depends on Red Hat release cycles and fast iteration from the community is gated by Red Hat engineering.

Removing AWX Operator from OperatorHub.io

With the above points in mind, Red Hat is taking action to remove AWX Operator from operatorhub.io on August 09, 2024.

You can sum up the motivation for removing the AWX Operator from operatorhub.io quite simply. Red Hat has put an obstacle in its own way by releasing the AWX operator on OperatorHub. This change is about Red Hat getting out of its own way.

Here is what is going to change:

  • AWX Operator will no longer be listed on the OperatorHub. If you depend on installing awx-operator via OperatorHub, please plan accordingly for this upcoming change.
  • The OLM subscription resource for this operator will be unhealthy.

Here is what is NOT going to change:

  • Existing installations of the AWX Operator should continue to work.

  • The awx-operator image will remain available.

  • The Ansible community can still continue to install AWX Operator following the documentation.

  • No changes to the license.

  • Contributions are always welcome.

Relocating the Helm chart code

Red Hat is moving the Helm chart code to a new repository in the ansible-community org in GitHub. With this move the code will reside somewhere it should always have been located, given the content and the fact that the community maintains it. This move will take place on or before August 09, 2024.

In a similar, but different, way to the removal of the operator from the OperatorHub, you can sum up the motivation for returning the Helm chart to the community. Red Hat wants to avoid creating obstacles for community development, especially once the work to refactor the AWX codebase begins in earnest. This change is about Red Hat getting out of the way of the community. As a consequence of this move, we expect there will be more enhancements and a faster release cadence.

Last year, Red Hat moved the source for the Ansible core and community package documentation to a new repository to provide greater ownership to content for the Steering Committee and to remove constraints on the documentation toolchain that came from the Ansible core CI/CD pipelines. It happened to be the case that the Red Hat engineering team and the Ansible community were pulling in the same direction, but just getting into each other’s way at times. After moving the documentation to a new repository, we saw that the community not only got more ownership of content, but the tooling around the docs repo was dramatically improved. Those tooling improvements, in turn, greatly reduced manual overhead and created a path to innovation and enhanced community contributions. We’re hoping to see the same kind of results for the Helm chart code.

Here is what is going to change:

To make this move a success, we need the help of the community. We are looking for community maintainers for the Helm chart. Respond to this post or open an issue in the new repository if you’re interested!

Updating the release process

As mentioned in the previous post, AWX releases are paused for the time being. Along with the changes I’ve outlined above, we also need to make some changes to the release process for when we resume AWX Operator releases. The changes we plan to make include:

Please note that this list of changes is not exhaustive and there might be some additional smaller changes.

Call to action

If you depend on OperatorHub as an installation method for the AWX Operator, plan accordingly for the removal taking place on August 09, 2024.

If you are a member of the community who is interested in helping maintain the Helm chart for the AWX Operator, we encourage you to get involved! You can respond to this post to express your interest or open an issue in the new repository to let us know.

Join the discussion

We value your input on these changes. You can share your thoughts asynchronously by replying to this forum post. I can also add a link for providing anonymous feedback if folks think that would be useful or preferable.

You can also come along to the next AWX community meeting, scheduled for August 13, and speak with us directly.

Stay updated

To stay in the loop with all things AWX, join the Ansible Forum, follow the News & Announcements and follow the awx tag.

Links in this AWX update series

Useful links

8 Likes

Now that’s a bit of a shock, now I have to review my deployment process…

What about the awx-operator-{bundle,catalog} images pushed to Quay.io? Will the following portion of the feature workflow stay? feature.yml#L42-L56

While I do not use the operatorhub.io catalog source or even the slightly downstream redhat-openshift-ecosystem/community-operators-prod catalog source, I do use the operator bundle images for building a private/custom catalog source.

My current deployment process involves building my own OLM catalog source by:

  • intersecting the tags of the available operator images and bundle images (to account for some missing tags in one source or the other)
  • render the intersected bundles into a private catalog using a combination of opm and a jinja2 template in ansible
  • build the rendered catalog image and push to private registry
  • install the private catalog source on k8s cluster
  • subscribe to awx-operator from the private catalog

I was already anticipating having to make possible changes due to the switch from SemVer to CalVer, but if you stop building the bundles… I guess I’ll have to figure out how to continue building those myself. Using the kustomize or helm install methods are far from preferable in my case.

Hi folks - the PR to move helm chart is ready for review at Remove Helm chart code by oraNod · Pull Request #1938 · ansible/awx-operator · GitHub

And you’ll need to go to awx-operator Helm charts | awx-operator-helm to install helm chart going foward.

Hi Denny,

Thanks for the question and I hope you don’t mind a bit of delay in the response. To be transparent, I’ve intentionally not replied until now to give others in the community some time to chime in. However, it’s not good to let questions go unanswered.

As stated in the post, the AWX operator image on Quay will remain available. To the best of my knowledge there are no plans to remove that or any of the other images, even if the OLM related bits are no longer published to OperatorHub.

I also asked around and got some more specific details. The kustomize and make deploy install methods will remain available. It’s true there are limitations to hooking those methods into infra-as-code style deployments, but this is a trade-off Red Hat is making to enable the development team to focus on improving the core parts of AWX.

1 Like

So if we already deploy via Helm chart, we merely need to update the helm source in our deployment?
From https://ansible.github.io/awx-operator/ to GitHub - ansible-community/awx-operator-helm ?

That’s correct @disi

2 Likes

There’s a new post in the AWX Modernization series at AWX UI and credential types transitioning to the new pluggable architecture.

1 Like

I’ve just posted the next forum post in this series, Transitioning authentication and authorization (RBAC) to the new AWX architecture

I made the change today and it worked like a charm. new URL: awx-operator Helm charts | awx-operator-helm

4 Likes

Great to hear, thanks for sharing.

1 Like