In the README.md for the AWX operator in GitHub, it contains the following verbiage:
Cluster-scope to Namespace-scope considerations
Starting with awx-operator 0.14.0, AWX can only be deployed in the namespace that the operator exists in. This is called a namespace-scoped operator. If you are upgrading from an earlier version, you will want to delete your existing awx-operator service account, role and role binding.
This suggests that it’s either not possible to deploy multiple AWX instances in the same K8s cluster in different namespaces, or that one would need to use a 1-to-1 ratio in operators-to-instances (i.e. one operator in every namespace which will house an instance).
Under the assumption that the latter would be the case, I attempted just such a deployment: two namespaces, each with their own operator (using Helm). However, this failed as the second operator installation would complain as follows:
Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole “awx-operator-metrics-reader” in namespace “” exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key “meta.helm.sh/release-name” must equal “awx-operator-two”: current value is “awx-operator”; annotation validation error: key “meta.helm.sh/release-namespace” must equal “awx-two”: current value is “awx”
(Yes, I know “awx” and “awx-two” aren’t the best names … this was a POC so bear with me :D).
So the question is: if I wished to deploy two AWX instances within the same K8s cluster, and taking into account the verbiage re: that the Operator can no longer deploy instances outside its namespace, how could I go about it?
To clarify - I tried running both instances within the same namespace as the operator, using separate volumes and DB instances and whatnot, but the earlier instance would cease to work and start getting “Error creating pod: Unauthorized” errors, which I interpret to be related to role binding shenanigans not playing nicely for two instances in the same namespace (hence the hope of using namespaces to split things apart and avoid said conflict).
Through interaction in the chat channel (through https://web.libera.chat/ , #ansible-awx), and thanks to TheRealHaoLiu[m], I’ve learned the following:
The helm chart for the Operator doesn’t support deploying multiple operators in the same cluster, even if they’re going into different namespaces, as it tries to clobber non-namespaced resources and this causes an unrecoverable deployment problem
Deploying multiple operators to the same cluster, each in their own namespace, is perfectly possible using Kustomize
Once this is achieved, deploying multiple instances of AWX within the same cluster is 100% possible using the CRD definitions like “normal”
So the problem was Helm, not AWX per-se. Instructions for deployment using Kustomize are available in the awx-operator GitHub page.
I was curious about this. I’m sorry I did not speak up sooner. I do have multiple AWX instances running on a single OpenShift cluster, and it is running with a 1-1 of operators to AWX instances, each in its own namespace. The ones that are working on our platform are 20.0.1 and installed by my predecessor. I’m kind of hoping to have a fully viable 21.10.2+ install before I get to sounding cocky.
The idea of having one operator for multiple AWXs seems appealing, but I’m not sure of the value. Operators are inexpensive, so I’m OK with there being many.
