Multi-instance deployment in cluster using Kustomize

Archives AWX Project
,
1

I have been experimenting with deploying multiple AWX instances in a
cluster, with each instance in its own namespace. I have been
following the process using Kustomize with AWX Operator v1.4.0 with
AWX v21.14.0.

I have seen that it is possible to install 2 different instances of
AWX alongside an instance of AWX Operator in 2 different namespaces.
However when I uninstall one of the AWX + AWX Operator instances using
Kustomize with "kustomization.yaml" as below, using the command
"kubectl delete -k ." that the other AWX is also uninstalled.

I suppose this is because the Operator installation for each deploys
non-namespace-scoped K8s objects and the uninstallation of one of the
Operator instances removes these "shared" objects, which triggers the
removal of the other AWX instance.

apiVersion: kustomize.config.k8s.io/v1beta1
images:
- name: quay.io/ansible/awx-operator
newTag: 1.4.0
kind: Kustomization
namespace: awx-dev25491
resources:
- github.com/ansible/awx-operator/config/default?ref=1.4.0
- awx_manifest.yml

I suppose that instead of installing the AWX instance with Kustomize I
could install it with the command "kubectl apply -f awx_manifest.yml"
and uninstall only that AWX instance with "kubectl delete -f
awx_manifest.yml" and then delete the namespace to get rid of the
namespace-scoped Operator objects. However that doesn't cleanly get
rid of all Operator objects and that even after uninstalling all AWX
instances and their namespaces, the non-namespace-scoped Operator
objects would need to be separately deleted.

Is there a clean way of deleting an Operator instance without
clobbering all AWX instances on that cluster?

2

Part of the problem here is that the AWX custom resource definition (CRD) is a global resource. So when you run “kubectl delete -k .”, when the operator is deleted it will delete the CRD in the process, thus deleting all of the child custom resources (CR) in the process.

customresourcedefinition.apiextensions.k8s.io “awxs.awx.ansible.com” deleted

So the trick will be finding a way to exclude deleting the 3 CRD’s upon teardown. You’ll have the same problem with make deploy and make undeploy. I’ll think on a potential solution here… The OLM install leaves the CRD behind when the operator is deleted for this reason (OperatorHub install).

Thanks,
AWX Team

3

You could do something like this to exclude things from the strategic merge of resources kustomize does before applying/deleting.

Example kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:

Find the latest tag here: https://github.com/ansible/awx-operator/releases

patches:

  • path: delete-cluster-resources/clusterrolebinding.yaml
  • path: delete-cluster-resources/clusterrole.yaml

- path: delete-cluster-resources/awx.yaml

- path: delete-cluster-resources/awxbackup.yaml

- path: delete-cluster-resources/awxrestore.yaml

Set the image tags to match the git version from above

images:

Specify a custom namespace in which to install AWX

namespace: awx

Example patch file in delete-cluster-resources/clusterrolebinding.yaml:

$patch: delete
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: awx-operator-proxy-rolebinding

Example patch file in delete-cluster-resources/clusterrole.yaml:

$patch: delete
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: awx-operator-proxy-role

However, I wasn’t able to use this approach to exclude deletion of the CRD resources because it is not guaranteed that they exist (because of cleanup). So I would recommend making a cleanup script for the time being.

Something like:

#!/usr/bin/env bash

kubectl delete serviceaccount/awx-operator-controller-manager -n namespace
kubectl delete role.rbac.authorization.k8s.io/awx-operator-awx-manager-role -n namespace
kubectl delete role.rbac.authorization.k8s.io/awx-operator-leader-election-role -n namespace
kubectl delete rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding -n namespace
kubectl delete rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding -n namespace
kubectl delete configmap/awx-operator-awx-manager-config -n namespace
kubectl delete service/awx-operator-controller-manager-metrics-service -n namespace
kubectl delete deployment.apps/awx-operator-controller-manager -n namespace

Hopefully this helps,
AWX Team