# Multi-instance deployment in cluster using Kustomize **URL:** \ **Category:** AWX Project\ **Tags:** kubernetes, awx\ **Created:** [April 20, 2023, 8:28am UTC](https://forum.ansible.com/t/multi-instance-deployment-in-cluster-using-kustomize/37909 "2023-04-20T08:28:33Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![iamroddo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/iamroddo/32/321_2.png) [@iamroddo](https://forum.ansible.com/u/iamroddo)\ **Post date:** [April 20, 2023, 8:28am UTC](https://forum.ansible.com/t/multi-instance-deployment-in-cluster-using-kustomize/37909/1 "2023-04-20T08:28:33Z")
I have been experimenting with deploying multiple AWX instances in a cluster, with each instance in its own namespace. I have been following the process using Kustomize with AWX Operator v1.4.0 with AWX v21.14.0. I have seen that it is possible to install 2 different instances of AWX alongside an instance of AWX Operator in 2 different namespaces. However when I uninstall one of the AWX + AWX Operator instances using Kustomize with "kustomization.yaml" as below, using the command "kubectl delete -k ." that the other AWX is also uninstalled. I suppose this is because the Operator installation for each deploys non-namespace-scoped K8s objects and the uninstallation of one of the Operator instances removes these "shared" objects, which triggers the removal of the other AWX instance. apiVersion: kustomize.config.k8s.io/v1beta1 images: - name: quay.io/ansible/awx-operator newTag: 1.4.0 kind: Kustomization namespace: awx-dev25491 resources: - github.com/ansible/awx-operator/config/default?ref=1.4.0 - awx\_manifest.yml I suppose that instead of installing the AWX instance with Kustomize I could install it with the command "kubectl apply -f awx\_manifest.yml" and uninstall only that AWX instance with "kubectl delete -f awx\_manifest.yml" and then delete the namespace to get rid of the namespace-scoped Operator objects. However that doesn't cleanly get rid of all Operator objects and that even after uninstalling all AWX instances and their namespaces, the non-namespace-scoped Operator objects would need to be separately deleted. Is there a clean way of deleting an Operator instance without clobbering all AWX instances on that cluster? ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [July 7, 2023, 5:45pm UTC](https://forum.ansible.com/t/multi-instance-deployment-in-cluster-using-kustomize/37909/2 "2023-07-07T17:45:33Z")
Part of the problem here is that the AWX custom resource definition (CRD) is a global resource. So when you run “kubectl delete -k .”, when the operator is deleted it will delete the CRD in the process, thus deleting all of the child custom resources (CR) in the process. _[customresourcedefinition.apiextensions.k8s.io](http://customresourcedefinition.apiextensions.k8s.io) “[awxs.awx.ansible.com](http://awxs.awx.ansible.com)” deleted_ So the trick will be finding a way to exclude deleting the 3 CRD’s upon teardown. You’ll have the same problem with `make deploy` and `make undeploy`. I’ll think on a potential solution here… The OLM install leaves the CRD behind when the operator is deleted for this reason (OperatorHub install). Thanks, AWX Team ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [July 7, 2023, 9:52pm UTC](https://forum.ansible.com/t/multi-instance-deployment-in-cluster-using-kustomize/37909/3 "2023-07-07T21:52:27Z")
You could do something like this to exclude things from the strategic merge of resources kustomize does before applying/deleting. Example `kustomization.yaml` apiVersion: [kustomize.config.k8s.io/v1beta1](http://kustomize.config.k8s.io/v1beta1) kind: Kustomization resources: # Find the latest tag here: [Releases · ansible/awx-operator · GitHub](https://github.com/ansible/awx-operator/releases) - [github.com/ansible/awx-operator/config/default?ref=1.4.0](http://github.com/ansible/awx-operator/config/default?ref=1.4.0) patches: - path: delete-cluster-resources/clusterrolebinding.yaml - path: delete-cluster-resources/clusterrole.yaml # - path: delete-cluster-resources/awx.yaml # - path: delete-cluster-resources/awxbackup.yaml # - path: delete-cluster-resources/awxrestore.yaml # Set the image tags to match the git version from above images: - name: [Quay](http://quay.io/ansible/awx-operator) newTag: 1.4.0 # Specify a custom namespace in which to install AWX namespace: awx Example patch file in `delete-cluster-resources/clusterrolebinding.yaml`: $patch: delete apiVersion: [rbac.authorization.k8s.io/v1](http://rbac.authorization.k8s.io/v1) kind: ClusterRoleBinding metadata: name: awx-operator-proxy-rolebinding Example patch file in `delete-cluster-resources/clusterrole.yaml`: $patch: delete apiVersion: [rbac.authorization.k8s.io/v1](http://rbac.authorization.k8s.io/v1) kind: ClusterRole metadata: name: awx-operator-proxy-role However, I wasn’t able to use this approach to exclude deletion of the CRD resources because it is not guaranteed that they exist (because of cleanup). So I would recommend making a cleanup script for the time being. Something like: #!/usr/bin/env bash kubectl delete serviceaccount/awx-operator-controller-manager -n namespace kubectl delete [role.rbac.authorization.k8s.io/awx-operator-awx-manager-role](http://role.rbac.authorization.k8s.io/awx-operator-awx-manager-role) -n namespace kubectl delete [role.rbac.authorization.k8s.io/awx-operator-leader-election-role](http://role.rbac.authorization.k8s.io/awx-operator-leader-election-role) -n namespace kubectl delete [rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding](http://rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding) -n namespace kubectl delete [rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding](http://rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding) -n namespace kubectl delete configmap/awx-operator-awx-manager-config -n namespace kubectl delete service/awx-operator-controller-manager-metrics-service -n namespace kubectl delete deployment.apps/awx-operator-controller-manager -n namespace Hopefully this helps, AWX Team