# AWX Backup Fails **URL:** \ **Category:** Get Help\ **Tags:** playbook, awx, kubernetes\ **Created:** [August 8, 2024, 11:57am UTC](https://forum.ansible.com/t/awx-backup-fails/7739 "2024-08-08T11:57:44Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 11:57am UTC](https://forum.ansible.com/t/awx-backup-fails/7739/1 "2024-08-08T11:57:45Z")
My AWX backup is failing. I am running AWX ver 23.7.0 I have applied my manifest and the job runs until it completes with an error. I don’t understand what the output from the ansible job is telling me. Can anyone assist with troubleshooting? Also see post: [Backup Posstgres DB from cluster #1 and restore DB to cluster#2](http://forum.ansible.com/t/backup-posstgres-db-from-cluster-1-and-restore-db-to-cluster-2/7643/5) ```auto --- apiVersion: awx.ansible.com/v1beta1 kind: AWXBackup metadata: name: awxbackup-2024-08-06 namespace: awx spec: deployment_name: awx-operator-controller-manager backup_pvc: awx-backup clean_backup_on_delete: true ``` ```PLAY localhost : ok=13 changed=1 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0 ---------- {"level":"error","ts":"2024-08-06T16:59:09Z","msg":"Reconciler error","controller":"awxbackup-controller","object":{"name":"awxbackup-08-05-2024","namespace":"awx"},"namespace":"awx","name":"awxbackup-08-05-2024","reconcileID":"746a5fdc-2c6d-4fef-9af0-71f4d4ba8722","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"} {"level":"info","ts":"2024-08-06T16:59:10Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-08-05-2024","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"68433906028693526","EventData.TaskArgs":""} ``` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 8, 2024, 1:12pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/2 "2024-08-08T13:12:59Z")
Hi, could you please share your full logs (or enough from the end of the logs) logs from operator? It is necessary to carefully search the complete logs to determine which tasks actually failed. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 1:36pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/3 "2024-08-08T13:36:52Z")
I will get them loaded here ASAP. I have to get approvals for logs from our security team. It may take a day to two. Thanks for your patience. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 8, 2024, 1:41pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/4 "2024-08-08T13:41:11Z")
If uploading the log here is a hassle, you can try to find the failed task on your side. The log should look a lot like the standard playbook log, with a line somewhere like `fatal: [localhost]: FAILED!`. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 8, 2024, 2:08pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/6 "2024-08-08T14:08:33Z")
Basically, what I’m trying to do is identify the failed tasks from the logs, check the implementation of those parts in the GitHub code, and guess the reasons for the failures. The role that works during backup starts from this main.yaml: [awx-operator/roles/backup/tasks/main.yml at 2.11.0 · ansible/awx-operator · GitHub](https://github.com/ansible/awx-operator/blob/2.11.0/roles/backup/tasks/main.yml) So, it’s really no different from troubleshooting when a regular playbook fails. The implementation of the role might be a bit complex, but I think you can do the same thing too! ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 2:16pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/7 "2024-08-08T14:16:12Z")
OK, I think I see the issue. The backup job is trying to pull a container postgres13 and getting a not ready status. ```auto {"message": "backoff pulling image" \ "postgres:13"\} ``` I am running a private repo, air-gapped system and needed to pull my images for AWX from there. How do I modify the job to get the containers pulled from it.? My private repo is something like this: `gsil-docker1.idm.gsil.org:5001/postgres:13` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 8, 2024, 2:28pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/8 "2024-08-08T14:28:56Z")
You can specify `postgres_image: gsil-docker1.idm.gsil.org:5001/postgres:13` in `spec` of your `AWXBackup` object. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 2:30pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/9 "2024-08-08T14:30:14Z")
Ok, great. Let me make that modification, watch the job and report back. Thanks! 😃 ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 2:59pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/10 "2024-08-08T14:59:32Z")
OK, I modified the manifest for my backup and added the postgres parameter as suggested. I am still seeing the image backoff error. I am trying to remember what I need to do in this case. I don’t completely trust my Kubernetes knowledge yet as I am still pretty new to Kubernetes. I tried to helm delete my deployment and all the pods stopped with the exception of the awxbackup-db-management. I can delete that pod but that didn’t help. Which component must I modify or remove? I would have thought applying the updated manifest would have done it… ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 3:16pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/11 "2024-08-08T15:16:42Z")
OK, I think I figured that part out. I had to delete the deployment followed by deleting the pod. Then I see that an automation job pod runs for a bit and then goes away. I presume that is my backup trying to run? ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 8, 2024, 3:28pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/12 "2024-08-08T15:28:40Z")
Ah my bad, correct `spec` is: ```yaml spec: ... postgres_image: gsil-docker1.idm.gsil.org:5001/postgres postgres_image_version: 13 ``` Ref.: [awx-operator/config/crd/bases/awx.ansible.com\_awxbackups.yaml at 2.11.0 · ansible/awx-operator · GitHub](https://github.com/ansible/awx-operator/blob/2.11.0/config/crd/bases/awx.ansible.com_awxbackups.yaml#L87-L92) ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 3:29pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/13 "2024-08-08T15:29:23Z")
no worries, I’ll make that change and apply it. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 3:42pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/14 "2024-08-08T15:42:31Z")
OK, I initially got an error when applying that and figured out it need to be single quoted. Otherwise kubernetes barked at me for an invalid spec file. ```auto spec: ... postgres_image: gsil-docker1.idm.gsil.org:5001/postgres postgres_image_version: '13' ``` Anyway, I don’t see any new files or folders at my defined pvc location. ` /var/lib/postgres/backup/` Hmmm… 🤔 any thoughts? ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 8, 2024, 3:48pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/15 "2024-08-08T15:48:59Z")
First of all check the operator’s log to see if PLAY RECAP succeeds with a failed of 0. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 8, 2024, 3:52pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/16 "2024-08-08T15:52:55Z")
OK, part of my issue is that fact that I misread the spec. I forgot to drop the 13 at the end of postgres\_image. I did that now. How do I check the operator log? Also, should I be seeing the operator listed as a deployment within my cluster now after having deleted it earlier in this post? ```auto [root@gsil-kube01 ~]# kubectl get all -n awx NAME READY STATUS RESTARTS AGE pod/awx-postgres-13-0 1/1 Running 0 60m pod/awx-task-667c9bbf9d-5sbzw 4/4 Running 0 60m pod/awx-task-667c9bbf9d-tfsng 4/4 Running 0 60m pod/awx-web-c7bcbd976-6q4v2 3/3 Running 0 59m pod/awx-web-c7bcbd976-gzzn2 3/3 Running 0 59m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-operator-controller-manager-metrics-service ClusterIP x.x.216.205 8443/TCP 60m service/awx-postgres-13 ClusterIP None 5432/TCP 60m service/awx-service NodePort x.x.18.162 80:30080/TCP 60m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx-task 2/2 2 2 60m deployment.apps/awx-web 2/2 2 2 59m NAME DESIRED CURRENT READY AGE replicaset.apps/awx-task-667c9bbf9d 2 2 2 60m replicaset.apps/awx-web-c7bcbd976 2 2 2 59m NAME READY AGE statefulset.apps/awx-postgres-13 1/1 60m ``` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [August 9, 2024, 1:42pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/17 "2024-08-09T13:42:20Z")
Since the backup role is executed by the operator, nothing will happen if the operator is not running. Here’s what you need to do: 1. Add the following spec to AWXBackup. ```yaml spec: ... postgres_image: gsil-docker1.idm.gsil.org:5001/postgres postgres_image_version: '13' ``` 2. Start the operator. 3. Observe the logs of the operator’s pod and make sure that the playbook succeeds with `failed=0`. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 9, 2024, 6:25pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/18 "2024-08-09T18:25:49Z")
> [@kurokobo](#): > > Add the following spec to AWXBackup. 1. Affirmative, that is what I have. 2. Yes I stopped and started AWX and the operator is running now. 3. I can see the logs of the operator pod and the job is failing. I have noted that for whatever reason I have two pods that are running as the backup job. That may be part of my issue. I was not expecting to see two pods running. One is for an 8/5 date, the other is for an 8/8 date. I suspect I have some old data that is fouling things up but I am unsure where to look. When I applied my backup manifest, where does that go? Here is what I am seeing right now: ```auto [root@gsil-kube01 ~]# kubectl get all -n awx NAME READY STATUS RESTARTS AGE pod/awx-operator-controller-manager-6ffc56f846-qrg8j 2/2 Running 0 70m pod/awx-postgres-13-0 1/1 Running 0 69m pod/awx-task-667c9bbf9d-k7k52 4/4 Running 0 69m pod/awx-task-667c9bbf9d-t5gfw 4/4 Running 0 69m pod/awx-web-c7bcbd976-bgn8j 3/3 Running 0 69m pod/awx-web-c7bcbd976-tqj82 3/3 Running 0 69m pod/awxbackup-08-05-2024-db-management 0/1 ImagePullBackOff 0 30s pod/awxbackup-08-08-2024-db-management 1/1 Terminating 0 9s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-operator-controller-manager-metrics-service ClusterIP 10.96.48.105 8443/TCP 70m service/awx-postgres-13 ClusterIP None 5432/TCP 69m service/awx-service NodePort 10.110.109.166 80:30080/TCP 69m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx-operator-controller-manager 1/1 1 1 70m deployment.apps/awx-task 2/2 2 2 69m deployment.apps/awx-web 2/2 2 2 69m NAME DESIRED CURRENT READY AGE replicaset.apps/awx-operator-controller-manager-6ffc56f846 1 1 1 70m replicaset.apps/awx-task-667c9bbf9d 2 2 2 69m replicaset.apps/awx-web-c7bcbd976 2 2 2 69m NAME READY AGE statefulset.apps/awx-postgres-13 1/1 69m [root@gsil-kube01 ~]# kubectl get secret -n awx NAME TYPE DATA AGE awx-admin-password Opaque 1 91d awx-app-credentials Opaque 3 72m awx-broadcast-websocket Opaque 1 91d awx-custom-certs Opaque 2 80d awx-ldap-password Opaque 1 80d awx-postgres-configuration Opaque 6 91d awx-receptor-ca kubernetes.io/tls 2 91d awx-receptor-work-signing Opaque 2 91d awx-secret-key Opaque 1 91d redhat-operators-pull-secret Opaque 1 72m sh.helm.release.v1.gsil-awx.v1 helm.sh/release.v1 1 73m ``` ```auto --- apiVersion: awx.ansible.com/v1beta1 kind: AWXBackup metadata: name: awxbackup-2024-08-06 namespace: awx spec: deployment_name: awx-operator-controller-manager backup_pvc: awx-backup clean_backup_on_delete: true postgres_image: gsil-docker1.idm.gsil.org:5001/postgres postgres_image_version: '13' ``` I looked at the top of my post and realized I forgot to mention that I am deploying AWX via helm chart and so much of the spec is located in the yaml file that I am deploying with the helm chart. Will that make a difference? Do I need to integrate the backup with the yaml file I am using at Helm deployment time? Right now, I am running this to start up AWX: `helm install -n awx gsil/gsil-awx /root/awx-operator/ -f awxvalues.yml` My backup-awx.yml exists as a separate file from awxvalues.yml. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [August 13, 2024, 1:14pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/19 "2024-08-13T13:14:29Z")
@kurokobo Can you comment further? See my previous reply. Thanks! ---
**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\ **Post date:** [August 13, 2024, 7:05pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/20 "2024-08-13T19:05:45Z")
@jeremytourville You likely have 2 backup pods in your previous post because you scheduled a backup job while the operator was offline and before fixing the image used in the spec. So the first pod is stuck with `ImagePullBackoff`, while the second one was `Terminating` (and hopefully succeeded?). The helm chart is probably not doing anything to cleanup the old backup job. You should be able to see your backups with `kubectl get -n awx awxbackups.awx.ansible.com`. Note that these are the CR’s for the operator to trigger backups, not the data itself. However, you can run `kubectl describe` to find out the status, which should also show where the backup lives. Sample output: ```yaml Status: Backup Claim: awx-snd-backup-claim Backup Directory: /backups/tower-openshift-backup-2024-04-17-165512 Conditions: Last Transition Time: 2024-04-17T16:55:38Z Reason: Status: False Type: Failure Last Transition Time: 2024-04-17T16:54:52Z Reason: Successful Status: True Type: Running Last Transition Time: 2024-08-04T16:41:45Z Reason: Successful Status: True Type: Successful ``` The real data is inside of the `Backup Claim` at the `Backup Directory`. As for your stuck pod, I would simply delete the related `awxbackups.awx.ansible.com` and `deployment` resources from kubernetes. ---
**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\ **Post date:** [September 18, 2024, 12:15pm UTC](https://forum.ansible.com/t/awx-backup-fails/7739/21 "2024-09-18T12:15:43Z")
I was able to clean up the back job. I still don’t see any data being created in the backup folder. Instead I exec’d into the Postgres pod and manually created a backup of the AWX DB. In my new cluster I created an awx restore job. I was able to transfer the .sql file to the new cluster so it is ready to be restored/ingested by Postgres. I am having issues with the correct folder structure being created. I created a storage.yml for my system. I don’t see the folder listed when I browse to the path via CLI. The PV and PVC say that everything is OK, no errors when running kubectl describe. The status for PV and PVC say bound. Is there something I need to add to the spec file? Here is my spec file that I am deploying with helm at install time: ```auto AWX: # enable use of awx-deploy template enabled: true name: awx spec: replicas: 2 service_type: NodePort nodeport_port: 30080 admin_user: admin hostname: awx.idm.gsil.org image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx image_version: 23.7.0 init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee init_container_image_version: latest ee_images: - name: AWX EE image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:23.7.0 ee_extra_env: | - name: RECEPTOR_KUBE_SUPPORT_RECONNECT value: enabled postgres_image: gsil-docker1.idm.gsil.org:5001/postgres postgres_image_version: "13" postgres_selector: | nodefor: psql control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:23.7.0 redis_image: gsil-docker1.idm.gsil.org:5001/redis redis_image_version: "7" bundle_cacert_secret: awx-custom-certs ldap_cacert_secret: awx-custom-certs ldap_password_secret: awx-ldap-password extra_settings: - setting: AUTH_LDAP_SERVER_URI ... more stuff here for LDAP settings... ``` This is my restore file: ```auto [root@gsil-kube04 ~]# cat restore-awx.yml --- apiVersion: awx.ansible.com/v1beta1 kind: AWXRestore metadata: name: restore1 namespace: awx spec: deployment_name: awx-operator-controller-manager restore_pvc: awx-backup backup_name: clean_backup_on_delete: true postgres_image: gsil-docker1.idm.gsil.org:5001/postgres postgres_image_version: '13' ``` Here is my storage configuration: ```auto [root@gsil-kube04 ~]# cat storage.yaml --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageClass.kubernetes.io/is-default-class: "true" name: local-storage namespace: awx provisioner: kubernetes.io/no-provisioner volumeBindingMode: Immediate #volumeBindingMode: WaitForFirstConsumer --- apiVersion: v1 kind: PersistentVolume metadata: name: postgres-pv namespace: awx spec: capacity: storage: 2Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /var/lib/postgresql/data nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - gsil-kube04.idm.gsil.org - gsil-kube05.idm.gsil.org - gsil-kube06.idm.gsil.org --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: postgres-13-awx-postgres-13-0 namespace: awx spec: storageClassName: local-storage accessModes: - ReadWriteMany resources: requests: storage: 2Gi --- apiVersion: v1 kind: PersistentVolume metadata: name: awx-backup namespace: awx spec: capacity: storage: 2Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete storageClassName: local-storage local: path: /var/lib/postgresql/backup nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - gsil-kube04.idm.gsil.org - gsil-kube05.idm.gsil.org - gsil-kube06.idm.gsil.org --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: awx-backup namespace: awx spec: storageClassName: local-storage accessModes: - ReadWriteMany resources: requests: storage: 2Gi ``` [Next page](https://forum.ansible.com/t/awx-backup-fails/7739.md?page=2)