Is an AWXRestore with PVC only possible? (without the AWXBackup resource)

[fuggla]

Hi,

• We backup AWX using AWXBackup.
• This results in a PVC and an AWXBackup.
• All PVCs are backed up to another site. The AWXBackup resource is not, and will be lost if complete dataloss/disaster occurs.

Can I run a AWXRestore without the AWXBackup, and only the PVC?

Hope this makes sense! Please tell me if I further details are needed.

$ ka get awxbackup
NAME                         AGE
awx-backup-2023-10-19-1458   18h
$ ka get pvc
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-13-awx-postgres-13-0   Bound    pvc-9b686407-a5d8-4c96-b0e8-4451546fc27a   8Gi        RWO            longhorn       8d
pvc-awx-backup                  Bound    pvc-e677c5ed-b799-4588-863d-aa0b378bf660   10Gi       RWO            longhorn       18h

awx-backup.bash

manifest="
apiVersion: awx.ansible.com/v1beta1
kind: AWXBackup
metadata:
  name: awx-backup-$(date +%Y-%m-%d-%H%M)
  namespace: awx
spec:
  deployment_name: awx
  backup_pvc: pvc-awx-backup
  backup_storage_class: longhorn
  backup_pvc_namespace: awx
  clean_backup_on_delete: true
"
kubectl apply -f - <<< $manifest

awx-restore.bash (this wouldn’t work[?] since the AWXBackup doesn’t exist)

manifest="
apiVersion: awx.ansible.com/v1beta1
kind: AWXRestore
metadata:
  name: restore-$1
  namespace: awx
spec:
  deployment_name: awx
  backup_name: $1
  backup_pvc_namespace: awx
  backup_pvc: pvc-awx-backup
"

kubectl apply -f - <<< $manifest

[kurokobo]

Hi,

You can achieve restoring without AWXBackup by specifying backup_pvc and backup_dir. See: https://github.com/ansible/awx-operator/tree/devel/roles/restore

As documented, the path for backup_dir can be gathered by:

$ kubectl get awxbackup awxbackup1 -o jsonpath="{.items[0].status.backupDirectory}"
/backups/tower-openshift-backup-2021-04-02-03:25:08

Or if you can dig into the PVC, there should be backup directory that named tower-openshift-backup-* and you can use this name for backup_dir as /backups/<backup directory>.

[fuggla]

Cool, that’s easier than I thought it would be. Thanks again for helping out, very much appreciated.

For anyone else dropping by here, there’s another somewhat similar discussion on Github: Documentation: AWX backup secheduled job · Issue #1605 · ansible/awx-operator · GitHub

[fuggla]

@kurokobo actually, I can’t seem to get this to work. I create an AWXRestore, but it never processes beyond this step:

EDIT: backupS not backup!

  backup_dir: /backups/tower-openshift-backup-2023-10-19-135407

TASK [restore : Check provided PVC exists] *************************************
task path: /opt/ansible/roles/restore/tasks/init.yml:40

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-10-23T12:14:56Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"pvc-awx-backup","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXRestore","event_type":"playbook_on_task_start","job":"7174255286884531933","EventData.Name":"restore : Check provided PVC exists"}
{"level":"info","ts":"2023-10-23T12:14:57Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/awx/persistentvolumeclaims/pvc-awx-backup","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"persistentvolumeclaims","Subresource":"","Name":"pvc-awx-backup","Parts":["persistentvolumeclaims","pvc-awx-backup"]}}
{"level":"info","ts":"2023-10-23T12:14:57Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"pvc-awx-backup","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXRestore","event_type":"playbook_on_task_start","job":"7174255286884531933","EventData.Name":"restore : Emit ocp event with error"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [restore : Emit ocp event with error] *************************************
task path: /opt/ansible/roles/restore/tasks/error_handling.yml:7

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-10-23T12:14:57Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"pvc-awx-backup","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXRestore","event_type":"playbook_on_task_start","job":"7174255286884531933","EventData.Name":"restore : Fail early if pvc is defined but does not exist"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [restore : Fail early if pvc is defined but does not exist] ***************
task path: /opt/ansible/roles/restore/tasks/init.yml:58

I mounted the PVC in a tmeporary pod, to verify the dir name.

ka exec persistent-volume -- ls /data
lost+found
tower-openshift-backup-2023-10-19-135407

Restore manifest:

kind: AWXRestore
metadata:
  name: pvc-awx-backup
  namespace: awx
spec:
  deployment_name: awx
  backup_dir: /backup/tower-openshift-backup-2023-10-19-135407
  backup_pvc_namespace: awx
  backup_pvc: pvc-awx-backup

Any idea what I’m doing wrong here?

[kurokobo]

Hi, could you provide full logs of Operator?

[kurokobo]

@fuggla

EDIT: backupS not backup!

Ah, is this root cause for you? Ignore my comment if it’s already solved.