AWX restore fails (postgres part succeeds however)

Archives AWX Project
1

Hello all

Note: My production environment currently used awx-operator 0.14. and I am using that too here.This because I want to test upgrading etc.

Whackup using awx-on-k3s it fails.

After it has completed (unsuccesfully) only the awx-postgres-0 and awx-operator-controler… containers are Running.

I the logging I see :
{“level”:“error”,“ts”:1659002201.076261,“logger”:“logging_event_handler”,“msg”:“”,“name”:“awx”,“namespace”:“awx”,“gvk”:“awx.ansible.com/v1beta1,Kind=AWX",“event_type”:“runner_on_failed”,“job”:"5920064687863440564”,“EventData.Task”:“ApplyResources”,“EventData.TaskArgs”:“”,“EventData.FailedTaskPath”:“/opt/ansible/roles/installer/tasks/resources_configuration.yml:20”,“error”:“[playbook task failed]”}

With this job number is looked in the operator container and found in directory /tmp/ansible-operator/runner/awx.ansible.com/v1beta1/AWX/awx/awx/artifacts/5920064687863440564/job_events file : 10-aa101db8-b0d9-4a20-ae26-c9a7e044a654.json which seems to reference this job…
When I format the file and browse through it I see:

',‘reason’:‘Invalid’,‘details’:{‘name’:
‘awx-projects-claim’,‘kind’:‘PersistentVolumeClaim’,‘causes’:[{‘reason’:‘FieldValueForbidden’,‘message’:'Forbidden: spec is immutable after creation except resources.requests for bound claims
core.PersistentVolumeClaimSpec{
t… // 2 identical fields

Definitions of the pv and the pvc.

(attachments)

210-aa101db8-b0d9-4a20-ae26-c9a7e044a654.json (70 KB)

2

Hi all,

I tried the restore with the awx-operator version 0.25.0.
Same result. The postgres container/pod is running and contains the correct data.
The pod with the redis, awx-web. etc containers does not start.

Can someone please advise help me out ?

Regards

3

Hi, could you provide the “awx_object” file that is in the backup directory you want to restore? e.g. /data/backup/tower-openshift-backup-YYYY-MM-dd-HH:mm:ss/awx_object

Perhaps there is a difference between the “project_*” params of AWX to be restored and the specs of pre-created PVC.

4

Hi,

This is the object.

Regards Hans-Peter

(attachments)

awx_object.txt (1.69 KB)

5

Hi,

I see that the parameter “projects_existing_claim” was removed in the source AWX.

If “projects_existing_claim” is not specified, AWX Operator will apply the PVC manifest to the cluster.
https://github.com/ansible/awx-operator/blob/0.14.0/roles/installer/templates/persistent.yaml.j2#L1
However, some fields in the PVC are immutable and cannot be modified, resulting in an error you’ve faced.

I think you have two choices:

  1. Add projects_existing_claim: awx-projects-claim to awx-object and restore.
    I don’t know why the file you attached is not in correct YAML format, but anyway, be careful with formatting in terms of indentation, line breaks, spaces, comma, etc.

2, Make the pre-created PVC specs complete as per the following template.
https://github.com/ansible/awx-operator/blob/0.14.0/roles/installer/templates/persistent.yaml.j2

Note that it is recommended to use the same version of Operator for the restore as for the backup.
It is also advisable to change the size of the PVC to be pre-created, since your “awx_object” uses 8Gi as the size of the project PVC

Regards,