AWX upgrade failing to perform migration for upgrade of AWX operator 2.10.0 -> 2.19.1

AWX upgrade failing to perform migration for upgrade of AWX operator 2.10.0 → 2.19.1.

Hello,

I’m in the process of trying to upgrade my AWX environment via the AWX operator. I’m going from version 2.10.0 of the operator to 2.19.1. However, it appears my upgrade is failing with errors for migrate_schema.yml:

awx-manager  TASK [Create kubernetes job to perform the migration] ********************************

awx-manager fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to retrieve requested object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"jobs.batch \\\\\"awx-migration-24.6.1\\\\\" is forbidden: User \\\\\"system:serviceaccount:awx:awx-operator-controller-manager\\\\\" cannot get resource \\\\\"jobs\\\\\" in API group \\\\\"batch\\\\\" in the namespace \\\\\"awx\\\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\"awx-migration-24.6.1\",\"group\":\"batch\",\"kind\":\"jobs\"},\"code\":403}\\n'", "reason": "Forbidden"}

awx-manager {"level":"error","ts":"2024-08-12T16:54:44Z","logger":"logging_event_handler","msg":"","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"746298583749467876","EventData.Task":"Create kubernetes job to perform the migration","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/installer/tasks/migrate_schema.yml:33","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"}
[...]

and

TASK [installer : Check for pending migrations] ********************************\r\ntask path: /opt/ansible/roles/installer/tasks/migrate_schema.yml:3\nok: [localhost] => {\"changed\": false, \"rc\": 0, \"return_code\": 0, \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"15\\n\", \"stdout_lines\": [\"15\"]}\n\r\nTASK [installer : Get version of controller for tracking] **********************\r\ntask path: /opt/ansible/roles/installer/tasks/migrate_schema.yml:15\nok: [localhost] => {\"changed\": false, \"rc\": 0, \"return_code\": 0, \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"24.6.1\\n\", \"stdout_lines\": [\"24.6.1\"]}\n\r\nTASK [installer : Sanitize instance version] ***********************************\r\ntask path: /opt/ansible/roles/installer/tasks/migrate_schema.yml:25\nok: [localhost] => {\"ansible_facts\": {\"version\": \"24.6.1\"}, \"changed\": false}\n\r\nTASK [installer : Create kubernetes job to perform the migration] **************\r\ntask path: /opt/ansible/roles/installer/tasks/migrate_schema.yml:33\nfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Failed to retrieve requested object: b'{\\\"kind\\\":\\\"Status\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"metadata\\\":{},\\\"status\\\":\\\"Failure\\\",\\\"message\\\":\\\"jobs.batch \\\\\\\\\\\"awx-migration-24.6.1\\\\\\\\\\\" is forbidden: User \\\\\\\\\\\"system:serviceaccount:awx:awx-operator-controller-manager\\\\\\\\\\\" cannot get resource \\\\\\\\\\\"jobs\\\\\\\\\\\" in API group \\\\\\\\\\\"batch\\\\\\\\\\\" in the namespace \\\\\\\\\\\"awx\\\\\\\\\\\"\\\",\\\"reason\\\":\\\"Forbidden\\\",\\\"details\\\":{\\\"name\\\":\\\"awx-migration-24.6.1\\\",\\\"group\\\":\\\"batch\\\",\\\"kind\\\":\\\"jobs\\\"},\\\"code\\\":403}\\\\n'\", \"reason\": \"Forbidden\"}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=77   changed=0    unreachable=0    failed=1    skipped=65   rescued=0    ignored=0   \n","job":"3843762477658756621","name":"awx","namespace":"awx","error":"exit status 2"

Please let me know if any other info is needed. Thanks!

I found this thread Reddit - Dive into anything and running awx-manage migrate --noinput from the initdb container seems to have done the trick to get the upgrade finished. I also may have messed up my kustomize.yml file by forgetting to update the tag for “resources” to match the tag under “images”.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.