Issues upgrading AWX 16.0.0 to 20.0.1

Archives AWX Project
, ,
1

Hi,

I inherited the following AWX installation:

  • Ubuntu 18.04 on an AWS EC2 instance
  • AWX 16.0.0 running in docker containers (awx_web and awx_task)
  • the AWX database is an AWS RDS PostgreSQL.

Now I try to upgrade AWX to latest 20.x (though 19.x would be fine too)

I tried to follow the steps described here: https://github.com/ansible/awx/blob/devel/tools/docker-compose/docs/data_migration.md
But I got stuck at the first step. I did a git pull locally and then I ran:

ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/migrate.yml -e “old_docker_compose_dir=~/.awx/awxcompose”

The difference between the procedure explained at the above link and my local setup is that the procedure is for PostgreSQL running containerized, but my PostgreSQL instance is an RDS one and I want to keep it like that.
Possibly because of that, I got the following errors:

TASK [migrate : Remove awx_postgres to ensure consistent start state] ***************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “docker rm -f awx_postgres\n”, “delta”: “0:00:00.045924”, “end”: “2022-03-11 09:32:51.594861”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-03-11 09:32:51.548937”, “stderr”: “Error: No such container: awx_postgres”, “stderr_lines”: [“Error: No such container: awx_postgres”], “stdout”: “”, “stdout_lines”: }
…ignoring

TASK [migrate : Start Local Docker database container] ******************************************************************************************************************************************************************************
ok: [localhost]

TASK [migrate : Wait for postgres to initialize] ************************************************************************************************************************************************************************************
ok: [localhost]

TASK [migrate : Database dump to local filesystem] **********************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “docker-compose -f ~/.awx/awxcompose/docker-compose.yml exec -T postgres pg_dumpall -U awx > awx_dump.sql\n”, “delta”: “0:00:00.687348”, “end”: “2022-03-11 09:32:56.891168”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-03-11 09:32:56.203820”, “stderr”: “”, “stderr_lines”: , “stdout”: “”, “stdout_lines”: }

PLAY RECAP **************************************************************************************************************************************************************************************************************************
localhost : ok=21 changed=2 unreachable=0 failed=1 skipped=1 rescued=0 ignored=1

I didn’t know what to do next so I abandoned this approach and I tried differently.

I stopped and deleted the awx_task and awx_web containers and I updated the docker-compose.yml file that was used to launch the stack.
Basically I changed the AWX version of the used images, from ansible/awx:16.0.0 to quay.io/ansible/awx:20.0.1.

Then I ran docker-compose -f docker-compose.yml up. The process started, the new images got downloaded but I got stuck in the following loop:

awx_task | [wait-for-migrations] Attempt 23 of 30
awx_web | [wait-for-migrations] Waiting 30 seconds before next attempt
awx_task | [wait-for-migrations] Waiting 30 seconds before next attempt
awx_web | [wait-for-migrations] Attempt 24 of 30
awx_task | [wait-for-migrations] Attempt 24 of 30
awx_task | [wait-for-migrations] Waiting 30 seconds before next attempt
awx_web | [wait-for-migrations] Waiting 30 seconds before next attempt

What I want to achieve is:

  • upgrade AWX to latest 20.x (but 19.x would be fine too)
  • do not loose my database content, because I have a lot of settings there. So I need to find a way to migrate it.
  • keep the database on RDS

I don’t mind to completely ditch the containerized approach and have an installation directly on the host, as long as I cover all the points above.
I don’t mind also to do the upgrades one by one instead of directly going to 20.x, if this would be a solution.
But honestly I’m completely stuck.
By the way, I still have the inventory file used at the initial installation, in case it is important.

So, any idea on how should I proceed?

Thank you,
Ciprian

2

Hey there,

We recommend doing backup and restore, you may have to do that through a couple versions between 16.0.x and 21.0.x to be certain the upgrade maintains the data integrity the way you want it. Docs for backup and restore are here (I’m sending the latest version but you can go back versions in the version drop down on the left hand side): https://docs.ansible.com/automation-controller/latest/html/administration/backup_restore.html

Also, a few other members of the community recently did this if you want to read a little more about their experience:
https://groups.google.com/g/awx-project/c/sYk_bfYL4vM

Cheers,
rebeccahhh + the AWX team

3

If you can get into your awx-task container, you can manually run the awx-manage migrate command there and see where the upgrade is hung at. When we migrated from 16.x to 19.x with an external db, what we had to do was to enable verbose logging on the database and run the awx-manage migrate to see all the commands being runned on the db by main.0144_event_partitions. That migration would always fail and so we had to manually run the steps and validate it. Then insert the event into the django_migrations table. rerun awx-manage migrate again to proceed thru the rest of the migration step. Once that was done the remainder of thing worked.

We tried the backup and restore but the db schema was different after a certain upgrade that the retore process would not apply the data from the backup. That’s why we had to log all the statement and figure out how to migrate the event steps that was failing and get the schema to the state the next event migration wanted it to be at. With that we were able to migrate our external db installed on docker to k8 still using the same external db.

Michael