# Issues upgrading AWX 16.0.0 to 20.0.1

**URL:** <https://forum.ansible.com/t/issues-upgrading-awx-16-0-0-to-20-0-1/36424>\
**Category:** AWX Project\
**Tags:** ubuntu, aws, awx\
**Created:** [March 11, 2022, 10:10am UTC](https://forum.ansible.com/t/issues-upgrading-awx-16-0-0-to-20-0-1/36424 "2022-03-11T10:10:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ciprian\_Stoica](https://avatars.discourse-cdn.com/v4/letter/c/b9bd4f/32.png) [@Ciprian\_Stoica](https://forum.ansible.com/u/Ciprian_Stoica)\
**Post date:** [March 11, 2022, 10:10am UTC](https://forum.ansible.com/t/issues-upgrading-awx-16-0-0-to-20-0-1/36424/1 "2022-03-11T10:10:13Z")

</div>

Hi,

I inherited the following AWX installation:

- Ubuntu 18.04 on an AWS EC2 instance
- AWX 16.0.0 running in docker containers (awx\_web and awx\_task)
- the AWX database is an AWS RDS PostgreSQL.

Now I try to upgrade AWX to latest 20.x (though 19.x would be fine too)

I tried to follow the steps described here: [https://github.com/ansible/awx/blob/devel/tools/docker-compose/docs/data\_migration.md](https://github.com/ansible/awx/blob/devel/tools/docker-compose/docs/data_migration.md)  
But I got stuck at the first step. I did a git pull locally and then I ran:

ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/migrate.yml -e “old\_docker\_compose\_dir=~/.awx/awxcompose”

The difference between the procedure explained at the above link and my local setup is that the procedure is for PostgreSQL running containerized, but my PostgreSQL instance is an RDS one and I want to keep it like that.  
Possibly because of that, I got the following errors:

TASK [migrate : Remove awx\_postgres to ensure consistent start state] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
fatal: [localhost]: FAILED! =\> {“changed”: true, “cmd”: “docker rm -f awx\_postgres\n”, “delta”: “0:00:00.045924”, “end”: “2022-03-11 09:32:51.594861”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-03-11 09:32:51.548937”, “stderr”: “Error: No such container: awx\_postgres”, “stderr\_lines”: [“Error: No such container: awx\_postgres”], “stdout”: “”, “stdout\_lines”: }  
…ignoring

TASK [migrate : Start Local Docker database container] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
ok: [localhost]

TASK [migrate : Wait for postgres to initialize] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
ok: [localhost]

TASK [migrate : Database dump to local filesystem] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
fatal: [localhost]: FAILED! =\> {“changed”: true, “cmd”: “docker-compose -f ~/.awx/awxcompose/docker-compose.yml exec -T postgres pg\_dumpall -U awx \> awx\_dump.sql\n”, “delta”: “0:00:00.687348”, “end”: “2022-03-11 09:32:56.891168”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2022-03-11 09:32:56.203820”, “stderr”: “”, “stderr\_lines”: , “stdout”: “”, “stdout\_lines”: }

PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
localhost : ok=21 changed=2 unreachable=0 failed=1 skipped=1 rescued=0 ignored=1

I didn’t know what to do next so I abandoned this approach and I tried differently.

I stopped and deleted the awx\_task and awx\_web containers and I updated the docker-compose.yml file that was used to launch the stack.  
Basically I changed the AWX version of the used images, from ansible/awx:16.0.0 to [quay.io/ansible/awx:20.0.1](http://quay.io/ansible/awx:20.0.1).

Then I ran docker-compose -f docker-compose.yml up. The process started, the new images got downloaded but I got stuck in the following loop:

awx\_task | [wait-for-migrations] Attempt 23 of 30  
awx\_web | [wait-for-migrations] Waiting 30 seconds before next attempt  
awx\_task | [wait-for-migrations] Waiting 30 seconds before next attempt  
awx\_web | [wait-for-migrations] Attempt 24 of 30  
awx\_task | [wait-for-migrations] Attempt 24 of 30  
awx\_task | [wait-for-migrations] Waiting 30 seconds before next attempt  
awx\_web | [wait-for-migrations] Waiting 30 seconds before next attempt

What I want to achieve is:

- upgrade AWX to latest 20.x (but 19.x would be fine too)
- do not loose my database content, because I have a lot of settings there. So I need to find a way to migrate it.
- keep the database on RDS

I don’t mind to completely ditch the containerized approach and have an installation directly on the host, as long as I cover all the points above.  
I don’t mind also to do the upgrades one by one instead of directly going to 20.x, if this would be a solution.  
But honestly I’m completely stuck.  
By the way, I still have the inventory file used at the initial installation, in case it is important.

So, any idea on how should I proceed?

Thank you,  
Ciprian

---

<div class="post-metadata">

**Author:** ![Rebeccah\_Hunter](https://avatars.discourse-cdn.com/v4/letter/r/4da419/32.png) [@Rebeccah\_Hunter](https://forum.ansible.com/u/Rebeccah_Hunter)\
**Post date:** [May 4, 2022, 8:00pm UTC](https://forum.ansible.com/t/issues-upgrading-awx-16-0-0-to-20-0-1/36424/2 "2022-05-04T20:00:06Z")

</div>

Hey there,

We recommend doing backup and restore, you may have to do that through a couple versions between 16.0.x and 21.0.x to be certain the upgrade maintains the data integrity the way you want it. Docs for backup and restore are here (I’m sending the latest version but you can go back versions in the version drop down on the left hand side): [https://docs.ansible.com/automation-controller/latest/html/administration/backup\_restore.html](https://docs.ansible.com/automation-controller/latest/html/administration/backup_restore.html)

Also, a few other members of the community recently did this if you want to read a little more about their experience:  
[https://groups.google.com/g/awx-project/c/sYk\_bfYL4vM](https://groups.google.com/g/awx-project/c/sYk_bfYL4vM)

Cheers,  
rebeccahhh + the AWX team

---

<div class="post-metadata">

**Author:** ![Michael\_Nhan](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@Michael\_Nhan](https://forum.ansible.com/u/Michael_Nhan)\
**Post date:** [May 5, 2022, 4:11pm UTC](https://forum.ansible.com/t/issues-upgrading-awx-16-0-0-to-20-0-1/36424/3 "2022-05-05T16:11:35Z")

</div>

If you can get into your awx-task container, you can manually run the awx-manage migrate command there and see where the upgrade is hung at. When we migrated from 16.x to 19.x with an external db, what we had to do was to enable verbose logging on the database and run the awx-manage migrate to see all the commands being runned on the db by main.0144\_event\_partitions. That migration would always fail and so we had to manually run the steps and validate it. Then insert the event into the django\_migrations table. rerun awx-manage migrate again to proceed thru the rest of the migration step. Once that was done the remainder of thing worked.

We tried the backup and restore but the db schema was different after a certain upgrade that the retore process would not apply the data from the backup. That’s why we had to log all the statement and figure out how to migrate the event steps that was failing and get the schema to the state the next event migration wanted it to be at. With that we were able to migrate our external db installed on docker to k8 still using the same external db.

Michael