Stuck on installing update to AWX using operator 2.19.1

Get Help
1

I had a working installation of AWX 22.3.0, but figured it’s over a year old; I should update to something newer.

I tried following various docs:

  1. Backup role allowed me to create a backup, though I’m not super confident that I’d be able to restore it.
  2. Upgrading doc advised deleting old Operator, so I did that.
  3. It then suggested using Basic Install to install the newer operator (I’m using 2.19.1).
  4. I ran make deploy, created kustomization.yml with my AWX deployment name and tag, and include the awx-demo.yml
  5. Ran kubectl apply -k .
  6. It seemed to get rid of my old awx-demo pods (task, web, etc.) and start its install process
  7. Install process creates a Postgres 15 pod (old DB was 13). New Postgres pod is stuck in “Pending” with “mkdir: cannot create directory ‘/var/lib/pgsql/data/userdata’: Permission denied”
  8. awx-manager logs seem to be looping, trying to install, “"StatefulSet" "awx-demo-postgres-15": Timed out waiting on resource”
  9. Looked into the Migrating data from an old AWX instance - Ansible AWX Operator Documentation and added awx-demo-postgres-configuration Secret, updated awx-demo.yml and re-ran kubectl apply -k. Also evicted the operator pod so it would be restarted.

Expected: AWX installed to newer version (24.6.1?) and running with my old credentials, templates, inventories, etc.

Actual: Operator is spinning on the install, no pods for awx-web, task, etc.

2

Sounds similar to this issue Postgres 15 pod: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied" · Issue #1770 · ansible/awx-operator · GitHub maybe this person’s chown fix will work for you Postgres 15 pod: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied" · Issue #1770 · ansible/awx-operator · GitHub or editing the spec like this person to do the chown Postgres 15 pod: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied" · Issue #1770 · ansible/awx-operator · GitHub

1 Like
3

Maybe just adding postgres_data_volume_init: true to your AWX’s spec solves your issue :wink:

4

@kurokobo Thanks! I did this, plus a few more steps and was able to get it working, with a few snags along the way. Full detail:

  1. Added postgres_data_volume_init: true to the awx-demo.yml and then re-ran kubectl apply -k .
  2. Evicted the old operator pod and the old postgres-15 pod. (Note that my old install was using a postgres-13 pod; I evicted the “new” postgres-15 pod that was failing.)
  3. It ran for a while, deleted the old postgres-13 pod, and then threw a different error, which was hidden due to no_log: true
  4. Added no_log: false to awx-demo.yml and then re-ran kubectl apply -k and evicted the operator pod.
  5. New error log showed “pg_dump: error: could not translate host name "awx-demo-postgres-13"” (of course this was because the install had deleted it earlier)
  6. Removed the old_postgres_configuration_secret from awx-demo.yml and deleted the Secret “awx-demo-old-postgres-configuration” and then re-ran kubectl apply -k and evicted the operator pod.

From there, it seemed to succeed.

Thinking about how the docs might be updated to avoid someone else getting stuck with this. Thoughts on adding, in Basic Install doc mention of postgres_data_volume_init: true ?

If that seems like a good practice for all installs, I’d be happy to make a git PR to add some detail on that to the docs. Thoughts?

5

Hi, sorry for my late reply.

The parameter postgres_data_volume_init is not necessarily required for all users. For example, it does not required on OpenShift, and even on Kubernetes, whether the issue arises depends on how the storage providing the PV behaves (how the default permissions of the PV are set). This is also mentioned in the documentation that explains this parameter. That is why this parameter is false by default.

I think adding more description about this setting in the documentation is a good idea, but it might be more natural to add it to the troubleshooting section rather than the basic install section.