Migrating AWX deployment to another cluster (k8s)

Get Help
,
1

Intro

We migrated AWX between Kubernetes clusters without using the AWX Operator’s AWXBackup/AWXRestore and without DB superuser privileges. This path relies on a manual pg_dump / pg_restore and preserving the original SECRET_KEY so encrypted credentials continue to work after the move. Below are the exact steps we used.

Prereqs

  • AWX deployed via awx-operator in both clusters (same operator version recommended).
  • Access to both clusters with kubectl.
  • Namespace assumed: awx.
  • You can create a temporary pgclient pod to run psql/pg_dump/pg_restore.

Source cluster — create the backup

  1. Discover Postgres connection from the AWX Secret
NS=awx
PGS=ansible-awx-postgres-configuration

DB=$(kubectl -n $NS get secret $PGS -o jsonpath='{.data.database}' | base64 -d)
HOST=$(kubectl -n $NS get secret $PGS -o jsonpath='{.data.host}'     | base64 -d)
USER=$(kubectl -n $NS get secret $PGS -o jsonpath='{.data.username}' | base64 -d)
PASS=$(kubectl -n $NS get secret $PGS -o jsonpath='{.data.password}' | base64 -d)
PORT=$(kubectl -n $NS get secret $PGS -o jsonpath='{.data.port}'     | base64 -d)
  1. Spin up a Postgres client pod
kubectl -n $NS run pgclient --image=quay.io/sclorg/postgresql-15-c9s:latest --restart=Never -- sleep 3600
kubectl -n $NS wait --for=condition=Ready pod/pgclient --timeout=180s
  1. Make the dump (custom format) and copy it locally
kubectl -n $NS exec -it pgclient -- bash -lc \
"PGPASSWORD='$PASS' pg_dump -h '$HOST' -p '$PORT' -U '$USER' -d '$DB' -F c -f /tmp/awx.dump"

kubectl -n $NS cp pgclient:/tmp/awx.dump ./awx.dump
  1. Export key artifacts
kubectl -n $NS get secret ansible-awx-secret-key -o yaml > secret-key.yaml      # must use in target
kubectl -n $NS get secret ansible-awx-admin-password -o yaml > admin-pass.yaml  # optional
kubectl -n $NS get awx -o yaml > awx-cr.yaml                                    # for reference only

Remember remove metadata.uid, resourceVersion, etc etc…

Target cluster — restore the backup (no DB admin rights needed)

  1. Confirm a clean AWX deploy runs (web/task/postgres Running; migration job Completed).
  2. Apply the SAME SECRET_KEY from source
kubectl -n awx apply -f secret-key.yaml
  1. Quiesce AWX to avoid writes during restore
kubectl -n awx patch awx ansible-awx --type merge -p '{"spec":{"web_replicas":0,"task_replicas":0}}'
kubectl -n awx get pods  # wait until web/task are down

or

kubectl -n awx scale deploy ansible-awx-task --replicas=0
kubectl -n awx scale deploy ansible-awx-web --replicas=0
  1. Upload the dump to a pgclient in the target
kubectl -n awx run pgclient --image=quay.io/sclorg/postgresql-15-c9s:latest --restart=Never -- sleep 3600 \
  2>/dev/null || true
kubectl -n awx wait --for=condition=Ready pod/pgclient --timeout=180s

kubectl -n awx cp ./awx.dump pgclient:/tmp/awx.dump
  1. Read Postgres connection from target Secret
PGS=ansible-awx-postgres-configuration
DB=$(kubectl -n awx get secret $PGS -o jsonpath='{.data.database}' | base64 -d)
HOST=$(kubectl -n awx get secret $PGS -o jsonpath='{.data.host}'     | base64 -d)
USER=$(kubectl -n awx get secret $PGS -o jsonpath='{.data.username}' | base64 -d)
PASS=$(kubectl -n awx get secret $PGS -o jsonpath='{.data.password}' | base64 -d)
PORT=$(kubectl -n awx get secret $PGS -o jsonpath='{.data.port}'     | base64 -d)
  1. Prepare the database WITHOUT superuser privileges
    Use DROP OWNED BY CURRENT_USER CASCADE; to remove everything owned by the awx role:
kubectl -n awx exec -it pgclient -- bash -lc \
"PGPASSWORD='$PASS' psql -h '$HOST' -p '$PORT' -U '$USER' -d '$DB' -v ON_ERROR_STOP=1 \
  -c 'DROP OWNED BY CURRENT_USER CASCADE;'"
  1. Restore the dump
kubectl -n awx exec -it pgclient -- bash -lc \
"PGPASSWORD='$PASS' pg_restore -h '$HOST' -p '$PORT' -U '$USER' -d '$DB' \
  --no-owner --no-privileges -j 4 /tmp/awx.dump"
  1. Bring AWX back up and validate
kubectl -n awx patch awx ansible-awx --type merge -p '{"spec":{"web_replicas":1,"task_replicas":1}}'
kubectl -n awx get pods
# If an 'ansible-awx-migration-*' job appears, it should finish as Completed.

Happy hacking!