init container in terminated state & web/task pods crashing!

Archives AWX Project
,
1

Hi,

My init container under awx-task pod have been ending up in Terminated state every time I spin it up. I have been using awx-ee:latest image to spin up the task-pod. Like following is an error sample. Also web and task pods keeps crashing and restarting once following reached 30 attempts. Any clue to get this resolved will be helpful. TIA
Issue 1:

State: Terminated
Reason: Completed

Issue2:

[wait-for-migrations] Waiting for database migrations…
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3 of 30
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4 of 30
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5 of 30

2

can you provide the output of
kubectl logs pod/ -c init
?

also which AWX version are you using?

thanks

AWX Team

3

Hi,

Thanks for your reply. I am using awx-operator:2.0.1 and awx & awx-ee: 22.2.0. I believe it has something to do with Postgres pod. init pod doesnt show any logs.
Below logs are from my postgres pod

PostgreSQL Database directory appears to contain a database; Skipping initialization

2023-05-24 21:28:03.437 UTC [1] LOG: starting PostgreSQL 13.10 (Debian 13.10-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-05-24 21:28:03.437 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2023-05-24 21:28:03.437 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2023-05-24 21:28:03.441 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2023-05-24 21:28:03.447 UTC [27] LOG: database system was shut down at 2023-05-24 21:26:14 UTC
2023-05-24 21:28:03.458 UTC [1] LOG: database system is ready to accept connections
2023-05-24 21:28:22.860 UTC [34] FATAL: password authentication failed for user “awx”
2023-05-24 21:28:22.860 UTC [34] DETAIL: Connection matched pg_hba.conf line 99: “host all all all scram-sha-256”

4

reference ticket for more logs: https://github.com/ansible/awx-operator/issues/1431

5

This line looks like an issue:
2023-05-24 21:28:22.860 UTC [34] FATAL: password authentication failed for user “awx”

It would indicate that the database password secret is either not set or no longer correct.
We have seen an issue with a race condition where a PVC could contain an old secret from an older install and you could errors like this even if your current kube secret is correct.

You might want to try to exec into the DB pod and then run a psql command to update the awx users password match the current kube secret.

All that being said it looks like you are getting some traction in the issue so we will let the discussion continue there.

-The AWX Team

6

Thanks for your reply. I am trying to include secrets as annotations in my Secret file as below. This was working yesterday and I was able to get the postgres pod running. However today it started asking for fields such as username, password etc and cannot get the postgres pod running.

stringData:
host: awx-postgres
port: “5432”
database: awx-postgres
username: awx
password: /path/to/internal/secretvault