postgres-0 status is CrashLoopBackOff

Hi,

I’ve been using AWX 19.3.0 / AWX operator 0.13.0 on minikube.
After deploying AWX, I was able to access it without any problems. But, I have lost access to AWX today.

The status of postgres-0 pod was CrashLoopBackOff.
The log output was “initdb: error: directory “/var/lib/postgresql/data/pgdata” exists but is not empty”.

Is there any way to recover?
I would be grateful if anyone could comment.

The following is the status of the pod and the output of describe and log.

[root@centos-tools ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
awx-demo-7bbb564887-m7t5n 4/4 Running 8 13d
awx-demo-postgres-0 0/1 CrashLoopBackOff 834 13d
awx-operator-69c646c48f-cbf5k 1/1 Running 1 13d

Hi Yasu

Few rhetorical queries

  1. Are you using an external postgres DB for AWX ??
    if external then where does “/var/lib/postgresql/data/pgdata” map to

  2. Is it a fresh installation or are you trying to migrate things ???

  3. please share the yamls which you are using to start the AWX

Regards
Abhinit

Hi Abhinit,

Thank you for your reply.

  1. Are you using an external postgres DB for AWX ??
    No, I using internal postgres DB.

  2. Is it a fresh installation or are you trying to migrate things ???
    It is the fresh installation.

  3. please share the yamls which you are using to start the AWX
    I used the following YAML.

kubectl apply -f https://raw.githubusercontent.com/ansible/awx-operator/0.13.0/deploy/awx-operator.yaml
kubectl apply -f awx-demo.yml

cat awx-demo.yml

Hi Yasu

As it’s is fresh setup I would rather request you to use an external postgres database which will be hosted on your minikube where the postgres database will store the data in your local minikube host ( in that way if you backup the filesystem your postgres is getting backed up)
Your awx will basically access the external postgres and will keep running.
If you are fine with the solution
I can share the configuration with you .

Regards
Abhinit

Hi,

Looking at postgres:12 image, the docker-entry.sh has the following:

docker_setup_env() {

file_env ‘POSTGRES_PASSWORD’

file_env ‘POSTGRES_USER’ ‘postgres’

file_env ‘POSTGRES_DB’ “$POSTGRES_USER”

file_env ‘POSTGRES_INITDB_ARGS’

default authentication method is md5

: “${POSTGRES_HOST_AUTH_METHOD:=md5}”

declare -g DATABASE_ALREADY_EXISTS

look specifically for PG_VERSION, as it is expected in the DB dir

if [ -s “$PGDATA/PG_VERSION” ]; then

DATABASE_ALREADY_EXISTS=‘true’

fi

}

This docker_setup_env() function gets called at the beginning of the _main() and the postgres goes to a initdb path since DATABASE_ALREADY_EXISTS is empty.

only run initialization on an empty data directory

if [ -z “$DATABASE_ALREADY_EXISTS” ]; then

docker_verify_minimum_env

check dir permissions to reduce likelihood of half-initialized database

ls /docker-entrypoint-initdb.d/ > /dev/null

docker_init_database_dir

pg_setup_hba_conf

Looks like in your case your PGDATA path exists, but the PG_VERSION file is not there or empty. I think your postgres db is corrupt. You need to rebuild the DB, and restore from a backup.

HTH,

Hi Abhinit,

Thank you for your reply.

I understand it.
Could you please share the configuration?

Best Regard,
Yasu

2021年9月6日月曜日 2:08:47 UTC+9 Abhinit Mallick: