I’ve been using AWX 19.3.0 / AWX operator 0.13.0 on minikube.
After deploying AWX, I was able to access it without any problems. But, I have lost access to AWX today.
The status of postgres-0 pod was CrashLoopBackOff.
The log output was “initdb: error: directory “/var/lib/postgresql/data/pgdata” exists but is not empty”.
Is there any way to recover?
I would be grateful if anyone could comment.
The following is the status of the pod and the output of describe and log.
[root@centos-tools ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
awx-demo-7bbb564887-m7t5n 4/4 Running 8 13d
awx-demo-postgres-0 0/1 CrashLoopBackOff 834 13d
awx-operator-69c646c48f-cbf5k 1/1 Running 1 13d
As it’s is fresh setup I would rather request you to use an external postgres database which will be hosted on your minikube where the postgres database will store the data in your local minikube host ( in that way if you backup the filesystem your postgres is getting backed up)
Your awx will basically access the external postgres and will keep running.
If you are fine with the solution
I can share the configuration with you .
Looking at postgres:12 image, the docker-entry.sh has the following:
docker_setup_env() {
file_env ‘POSTGRES_PASSWORD’
file_env ‘POSTGRES_USER’ ‘postgres’
file_env ‘POSTGRES_DB’ “$POSTGRES_USER”
file_env ‘POSTGRES_INITDB_ARGS’
default authentication method is md5
: “${POSTGRES_HOST_AUTH_METHOD:=md5}”
declare -g DATABASE_ALREADY_EXISTS
look specifically for PG_VERSION, as it is expected in the DB dir
if [ -s “$PGDATA/PG_VERSION” ]; then
DATABASE_ALREADY_EXISTS=‘true’
fi
}
This docker_setup_env() function gets called at the beginning of the _main() and the postgres goes to a initdb path since DATABASE_ALREADY_EXISTS is empty.
only run initialization on an empty data directory
if [ -z “$DATABASE_ALREADY_EXISTS” ]; then
docker_verify_minimum_env
check dir permissions to reduce likelihood of half-initialized database
ls /docker-entrypoint-initdb.d/ > /dev/null
docker_init_database_dir
pg_setup_hba_conf
Looks like in your case your PGDATA path exists, but the PG_VERSION file is not there or empty. I think your postgres db is corrupt. You need to rebuild the DB, and restore from a backup.