Refresh postgres database?

I’m experimenting with 21.10.2 with postgres 13 on OpenShift. I hung my installation in Terminating mode last week when I was trying to delete and reinstall it, and had to forcefully delete it. Now, when I reinstall it, it seems like the postgres database has issues. When I do an awx-manage migrate, I get inconsistent results. Sometimes it runs into a dozen errors before I give up and reinstall it, sometimes it runs into no errors but the final running AWX install has no objects in it. A clean AWX install has Demo projects, inventories and the like and at the very least an admin superuser. When this install error happens, I have none of those things. And once in a while, it installs perfectly and I celebrate that it’s fixed, only to find out the next reinstall fails again.

Is there any way to do a full purge and refresh of the database?

Thank you,

Kevin

are you using an external db or the one that is auto deployed with AWX?

awx-manage migrate, I get inconsistent results.

what errors are you seeing in the migration step?

AWX Team

I’m using the internal postgres automatically deployed with AWX. The install acts differently every time. So, I just did a delete and reran the full install script. It created everything perfectly, but the site would not come up. The route was not published in OpenShift. I deleted the AWX pod, and let it recreate itself and that time it worked. In fact, it worked perfectly. I have all the missing components. So, I’m not able to demonstrate this time what happens when it’s broken.

When the site was not browseable, I ran the command at the web node the first time and received: (After getting this I did the pod refresh, and everything is working like a dream now.)

h-5.1$ awx-manage migrate --noinput
Operations to perform:
Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit
Running migrations:
Applying main.0002_squashed_v300_release…Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 82, in _execute
return self.cursor.execute(sql)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint “pg_type_typname_nsp_index”
DETAIL: Key (typname, typnamespace)=(main_towersettings_id_seq, 2200) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py”, line 201, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py”, line 419, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/init.py”, line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py”, line 354, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py”, line 398, in execute
output = self.handle(*args, **options)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py”, line 89, in wrapped
res = handle_func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py”, line 244, in handle
post_migrate_state = executor.migrate(
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py”, line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py”, line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py”, line 227, in apply_migration
state = migration.apply(state, schema_editor)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py”, line 126, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/models.py”, line 92, in database_forwards
schema_editor.create_model(model)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py”, line 345, in create_model
self.execute(sql, params or None)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py”, line 145, in execute
cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 84, in _execute
return self.cursor.execute(sql, params)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py”, line 90, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 82, in _execute
return self.cursor.execute(sql)
django.db.utils.IntegrityError: duplicate key value violates unique constraint “pg_type_typname_nsp_index”
DETAIL: Key (typname, typnamespace)=(main_towersettings_id_seq, 2200) already exists.

sh-5.1$

Just deleted and rebuild again. After the fresh start, I have this in the Web log:
[wait-for-migrations] Waiting for database migrations… [wait-for-migrations] Attempt 1 of 30 [wait-for-migrations] Waiting 0.5 seconds before next attempt [wait-for-migrations] Attempt 2 of 30 [wait-for-migrations] Waiting 1 seconds before next attempt [wait-for-migrations] Attempt 3 of 30 [wait-for-migrations] Waiting 2 seconds before next attempt [wait-for-migrations] Attempt 4 of 30

So, doing the migration I get:
sh-5.1$ awx-manage migrate --noinput
Operations to perform:
Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso, taggit
Running migrations:
Applying main.0144_event_partitions…Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py”, line 82, in _execute
return self.cursor.execute(sql)
psycopg2.errors.DuplicateTable: relation “_unpartitioned_main_jobevent” already exists

After that, the site comes up, and it’s in good order. I have all the doohickeys, including the admin user and the EEs. So, it worked twice for me, both times after attempting the migration and getting an error.

Is there any chance if I run the migration multiple times it could hose the database? Because when I see that first error, I have retried as a rule.

Kevin

Driving me nuts, here. It’s nothing to do with leftover data. I just created a new AWX namespace, and had the same issue. Deleted it, recreated it, and had the same issue. Why does it not always work and why does it sometimes work? It has to be something I’m doing, but I have no idea what.

A list of all the errors in the database startup is:
Search “Error:” (58 hits in 1 file of 1 searched)
Line 57: 2023-01-05 14:16:09.323 UTC [60] ERROR: relation “conf_setting” does not exist at character 158
Line 59: 2023-01-05 14:16:09.931 UTC [61] ERROR: relation “conf_setting” does not exist at character 158
Line 61: 2023-01-05 14:16:12.079 UTC [64] ERROR: relation “conf_setting” does not exist at character 158
Line 63: 2023-01-05 14:16:12.358 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 65: 2023-01-05 14:16:12.360 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 67: 2023-01-05 14:16:12.371 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 69: 2023-01-05 14:16:12.372 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 71: 2023-01-05 14:16:12.381 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 73: 2023-01-05 14:16:12.406 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 75: 2023-01-05 14:16:12.408 UTC [64] ERROR: relation “django_migrations” does not exist at character 124
Line 77: 2023-01-05 14:16:12.643 UTC [65] ERROR: relation “conf_setting” does not exist at character 158
Line 79: 2023-01-05 14:16:12.970 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 81: 2023-01-05 14:16:12.972 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 83: 2023-01-05 14:16:13.128 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 85: 2023-01-05 14:16:13.130 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 87: 2023-01-05 14:16:13.132 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 89: 2023-01-05 14:16:13.143 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 91: 2023-01-05 14:16:13.144 UTC [65] ERROR: relation “django_migrations” does not exist at character 124
Line 93: 2023-01-05 14:16:14.149 UTC [67] ERROR: relation “conf_setting” does not exist at character 158
Line 95: 2023-01-05 14:16:14.380 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 97: 2023-01-05 14:16:14.389 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 99: 2023-01-05 14:16:14.390 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 101: 2023-01-05 14:16:14.416 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 103: 2023-01-05 14:16:14.418 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 105: 2023-01-05 14:16:14.561 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 107: 2023-01-05 14:16:14.563 UTC [67] ERROR: relation “django_migrations” does not exist at character 124
Line 109: 2023-01-05 14:16:15.534 UTC [70] ERROR: relation “conf_setting” does not exist at character 158
Line 111: 2023-01-05 14:16:16.286 UTC [71] ERROR: relation “conf_setting” does not exist at character 158
Line 113: 2023-01-05 14:16:18.248 UTC [75] ERROR: relation “conf_setting” does not exist at character 158
Line 115: 2023-01-05 14:16:18.354 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 117: 2023-01-05 14:16:18.371 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 119: 2023-01-05 14:16:18.372 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 121: 2023-01-05 14:16:18.384 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 123: 2023-01-05 14:16:18.386 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 125: 2023-01-05 14:16:18.501 UTC [75] ERROR: relation “django_migrations” does not exist at character 124
Line 127: 2023-01-05 14:16:18.510 UTC [75] ERROR: relation “django_migrations” does not exist at character 124
Line 129: 2023-01-05 14:16:18.515 UTC [75] ERROR: relation “django_migrations” does not exist at character 124
Line 131: 2023-01-05 14:16:18.517 UTC [75] ERROR: relation “django_migrations” does not exist at character 124
Line 133: 2023-01-05 14:16:18.520 UTC [75] ERROR: relation “django_migrations” does not exist at character 124
Line 135: 2023-01-05 14:16:18.536 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 137: 2023-01-05 14:16:18.538 UTC [76] ERROR: relation “django_migrations” does not exist at character 124
Line 139: 2023-01-05 14:16:18.936 UTC [77] ERROR: relation “conf_setting” does not exist at character 158
Line 141: 2023-01-05 14:16:22.253 UTC [80] ERROR: relation “conf_setting” does not exist at character 158
Line 143: 2023-01-05 14:16:23.054 UTC [81] ERROR: relation “conf_setting” does not exist at character 158
Line 145: 2023-01-05 14:16:25.052 UTC [84] ERROR: relation “conf_setting” does not exist at character 158
Line 147: 2023-01-05 14:16:25.833 UTC [85] ERROR: relation “conf_setting” does not exist at character 158
Line 149: 2023-01-05 14:16:30.196 UTC [88] ERROR: relation “conf_setting” does not exist at character 158
Line 151: 2023-01-05 14:16:30.800 UTC [89] ERROR: relation “conf_setting” does not exist at character 158
Line 153: 2023-01-05 14:16:33.031 UTC [92] ERROR: relation “conf_setting” does not exist at character 158
Line 155: 2023-01-05 14:16:33.666 UTC [93] ERROR: relation “conf_setting” does not exist at character 158
Line 157: 2023-01-05 14:16:40.339 UTC [97] ERROR: relation “conf_setting” does not exist at character 158
Line 159: 2023-01-05 14:16:40.647 UTC [98] ERROR: relation “conf_setting” does not exist at character 158
Line 161: 2023-01-05 14:16:43.035 UTC [101] ERROR: relation “conf_setting” does not exist at character 158
Line 163: 2023-01-05 14:16:43.415 UTC [102] ERROR: relation “conf_setting” does not exist at character 158
Line 165: 2023-01-05 14:16:54.027 UTC [105] ERROR: relation “conf_setting” does not exist at character 158
Line 167: 2023-01-05 14:16:54.276 UTC [106] ERROR: relation “conf_setting” does not exist at character 158
Line 169: 2023-01-05 14:16:56.351 UTC [109] ERROR: relation “conf_setting” does not exist at character 158
Line 171: 2023-01-05 14:16:56.497 UTC [110] ERROR: relation “conf_setting” does not exist at character 158

In the database log, I see this:
chmod: changing permissions of ‘/var/run/postgresql’: Operation not permitted
The files belonging to this database system will be owned by user “1005300000”.
This user must also own the server process.

The database cluster will be initialized with locale “en_US.utf8”.
The default database encoding has accordingly been set to “UTF8”.
The default text search configuration will be set to “english”.

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata … ok
creating subdirectories … ok
selecting dynamic shared memory implementation … posix
selecting default max_connections … 100
selecting default shared_buffers … 128MB
selecting default time zone … Etc/UTC
creating configuration files … ok
running bootstrap script … ok
performing post-bootstrap initialization … ok
syncing data to disk … ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/data/pgdata -l logfile start

initdb: warning: enabling “trust” authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
–auth-local and --auth-host, the next time you run initdb.
waiting for server to start…2023-01-05 14:15:36.006 UTC [36] LOG: starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-05 14:15:36.008 UTC [36] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2023-01-05 14:15:36.018 UTC [37] LOG: database system was shut down at 2023-01-05 14:15:35 UTC
2023-01-05 14:15:36.025 UTC [36] LOG: database system is ready to accept connections
done
server started
CREATE DATABASE

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2023-01-05 14:15:36.308 UTC [36] LOG: received fast shutdown request
waiting for server to shut down…2023-01-05 14:15:36.313 UTC [36] LOG: aborting any active transactions
2023-01-05 14:15:36.315 UTC [36] LOG: background worker “logical replication launcher” (PID 43) exited with exit code 1
2023-01-05 14:15:36.315 UTC [38] LOG: shutting down
2023-01-05 14:15:36.335 UTC [36] LOG: database system is shut down
done
server stopped

PostgreSQL init process complete; ready for start up.

2023-01-05 14:15:36.443 UTC [1] LOG: starting PostgreSQL 13.9 (Debian 13.9-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2023-01-05 14:15:36.443 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2023-01-05 14:15:36.443 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2023-01-05 14:15:36.449 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2023-01-05 14:15:36.456 UTC [51] LOG: database system was shut down at 2023-01-05 14:15:36 UTC
2023-01-05 14:15:36.464 UTC [1] LOG: database system is ready to accept connections
2023-01-05 14:16:09.323 UTC [60] ERROR: relation “conf_setting” does not exist at character 158

Could this just be a matter of impatience? I left this one sit, untouched, for an hour and when I came back it was fully built and working.

When first installing, until migrations are completed, there could be errors in the services that depend on the database – this is to be expected. After migrations complete, things should sort themselves out and run successfully. Is this what you are seeing?

You can always check the status of migrations by shelling into a running container and running awx-manage showmigrations.

AWX Team

That helps. Thank you!

Kevin