Getting wait-for-migrations log inside awx-task and awx-web pods

Archives AWX Project
,
1

I have deployed awx-operator on kubernetes cluster and everything seems smooth from outside, every service and pods were running perfectly. Then when i walked through logs of the awx pods i saw that the pods were waiting for database migration . this is a fresh installation and it is looping through waiting for the migration. Any idea on what exactly is the issue.

kubectl logs pod/awx-web-6b4767c86f-lsjnp

[wait-for-migrations] Waiting for database migrations…
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3 of 30
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4 of 30
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5 of 30
[wait-for-migrations] Waiting 8 seconds before next attempt
[wait-for-migrations] Attempt 6 of 30
[wait-for-migrations] Waiting 16 seconds before next attempt
[wait-for-migrations] Attempt 7 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 8 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 9 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 10 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 11 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 12 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 13 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt

2

Hello,
Migrations may take a bit of time to run. Did this eventually complete? If not, you can shell into your task container and run awx-manage showmigrations to see where it might be getting caught up. If that is the case, and migrations are not resolving, please provide us with the output of awx-manage showmigrations.

  • AWX Team
3

bash-5.1$ awx-manage migrate
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 289, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 270, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/init.py”, line 184, in manage
if (connection.pg_version // 10000) < 12:
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py”, line 15, in getattr
return getattr(self._connections[self._alias], item)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py”, line 57, in get
res = instance.dict[self.name] = self.func(instance)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 436, in pg_version
with self.temporary_connection():
File “/usr/lib64/python3.9/contextlib.py”, line 119, in enter
return next(self.gen)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 705, in temporary_connection
with self.cursor() as cursor:
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 330, in cursor
return self._cursor()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 306, in _cursor
self.ensure_connection()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 289, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py”, line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 289, in ensure_connection
self.connect()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 270, in connect
self.connection = self.get_new_connection(conn_params)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py”, line 26, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py”, line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 728, in connect
raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known

4

Hello,
If you are using an external database, could you please provide your secret Posgress configuration spec and your AWX spec? Please be sure to redact any sensitive information. This will better enable us to assist you.
-AWX Team

5

I am doing basic installation as documented in awx operator repo.
https://github.com/ansible/awx-operator/blob/devel/docs/installation/basic-install.md
I just cloned the repo and run 2 simple commands
`
$ make deploy # from root of repo
&&
$ kubectl apply -f awx-demo ## this also from root