I had a working AWX instance last week, now the awx-web pod doesn’t start anymore - “CrashLoopBackOff”, here is the output of
kubectl -n awx logs -f awx-web-5cb9b5b89b-z4x6h:
2024-05-31 06:27:21,905 INFO RPC interface 'supervisor' initialized
2024-05-31 06:27:21,905 INFO RPC interface 'supervisor' initialized
2024-05-31 06:27:21,905 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-05-31 06:27:21,905 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-05-31 06:27:21,906 INFO supervisord started with pid 7
2024-05-31 06:27:21,906 INFO supervisord started with pid 7
2024-05-31 06:27:22,910 INFO spawned: 'superwatcher' with pid 13
2024-05-31 06:27:22,910 INFO spawned: 'superwatcher' with pid 13
2024-05-31 06:27:22,915 INFO spawned: 'nginx' with pid 14
2024-05-31 06:27:22,915 INFO spawned: 'nginx' with pid 14
2024-05-31 06:27:22,919 INFO spawned: 'uwsgi' with pid 15
2024-05-31 06:27:22,919 INFO spawned: 'uwsgi' with pid 15
2024-05-31 06:27:22,921 INFO spawned: 'daphne' with pid 16
2024-05-31 06:27:22,921 INFO spawned: 'daphne' with pid 16
2024-05-31 06:27:22,923 INFO spawned: 'awx-cache-clear' with pid 17
2024-05-31 06:27:22,923 INFO spawned: 'awx-cache-clear' with pid 17
2024-05-31 06:27:22,925 INFO spawned: 'ws-heartbeat' with pid 18
2024-05-31 06:27:22,925 INFO spawned: 'ws-heartbeat' with pid 18
READY
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.21 (64bit) on [Fri May 31 06:27:22 2024] ***
compiled with version: 11.4.1 20230605 (Red Hat 11.4.1-2) on 02 January 2024 18:59:48
os: Linux-5.14.0-427.18.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Mon May 27 16:35:12 UTC 2024
nodename: awx-web-5cb9b5b89b-z4x6h
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8050 fd 3
Python version: 3.9.18 (main, Sep 7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0xd33620
your server socket listen backlog is limited to 128 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 15)
spawned uWSGI worker 1 (pid: 20, cores: 1)
spawned uWSGI worker 2 (pid: 21, cores: 1)
spawned uWSGI worker 3 (pid: 22, cores: 1)
spawned uWSGI worker 4 (pid: 23, cores: 1)
spawned uWSGI worker 5 (pid: 24, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known
2024-05-31 06:27:27,481 WARN exited: awx-cache-clear (exit status 1; not expected)
2024-05-31 06:27:27,481 WARN exited: awx-cache-clear (exit status 1; not expected)
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in <module>
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
if (connection.pg_version // 10000) < 12:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
return getattr(self._connections[self._alias], item)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
with self.temporary_connection():
File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
with self.cursor() as cursor:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known
2024-05-31 06:27:27,634 WARN exited: ws-heartbeat (exit status 1; not expected)
2024-05-31 06:27:27,634 WARN exited: ws-heartbeat (exit status 1; not expected)
2024-05-31 06:27:29,640 INFO spawned: 'awx-cache-clear' with pid 59
2024-05-31 06:27:29,640 INFO spawned: 'awx-cache-clear' with pid 59
2024-05-31 06:27:29,644 INFO spawned: 'ws-heartbeat' with pid 60
2024-05-31 06:27:29,644 INFO spawned: 'ws-heartbeat' with pid 60
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in <module>
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
if (connection.pg_version // 10000) < 12:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
return getattr(self._connections[self._alias], item)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
with self.temporary_connection():
File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
with self.cursor() as cursor:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in <module>
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
if (connection.pg_version // 10000) < 12:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
return getattr(self._connections[self._alias], item)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
with self.temporary_connection():
File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
with self.cursor() as cursor:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known
2024-05-31 06:27:30,459 WARN exited: awx-cache-clear (exit status 1; not expected)
2024-05-31 06:27:30,459 WARN exited: awx-cache-clear (exit status 1; not expected)
2024-05-31 06:27:30,473 WARN exited: ws-heartbeat (exit status 1; not expected)
2024-05-31 06:27:30,473 WARN exited: ws-heartbeat (exit status 1; not expected)
2024-05-31 06:27:33,479 INFO spawned: 'awx-cache-clear' with pid 65
2024-05-31 06:27:33,479 INFO spawned: 'awx-cache-clear' with pid 65
2024-05-31 06:27:33,484 INFO spawned: 'ws-heartbeat' with pid 66
2024-05-31 06:27:33,484 INFO spawned: 'ws-heartbeat' with pid 66
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in <module>
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
if (connection.pg_version // 10000) < 12:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
return getattr(self._connections[self._alias], item)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
with self.temporary_connection():
File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
with self.cursor() as cursor:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
psycopg.OperationalError: connection is bad: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in <module>
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 159, in manage
if (connection.pg_version // 10000) < 12:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/connection.py", line 15, in __getattr__
return getattr(self._connections[self._alias], item)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/functional.py", line 57, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 436, in pg_version
with self.temporary_connection():
File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 705, in temporary_connection
with self.cursor() as cursor:
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 330, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 306, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 289, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py", line 270, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/base.py", line 275, in get_new_connection
connection = self.Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py", line 728, in connect
raise ex.with_traceback(None)
django.db.utils.OperationalError: connection is bad: Name or service not known
2024-05-31 06:27:34,341 WARN exited: awx-cache-clear (exit status 1; not expected)
2024-05-31 06:27:34,341 WARN exited: awx-cache-clear (exit status 1; not expected)
2024-05-31 06:27:34,342 INFO gave up: awx-cache-clear entered FATAL state, too many start retries too quickly
2024-05-31 06:27:34,342 INFO gave up: awx-cache-clear entered FATAL state, too many start retries too quickly
Processing Event: ver:3.0 server:supervisor serial:0 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:72
2024-05-31 06:27:34,343 WARN exited: ws-heartbeat (exit status 1; not expected)
2024-05-31 06:27:34,343 WARN exited: ws-heartbeat (exit status 1; not expected)
2024-05-31 06:27:35,344 INFO gave up: ws-heartbeat entered FATAL state, too many start retries too quickly
2024-05-31 06:27:35,344 INFO gave up: ws-heartbeat entered FATAL state, too many start retries too quickly
2024-05-31 06:27:35,344 WARN received SIGQUIT indicating exit request
2024-05-31 06:27:35,344 WARN received SIGQUIT indicating exit request
2024-05-31 06:27:35,344 INFO waiting for superwatcher, nginx, uwsgi, daphne to die
2024-05-31 06:27:35,344 INFO waiting for superwatcher, nginx, uwsgi, daphne to die
...brutally killing workers...
2024-05-31 06:27:35,358 INFO stopped: nginx (exit status 0)
2024-05-31 06:27:35,358 INFO stopped: nginx (exit status 0)
2024-05-31 06:27:35,400 WARNING [-] awx.conf.settings Database settings are not available, using defaults. error: connection is bad: Name or service not known
2024-05-31 06:27:35,400 WARNING Database settings are not available, using defaults. error: connection is bad: Name or service not known
2024-05-31 06:27:35,345 INFO [-] daphne.server Killed 0 pending application instances
2024-05-31 06:27:35,345 INFO Killed 0 pending application instances
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
binary reloading uWSGI...
chdir() to /var/lib/awx
closing all non-uwsgi socket fds > 2 (max_fd = 1048576)...
found fd 3 mapped to socket 0 (127.0.0.1:8050)
2024-05-31 06:27:36,346 INFO stopped: daphne (exit status 0)
2024-05-31 06:27:36,346 INFO stopped: daphne (exit status 0)
running /var/lib/awx/venv/awx/bin/uwsgi
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.21 (64bit) on [Fri May 31 06:27:36 2024] ***
compiled with version: 11.4.1 20230605 (Red Hat 11.4.1-2) on 02 January 2024 18:59:48
os: Linux-5.14.0-427.18.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Mon May 27 16:35:12 UTC 2024
nodename: awx-web-5cb9b5b89b-z4x6h
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address 127.0.0.1:8050 fd 3
Python version: 3.9.18 (main, Sep 7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1747ef0
your server socket listen backlog is limited to 128 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 15)
spawned uWSGI worker 1 (pid: 72, cores: 1)
spawned uWSGI worker 2 (pid: 73, cores: 1)
spawned uWSGI worker 3 (pid: 74, cores: 1)
spawned uWSGI worker 4 (pid: 75, cores: 1)
spawned uWSGI worker 5 (pid: 76, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2024-05-31 06:27:38,860 INFO waiting for superwatcher, uwsgi to die
2024-05-31 06:27:38,860 INFO waiting for superwatcher, uwsgi to die
WSGI app 0 (mountpoint='/') ready in 4 seconds on interpreter 0x1747ef0 pid: 75 (default app)
WSGI app 0 (mountpoint='/') ready in 4 seconds on interpreter 0x1747ef0 pid: 73 (default app)
WSGI app 0 (mountpoint='/') ready in 4 seconds on interpreter 0x1747ef0 pid: 72 (default app)
WSGI app 0 (mountpoint='/') ready in 4 seconds on interpreter 0x1747ef0 pid: 76 (default app)
WSGI app 0 (mountpoint='/') ready in 4 seconds on interpreter 0x1747ef0 pid: 74 (default app)
2024-05-31 06:27:42,533 INFO waiting for superwatcher, uwsgi to die
2024-05-31 06:27:42,533 INFO waiting for superwatcher, uwsgi to die
2024-05-31 06:27:45,538 WARN killing 'uwsgi' (15) with SIGKILL
2024-05-31 06:27:45,538 WARN killing 'uwsgi' (15) with SIGKILL
2024-05-31 06:27:45,538 INFO waiting for superwatcher, uwsgi to die
2024-05-31 06:27:45,538 INFO waiting for superwatcher, uwsgi to die
2024-05-31 06:27:45,560 WARN stopped: uwsgi (terminated by SIGKILL)
2024-05-31 06:27:45,560 WARN stopped: uwsgi (terminated by SIGKILL)
2024-05-31 06:27:45,560 WARN stopped: superwatcher (terminated by SIGTERM)
2024-05-31 06:27:45,560 WARN stopped: superwatcher (terminated by SIGTERM)
I have also tried an update to the next Operator version.
Currently installed version: 2.10.0
Do any of you have any ideas? Maybe a certificate issue?
Thanks in advance,
Lukas