Install AWX 24.0.0 using AWX Operator. K8s Version: v1.29.1
When the installation finishes, try to run the default job: Demo Job Template and it remains pending.
The awx-task container logs say the following:
`[wait-for-migrations] Waiting for database migrations…
[wait-for-migrations] Attempt 1
Instance Group already registered controlplane
Instance Group already registered default
Successfully registered instance None
(changed: True)
2024-03-25 01:52:48,479 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-25 01:52:48,479 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-03-25 01:52:48,486 INFO RPC interface ‘supervisor’ initialized
2024-03-25 01:52:48,486 INFO RPC interface ‘supervisor’ initialized
2024-03-25 01:52:48,486 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2024-03-25 01:52:48,486 CRIT Server ‘unix_http_server’ running without any HTTP authentication checking
2024-03-25 01:52:48,486 INFO supervisord started with pid 7
2024-03-25 01:52:48,486 INFO supervisord started with pid 7
2024-03-25 01:52:49,491 INFO spawned: ‘superwatcher’ with pid 30
2024-03-25 01:52:49,491 INFO spawned: ‘superwatcher’ with pid 30
2024-03-25 01:52:49,494 INFO spawned: ‘dispatcher’ with pid 31
2024-03-25 01:52:49,494 INFO spawned: ‘dispatcher’ with pid 31
2024-03-25 01:52:49,497 INFO spawned: ‘callback-receiver’ with pid 32
2024-03-25 01:52:49,497 INFO spawned: ‘callback-receiver’ with pid 32
2024-03-25 01:52:49,500 INFO spawned: ‘wsrelay’ with pid 33
2024-03-25 01:52:49,500 INFO spawned: ‘wsrelay’ with pid 33
READY
2024-03-25 01:52:50,503 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-25 01:52:50,503 INFO success: superwatcher entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-03-25 01:52:53,200 INFO [-] awx.main.dispatch Running worker dispatcher listening to queues [‘tower_broadcast_all’, ‘tower_settings_change’, ‘awx-task-788785d975-nx2z8’]
2024-03-25 01:52:53,202 INFO [-] awx.main.commands.run_callback_receiver Callback receiver started with pid=32
2024-03-25 01:52:53,273 INFO [-] awx.main.wsrelay Active instance with hostname awx-task-788785d975-nx2z8 is registered.
2024-03-25 01:52:53,529 WARNING [-] awx.main.tasks.system Rejoining the cluster as instance awx-task-788785d975-nx2z8. Prior last_seen None
2024-03-25 01:52:53,572 INFO [-] awx.main.dispatch Dispatcher listener connection established
2024-03-25 01:52:53,572 WARNING [-] awx.main.dispatch.periodic Scheduler next run unexpectedly over 20 seconds in future: 25.627272129058838
2024-03-25 01:53:19,605 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-03-25 01:53:19,605 INFO success: dispatcher entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-03-25 01:53:19,605 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-03-25 01:53:19,605 INFO success: callback-receiver entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-03-25 01:53:19,605 INFO success: wsrelay entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-03-25 01:53:19,605 INFO success: wsrelay entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
2024-03-25 01:53:56,408 WARNING [-] awx.main.tasks.system Heartbeat skew - interval=62.8150, expected=60
2024-03-25 01:54:02,325 ERROR [-] awx.main.dispatch Worker failed to run task awx.main.tasks.system.awx_k8s_reaper(*, **{}
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py”, line 98, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/server_cursor.py”, line 321, in fetchmany
recs = self._conn.wait(self._fetch_gen(size))
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 957, in wait
return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/waiting.py”, line 283, in wait_epoll
s = gen.send(ready)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/server_cursor.py”, line 172, in _fetch_gen
res = yield from self._conn._exec_command(query, result_format=self._format)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 466, in _exec_command
raise e.error_from_result(result, encoding=pgconn_encoding(self.pgconn))
psycopg.errors.InvalidCursorName: cursor “_django_curs_140304504547136_sync_1” does not exist
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 2093, in cursor_iter
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 2093, in
for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py”, line 98, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py”, line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py”, line 98, in inner
return func(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/server_cursor.py”, line 321, in fetchmany
recs = self._conn.wait(self._fetch_gen(size))
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 957, in wait
return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/waiting.py”, line 283, in wait_epoll
s = gen.send(ready)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/server_cursor.py”, line 172, in _fetch_gen
res = yield from self._conn._exec_command(query, result_format=self._format)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 466, in _exec_command
raise e.error_from_result(result, encoding=pgconn_encoding(self.pgconn))
django.db.utils.ProgrammingError: cursor “_django_curs_140304504547136_sync_1” does not exist
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/worker/task.py”, line 103, in perform_work
result = self.run_callable(body)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/worker/task.py”, line 78, in run_callable
return _call(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/system.py”, line 700, in awx_k8s_reaper
for group in InstanceGroup.objects.filter(is_container_group=True).iterator():
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py”, line 516, in _iterator
yield from iterable
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/query.py”, line 121, in iter
for row in compiler.results_iter(results):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 1496, in apply_converters
for row in map(list, rows):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/models/sql/compiler.py”, line 2096, in cursor_iter
cursor.close()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/server_cursor.py”, line 269, in close
self._conn.wait(self._close_gen())
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 957, in wait
return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/waiting.py”, line 283, in wait_epoll
s = gen.send(ready)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/server_cursor.py”, line 158, in _close_gen
yield from self._conn._exec_command(query)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 466, in _exec_command
raise e.error_from_result(result, encoding=pgconn_encoding(self.pgconn))
psycopg.errors.InvalidCursorName: cursor “_django_curs_140304504547136_sync_1” does not exist
2024-03-25 01:59:56,379 ERROR [-] awx.main.dispatch project_update 1 (failed) is no longer waiting; reaping
`
AWX version
24.0.0
How i can fix it?