Hi Folks,
Need some help. We are doing some customization for AWX/Tower .
For creating build we are following, https://github.com/ansible/awx/blob/devel/CONTRIBUTING.md .
Now, while running the job it is stuck and not getting started(Screenshot below). It is my first job, so no point of multiple jobs queued. Also, I have waited for almost half an hour.
AWX Version : 15.0.1
Any help would be much appreciated
Thanks.
Have you looked at the logs? Has any jobs worked? What customization’s have you made?
Hi.
We have the same issue… for diagnostics we did:
bash-4.4# awx-manage run_dispatcher --status
in the aws_task container and there is:
2020-12-17 12:44:15,779 ERROR awx.main.dispatch dispatcher did not reply within 5s
Traceback (most recent call last):
File “/usr/bin/awx-manage”, line 8, in
sys.exit(manage())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py”, line 154, in manage
execute_from_command_line(sys.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 381, in execute_from_command_line
utility.execute()
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py”, line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 323, in run_from_argv
self.execute(*args, **cmd_options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py”, line 364, in execute
output = self.handle(*args, **options)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/management/commands/run_dispatcher.py”, line 37, in handle
print(Control(‘dispatcher’).status())
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/control.py”, line 24, in status
return self.control_with_reply(‘status’, *args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/dispatch/control.py”, line 46, in control_with_reply
raise RuntimeError(f"{self.service} did not reply within {timeout}s")
RuntimeError: dispatcher did not reply within 5s
but, for example
bash-4.4# supervisorctl status
awx-config-watcher RUNNING pid 81, uptime 0:15:25
tower-processes:awx-rsyslogd RUNNING pid 147, uptime 0:05:03
tower-processes:daphne RUNNING pid 84, uptime 0:15:25
tower-processes:nginx RUNNING pid 82, uptime 0:15:25
tower-processes:uwsgi RUNNING pid 83, uptime 0:15:25
tower-processes:wsbroadcast RUNNING pid 85, uptime 0:15:25
looks like OK.
anyway, the logs are empty:
less /var/log/tower/dispatcher.log
2020-12-18 09:48:35,223 WARNING awx.main.dispatch.periodic PID:212 periodic beat started
2020-12-18 09:48:35,258 WARNING awx.main.dispatch PID:206 scaling up worker pid:213
2020-12-18 09:48:35,264 WARNING awx.main.dispatch PID:206 scaling up worker pid:214
2020-12-18 09:48:35,269 WARNING awx.main.dispatch PID:206 scaling up worker pid:215
2020-12-18 09:48:35,275 WARNING awx.main.dispatch PID:206 scaling up worker pid:216
2020-12-18 09:48:35,278 WARNING awx.main.dispatch PID:206 Running worker dispatcher listening to
queues [‘tower_broadcast_all’, ‘awx’]
What diagnostic methods are available for the pending status?
суббота, 21 ноября 2020 г. в 03:47:06 UTC+5, wme...@why-me.com:
Hi. We found that awx used the PubSub and it is incompatible with pgbound. Swithcing to default PG port solved our issue.
пятница, 18 декабря 2020 г. в 14:58:38 UTC+5, Дмитрий Рубцов:
Hi,
We identified our Redis socket was not getting opened.
AWX Jobs get queued in redis, if the socket will not available then jobs will not run.
Can you please check whether
Hi Дмитрий,
That’s great.
I saw your mail so late. Apologies.