Hi,
we’ve had an issue with AWX Ansible over the time with disappearing job outputs in UI. Currently we’re running AWX 8.0.0 with external Postgresql 10. We are getting rid of the issue by stopping the docker containers and database and then restarting database and after that composers. During the time when the problem is occurring we’re facing in awx_task containers logs these messages below but not exactly at same timestamps with the start/stop timestamps of the jobs (close though). Could these be related?
2019-11-21 01:59:45,472 WARNING awx.main.commands.run_callback_receiver could not write to queue 2
2019-11-21 01:59:45,472 WARNING awx.main.commands.run_callback_receiver could not write to queue 2
2019-11-21 01:59:45,489 WARNING awx.main.commands.run_callback_receiver detail: Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 263, in write
self.workers[queue_actual].put(body)
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 81, in put
self.calculate_managed_tasks()
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 127, in calculate_managed_tasks
del self.managed_tasks[uuid]
KeyError: ‘0242ac12-0005-08f5-8889-00000000037f_159’
2019-11-21 02:33:15,470 WARNING awx.main.commands.run_callback_receiver detail: Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 263, in write
self.workers[queue_actual].put(body)
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 81, in put
self.calculate_managed_tasks()
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 127, in calculate_managed_tasks
del self.managed_tasks[uuid]
KeyError: ‘bc56993f-e059-40fb-a9cf-bc6d82494e76’
2019-11-21 02:33:15,470 WARNING awx.main.commands.run_callback_receiver detail: Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 263, in write
self.workers[queue_actual].put(body)
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 81, in put
self.calculate_managed_tasks()
File “/var/lib/awx/venv/awx/lib64/python3.6/site-packages/awx/main/dispatch/pool.py”, line 127, in calculate_managed_tasks
del self.managed_tasks[uuid]
KeyError: ‘bc56993f-e059-40fb-a9cf-bc6d82494e76’
best regards,
Jarkko Perttula