We’ve been using AWX successfully for awhile now, but something has happened and now every single job is failing. When trying to launch jobs (Templates, Project updates, anything), I just see this error:
stdout capture is missing
If I tail the awx_task logs during execution, I see the following:
[2018-05-31 14:38:32,072: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run_job_launch[872b1428-1222-45d2-a4ff-142267740786]
[2018-05-31 14:38:32,073: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x3bebcf8> (args:(‘awx.main.scheduler.tasks.run_job_launch’, ‘872b1428-1222-45d2-a4ff-142267740786’, {‘origin’: ‘gen144@awxweb’, ‘lang’: ‘py’, ‘task’: ‘awx.main.scheduler.tasks.run_job_launch’, ‘group’: None, ‘root_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, u’delivery_info’: {u’priority’: 0, u’redelivered’: False, u’routing_key’: u’tower_scheduler.job.launch’, u’exchange’: u’scheduler’}, ‘expires’: None, u’correlation_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, ‘retries’: 0, ‘timelimit’: [None, None], ‘argsrepr’: ‘(26439,)’, ‘eta’: None, ‘parent_id’: None, u’reply_to’: ‘4ac6ad8f-40b2-3532-ae8b-c82596cd05b5’, ‘id’: ‘872b1428-1222-45d2-a4ff-142267740786’, ‘kwargsrepr’: ‘{}’}, u’[[26439], {}, {“chord”: null, “callbacks”: null, “errbacks”: null, “chain”: null}]', ‘application/json’, ‘utf-8’) kwargs:{})
[2018-05-31 14:38:32,074: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run_job_launch[872b1428-1222-45d2-a4ff-142267740786] pid:15838
2018-05-31 14:38:32,094 DEBUG awx.main.scheduler Starting Scheduler
2018-05-31 14:38:32,094 DEBUG awx.main.scheduler Starting Scheduler
[2018-05-31 14:38:32,094: DEBUG/ForkPoolWorker-22499] Starting Scheduler
2018-05-31 14:38:32,191 DEBUG awx.main.scheduler Starting project_update 26439 (pending) in group tower (remaining_capacity=2225)
2018-05-31 14:38:32,191 DEBUG awx.main.scheduler Starting project_update 26439 (pending) in group tower (remaining_capacity=2225)
[2018-05-31 14:38:32,191: DEBUG/ForkPoolWorker-22499] Starting project_update 26439 (pending) in group tower (remaining_capacity=2225)
2018-05-31 14:38:32,198 INFO awx.main.scheduler Submitting project_update 26439 (waiting) to instance group 1.
2018-05-31 14:38:32,198 INFO awx.main.scheduler Submitting project_update 26439 (waiting) to instance group 1.
[2018-05-31 14:38:32,198: INFO/ForkPoolWorker-22499] Submitting project_update 26439 (waiting) to instance group 1.
2018-05-31 14:38:32,244 DEBUG awx.main.scheduler project_update 26439 (waiting) consumed 20 capacity units from tower with prior total of 0
2018-05-31 14:38:32,244 DEBUG awx.main.scheduler project_update 26439 (waiting) consumed 20 capacity units from tower with prior total of 0
[2018-05-31 14:38:32,244: DEBUG/ForkPoolWorker-22499] project_update 26439 (waiting) consumed 20 capacity units from tower with prior total of 0
[2018-05-31 14:38:32,262: DEBUG/ForkPoolWorker-22499] Start from server, version: 0.9, properties: {‘information’: ‘Licensed under the MPL. See http://www.rabbitmq.com/’, ‘product’: ‘RabbitMQ’, ‘copyright’: ‘Copyright (C) 2007-2017 Pivotal Software, Inc.’, ‘capabilities’: {‘exchange_exchange_bindings’: True, ‘connection.blocked’: True, ‘authentication_failure_close’: True, ‘direct_reply_to’: True, ‘basic.nack’: True, ‘per_consumer_qos’: True, ‘consumer_priorities’: True, ‘consumer_cancel_notify’: True, ‘publisher_confirms’: True}, ‘cluster_name’: ‘rabbit@0a15ad2feb7d’, ‘platform’: ‘Erlang/OTP 19.2.1’, ‘version’: ‘3.6.14’}, mechanisms: [‘PLAIN’, ‘AMQPLAIN’], locales: [u’en_US’]
[2018-05-31 14:38:32,263: DEBUG/ForkPoolWorker-22499] using channel_id: 1
[2018-05-31 14:38:32,264: DEBUG/ForkPoolWorker-22499] Channel open
[2018-05-31 14:38:32,297: INFO/MainProcess] Received task: awx.main.tasks.run_project_update[4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c]
[2018-05-31 14:38:32,298: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x3bebcf8> (args:(‘awx.main.tasks.run_project_update’, ‘4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c’, {‘origin’: ‘gen15838@awx’, ‘lang’: ‘py’, ‘task’: ‘awx.main.tasks.run_project_update’, ‘group’: None, ‘root_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, u’delivery_info’: {u’priority’: None, u’redelivered’: False, u’routing_key’: u’tower’, u’exchange’: u’‘}, ‘expires’: None, u’correlation_id’: ‘4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c’, ‘retries’: 0, ‘timelimit’: [None, None], ‘argsrepr’: ‘[26439]’, ‘eta’: None, ‘parent_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, u’reply_to’: ‘f021af83-1abb-3d85-b658-d49f1be8693c’, ‘id’: ‘4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c’, ‘kwargsrepr’: ‘{}’}, u’[[26439], {}, {“chord”: null, “callbacks”: [{“chord_size”: null, “task”: “awx.main.tasks.handle_work_success”, “subtask_type”: null, “kwargs”: {“task_actual”: {“type”: “project_update”, “id”: 26439}}, “args”: , “options”: {}, “immutable”: false}], “errbacks”: [{“chord_size”: null, “task”: “awx.main.tasks.handle_work_error”, “subtask_type”: null, “kwargs”:… kwargs:{})
[2018-05-31 14:38:32,299: INFO/ForkPoolWorker-22499] Task awx.main.scheduler.tasks.run_job_launch[872b1428-1222-45d2-a4ff-142267740786] succeeded in 0.224176254123s: None
[2018-05-31 14:38:32,299: DEBUG/MainProcess] Task accepted: awx.main.tasks.run_project_update[4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c] pid:15837
2018-05-31 14:38:33,448 INFO awx.main.tasks project_update 26439 (running) Could not find scm revision in check
2018-05-31 14:38:33,448 INFO awx.main.tasks project_update 26439 (running) Could not find scm revision in check
[2018-05-31 14:38:33,448: INFO/ForkPoolWorker-22498] project_update 26439 (running) Could not find scm revision in check
[2018-05-31 14:38:33,653: WARNING/ForkPoolWorker-22498] /var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: TypeError(‘handle_work_error() takes at least 4 arguments (4 given)’,):
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 388, in trace_task
I, R, state, retval = on_error(task_request, exc, uuid)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 330, in on_error
task, request, eager=eager, call_errbacks=call_errbacks,
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 164, in handle_error_state
call_errbacks=call_errbacks)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 209, in handle_failure
call_errbacks=call_errbacks,
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 168, in mark_as_failure
self._call_task_errbacks(request, exc, traceback)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 175, in _call_task_errbacks
errback(request, exc, traceback)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/canvas.py”, line 178, in call
return self.type(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 630, in protected_call
return orig(self, *args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/task.py”, line 380, in call
return self.run(*args, **kwargs)
TypeError: handle_work_error() takes at least 4 arguments (4 given)
exc, exc_info.traceback)))
[2018-05-31 14:38:33,675: ERROR/MainProcess] Pool callback raised exception: TypeError(‘handle_work_error() takes at least 4 arguments (4 given)’,)
Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/billiard/pool.py”, line 1747, in safe_apply_callback
fun(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/request.py”, line 366, in on_failure
self.id, exc, request=self, store_result=self.store_errors,
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 168, in mark_as_failure
self._call_task_errbacks(request, exc, traceback)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 175, in _call_task_errbacks
errback(request, exc, traceback)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/canvas.py”, line 178, in call
return self.type(*args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 630, in protected_call
return orig(self, *args, **kwargs)
File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/task.py”, line 380, in call
return self.run(*args, **kwargs)
TypeError: handle_work_error() takes at least 4 arguments (4 given)
[
Any ideas?