# All Jobs Started Failing

**URL:** <https://forum.ansible.com/t/all-jobs-started-failing/27676>\
**Category:** AWX Project\
**Tags:** awx\
**Created:** [May 31, 2018, 2:40pm UTC](https://forum.ansible.com/t/all-jobs-started-failing/27676 "2018-05-31T14:40:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cody\_John](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@Cody\_John](https://forum.ansible.com/u/Cody_John)\
**Post date:** [May 31, 2018, 2:40pm UTC](https://forum.ansible.com/t/all-jobs-started-failing/27676/1 "2018-05-31T14:40:08Z")

</div>

We’ve been using AWX successfully for awhile now, but something has happened and now every single job is failing. When trying to launch jobs (Templates, Project updates, anything), I just see this error:

stdout capture is missing

If I tail the awx\_task logs during execution, I see the following:

[2018-05-31 14:38:32,072: INFO/MainProcess] Received task: awx.main.scheduler.tasks.run\_job\_launch[872b1428-1222-45d2-a4ff-142267740786]

[2018-05-31 14:38:32,073: DEBUG/MainProcess] TaskPool: Apply \<function \_fast\_trace\_task at 0x3bebcf8\> (args:(‘awx.main.scheduler.tasks.run\_job\_launch’, ‘872b1428-1222-45d2-a4ff-142267740786’, {‘origin’: ‘gen144@awxweb’, ‘lang’: ‘py’, ‘task’: ‘awx.main.scheduler.tasks.run\_job\_launch’, ‘group’: None, ‘root\_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, u’delivery\_info’: {u’priority’: 0, u’redelivered’: False, u’routing\_key’: u’tower\_scheduler.job.launch’, u’exchange’: u’scheduler’}, ‘expires’: None, u’correlation\_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, ‘retries’: 0, ‘timelimit’: [None, None], ‘argsrepr’: ‘(26439,)’, ‘eta’: None, ‘parent\_id’: None, u’reply\_to’: ‘4ac6ad8f-40b2-3532-ae8b-c82596cd05b5’, ‘id’: ‘872b1428-1222-45d2-a4ff-142267740786’, ‘kwargsrepr’: ‘{}’}, u’[[26439], {}, {“chord”: null, “callbacks”: null, “errbacks”: null, “chain”: null}]', ‘application/json’, ‘utf-8’) kwargs:{})

[2018-05-31 14:38:32,074: DEBUG/MainProcess] Task accepted: awx.main.scheduler.tasks.run\_job\_launch[872b1428-1222-45d2-a4ff-142267740786] pid:15838

2018-05-31 14:38:32,094 DEBUG awx.main.scheduler Starting Scheduler

2018-05-31 14:38:32,094 DEBUG awx.main.scheduler Starting Scheduler

[2018-05-31 14:38:32,094: DEBUG/ForkPoolWorker-22499] Starting Scheduler

2018-05-31 14:38:32,191 DEBUG awx.main.scheduler Starting project\_update 26439 (pending) in group tower (remaining\_capacity=2225)

2018-05-31 14:38:32,191 DEBUG awx.main.scheduler Starting project\_update 26439 (pending) in group tower (remaining\_capacity=2225)

[2018-05-31 14:38:32,191: DEBUG/ForkPoolWorker-22499] Starting project\_update 26439 (pending) in group tower (remaining\_capacity=2225)

2018-05-31 14:38:32,198 INFO awx.main.scheduler Submitting project\_update 26439 (waiting) to instance group 1.

2018-05-31 14:38:32,198 INFO awx.main.scheduler Submitting project\_update 26439 (waiting) to instance group 1.

[2018-05-31 14:38:32,198: INFO/ForkPoolWorker-22499] Submitting project\_update 26439 (waiting) to instance group 1.

2018-05-31 14:38:32,244 DEBUG awx.main.scheduler project\_update 26439 (waiting) consumed 20 capacity units from tower with prior total of 0

2018-05-31 14:38:32,244 DEBUG awx.main.scheduler project\_update 26439 (waiting) consumed 20 capacity units from tower with prior total of 0

[2018-05-31 14:38:32,244: DEBUG/ForkPoolWorker-22499] project\_update 26439 (waiting) consumed 20 capacity units from tower with prior total of 0

[2018-05-31 14:38:32,262: DEBUG/ForkPoolWorker-22499] Start from server, version: 0.9, properties: {‘information’: ‘Licensed under the MPL. See [http://www.rabbitmq.com/](http://www.rabbitmq.com/)’, ‘product’: ‘RabbitMQ’, ‘copyright’: ‘Copyright (C) 2007-2017 Pivotal Software, Inc.’, ‘capabilities’: {‘exchange\_exchange\_bindings’: True, ‘connection.blocked’: True, ‘authentication\_failure\_close’: True, ‘direct\_reply\_to’: True, ‘basic.nack’: True, ‘per\_consumer\_qos’: True, ‘consumer\_priorities’: True, ‘consumer\_cancel\_notify’: True, ‘publisher\_confirms’: True}, ‘cluster\_name’: ‘rabbit@0a15ad2feb7d’, ‘platform’: ‘Erlang/OTP 19.2.1’, ‘version’: ‘3.6.14’}, mechanisms: [‘PLAIN’, ‘AMQPLAIN’], locales: [u’en\_US’]

[2018-05-31 14:38:32,263: DEBUG/ForkPoolWorker-22499] using channel\_id: 1

[2018-05-31 14:38:32,264: DEBUG/ForkPoolWorker-22499] Channel open

[2018-05-31 14:38:32,297: INFO/MainProcess] Received task: awx.main.tasks.run\_project\_update[4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c]

[2018-05-31 14:38:32,298: DEBUG/MainProcess] TaskPool: Apply \<function \_fast\_trace\_task at 0x3bebcf8\> (args:(‘awx.main.tasks.run\_project\_update’, ‘4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c’, {‘origin’: ‘gen15838@awx’, ‘lang’: ‘py’, ‘task’: ‘awx.main.tasks.run\_project\_update’, ‘group’: None, ‘root\_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, u’delivery\_info’: {u’priority’: None, u’redelivered’: False, u’routing\_key’: u’tower’, u’exchange’: u’‘}, ‘expires’: None, u’correlation\_id’: ‘4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c’, ‘retries’: 0, ‘timelimit’: [None, None], ‘argsrepr’: ‘[26439]’, ‘eta’: None, ‘parent\_id’: ‘872b1428-1222-45d2-a4ff-142267740786’, u’reply\_to’: ‘f021af83-1abb-3d85-b658-d49f1be8693c’, ‘id’: ‘4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c’, ‘kwargsrepr’: ‘{}’}, u’[[26439], {}, {“chord”: null, “callbacks”: [{“chord\_size”: null, “task”: “awx.main.tasks.handle\_work\_success”, “subtask\_type”: null, “kwargs”: {“task\_actual”: {“type”: “project\_update”, “id”: 26439}}, “args”: , “options”: {}, “immutable”: false}], “errbacks”: [{“chord\_size”: null, “task”: “awx.main.tasks.handle\_work\_error”, “subtask\_type”: null, “kwargs”:… kwargs:{})

[2018-05-31 14:38:32,299: INFO/ForkPoolWorker-22499] Task awx.main.scheduler.tasks.run\_job\_launch[872b1428-1222-45d2-a4ff-142267740786] succeeded in 0.224176254123s: None

[2018-05-31 14:38:32,299: DEBUG/MainProcess] Task accepted: awx.main.tasks.run\_project\_update[4f26a8b4-8ee1-4fd7-8a33-de3e4deafd6c] pid:15837

2018-05-31 14:38:33,448 INFO awx.main.tasks project\_update 26439 (running) Could not find scm revision in check

2018-05-31 14:38:33,448 INFO awx.main.tasks project\_update 26439 (running) Could not find scm revision in check

[2018-05-31 14:38:33,448: INFO/ForkPoolWorker-22498] project\_update 26439 (running) Could not find scm revision in check

[2018-05-31 14:38:33,653: WARNING/ForkPoolWorker-22498] /var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: TypeError(‘handle\_work\_error() takes at least 4 arguments (4 given)’,):

Traceback (most recent call last):

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 388, in trace\_task

I, R, state, retval = on\_error(task\_request, exc, uuid)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 330, in on\_error

task, request, eager=eager, call\_errbacks=call\_errbacks,

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 164, in handle\_error\_state

call\_errbacks=call\_errbacks)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 209, in handle\_failure

call\_errbacks=call\_errbacks,

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 168, in mark\_as\_failure

self.\_call\_task\_errbacks(request, exc, traceback)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 175, in \_call\_task\_errbacks

errback(request, exc, traceback)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/canvas.py”, line 178, in **call**

return self.type(\*args, \*\*kwargs)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 630, in **protected\_call**

return orig(self, \*args, \*\*kwargs)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/task.py”, line 380, in **call**

return self.run(\*args, \*\*kwargs)

TypeError: handle\_work\_error() takes at least 4 arguments (4 given)

exc, exc\_info.traceback)))

[2018-05-31 14:38:33,675: ERROR/MainProcess] Pool callback raised exception: TypeError(‘handle\_work\_error() takes at least 4 arguments (4 given)’,)

Traceback (most recent call last):

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/billiard/pool.py”, line 1747, in safe\_apply\_callback

fun(\*args, \*\*kwargs)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/request.py”, line 366, in on\_failure

self.id, exc, request=self, store\_result=self.store\_errors,

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 168, in mark\_as\_failure

self.\_call\_task\_errbacks(request, exc, traceback)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/backends/base.py”, line 175, in \_call\_task\_errbacks

errback(request, exc, traceback)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/canvas.py”, line 178, in **call**

return self.type(\*args, \*\*kwargs)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/trace.py”, line 630, in **protected\_call**

return orig(self, \*args, \*\*kwargs)

File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/app/task.py”, line 380, in **call**

return self.run(\*args, \*\*kwargs)

TypeError: handle\_work\_error() takes at least 4 arguments (4 given)

[

Any ideas?

---

<div class="post-metadata">

**Author:** ![AlanCoding](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/alancoding/32/301_2.png) [@AlanCoding](https://forum.ansible.com/u/AlanCoding)\
**Post date:** [May 31, 2018, 2:51pm UTC](https://forum.ansible.com/t/all-jobs-started-failing/27676/2 "2018-05-31T14:51:03Z")

</div>

This looks like it contains an error that was fixed with:

[https://github.com/ansible/awx/pull/921](https://github.com/ansible/awx/pull/921)

Are you using the most recent source?

---

<div class="post-metadata">

**Author:** ![Cody\_John](https://avatars.discourse-cdn.com/v4/letter/c/8e7dd6/32.png) [@Cody\_John](https://forum.ansible.com/u/Cody_John)\
**Post date:** [May 31, 2018, 5:14pm UTC](https://forum.ansible.com/t/all-jobs-started-failing/27676/3 "2018-05-31T17:14:56Z")

</div>

No, I’m using 1.0.1.231. From what I understand, upgrades are going to break my database…

---

<div class="post-metadata">

**Author:** ![jowestco](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jowestco](https://forum.ansible.com/u/jowestco)\
**Post date:** [May 31, 2018, 5:16pm UTC](https://forum.ansible.com/t/all-jobs-started-failing/27676/4 "2018-05-31T17:16:49Z")

</div>

You can try a data migration: [https://github.com/ansible/awx/blob/devel/DATA\_MIGRATION.md](https://github.com/ansible/awx/blob/devel/DATA_MIGRATION.md)

-John