Hello,
I’m having issues with the tasks showing as “failed” in the UI but they are still running in the background. According to UI they are failed, but the tasks are still running in the background and when all the processes catch up later it some how will update the status to sucucessful. However this process can take an hour or two to show the correct status.
These jobs tend to be jobs that are OVER 5 MIN… I haven’t seen this on jobs less than 5 minute run times.
Log from “docker logs -f awx_task”
[2018-02-09 03:23:16,883: DEBUG/ForkPoolWorker-5512] using channel_id: 1 [2018-02-09 03:23:16,884: DEBUG/ForkPoolWorker-5512] Channel open [2018-02-09 03:23:16,887: DEBUG/MainProcess] pidbox received method active_queues() [reply_to:{u'routing_key': u'86ab3717-36cc-3dc5-964e-37b60d736f6c', u'exchange': u'reply.celery.pidbox'} ticket:52a253c6-eb16-48c1-8811-9d59999396ab] [2018-02-09 03:23:16,889: ERROR/MainProcess] Control command error: error(104, 'Connection reset by peer') Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/celery/worker/pidbox.py", line 42, in on_message self.node.handle_message(body, message) File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 129, in handle_message return self.dispatch(**body) File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 112, in dispatch ticket=ticket) File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 135, in reply serializer=self.mailbox.serializer) File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/pidbox.py", line 265, in _publish_reply **opts File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 181, in publish exchange_name, declare, File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/messaging.py", line 203, in _publish mandatory=mandatory, immediate=immediate, File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py", line 1734, in _basic_publish (0, exchange, routing_key, mandatory, immediate), msg File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract_channel.py", line 50, in send_method conn.frame_writer(1, self.channel_id, sig, args, content) File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method_framing.py", line 166, in write_frame write(view[:offset]) File "/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py", line 258, in write self._write(s) File "/usr/lib64/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 104] Connection reset by peer [2018-02-09 03:23:16,892: DEBUG/MainProcess] Closed channel #3 [2018-02-09 03:23:16,892: DEBUG/MainProcess] using channel_id: 3 [2018-02-09 03:23:16,892: DEBUG/MainProcess] Channel open [2018-02-09 03:23:17,906: DEBUG/MainProcess] pidbox received method add_consumer(queue=u'Tower Servers', exchange=None, routing_key=None, exchange_type=u'direct') [reply_to:{u'routing_key': u'86ab3717-36cc-3dc5-964e-37b60d736f6c', u'exchange': u'reply.celery.pidbox'} ticket:52302123-3dca-4a2a-ad1d-5351e84fa669]