# Websockets not working after a little while

**URL:** <https://forum.ansible.com/t/websockets-not-working-after-a-little-while/27945>\
**Category:** AWX Project\
**Tags:** windows, awx\
**Created:** [July 4, 2018, 1:55am UTC](https://forum.ansible.com/t/websockets-not-working-after-a-little-while/27945 "2018-07-04T01:55:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![William\_Easton](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@William\_Easton](https://forum.ansible.com/u/William_Easton)\
**Post date:** [July 4, 2018, 1:55am UTC](https://forum.ansible.com/t/websockets-not-working-after-a-little-while/27945/1 "2018-07-04T01:55:19Z")

</div>

Hi,

I’m having an issue where websockets stop working ~20-60 minutes after the application has been deployed. This impacts the task containers ability to post job stdout as well as removes the ability to view job details in the UI.

I saw that an issue was opened here: [https://github.com/ansible/awx/issues/1861](https://github.com/ansible/awx/issues/1861) and have added comments with my findings as I go.

I have enabled verbose logging on Daphne but I’m a little perplexed.

Daphne is showing that the websocket is opened:

> 2018-06-27 03:19:21,372 DEBUG Upgraded connection daphne.response.XbupPxYRcS!aPmLgJGDZd to WebSocket daphne.response.XbupPxYRcS!hTzJudfDoM

Then suddenly nginx reports that the client closed the connection

> 10.255.0.2 - - [27/Jun/2018:03:19:24 +0000] “GET /websocket/ HTTP/1.1” 499 0 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17682”

And then daphne reports that the websocket has closed

> 2018-06-27 03:19:25,571 DEBUG WebSocket closed for daphne.response.XbupPxYRcS!hTzJudfDoM

The browser itself reports:

> WebSocket connection to ‘wss://…/websocket/’ failed: WebSocket is closed before the connection is established.

And the Task container reports (when running the job):

> [2018-07-02 19:03:47,717: DEBUG/Worker-4] using channel\_id: 2  
> 2018-07-02 19:03:47,718 ERROR awx.main.models.unified\_jobs job 15 (running) failed to emit channel msg about status change  
> Traceback (most recent call last):  
> File “/usr/lib/python2.7/site-packages/awx/main/models/unified\_jobs.py”, line 1169, in \_websocket\_emit\_status  
> emit\_channel\_notification(‘jobs-status\_changed’, status\_data)  
> File “/usr/lib/python2.7/site-packages/awx/main/consumers.py”, line 70, in emit\_channel\_notification  
> Group(group).send({“text”: json.dumps(payload, cls=DjangoJSONEncoder)})  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/channels/channel.py”, line 88, in send  
> self.channel\_layer.send\_group(self.name, content)  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/asgi\_amqp/core.py”, line 190, in send\_group  
> self.send(channel, message)  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/asgi\_amqp/core.py”, line 95, in send  
> self.recover()  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/asgi\_amqp/core.py”, line 77, in recover  
> self.tdata.consumer.revive(self.tdata.connection.channel())  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/connection.py”, line 255, in channel  
> chan = self.transport.create\_channel(self.connection)  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/kombu/transport/pyamqp.py”, line 92, in create\_channel  
> return connection.channel()  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/connection.py”, line 282, in channel  
> return self.Channel(self, channel\_id)  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py”, line 101, in **init**  
> self.\_x\_open()  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/channel.py”, line 427, in \_x\_open  
> self.\_send\_method((20, 10), args)  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/abstract\_channel.py”, line 56, in \_send\_method  
> self.channel\_id, method\_sig, args, content,  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/method\_framing.py”, line 221, in write\_method  
> write\_frame(1, channel, payload)  
> File “/var/lib/awx/venv/awx/lib/python2.7/site-packages/amqp/transport.py”, line 182, in write\_frame  
> frame\_type, channel, size, payload, 0xce,  
> File “/usr/lib64/python2.7/socket.py”, line 224, in meth  
> return getattr(self.\_sock,name)(\*args)  
> error: [Errno 104] Connection reset by peer

Can anyone help with what the next troubleshooting steps might be or with any wisdom on additional logging that could be enabled?

---

<div class="post-metadata">

**Author:** ![Christer\_Hemgren](https://avatars.discourse-cdn.com/v4/letter/c/97f17d/32.png) [@Christer\_Hemgren](https://forum.ansible.com/u/Christer_Hemgren)\
**Post date:** [July 6, 2018, 9:03pm UTC](https://forum.ansible.com/t/websockets-not-working-after-a-little-while/27945/2 "2018-07-06T21:03:11Z")

</div>

We can see similar behavior on awx 1.0.6.x.

We use ha-proxy in a separate continer in front for ssl.  
Reload of ha-proxy, awx web and awx task continer use to solve ower issue.

---

<div class="post-metadata">

**Author:** ![William\_Easton](https://avatars.discourse-cdn.com/v4/letter/w/90db22/32.png) [@William\_Easton](https://forum.ansible.com/u/William_Easton)\
**Post date:** [July 6, 2018, 10:45pm UTC](https://forum.ansible.com/t/websockets-not-working-after-a-little-while/27945/3 "2018-07-06T22:45:32Z")

</div>

Does it fix it permanently or does it come back as an issue shortly after?

I’m having trouble figuring out what to test/try next. It works after a restart of the containers for some short period of time (normally less than an hour) and then stops working at some point but it doesnt seem to make sense when and why it stops working.

---

<div class="post-metadata">

**Author:** ![Christer\_Hemgren](https://avatars.discourse-cdn.com/v4/letter/c/97f17d/32.png) [@Christer\_Hemgren](https://forum.ansible.com/u/Christer_Hemgren)\
**Post date:** [July 7, 2018, 8:24pm UTC](https://forum.ansible.com/t/websockets-not-working-after-a-little-while/27945/4 "2018-07-07T20:24:02Z")

</div>

No, we use to need to reload every week.
