# Receptor error when starting jobs in new AWX deployment in AKS **URL:** \ **Category:** Get Help\ **Tags:** awx, kubernetes\ **Created:** [April 18, 2024, 9:01am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015 "2024-04-18T09:01:24Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 18, 2024, 9:01am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/1 "2024-04-18T09:01:24Z")
### Bug Summary Hi Team, I am testing AWX in AKS cluster. This is a private cluster, so I am unable to do this with AWX operator. So I created a different cluster in public, used that deployments, secret, config maps, service account, role bindings and put this in my private cluster. I was able to bring up the AWX in this cluster. However when I run the jobs, I see this error: Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/[receptor.py]”, line 372, in _run\_internal resultsock = receptor\_ctl.get\_work\_results(self.unit\_id, return\_sockfile=True) File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/receptorctl/socket_ [interface.py]”, line 248, in get\_work\_results self.writestr(f"work results {unit\_id}\n") File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/receptorctl/socket\_[interface.py]”, line 49, in writestr self.\_sockfile.flush() File “/usr/lib64/python3.9/[socket.py](http://socket.py/)”, line 722, in write return self.\_sock.send(b) BrokenPipeError: [Errno 32] Broken pipe In the **AWX task** logs, I see: 2024-04-18 06:56:20,999 ERROR [9c1dd129d187483c8fd7d2d13aa2739c] awx.main.tasks.receptor An error was encountered while getting status for work unit iv1jPZsF Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py”, line 356, in \_run\_internal ConnectionResetError: [Errno 104] Connection reset by peer When I investigated further, in receptor, I see this: { “iv1jPZsF”: { “Detail”: “Pod created”, “ExtraData”: { “Command”: “”, “Image”: “”, “KubeConfig”: “”, “KubeNamespace”: “”, “KubePod”: “”, “Params”: “”, “PodName”: “automation-job-44-69vvn” }, “State”: 0, “StateName”: “Pending”, “StdoutSize”: 0, “WorkType”: “kubernetes-incluster-auth” } } bash-5.1$ /var/lib/awx/venv/awx/bin/receptorctl --socket /run/receptor/receptor.sock work list ERROR: [Errno 111] Connection refused By any chance do you know what could be happening here? Any pointers would be really helpful. I am using AWX 21.4 as that is what we are using in our Prod clusters. We have deployed this in AKS clusters. **I see this error in AWX EE** : DEBUG 2024/04/18 08:41:38 Client connected to control service @ DEBUG 2024/04/18 08:41:38 Control service closed DEBUG 2024/04/18 08:41:38 Client disconnected from control service @ DEBUG 2024/04/18 08:41:38 Client connected to control service @ DEBUG 2024/04/18 08:41:40 Stdout complete - closing channel for: BDE6e0Ef WARNING 2024/04/18 08:41:40 Could not read in control service: read unix /var/run/receptor/receptor.sock-\>@: use of closed network connection DEBUG 2024/04/18 08:41:40 Client disconnected from control service @ WARNING 2024/04/18 08:41:40 Could not close connection: close unix /var/run/receptor/receptor.sock-\>@: use of closed network connection DEBUG 2024/04/18 08:41:40 Client connected to control service @ DEBUG 2024/04/18 08:41:40 Control service closed DEBUG 2024/04/18 08:41:40 Client disconnected from control service @ DEBUG 2024/04/18 08:41:40 Client connected to control service @ DEBUG 2024/04/18 08:41:40 Control service closed DEBUG 2024/04/18 08:41:40 Client disconnected from control service @ DEBUG 2024/04/18 08:41:40 Client connected to control service @ panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11ba558] and **then I see error on awx\_task** saying 2024-04-18 08:41:40,986 DEBUG [a6dbacaa718a4cc6a108738b0a75349b] awx.analytics.job\_lifecycle inventoryupdate-64 work unit id assigned 2024-04-18 08:41:43,453 INFO [a6dbacaa718a4cc6a108738b0a75349b] awx.main.commands.run\_callback\_receiver Starting EOF event processing for Job 64 2024-04-18 08:41:43,453 INFO [a6dbacaa718a4cc6a108738b0a75349b] awx.main.commands.run\_callback\_receiver Starting EOF event processing for Job 64 2024-04-18 08:41:43,469 ERROR [a6dbacaa718a4cc6a108738b0a75349b] awx.main.tasks.receptor An error was encountered while getting status for work unit BurUgPkP Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py”, line 356, in \_run\_internal unit\_status = receptor\_ctl.simple\_command(f’work status {self.unit\_id}') In AWX Task receptor I see this: { “iv1jPZsF”: { “Detail”: “Pod created”, “ExtraData”: { “Command”: “”, “Image”: “”, “KubeConfig”: “”, “KubeNamespace”: “”, “KubePod”: “”, “Params”: “”, “PodName”: “automation-job-44-69vvn” }, “State”: 0, “StateName”: “Pending”, “StdoutSize”: 0, “WorkType”: “kubernetes-incluster-auth” } } bash-5.1$ /var/lib/awx/venv/awx/bin/receptorctl --socket /run/receptor/receptor.sock work list ERROR: [Errno 111] Connection refused Please find the github issue raised for this: [Receptor error when starting jobs in new AWX deployment in AKS. · Issue #1831 · ansible/awx-operator · GitHub](https://github.com/ansible/awx-operator/issues/1831) Thanks and Regards, Mani ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 18, 2024, 9:39am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/2 "2024-04-18T09:39:52Z")
We have configured service mesh istio in our system. I am wondering whether this is causing any issues! ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [April 18, 2024, 3:58pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/3 "2024-04-18T15:58:12Z")
Which image is used for `awx-ee` container in `awx-task` pod? If it is `awx-ee:latest`, is it the old `latest` exported from the running Prod environment or the new `latest` newly pulled? ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 18, 2024, 4:24pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/4 "2024-04-18T16:24:38Z")
We are retrieving the image from our ACR. We are referencing with AWX EE 2.12.5, because AWX version is 21.4. We went with AWX EE 2.12.5 because that was the recommended version for AWX 21.4. ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 18, 2024, 4:27pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/5 "2024-04-18T16:27:20Z")
I see the image getting pulled, and I see that the underlying receptor interaction between awx task and awx ee is having some issue on the initial connection, which is strange. Which was why I was thinking that it could be due to some underlying AKS networking or Istio service mesh. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [April 18, 2024, 4:45pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/6 "2024-04-18T16:45:02Z")
> [@mani3887](#): > > AWX EE 2.12.5 because that was the recommended version for AWX 21.4 There is no such tag `2.12.5`: [Quay](https://quay.io/repository/ansible/awx-ee) Where your EE image came from? Isn’t it mirror of official `quay.io/ansible/awx-ee` or customized using ansible-builder? Which version of Receptor running in the `awx-ee` container in `awx-task` pod? ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 18, 2024, 4:58pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/7 "2024-04-18T16:58:31Z")
Yes that is accurate. I am using ansible-core 2.12.5 and building a custom AWX EE image. Can you please let me know how I can check the version for receptor in awx-task and awx-ee pods? ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 23, 2024, 5:15pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/8 "2024-04-23T17:15:26Z")
Hi Kurokobo, Can you please let me know how I can check the version for receptor in awx-task and awx-ee pods? ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [April 24, 2024, 1:38am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/9 "2024-04-24T01:38:18Z")
Just run `receptor --version` in the `awx-ee` container in `awx-task` pod. By the way for this type of extremely simple (and easy) questions, you can get the answer faster (in 5 minutes) by referring the official receptor docs, asking ChatGPT, or Google than just wait for 5 days with doing nothing. You may also need to make an effort to find out the solutions for yourself. Thanks for your understanding. ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 29, 2024, 6:22pm UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/10 "2024-04-29T18:22:03Z")
Thanks for your response and help. Apologies for the delayed response. We figured out the issue. The issue was Istio was messing the connectivity between AWX and AWX EE pods. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [April 30, 2024, 12:58am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/11 "2024-04-30T00:58:11Z")
@mani3887 Hi, thanks for updating, glad to hear that the issue has been solved. It’s okay to close this thread but if possible, could you update here with how you discovered the root cause and how you resolved it? The Istio is one of the major CNI plugin so someone would face the same issue in the future, and this thread could be a good references to troubleshoot them if you can provide additional information. Thanks! ---
**Author:** ![mani3887](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@mani3887](https://forum.ansible.com/u/mani3887)\ **Post date:** [April 30, 2024, 2:15am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/12 "2024-04-30T02:15:19Z")
I observed that the AWX task was transmitting messages to the Receptor websocket connection, and AWX EE received these messages. However, the connection closed when AWX EE attempted to respond. AWX is deployed in an AKS cluster using the kubenet plugin. I suspected that an external process was terminating this connection. During my investigation with Azure support for the kubenet plugin, they suggested that using the kubenet plugin might introduce an additional hop. I also considered Istio as a potential cause, since I had installed a sidecar with AWX. After disabling the sidecar, I noticed the problem was resolved. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [April 30, 2024, 11:54am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/13 "2024-04-30T11:54:17Z")
@mani3887 Thanks for your controbution 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\ **Post date:** [May 30, 2024, 11:55am UTC](https://forum.ansible.com/t/receptor-error-when-starting-jobs-in-new-awx-deployment-in-aks/5015/14 "2024-05-30T11:55:06Z")
Closed on May 30, 2024, 11:55am UTC This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.