# AWX job terminated unexpectedly **URL:** \ **Category:** AWX Project\ **Tags:** awx\ **Created:** [October 20, 2022, 1:41pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252 "2022-10-20T13:41:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [October 20, 2022, 1:41pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/1 "2022-10-20T13:41:42Z")
Hello, I’m having an issue with a job template on AWX that terminated unexpectedly. AKS version: 1.23.12 AWX version: 21.6.0 It’s a long automation but after 31m and 7000 tasks executed fails each time as you can see from pic. ![AWX_err.PNG](https://us1.discourse-cdn.com/ansible/original/2X/e/efb36142ef03834d4490a49c18b354a08c2a7d9d.png) No errors displayed in Stdout in AWX ui. Running: **_kubectl logs awx-c7647dbf5-2mt88 -n awx -c awx-task | grep error_** I see this error: _**2022-10-20 11:24:53,613 WARNING [a5fc6bd17e2146a883b741b0bd8a3bbd] awx.main.dispatch job 142359 (error) encountered an**_ _**error (rc=None), please see task stdout for details.**_ I’ve tried to get logs directly from automation pod: **_kubectl logs automation-job-142359-nxxpb -n awx -f_** The automation stops without errors displayed and automation pod destroyed from aks. This is the last line before the crash: **_{“uuid”: “5fce5d17-9f63-4705-bf05-adf9b4b2a73b”, “counter”: 26487, “stdout”: “”, “start\_line”: 24454, “end\_line”: 24454, “runner\_ident”: “142359”, “event”: “runner\_on\_start”, “job\_id”: 142359, “pid”: 21, “created”: “2022-10-20T11:19:39.658349”, “parent\_uuid”: “02901256-661c-2a55-e472-000000001bc8”, “event\_data”: {“playbook”: “log\_query\_for\_all\_cids\_from\_appservers.yml”, “playbook\_uuid”: “ef3e2675-daf8-4ac4-84db-9b797d3ffc6e”, “play”: “log query on tse db for all customers”, “play\_uuid”: “02901256-661c-2a55-e472-000000000066”, “play\_pattern”: “res\_alyante\_srv\_grp\_prod”, “task”: “set facts”, “task\_uuid”: “02901256-661c-2a55-e472-000000001bc8”, “task\_action”: “set\_fact”, “task\_args”: “”, “task\_path”: “/runner/project/roles/common/exec\_query\_with\_logs\_on\_win\_vm/tasks/log\_query.yml:2”, “role”: “exec\_query\_with\_logs\_on\_win\_vm”, “host”: “aly110-srv”, “uuid”: “5fce5d17-9f63-4705-bf05-adf9b4b2a73b”}}_** What could be the problem? Why the automation-job pod is destroyed unexpectedly? Is possible something like a timeout after about 31min? Who runs the command for delete automation-job pod? Thank you for your help. ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [October 20, 2022, 3:14pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/2 "2022-10-20T15:14:05Z")
I’ve done other test we a simple role like this: **_- name: loop date for 60 min_** **_shell: |_** **_date_** **_register: result_** **_until: result.stdout == “LOOP”_** **_retries: 60_** **_delay: 60_** It’s not an issue due to a timeout… it worked fine for 60 minutes. So what is the problem? Tasks number? Thank you for your help. ---
**Author:** ![kurokobo1](https://avatars.discourse-cdn.com/v4/letter/k/f0a364/32.png) [@kurokobo1](https://forum.ansible.com/u/kurokobo1)\ **Post date:** [October 20, 2022, 6:46pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/3 "2022-10-20T18:46:23Z")
Hi, There seems to be a huge logs, so I guess your issue might be caused by [https://github.com/ansible/awx/issues/11338](https://github.com/ansible/awx/issues/11338) ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [October 21, 2022, 10:17am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/4 "2022-10-21T10:17:31Z")
You’re right!!! I don’t know how much time I wasted looking for a solution ☹ I’ve increased from 50MB (default in my aks nodepool ) to 100MB and the job templated completed without crash. For verify aks nodepool configuration: **_kubectl proxy_** **_curl -sSL “[http://localhost:8001/api/v1/nodes/NODE-NAME/proxy/configz](http://localhost:8001/api/v1/nodes/NODE-NAME/proxy/configz)” | python3 -m json.tool_** The question now is… which is the right value for log rotation? 100, 200, 500? Is there no possibility that this thing will fixed in a future AWX release? This is only a workaround… it works but it’s not a stable solution. Thank you for you help! ---
**Author:** ![kurokobo1](https://avatars.discourse-cdn.com/v4/letter/k/f0a364/32.png) [@kurokobo1](https://forum.ansible.com/u/kurokobo1)\ **Post date:** [October 21, 2022, 10:40pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/5 "2022-10-21T22:40:37Z")
Congrats 🙂 > The question now is… which is the right value for log rotation? 100, 200, 500? I think there is no right value. That configuration affects not only logs for EE but also logs for all pods on the cluster, so it totally depends on the size of the log for your job and capacity of the node. > Is there no possibility that this thing will fixed in a future AWX release? This is only a workaround… it works but it’s not a stable solution. The root cause is not in the AWX implementation, but in the behavior of the Kubernetes API. Refer [https://github.com/kubernetes/kubernetes/issues/59902](https://github.com/kubernetes/kubernetes/issues/59902) I agree with you that this issue should be fixed, but sadly seems take long time since the issue is kept opened for five years… Regards, ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [October 24, 2022, 8:32am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/6 "2022-10-24T08:32:40Z")
I have an AKS cluster with 3 nodes dedicated to AWX so there should not be problems of capacity for the nodes. Maybe I will try 200MB or 500MB and I hope will be enough. The very strange thing is that I’m migrating my automation from a very old AWX version (3.0.1.0) and same templates worked without errors. The 3.0.1.0 AWX version is not affected to this AKS cluster issue? Regards, Elia ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [October 25, 2022, 10:32am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/7 "2022-10-25T10:32:05Z")
For a specif template with about 100000 log lines I have increased log rotation to 500MB and now it’s working. How is it possible with AWX 3.0.1.0 older version there isn’t any issue due to log rotation on aks? Thank you for your help. ---
**Author:** ![kurokobo1](https://avatars.discourse-cdn.com/v4/letter/k/f0a364/32.png) [@kurokobo1](https://forum.ansible.com/u/kurokobo1)\ **Post date:** [October 25, 2022, 11:31pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/8 "2022-10-25T23:31:16Z")
Hi, > How is it possible with AWX 3.0.1.0 older version there isn’t any issue due to log rotation on aks? The root cause of the issue is in Kubernetes API, and the recent AWX (18+) is designed to use that API to gather logs from EE. In my understanding, in the older AWX 3.x, no Kubernetes API is required to run job. There is no EE and your playbook run directly under virtualenv on AWX container. I think that’s why this issue affects recent AWX only. Regards, ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [October 26, 2022, 8:59am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/9 "2022-10-26T08:59:00Z")
Hi, it makes sense what you say. Thanks for the explanation! ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [December 2, 2022, 11:23am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/10 "2022-12-02T11:23:23Z")
Hello, I’m having a similar issue with job terminated unexpectedly but this time is not due to log rotation that I have solved last time on my aks cluster. In AWX I see the job (id: 144294) failed but I can’t visualize the job details, AWX UI try to load the log without success: ![1.PNG](https://us1.discourse-cdn.com/ansible/original/2X/7/7be0e20117756afeeb75c28383483d794b451a37.png) So direct in awx-task container **_2022-12-02 01:15:09,647 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 pre run_** **_2022-12-02 01:15:09,861 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 preparing playbook_** **_2022-12-02 01:15:09,996 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 running playbook_** **_2022-12-02 01:15:10,332 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 work unit id received_** **_2022-12-02 01:15:10,370 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 work unit id assigned_** **_2022-12-02 02:01:11,660 INFO [544931daa2964f35a9686d3004f8628b] awx.main.commands.run\_callback\_receiver Starting EOF event processing for Job 144294_** **_2022-12-02 02:06:17,396 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 post run_** **_2022-12-02 02:06:55,686 INFO [544931daa2964f35a9686d3004f8628b] awx.analytics.job\_lifecycle job-144294 finalize run_** **_2022-12-02 02:06:55,756 WARNING [544931daa2964f35a9686d3004f8628b] awx.main.dispatch job 144294 (error) encountered an error (rc=None), please see task stdout for details._** in awx-ee container _ **ERROR 2022/12/02 02:06:20 Exceeded retries for reading stdout /tmp/receptor/awx-c7647dbf5-85g85/FF2xD2Oc/stdout** _ I’ve also checked stdout directly from awx db in “awx\_jobevent” table but there are not errors for this job id… probably the pod was terminated so I found last command executed. Any ideas? Thank you very much. Elia ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [December 2, 2022, 6:17pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/11 "2022-12-02T18:17:15Z")
Hi, is this a consistent failure you are seeing in your deployment? based on logs, seems the job is running for about 45 minutes to completion. For debugging purposes, you could set RECEPTOR\_RELEASE\_WORK in the django settings (see below), and after encountering this issue, check the receptor work unit by execing into the awx pod, awx-ee container, and cat the _ **/tmp/receptor/awx-c7647dbf5-85g85/FF2xD2Oc/status** _ file ```auto extra_settings: - setting: RECEPTOR_RELEASE_WORK value: "False" ``` If you provide that info we can help further, thanks! AWX Team ---
**Author:** ![ponnurangam\_h](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@ponnurangam\_h](https://forum.ansible.com/u/ponnurangam_h)\ **Post date:** [July 14, 2023, 3:08pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/12 "2023-07-14T15:08:33Z")
Hi We had the same issue with EKS and fixed with by increasing the log size, but want to know how you did at template leave. so that we can use only for a particular template thanks in advance ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [July 14, 2023, 6:18pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/13 "2023-07-14T18:18:28Z")
increasing log size is a k8s cluster level setting (and will apply to all job pods launched), and cannot be set per AWX template. What reason did you have for wanting this at the template level? AWX Team ---
**Author:** ![ponnurangam\_h](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@ponnurangam\_h](https://forum.ansible.com/u/ponnurangam_h)\ **Post date:** [July 14, 2023, 6:22pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/14 "2023-07-14T18:22:47Z")
Thanks for the clarity, That is what we did for the target node using tolerance or affinity rules, but our dev environment goes for reboots overnight as budget constraints, this creates problem with configuration inconsistencies. This is the case with EKS ---
**Author:** ![Wei-Yen\_Tan](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/wei-yen_tan/32/4111_2.png) [@Wei-Yen\_Tan](https://forum.ansible.com/u/Wei-Yen_Tan)\ **Post date:** [July 14, 2023, 7:35pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/15 "2023-07-14T19:35:57Z")
Interested in this and forgive my ignorance but how did you increase the k8slogs? ---
**Author:** ![ponnurangam\_h](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@ponnurangam\_h](https://forum.ansible.com/u/ponnurangam_h)\ **Post date:** [July 15, 2023, 3:35am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/16 "2023-07-15T03:35:19Z")
We modified a node level kubelet configurations and updated the maxcontainerlogsize to 100mib from default 10. Adding to we made sure the automation pods will only gets created on the same node using tolerance and node affinity rules through our deployment but the same in a different EKS region was good with similar setup, still not sure why this is happening ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [July 17, 2023, 7:18am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/17 "2023-07-17T07:18:56Z")
We also modified the “maxcontainerlogsize” from 10 to 500M in AKS 1.25. The problem occurs less often but is not solved. I’m confused about the RECEPTOR\_KUBE\_SUPPORT\_RECONNECT flag in the awx spec… What value should be set in this case to prevent the issue? Enabled or disabled? Thank you very much for help. Elia ---
**Author:** ![ponnurangam\_h](https://avatars.discourse-cdn.com/v4/letter/p/71e660/32.png) [@ponnurangam\_h](https://forum.ansible.com/u/ponnurangam_h)\ **Post date:** [July 17, 2023, 7:25am UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/18 "2023-07-17T07:25:01Z")
We set it as disabled ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [July 19, 2023, 6:45pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/19 "2023-07-19T18:45:52Z")
RECEPTOR\_KUBE\_SUPPORT\_RECONNECT should be enabled if you k8s kubelet version supports it. See here for the details of the versions [https://github.com/ansible/receptor/pull/683](https://github.com/ansible/receptor/pull/683) AWX Team ---
**Author:** ![epi82](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@epi82](https://forum.ansible.com/u/epi82)\ **Post date:** [July 19, 2023, 7:10pm UTC](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252/20 "2023-07-19T19:10:37Z")
We are in aks 1.25.6 so it should be supported. By the way this flag enable didn’t solved our issue for unexpected jobs terminations. Elia [Next page](https://forum.ansible.com/t/awx-job-terminated-unexpectedly/37252.md?page=2)