# Scheduled jobs failling to execute

**URL:** <https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984>\
**Category:** AWX Project\
**Tags:** awx\
**Created:** [August 12, 2022, 4:51pm UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984 "2022-08-12T16:51:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\
**Post date:** [August 12, 2022, 4:51pm UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/1 "2022-08-12T16:51:05Z")

</div>

Hi team,

I recently deploy version 21.4 of AWX using the 0.26 operator on a single node K3s cluster.  
All seem to be working as expected, some scheduled jobs fail with this error:

"Stream error running pod: stdin: error dialing backend: EOF, stdout: http2: response body closed

Job is scheduled to run every 5 minutes and runs a simple playbook to execute a python script.  
More concerning though is that this also occurs randomly when executing scheduled backup jobs.

Can anyone help identify what might be causing this issue.

Kind regards,  
Michael

---

<div class="post-metadata">

**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\
**Post date:** [August 17, 2022, 2:53pm UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/2 "2022-08-17T14:53:22Z")

</div>

Hey there,  
It sounds like there have been a couple other people to run into similar issues. It might be that your cluster is being restarted since that error is a k3s error.  
[https://github.com/ansible/awx/issues/10219](https://github.com/ansible/awx/issues/10219) ← the last comment in here has an explanation of what was going on with their setup, maybe it will help you out.  
AWX Team

---

<div class="post-metadata">

**Author:** ![Wei-Yen\_Tan](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/wei-yen_tan/32/4111_2.png) [@Wei-Yen\_Tan](https://forum.ansible.com/u/Wei-Yen_Tan)\
**Post date:** [August 17, 2022, 3:28pm UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/3 "2022-08-17T15:28:02Z")

</div>

What version of k3s? And have You has multiple upgraded iterations of awx?

---

<div class="post-metadata">

**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\
**Post date:** [August 17, 2022, 9:28pm UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/4 "2022-08-17T21:28:19Z")

</div>

Hi,

I did see that github thread, I ran “watch kubectl get pods -n awx” while the tasks are running and all the pods are up. I also see the automation-job-xxxx container being created and started, but after a few seconds it terminates with this error.

Would the pods still stay up while the cluster is restarted?

Thanks for the help,

Michael

> **(attachments)**
>
> ![cityscoot.gif](https://us1.discourse-cdn.com/ansible/original/2X/c/c6231c19f061c3cbf9e4803886a1aa273a2ab6c1.gif)

---

<div class="post-metadata">

**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\
**Post date:** [August 19, 2022, 9:25am UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/5 "2022-08-19T09:25:46Z")

</div>

Hi Team,

I have been monitoring the k3s logs and noticed the following error when jobs fail:

log.go:195] http: TLS handshake error from 127.0.0.1:39512: tls: first record does not look like a TLS handshake

 ![](https://us1.discourse-cdn.com/ansible/original/2X/f/ffd840da2fc48c08a347f1fa581d7c8e11d39c18.png)

Has anyone seen this behavior?

Any pointers to what I should check?

Regards,

> **(attachments)**
>
> ![cityscoot.gif](https://us1.discourse-cdn.com/ansible/original/2X/c/c6231c19f061c3cbf9e4803886a1aa273a2ab6c1.gif)

---

<div class="post-metadata">

**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\
**Post date:** [August 19, 2022, 5:16pm UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/6 "2022-08-19T17:16:57Z")

</div>

Hello,  
We were able to locate a GitHub issue that seems similar to yours: [https://github.com/kubernetes/dashboard/issues/2895](https://github.com/kubernetes/dashboard/issues/2895). This resolution may prove to be helpful to you, as what you have described appears to be an issue with the Kubernetes configuration.

Thanks,  
AWX Team

---

<div class="post-metadata">

**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\
**Post date:** [August 22, 2022, 9:54am UTC](https://forum.ansible.com/t/scheduled-jobs-failling-to-execute/36984/7 "2022-08-22T09:54:28Z")

</div>

Hi Team,

I was able to resolve this problem by downgrading to an earlier version of k3s (v1.21.9+k3s1).

Below the commands I used :

to uninstall the version (latest) I was running, all persistent data is lost 🙁

# /usr/local/bin/k3s-uninstall.sh

to install the version which works perfect with the latest version of the operator 0.26.0 and AWX 21.4.0

# curl -sfL [https://get.k3s.io](https://get.k3s.io) | INSTALL\_K3S\_VERSION={v1.21.9+k3s1} sh -

Hope this helps other.

Thanks for your time and help.

Regards,

> **(attachments)**
>
> ![cityscoot.gif](https://us1.discourse-cdn.com/ansible/original/2X/c/c6231c19f061c3cbf9e4803886a1aa273a2ab6c1.gif)
