RuntimeError: Error streaming stdin to pod

Archives AWX Project
,
1

When running a playbook, I get this error…

RuntimeError: Error streaming stdin to pod awx/automation-job-96-p897n. Error: error sending request: Post “https://10.0.0.1:443/api/v1/namespaces/awx/pods/automation-job-96-p897n/attach?container=worker&stdin=true”: read tcp 10.231.80.16:46144->10.0.0.1:443: read: connection timed out

  • 10.231.80.16:46144 Is the IP of the task POD
  • 10.0.0.1 is the K8S POD
  • I have been able to test connectivity between the POD and the K8S server
  • The automation-job-96-p897n is created and runs for the duration of the job
    Contents of the playbook:
2

so did the job run successful or end in a failed / error state? In your awx-ee container logs do you see “retry” messages for this job? e.g.

Error streaming stdin to pod … Will retry 5 more times.

does this problem happen sporadically or every time?

what k8s cluster are you on (k3s, k3d, etc)

AWX Team

3

Thanks for your help.
We are running AWX on Azure AKS.The job fails every time.

Here are the output logs from the job…

Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py”, line 435, in _run_internal lines = resultfile.readlines() OSError: read() should have returned a bytes object, not ‘NoneType’ During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py”, line 597, in run res = receptor_job.run() File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py”, line 317, in run res = self._run_internal(receptor_ctl) File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py”, line 444, in _run_internal raise RuntimeError(detail) RuntimeError: Error streaming stdin to pod awx/automation-job-173-8rsrb. Error: error sending request: Post “https://10.0.0.1:443/api/v1/namespaces/awx/pods/automation-job-173-8rsrb/attach?container=worker&stdin=true”: read tcp 10.231.80.47:44656->10.0.0.1:443: read: connection timed out

4

At first glance this seems like a kubernetes level problem, not specifically AWX.

So the automation-job pods are staying up indefinitely, even after these stdin POST timeout errors?

does kubectl logs on that automation job pod show you stdout? do you see ansible events?

Also, as a side note, you may want to enabled RECEPTOR_KUBE_SUPPORT_RECONNECT, see this PR https://github.com/ansible/receptor/pull/683 But I don’t think it is related to your current issue

AWX Team