AWX: ssh debug: Mux_client_read_packet: read header failed: Broken pipe

Get Help
,
1

My team has installed AWX (Kurokobo - awx-on-k3s) , we have worked with Windows servers without problems, however, it does not work with any Linux distribution, since in all our tests we get the error:
mux_client_read_packet: header read failed: broken pipe

We have tested with these parameters:
ansible_network_cli_ssh_type: paramiko
ansible_ssh_common_args: ‘-o StrictHostKeyChecking=no’

However, the problem has not been solved.

I would greatly appreciate any ideas on how to solve this problem.

I suspect that it is not the SSH connection, since from the same AWX container the connection to the Linux servers was established without problems.

module_stdout: ‘’
module_stderr: “OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for ‘final all’ host 192.168.1.33 originally 192.168.1.33\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched ‘final’\r\ndebug2: match not found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug1: configuration requests final Match pass\r\ndebug2: resolve_canonicalize: hostname 192.168.1.33 is address\r\ndebug1: re-parsing configuration\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0\r\ndebug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf\r\ndebug2: checking match for ‘final all’ host 192.168.1.33 originally 192.168.1.33\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: matched ‘final’\r\ndebug2: match found\r\ndebug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1\r\ndebug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config\r\ndebug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]\r\ndebug3: kex names ok: [curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]\r\ndebug3: expanded UserKnownHostsFile ‘~/.ssh/known_hosts’ → ‘/runner/.ssh/known_hosts’\r\ndebug3: expanded UserKnownHostsFile ‘~/.ssh/known_hosts2’ → ‘/runner/.ssh/known_hosts2’\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 34\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to 192.168.1.33 closed.\r\n”
msg: |-
MODULE FAILURE
See stdout/stderr for the exact error
rc: 1
_ansible_no_log: false
changed: false

2

@MartinGM
I noticed that you posted the same question on my repo. Just a heads up, that’s not really the best way to go about it. Please check my comment on my repo:

Please do not post the same question in different places unless you explicitly mention that you have posted it elsewhere and provide the link to it. Even if a question you submitted to a particular place receives a response, others may not be aware of it from different places. Someone unaware of this might spend their time in good faith trying to help you elsewhere, which is not ideal.
No Working with Linux(Centos/Ubuntu): Mux_client_read_packet: read header failed: Broken pipe · Issue #394 · kurokobo/awx-on-k3s · GitHub

Could you please share your actual task which is failed in your playbook? I believe you are using command module.

Also:

  • Test the connection with ansible.builtin.ping module.
  • Ensure the python (/usr/bin/python3) is available on the target nodes.

The error message you’ve mentioned is not critical and may be safely ignored. As you can see from the logs, the SSH connection was successful multiple times. Only the SSH to invoke python script on the target node with sudo was failed.

:point_down: Exit with 0 (Success)

<192.168.1.33> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="-awx-idx"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/7fe6eec243"' 192.168.1.33 '/bin/sh -c '"'"'echo ~-awx-idx && sleep 0'"'"''
<192.168.1.33> (0, b'/home/-awx-idx\\n', b'OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024

<192.168.1.33> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="-awx-idx"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/7fe6eec243"' 192.168.1.33 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/-awx-idx/.ansible/tmp `"&& mkdir "` echo /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409 `" && echo ansible-tmp-1727424460.7293117-27-219517984534409="` echo /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409 `" ) && sleep 0'"'"''
<192.168.1.33> (0, b'ansible-tmp-1727424460.7293117-27-219517984534409=/home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409\\n', b"OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024

<192.168.1.33> SSH: EXEC sshpass -d12 sftp -o BatchMode=no -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="-awx-idx"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/7fe6eec243"' '[192.168.1.33]'
<192.168.1.33> (0, b'sftp> put /runner/.ansible/tmp/ansible-local-22f7z4o24r/tmpdcpqh9cw /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409/AnsiballZ_command.py\\n', b'OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024

<192.168.1.33> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="-awx-idx"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/7fe6eec243"' 192.168.1.33 '/bin/sh -c '"'"'chmod u+x /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409/ /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409/AnsiballZ_command.py && sleep 0'"'"''
<192.168.1.33> (0, b'', b"OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024

<192.168.1.33> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="-awx-idx"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/7fe6eec243"' 192.168.1.33 '/bin/sh -c '"'"'rm -f -r /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409/ > /dev/null 2>&1 && sleep 0'"'"''
<192.168.1.33> (0, b'', b"OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024

:point_down: Exit with 1 (Failure)

<192.168.1.33> SSH: EXEC sshpass -d12 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="-awx-idx"' -o ConnectTimeout=10 -o 'ControlPath="/runner/cp/7fe6eec243"' -tt 192.168.1.33 '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-adbixplamxegrlniuihqpxmigiapopvm ; /usr/bin/python3 /home/-awx-idx/.ansible/tmp/ansible-tmp-1727424460.7293117-27-219517984534409/AnsiballZ_command.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<192.168.1.33> (1, b'', b"OpenSSH_8.7p1, OpenSSL 3.2.2 4 Jun 2024
3

Thank you for your suggestion regarding the duplicity of the post, it is the first time I create a post.

Currently my team uses an AWX, which was developed a couple of years ago (I don’t know its source), and where we have been working on our templates for both Windows and Unix. But when trying to deploy a new version of AWX, during the migration of the templates, only the Unix playbooks present the problem that I showed in the previous post.

Answering your question, the module we use from ansible is: Shell.

The playbook starts by executing this task:

  • name: Get information about Unix System
    ansible.builtin.shell: uname
    register : check_unix
4

Hi Martin,

Are yo able to resolve this issue? I’ve the same problem

5

Hmm could you try adding following lines in your ansible.cfg? If you don’t have ansible.cfg yet, create and place it at the root of the project.

Attempt: Disable ControlMaster

[ssh_connection]
ssh_args = -o ControlMaster=no

Attempt: Use paramiko

[defaults]
transport = paramiko
1 Like
6

Thank you!! The templates already work.

#Unix Server: Disable ControlMaster
[ssh_connection]
ssh_args = -o ControlMaster=no

#Unix Server: Use paramiko
[defaults]
transport = paramiko