# Occasional SSH issue when running a playbook. (SSH command stdout empty.)

**URL:** <https://forum.ansible.com/t/occasional-ssh-issue-when-running-a-playbook-ssh-command-stdout-empty/16729>\
**Category:** Ansible Project\
**Created:** [November 10, 2014, 12:34am UTC](https://forum.ansible.com/t/occasional-ssh-issue-when-running-a-playbook-ssh-command-stdout-empty/16729 "2014-11-10T00:34:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paavo](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@Paavo](https://forum.ansible.com/u/Paavo)\
**Post date:** [November 10, 2014, 12:34am UTC](https://forum.ansible.com/t/occasional-ssh-issue-when-running-a-playbook-ssh-command-stdout-empty/16729/1 "2014-11-10T00:34:01Z")

</div>

Hi, Everyone.

I’m trying to run an Ansible playbook on two servers, and occasionally either one or both of them fail a task with the error “/usr/bin/python: can’t open file ‘command’: [Errno 13] Permission denied”. I started looking into why this was happening (and why it was only happening some of the time). I traced the error to the temporary directory where Ansible copies its command file sometimes being empty instead of the normal random directory created in /tmp (runner/ **init**.py:Runner.\_make\_tmp\_path()). Looking into it further, it looks like sometimes the mkdir && echo command used to create the temporary directory is assumed to be successful, but the output, which should be the created directory, is empty (runner/connection\_plugins/ssh.py:Connection.\_communicate()).

The mkdir && echo commands looks identical to me:

EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=“/home/user/.ansible/cp/ansible-ssh-%h-%p-%r” -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=30 server1 /bin/sh -c 'mkdir -p /tmp/ansible-tmp-1415576481.52-184054006705534 && chmod a+rx /tmp/ansible-tmp-1415576481.52-184054006705534 && echo /tmp/ansible-tmp-1415576481.52-18405400670

EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=“/home/user/.ansible/cp/ansible-ssh-%h-%p-%r” -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=30 server2 /bin/sh -c ‘mkdir -p /tmp/ansible-tmp-1415576481.52-82470479460561 && chmod a+rx /tmp/ansible-tmp-1415576481.52-82470479460561 && echo /tmp/ansible-tmp-1415576481.52-82470479460561’

But when I print out the stderr output from the commands, they are very different:

Server 1 (failed):

OpenSSH\_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013  
debug1: Reading configuration data /etc/ssh/ssh\_config  
debug1: Applying options for \*  
debug1: auto-mux: Trying existing master  
debug1: Control socket “/home/user/.ansible/cp/ansible-ssh-server1-22-user” does not exist  
debug2: ssh\_connect: needpriv 0  
debug1: Connecting to server1 [[server1 IP]] port 22.  
debug2: fd 3 setting O\_NONBLOCK  
debug1: fd 3 clearing O\_NONBLOCK  
debug1: Connection established.  
debug3: timeout: 30000 ms remain after connect  
debug1: identity file /home/user/.ssh/identity type -1  
debug1: identity file /home/user/.ssh/identity-cert type -1  
debug3: Not a RSA1 key file /home/user/.ssh/id\_rsa.  
debug2: key\_type\_from\_name: unknown key type ‘-----BEGIN’  
debug3: key\_read: missing keytype  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug3: key\_read: missing whitespace  
debug2: key\_type\_from\_name: unknown key type ‘-----END’  
debug3: key\_read: missing keytype  
debug1: identity file /home/user/.ssh/id\_rsa type 1  
debug1: identity file /home/user/.ssh/id\_rsa-cert type -1  
debug1: identity file /home/user/.ssh/id\_dsa type -1  
debug1: identity file /home/user/.ssh/id\_dsa-cert type -1  
debug1: identity file /home/user/.ssh/id\_ecdsa type -1  
debug1: identity file /home/user/.ssh/id\_ecdsa-cert type -1  
debug1: Remote protocol version 2.0, remote software version OpenSSH\_5.3  
debug1: match: OpenSSH\_5.3 pat OpenSSH\*  
debug1: Enabling compatibility mode for protocol 2.0  
debug1: Local version string SSH-2.0-OpenSSH\_5.3  
debug2: fd 3 setting O\_NONBLOCK  
debug1: SSH2\_MSG\_KEXINIT sent  
debug3: Wrote 960 bytes for a total of 981  
debug1: SSH2\_MSG\_KEXINIT received  
debug2: kex\_parse\_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1  
debug2: kex\_parse\_kexinit: [ssh-rsa-cert-v01@openssh.com](mailto:ssh-rsa-cert-v01@openssh.com),ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss  
debug2: kex\_parse\_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se  
debug2: kex\_parse\_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se  
debug2: kex\_parse\_kexinit: [hmac-md5,hmac-sha1,umac-64@openssh.com](mailto:hmac-md5,hmac-sha1,umac-64@openssh.com),hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96  
debug2: kex\_parse\_kexinit: [hmac-md5,hmac-sha1,umac-64@openssh.com](mailto:hmac-md5,hmac-sha1,umac-64@openssh.com),hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96  
debug2: kex\_parse\_kexinit: [zlib@openssh.com](mailto:zlib@openssh.com),zlib,none  
debug2: kex\_parse\_kexinit: [zlib@openssh.com](mailto:zlib@openssh.com),zlib,none  
debug2: kex\_parse\_kexinit:  
debug2: kex\_parse\_kexinit:  
debug2: kex\_parse\_kexinit: first\_kex\_follows 0  
debug2: kex\_parse\_kexinit: reserved 0  
debug2: kex\_parse\_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1  
debug2: kex\_parse\_kexinit: ssh-rsa,ssh-dss  
debug2: kex\_parse\_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se  
debug2: kex\_parse\_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se  
debug2: kex\_parse\_kexinit: [hmac-md5,hmac-sha1,umac-64@openssh.com](mailto:hmac-md5,hmac-sha1,umac-64@openssh.com),hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96  
debug2: kex\_parse\_kexinit: [hmac-md5,hmac-sha1,umac-64@openssh.com](mailto:hmac-md5,hmac-sha1,umac-64@openssh.com),hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96  
debug2: kex\_parse\_kexinit: [none,zlib@openssh.com](mailto:none,zlib@openssh.com)  
debug2: kex\_parse\_kexinit: [none,zlib@openssh.com](mailto:none,zlib@openssh.com)  
debug2: kex\_parse\_kexinit:  
debug2: kex\_parse\_kexinit:  
debug2: kex\_parse\_kexinit: first\_kex\_follows 0  
debug2: kex\_parse\_kexinit: reserved 0  
debug2: mac\_setup: found hmac-md5  
debug1: kex: server-\>client aes128-ctr hmac-md5 [zlib@openssh.com](mailto:zlib@openssh.com)  
debug2: mac\_setup: found hmac-md5  
debug1: kex: client-\>server aes128-ctr hmac-md5 [zlib@openssh.com](mailto:zlib@openssh.com)  
debug1: SSH2\_MSG\_KEX\_DH\_GEX\_REQUEST(1024\<1024\<8192) sent  
debug1: expecting SSH2\_MSG\_KEX\_DH\_GEX\_GROUP  
debug3: Wrote 24 bytes for a total of 1005  
debug2: dh\_gen\_key: priv key bits set: 146/256  
debug2: bits set: 521/1024  
debug1: SSH2\_MSG\_KEX\_DH\_GEX\_INIT sent  
debug1: expecting SSH2\_MSG\_KEX\_DH\_GEX\_REPLY  
debug3: Wrote 144 bytes for a total of 1149  
debug3: check\_host\_in\_hostfile: host server1 filename /home/user/.ssh/known\_hosts  
debug3: check\_host\_in\_hostfile: host server1 filename /home/user/.ssh/known\_hosts  
debug3: check\_host\_in\_hostfile: match line 9  
debug3: check\_host\_in\_hostfile: host [server1 IP] filename /home/user/.ssh/known\_hosts  
debug3: check\_host\_in\_hostfile: host [server1 IP] filename /home/user/.ssh/known\_hosts  
debug3: check\_host\_in\_hostfile: match line 10  
debug1: Host ‘server1’ is known and matches the RSA host key.  
debug1: Found key in /home/user/.ssh/known\_hosts:9  
debug2: bits set: 501/1024  
debug1: ssh\_rsa\_verify: signature correct  
debug2: kex\_derive\_keys  
debug2: set\_newkeys: mode 1  
debug1: SSH2\_MSG\_NEWKEYS sent  
debug1: expecting SSH2\_MSG\_NEWKEYS  
debug3: Wrote 16 bytes for a total of 1165  
debug2: set\_newkeys: mode 0  
debug1: SSH2\_MSG\_NEWKEYS received  
debug1: SSH2\_MSG\_SERVICE\_REQUEST sent  
debug3: Wrote 48 bytes for a total of 1213  
debug2: service\_accept: ssh-userauth  
debug1: SSH2\_MSG\_SERVICE\_ACCEPT received  
debug2: key: /home/user/.ssh/identity ((nil))  
debug2: key: /home/user/.ssh/id\_rsa (0x7f4dc338f2d0)  
debug2: key: /home/user/.ssh/id\_dsa ((nil))  
debug2: key: /home/user/.ssh/id\_ecdsa ((nil))  
debug3: Wrote 64 bytes for a total of 1277  
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password  
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password  
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey  
debug3: authmethod\_lookup gssapi-with-mic  
debug3: remaining preferred: gssapi-keyex,hostbased,publickey  
debug3: authmethod\_is\_enabled gssapi-with-mic  
debug1: Next authentication method: gssapi-with-mic  
debug3: Trying to reverse map address [server1 IP].  
debug1: Unspecified GSS failure. Minor code may provide more information  
Credentials cache file ‘/tmp/krb5cc\_12444’ not found

debug1: Unspecified GSS failure. Minor code may provide more information  
Credentials cache file ‘/tmp/krb5cc\_12444’ not found

debug1: Unspecified GSS failure. Minor code may provide more information

debug1: Unspecified GSS failure. Minor code may provide more information  
Credentials cache file ‘/tmp/krb5cc\_12444’ not found

debug2: we did not send a packet, disable method  
debug3: authmethod\_lookup gssapi-keyex  
debug3: remaining preferred: hostbased,publickey  
debug3: authmethod\_is\_enabled gssapi-keyex  
debug1: Next authentication method: gssapi-keyex  
debug1: No valid Key exchange context  
debug2: we did not send a packet, disable method  
debug3: authmethod\_lookup publickey  
debug3: remaining preferred: ,publickey  
debug3: authmethod\_is\_enabled publickey  
debug1: Next authentication method: publickey  
debug1: Trying private key: /home/user/.ssh/identity  
debug3: no such identity: /home/user/.ssh/identity  
debug1: Offering public key: /home/user/.ssh/id\_rsa  
debug3: send\_pubkey\_test  
debug2: we sent a publickey packet, wait for reply  
debug3: Wrote 368 bytes for a total of 1645  
debug1: Server accepts key: pkalg ssh-rsa blen 277  
debug2: input\_userauth\_pk\_ok: SHA1 fp d3:31:a6:e5:17:09:6c:cb:aa:f1:02:85:5f:fc:e9:94:56:e5:3e:d1  
debug3: sign\_and\_send\_pubkey: RSA d3:31:a6:e5:17:09:6c:cb:aa:f1:02:85:5f:fc:e9:94:56:e5:3e:d1  
debug1: read PEM private key done: type RSA  
debug3: Wrote 640 bytes for a total of 2285  
debug1: Enabling compression at level 6.  
debug1: Authentication succeeded (publickey).  
debug1: setting up multiplex master socket  
debug3: muxserver\_listen: temporary control path /home/user/.ansible/cp/ansible-ssh-server1-22-user.FZ7QTgXgs6oGvNas  
debug2: fd 4 setting O\_NONBLOCK  
debug3: fd 4 is O\_NONBLOCK  
debug3: fd 4 is O\_NONBLOCK  
debug1: channel 0: new [/home/user/.ansible/cp/ansible-ssh-server1-22-user.FZ7QTgXgs6oGvNas]  
debug3: muxserver\_listen: mux listener channel 0 fd 4  
debug1: Entering interactive session.  
debug2: set\_control\_persist\_exit\_time: schedule exit in 60 seconds

Server 2 (ok):

OpenSSH\_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013  
debug1: Reading configuration data /etc/ssh/ssh\_config  
debug1: Applying options for \*  
debug1: auto-mux: Trying existing master  
debug2: fd 3 setting O\_NONBLOCK  
debug2: mux\_client\_hello\_exchange: master version 4  
debug3: mux\_client\_request\_forwards: requesting forwardings: 0 local, 0 remote  
debug3: mux\_client\_request\_session: entering  
debug3: mux\_client\_request\_alive: entering  
debug3: mux\_client\_request\_alive: done pid = 18226  
debug3: mux\_client\_request\_session: session request sent  
debug1: mux\_client\_request\_session: master session id: 2  
debug1: mux\_client\_request\_session: master session id: 2  
debug3: mux\_client\_read\_packet: read header failed: Broken pipe  
debug2: Received exit status from master 0  
Shared connection to server2 closed.

I’m not familiar with SSH, so I don’t understand much what’s going on in the failing case. It looks to me like it’s trying out different authentication methods, and eventually is successful. (For authentication, I have keys set up in authorized\_keys in both servers.)

Version info:

control:  
CentOS release 6.6 (Final)  
Python 2.7.3  
openssh.x86\_64 5.3p1-104.el6

server1:  
CentOS release 6.2 (Final)  
Python 2.6.6  
openssh.x86\_64 5.3p1-70.el6

server2:  
CentOS release 6.2 (Final)  
Python 2.6.6  
openssh.x86\_64 5.3p1-70.el6

I hope someone could shed some light on this, or at least be able to propose a workaround, so I could get it to work.

Paavo

---

<div class="post-metadata">

**Author:** ![Paavo](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@Paavo](https://forum.ansible.com/u/Paavo)\
**Post date:** [November 11, 2014, 1:42am UTC](https://forum.ansible.com/t/occasional-ssh-issue-when-running-a-playbook-ssh-command-stdout-empty/16729/2 "2014-11-11T01:42:28Z")

</div>

Hey again.

As a workaround, I have now decided to force Ansible to use Paramiko instead of OpenSSH, and my playbooks work.

Paavo