SSH connection problem

Archives Ansible Project
1

Hi

I’m having some issues getting ansible to connect using SSH, but it only seems to affect one particular playbook. The odd thing is I can connect manually using ssh, but ansible fails with a timed out error. Other playbooks/hosts work absolutely fine with exactly the same config.

Here’s the output I get - a lot less than I’d normally expect for a failure (no ssh debug output etc):

`
-----@localhost:~/github/--------/ansible$ ansible-playbook -i hosts playbook.yml -vvvv

PLAY [all] ********************************************************************

GATHERING FACTS ***************************************************************
<ec2--------------.eu-west-1.compute.amazonaws.com> ESTABLISH CONNECTION FOR USER: ec2-user on PORT 22 TO ec2--------------.eu-west-1.compute.amazonaws.com
fatal: [ec2--------------.eu-west-1.compute.amazonaws.com] => {‘msg’: ‘FAILED: timed out’, ‘failed’: True}

TASK: [common | install wget] *************************************************
FATAL: no hosts matched or all hosts have already failed – aborting

PLAY RECAP ********************************************************************
to retry, use: --limit @/home/-------/playbook.retry

ec2--------------.eu-west-1.compute.amazonaws.com : ok=0 changed=0 unreachable=1 failed=0

`

But it works fine using ssh directly:

`
-------@localhost:~/github/---------/ansible$ ssh ec2-user@ec2--------------.eu-west-1.compute.amazonaws.com

Last login: Thu May 15 05:15:13 2014 from —.–.–.–
[ec2-user@ip-------------- ~]$ exit
logout
Connection to ec2--------------.eu-west-1.compute.amazonaws.com closed.
`

SSH config is set to use corkscrew to proxy over http, which works fine on other playbooks/sites and from command line:

`
KeepAlive yes
ServerAliveInterval 60

Host *.compute.amazonaws.com
IdentityFile ~/.ssh/-----------.pem
ProxyCommand /usr/bin/corkscrew -------.--------- 8080 %h %p
StrictHostKeyChecking no
`

Tried all the usual stuff, swapping hosts between working/failing playbooks, rebooting, etc - out of ideas!

Any suggestions?

Thanks
Ian

2

Please show the output from that first task as it’s trying to connect with “-vvvv” on the command line, and also the SSH debug from the other so we can see what ports it is trying to talk to there, etc.

It also doesn’t matter here I’m guessing, but it’s always useful to indicate your ansible version when asking a question.

Your host OS also matters, “-c ssh” will read your SSH config, paramiko will not.

Thanks!

3

Thanks for the reply - not too sure what you mean exactly! The output I pasted was the “broken” one running with -vvvv, and this is the output from a working ansible playbook using -vvvv (cut just before it starts running any tasks, but the connection was successful)

All using ansible 1.6. Host OS is Ubuntu 13.10. Both ansible and host OS are the same whether it works or not!

The bit that I dont get is why it works fine with other ansible playbooks, and that the debug output doesn’t even appear to be trying an actual SSH connection (no ssh debug output, just a connection timed out). Any other failure normally comes with a pile of SSH debug like the example below, just get nothing on this one.

`
-------@----------:~/github/--------------$ ansible-playbook -i aws_hosts site.yml -vvvv

PLAY [all] ********************************************************************

GATHERING FACTS ***************************************************************
<ec2-------------.eu-west-1.compute.amazonaws.com> ESTABLISH CONNECTION FOR USER: ec2-user
<ec2-------------.eu-west-1.compute.amazonaws.com> REMOTE_MODULE setup
<ec2-------------.eu-west-1.compute.amazonaws.com> EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/home/-------/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ec2-user’, ‘-o’, ‘ConnectTimeout=10’, ‘ec2-------------.eu-west-1.compute.amazonaws.com’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-tmp-1400161941.7-189110912378024 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1400161941.7-189110912378024 && echo $HOME/.ansible/tmp/ansible-tmp-1400161941.7-189110912378024’”]
<ec2-------------.eu-west-1.compute.amazonaws.com> PUT /tmp/tmpe6lXpq TO /home/ec2-user/.ansible/tmp/ansible-tmp-1400161941.7-189110912378024/setup
<ec2-------------.eu-west-1.compute.amazonaws.com> EXEC [‘ssh’, ‘-C’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/home/-------/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ec2-user’, ‘-o’, ‘ConnectTimeout=10’, ‘ec2-------------.eu-west-1.compute.amazonaws.com’, ‘/bin/sh -c 'sudo -k && sudo -H -S -p “[sudo via ansible, key=fvfeyzswxnomddeasiytndsflxlotzve] password: " -u root /bin/sh -c '”'“'echo SUDO-SUCCESS-fvfeyzswxnomddeasiytndsflxlotzve; /usr/bin/python /home/ec2-user/.ansible/tmp/ansible-tmp-1400161941.7-189110912378024/setup; rm -rf /home/ec2-user/.ansible/tmp/ansible-tmp-1400161941.7-189110912378024/ >/dev/null 2>&1'”'"''’]
ok: [ec2--------------.eu-west-1.compute.amazonaws.com]

`

If I connect using ssh command, this is the debug output from a successful connect:

`
-------@------------:~/github/-------------$ ssh ec2-user@ec2--------------.eu-west-1.compute.amazonaws.com -vvvv
OpenSSH_6.2p2 Ubuntu-6ubuntu0.4, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/-------/.ssh/config
debug1: /home/-------/.ssh/config line 9: Applying options for .compute.amazonaws.com
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Executing proxy command: exec /usr/bin/corkscrew -------.--------.-- 8080 ec2--------------.eu-west-1.compute.amazonaws.com 22
debug1: permanently_drop_suid: 1000
debug3: Incorrect RSA1 identifier
debug3: Could not load “/home/-------/.ssh/-----------.pem” as a RSA1 public key
debug1: identity file /home/-------/.ssh/-----------.pem type -1
debug1: identity file /home/-------/.ssh/-----------.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host “ec2--------------.eu-west-1.compute.amazonaws.com” from file “/home/-------/.ssh/known_hosts”
debug3: load_hostkeys: found key type RSA in file /home/-------/.ssh/known_hosts:74
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 126/256
debug2: bits set: 502/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA ab:80:6d:67:d9:32:35:9e:cf:8c:e3:40:bf:3f:f4:a7
debug3: load_hostkeys: loading entries for host “ec2--------------.eu-west-1.compute.amazonaws.com” from file “/home/-------/.ssh/known_hosts”
debug3: load_hostkeys: found key type RSA in file /home/-------/.ssh/known_hosts:74
debug3: load_hostkeys: loaded 1 keys
debug1: Host ‘ec2---------------.eu-west-1.compute.amazonaws.com’ is known and matches the RSA host key.
debug1: Found key in /home/-------/.ssh/known_hosts:74
debug2: bits set: 504/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: ------@--------- (0x------------),
debug2: key: -------@---------------- (0x-------------),
debug2: key: /home/--------/.ssh/------------.pem ((nil)), explicit
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file ‘/tmp/krb5cc_1000’ not found

debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file ‘/tmp/krb5cc_1000’ not found

debug1: Unspecified GSS failure. Minor code may provide more information

debug1: Unspecified GSS failure. Minor code may provide more information
Credentials cache file ‘/tmp/krb5cc_1000’ not found

debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: -----@---------
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering RSA public key: -------@----------------
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /home/--------/.ssh/------------.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA e5:df:37:7f:6b:1a:69:0e:01:89:ea:5e:6e:77:2a:28
debug2: we sent a publickey packet, wait for reply
debug1: Authentication succeeded (publickey).
Authenticated to ec2-------------------.eu-west-1.compute.amazonaws.com (via proxy).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_VTNR
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env VTE_VERSION
debug3: Ignored env CLUTTER_DISABLE_XINPUT
debug3: Ignored env GJS_DEBUG_OUTPUT
debug3: Ignored env WINDOWID
debug3: Ignored env OLDPWD
debug3: Ignored env GNOME_KEYRING_CONTROL
debug3: Ignored env GJS_DEBUG_TOPICS
debug3: Ignored env GTK_MODULES
debug3: Ignored env USER
debug3: Ignored env http_proxy
debug3: Ignored env LS_COLORS
debug3: Ignored env XDG_SESSION_PATH
debug3: Ignored env XDG_SEAT_PATH
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env ftp_proxy
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env DEFAULTS_PATH
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env PATH
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env PWD
debug3: Ignored env GNOME_KEYRING_PID
debug1: Sending env LANG = en_GB.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env GDM_LANG
debug3: Ignored env MANDATORY_PATH
debug3: Ignored env https_proxy
debug3: Ignored env GDMSESSION
debug3: Ignored env CINNAMON_VERSION
debug3: Ignored env SHLVL
debug3: Ignored env XDG_SEAT
debug3: Ignored env HOME
debug3: Ignored env LANGUAGE
debug3: Ignored env no_proxy
debug3: Ignored env GNOME_DESKTOP_SESSION_ID
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env GOPATH
debug3: Ignored env LESSOPEN
debug3: Ignored env TEXTDOMAIN
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env DISPLAY
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env LESSCLOSE
debug3: Ignored env TEXTDOMAINDIR
debug3: Ignored env COLORTERM
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Thu May 15 05:27:41 2014 from —.–.–.–
[ec2-user@ip------------------ ~]$ exitdebug2: channel 0: rcvd eof
debug2: channel 0: output open → drain
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug2: channel 0: rcvd eow
debug2: channel 0: close_read
debug2: channel 0: input open → closed
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close

logout
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain → closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)

Connection to ec2-----------------.eu-west-1.compute.amazonaws.com closed.
Transferred: sent 4144, received 2544 bytes, in 2.2 seconds
Bytes per second: sent 1914.9, received 1175.6
debug1: Exit status 0
`

Thanks
Ian

4

Are you creating the ec2 instances during this step or just before? It could be that they’re started but not yet accepting connections?

5

The instances are already created - it probably doesn’t help that I masked all the IPs, but the EC2 instance ansible is trying to connect to is the same one I can successfully SSH to.

Just tried “-c ssh” which Michael mentioned - that fixed it, but I’m still not sure why it fails if I don’t provide that, while another playbook works fine without it.

Thanks both :slight_smile:

6

Ahh, I see. Some distros like RHEL6/CentOS6 have slightly older versions of SSH that do not perform as well as paramiko. The default connection type (or transport) is “smart”, which will use paramiko for the connection if it detects an older version of SSH. As Michael said above, paramiko does not support reading the ssh_config, so the “-c ssh” forces ansible to use the ssh connection type.

You can make this permanent by changing the "transport = " setting in your ansible.cfg to “transport = smart”.

7

Did you mean “transport = ssh”?

Adam

8

smart will select “ssh” if you have ControlMaster capability on our control machine, and will default to paramiko only if it does not.

9

Yes, but if he’s passing -c ssh and it fixes the issue then he might want to change from the default to transport=ssh. Otherwise it might be using paramiko which may not be what he wants.

Adam

10

Yes I did mean “transport = ssh”, sorry for the confusion there.

11

Thanks all, I’d seen the “transport” option but hadn’t realised exactly what it meant!