Unable to reach to EC2 instance using dynamic inventory with a single region

Archives Ansible Project
1

Hi all,

I am pretty new to Ansible. I am using ec2.py to connect to my EC2 instances and run my ansible scripts on them. Previously, it was working fine when I was using “all” regions. But now when I change the region to a specific one region (any of the available ones), in ec2.ini, it gives me below error for the first time and after second time onward, it works well.

`

UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}

`

What I do is use ec2 module to create a instance and then wait_for port 22 on each host to be up. It wait successfully and moves forward but in the immediate next step when I try to connect to the instance, it gives above error first time. Here is my wait_for task: (ec2_server is the variable registered from ec2 module)

`

name: wait for ssh server to be running
wait_for: host={{ item.public_dns_name }} port=22
with_items: “{{ec2_server.instances | default()}}”
when: item.state == ‘running’

`

I ping in between to the instance using private ip and the ping succeeds. I suspect it is some timing issue or DNS name resolution issue with AWS. Has anyone faced such problem before?

Any help would be appreciated. Thanks !

2

Would you mind to post your ec2.ini with the credentials taken out? Also, what is the output of ec2.py when you run it alone?

3

Thanks @Alexander for your reply. Here is ec2.ini

`

Ansible EC2 external inventory script settings

4

Hi Nirav,

I have a vague recollection of at some point having had a similar issue with dynamic inventory that was solved by placing the availability zone in the region field instead of the region. My memory is far from a reliable source of information so if you decide to give that a shot don’t be surprised if it doesn’t work.

With that said, I just re-read your original post and your statement that this is only a problem on the first run stuck out because the script uses caching to avoid hitting the AWS api all the time. The related docs are here:

http://docs.ansible.com/ansible/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script

Though I’m sure you’ve read through those several times already, the bit I believe affects your specific situation is right at the bottom of that section:

Note that the AWS inventory script will cache results to avoid repeated API calls, and this cache setting is configurable in ec2.ini. To explicitly clear the cache, you can run the ec2.py script with the --refresh-cache parameter:

# ./ec2.py --refresh-cache

Have you tried adding the --refresh-cache option to your first run of the ec2.py script? If that works, then bully! but if it doesn’t, would you mind posting the related results from the first run of your playbook with the error and then also the related portions of the second one where it works?

Thanks,
Alex

5

Hi Alex,

  1. I am not sure I understood “placing the availability zone in the region field instead of the region” correctly. Because when I exported “us-west-2a” as AWS_DEFAULT_REGION (which my ec2.py is using to filter region), it gave me error like this:

`

ERROR! The file inventory/ec2.py is marked as executable, but failed to execute correctly. If this is not supposed to be an executable script, correct this with chmod -x inventory/ec2.py.
Inventory script (inventory/ec2.py) had an execution error: region name: us-west-2a likely not supported, or AWS is down. connection to region failed.
inventory/ec2.py:3: Error parsing host definition ‘’‘’': No closing quotation

`

Let me know where can I place AZ instead of region.

  1. Yes, I tried executing
    ./inventory/ec2.py --refresh-cache
    between instance creation playbook and ping command, but with no luck ! And IMHO it doesn’t matter to refresh cache as I have “cache_max_age = 0” in ec2.ini. (Correct me if I am wrong.)

For the first time when I ping doesn’t succeed, verbose and console output is:

[Private_IP1 and Private_IP2 are the instances already running and matching the filter in ec2.py, Private_IP3 is the IP of the instance just launched and waited for port 22 successfully]

`

Using /etc/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP3> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP3> SSH: PlayContext set ssh_common_args: ()
<Private_IP3> SSH: PlayContext set ssh_extra_args: ()
<Private_IP1> SSH: PlayContext set ssh_common_args: ()
<Private_IP1> SSH: PlayContext set ssh_extra_args: ()
<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP3> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP3 ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-147953530428586” && echo ansible-tmp-1474528176.32-147953530428586=“echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-147953530428586” ) && sleep 0’“'”‘’
<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP1 ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719” && echo ansible-tmp-1474528176.32-242806970585719=“echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719” ) && sleep 0’“'”‘’
<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP2> SSH: PlayContext set ssh_common_args: ()
<Private_IP2> SSH: PlayContext set ssh_extra_args: ()
<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP2 ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666” && echo ansible-tmp-1474528176.32-96061508357666=“echo $HOME/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666” ) && sleep 0’“'”‘’
<Private_IP1> PUT /tmp/tmplSwpxr TO /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719/ping
<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP1> SSH: PlayContext set ssh_common_args: ()
<Private_IP1> SSH: PlayContext set sftp_extra_args: ()
<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP1> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r ‘[Private_IP1]’
<Private_IP2> PUT /tmp/tmptkiK31 TO /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666/ping
<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP2> SSH: PlayContext set ssh_common_args: ()
<Private_IP2> SSH: PlayContext set sftp_extra_args: ()
<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP2> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r ‘[Private_IP2]’
<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP1> SSH: PlayContext set ssh_common_args: ()
<Private_IP1> SSH: PlayContext set ssh_extra_args: ()
<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP1 ‘/bin/sh -c ‘"’“‘LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719/ping; rm -rf “/home/centos/.ansible/tmp/ansible-tmp-1474528176.32-242806970585719/” > /dev/null 2>&1 && sleep 0’”’“‘’
<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=”/home/ubuntu/.ssh/MyKey.pem")
<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP2> SSH: PlayContext set ssh_common_args: ()
<Private_IP2> SSH: PlayContext set ssh_extra_args: ()
<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP2 ‘/bin/sh -c ‘"’“‘LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666/ping; rm -rf “/home/centos/.ansible/tmp/ansible-tmp-1474528176.32-96061508357666/” > /dev/null 2>&1 && sleep 0’”’"‘’
Private_IP1 | SUCCESS => {
“changed”: false,
“invocation”: {
“module_args”: {
“data”: null
},
“module_name”: “ping”
},
“ping”: “pong”
}
Private_IP2 | SUCCESS => {
“changed”: false,
“invocation”: {
“module_args”: {
“data”: null
},
“module_name”: “ping”
},
“ping”: “pong”
}
Private_IP3 | UNREACHABLE! => {
“changed”: false,
“msg”: “Failed to connect to the host via ssh.”,
“unreachable”: true
}

`

On the second time (after half a min max) when it succeeds, the logs are:

`

Using /etc/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<Private_IP3> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP1> SSH: PlayContext set ssh_common_args: ()
<Private_IP3> SSH: PlayContext set ssh_common_args: ()
<Private_IP3> SSH: PlayContext set ssh_extra_args: ()
<Private_IP1> SSH: PlayContext set ssh_extra_args: ()
<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP1 ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439” && echo ansible-tmp-1474528970.94-42038548997439=“echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439” ) && sleep 0’“'”‘’
<Private_IP3> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP3 ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121” && echo ansible-tmp-1474528970.94-11790883317121=“echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121” ) && sleep 0’“'”‘’
<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP2> SSH: PlayContext set ssh_common_args: ()
<Private_IP2> SSH: PlayContext set ssh_extra_args: ()
<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r Private_IP2 ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115” && echo ansible-tmp-1474528970.94-115245318606115=“echo $HOME/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115” ) && sleep 0’“'”‘’
<Private_IP1> PUT /tmp/tmpIzmsRg TO /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439/ping
<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP1> SSH: PlayContext set ssh_common_args: ()
<Private_IP1> SSH: PlayContext set sftp_extra_args: ()
<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP1> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r ‘[Private_IP1]’
<Private_IP2> PUT /tmp/tmppBD_UL TO /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115/ping
<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP2> SSH: PlayContext set ssh_common_args: ()
<Private_IP2> SSH: PlayContext set sftp_extra_args: ()
<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP2> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r ‘[Private_IP2]’
<Private_IP1> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP1> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP1> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP1> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP1> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP1> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP1> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP1> SSH: PlayContext set ssh_common_args: ()
<Private_IP1> SSH: PlayContext set ssh_extra_args: ()
<Private_IP1> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP1 ‘/bin/sh -c ‘"’“‘LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439/ping; rm -rf “/home/centos/.ansible/tmp/ansible-tmp-1474528970.94-42038548997439/” > /dev/null 2>&1 && sleep 0’”’“‘’
<Private_IP2> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP2> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP2> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP2> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=”/home/ubuntu/.ssh/MyKey.pem")
<Private_IP2> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP2> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP2> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP2> SSH: PlayContext set ssh_common_args: ()
<Private_IP2> SSH: PlayContext set ssh_extra_args: ()
<Private_IP2> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP2> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP2 ‘/bin/sh -c ‘"’“‘LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115/ping; rm -rf “/home/centos/.ansible/tmp/ansible-tmp-1474528970.94-115245318606115/” > /dev/null 2>&1 && sleep 0’”’“‘’
Private_IP1 | SUCCESS => {
“changed”: false,
“invocation”: {
“module_args”: {
“data”: null
},
“module_name”: “ping”
},
“ping”: “pong”
}
Private_IP2 | SUCCESS => {
“changed”: false,
“invocation”: {
“module_args”: {
“data”: null
},
“module_name”: “ping”
},
“ping”: “pong”
}
<Private_IP3> PUT /tmp/tmpQBmgof TO /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121/ping
<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=”/home/ubuntu/.ssh/MyKey.pem")
<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP3> SSH: PlayContext set ssh_common_args: ()
<Private_IP3> SSH: PlayContext set sftp_extra_args: ()
<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP3> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r ‘[Private_IP3]’
<Private_IP3> ESTABLISH SSH CONNECTION FOR USER: centos
<Private_IP3> SSH: ansible.cfg set ssh_args: (-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<Private_IP3> SSH: ANSIBLE_HOST_KEY_CHECKING/host_key_checking disabled: (-o)(StrictHostKeyChecking=no)
<Private_IP3> SSH: ANSIBLE_PRIVATE_KEY_FILE/private_key_file/ansible_ssh_private_key_file set: (-o)(IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”)
<Private_IP3> SSH: ansible_password/ansible_ssh_pass not set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<Private_IP3> SSH: ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set: (-o)(User=centos)
<Private_IP3> SSH: ANSIBLE_TIMEOUT/timeout set: (-o)(ConnectTimeout=10)
<Private_IP3> SSH: PlayContext set ssh_common_args: ()
<Private_IP3> SSH: PlayContext set ssh_extra_args: ()
<Private_IP3> SSH: found only ControlPersist; added ControlPath: (-o)(ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r)
<Private_IP3> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o ‘IdentityFile=“/home/ubuntu/.ssh/MyKey.pem”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=centos -o ConnectTimeout=10 -o ControlPath=/home/ubuntu/.ansible/cp/ansible-ssh-%h-%p-%r -tt Private_IP3 ‘/bin/sh -c ‘"’“‘LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/centos/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121/ping; rm -rf “/home/centos/.ansible/tmp/ansible-tmp-1474528970.94-11790883317121/” > /dev/null 2>&1 && sleep 0’”’"‘’
Private_IP3 | SUCCESS => {
“changed”: false,
“invocation”: {
“module_args”: {
“data”: null
},
“module_name”: “ping”
},
“ping”: “pong”
}

`

Let me know if you find something.

Thanks again,
Nirav

6

Hi Nirav,

Sorry about the lack of clarity in my suggestion. I was referring to your ec2.ini, specifically:

`

AWS regions to make calls to. Set this to ‘all’ to make request to all regions

in AWS and merge the results together. Alternatively, set this to a comma

separated list of regions. E.g. ‘us-east-1,us-west-1,us-west-2’

regions = all
regions_exclude = us-gov-west-1,cn-north-1
`

would become:

`

AWS regions to make calls to. Set this to ‘all’ to make request to all regions

in AWS and merge the results together. Alternatively, set this to a comma

separated list of regions. E.g. ‘us-east-1,us-west-1,us-west-2’

regions = us-west-2a
regions_exclude = us-gov-west-1,cn-north-1

`

If you were using that particular availability zone. I’d be surprised if that worked, however, what happens when you unset the environment variable and set the region in the ec2.ini file?

Alex

7

Hi Alex,

Replacing availability zone in ec2.ini also gave me the same error:

`

ERROR! The file inventory/ec2.py is marked as executable, but failed to execute correctly. If this is not supposed to be an executable script, correct this with chmod -x inventory/ec2.py.
Inventory script (inventory/ec2.py) had an execution error: region name: us-west-2a likely not supported, or AWS is down. connection to region failed.
inventory/ec2.py:3: Error parsing host definition ‘’‘’': No closing quotation

`

Unsetting the environment variable and setting region in ec2.ini does not help and gives the same Unreachable error. :frowning:

Thanks,
Nirav

8

Hi Nirav,

This is a tough nut to crack. Which version of Ansible are you using?

Which region is it specifically that is failing? Or is it just any region that is specified over ‘any’ region?

Alex

9

Hi Alex

Sorry for late reply.

I am using Ansible 2.1.1.0.

It happens for any available region over AWS.

One more finding:

  • If I remove port in wait_for, it works ok.
10

Hi Alex / everyone,

Another finding:

  • This also works fine:

`
wait_for: host={{ item.public_dns_name }} port=22 delay=60 timeout=320 state=started

`

I want to understand why

`
wait_for: host={{ item.public_dns_name }} port=22

`

does not work when I query for a particular region? Does it return before port 22 is actually open or before machine is actually SSH connect-able?

Just FYI, I am using a custom AMI built on top of Official CentOS 7 AMI (If this helps or if it is an OS specific issue).

Thanks,
Nirav

11

Hi,

While browsing the integration tests of Ansible modules, I cam across this:
https://github.com/ansible/ansible/blob/7f8e8ddca9cf103c05bf39d68ebb2e2ded4067f2/test/utils/ansible-playbook_integration_runner/ec2.yml

It has a task named “Wait a little longer for centos”. So anyone know why is it there? My target hosts run CentOS also, if this is the issue for the behavior mentioned in my question.

Thanks,
Nirav

12

I could not find the real reason why the SSH on EC2 instance running CentOS was not able to connect. The workaround I applied was to write a module which try to connect host over SSH for given number of times and return success when it is successful so that further plays won’t fail because of unreachable error.

wait_for_SSH.py

`

#!/usr/bin/python
‘’’
module: wait_for_SSH
short_description: Waits for a host to be SSH connectable.
description:

  • Tries to Ansible ping(not ICMP ping) to host as per the passed parameters
    options:
    host:
    description:
  • A resolvable hostname or IP address to ansible ping
    required: true
    retries:
    description:
  • maximum number of times to retry
    required: false
    default: 10
    delay:
    description:
  • number of seconds to wait between two consecutive pings
    required: false
    default: 5
    ‘’’
    from ansible.module_utils.basic import *
    from subprocess import call
    import time

def validate_params(module, retries, delay):
if retries < 0:
module.fail_json(msg=“retries should be greater than 0”)
if delay < 0:
module.fail_json(msg=“delay should be greater than 0”)
return

def main():
fields = {
“host”: {“required”: True, “type”: “str”},
“retries”: {“required”: False, “type”: “int”, “default”: 10},
“delay”: {“required”: False, “type”: “int”, “default”: 5}
}
module = AnsibleModule(argument_spec=fields)
host = module.params[‘host’]
retries = module.params[‘retries’]
delay = module.params[‘delay’]
count = 0
output = 1
validate_params(module, retries, delay)
while (count < retries) and (output != 0):
if delay:
time.sleep(delay)
output = call([“ansible”, “all”, “-i”, “,”+host, “-m”, “ping”])
count += 1
response = {“output” : output}
module.exit_json(changed=False, output=response)

if name == ‘main’:
main()

`

And executed it after wait_for port 22.

`

…

Wait only for running instances because ‘ec2_server’ might contain terminated instances to fulfil exact_count condition

  • name: wait for ssh server to be running
    wait_for: host={{ item.public_dns_name }} port=22 search_regex=OpenSSH
    with_items: “{{ec2_server.instances | default()}}”
    when: item.state == ‘running’

wait_for_SSH is our custom module which tries to Ansible ping on created instances until it is successful as per retries specified(bug AD-3)

  • name: Ensure SSH is running
    wait_for_SSH:
    host: “{{item.private_ip}}”
    register: moduleoutput
    with_items: “{{ec2_server.instances | default()}}”
    when: item.state == ‘running’
    …<>…

`

Thanks,
Nirav