# EC2 slow cloud-init, Ansible SSH connection fails due to race condition (wait\_for is not good enough)

**URL:** <https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345>\
**Category:** Ansible Project\
**Created:** [January 25, 2016, 9:03pm UTC](https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345 "2016-01-25T21:03:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![James\_Cuzella](https://avatars.discourse-cdn.com/v4/letter/j/c68b51/32.png) [@James\_Cuzella](https://forum.ansible.com/u/James_Cuzella)\
**Post date:** [January 25, 2016, 9:03pm UTC](https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345/1 "2016-01-25T21:03:56Z")

</div>

Hello,

I believe I’ve found an interesting race condition during EC2 instance creation due to a slow-running cloud-init process. The issue is that cloud-init appears to create the initial login user & installs the public SSH key onto a newly started EC2 instance, then restarts sshd. It takes a while to do this, and creates a race condition where Ansible cannot connect to the host and fails the playbook run. In my playbook, I’m using the ec2 module, followed by add\_host, and then wait\_for to wait for the SSH port to be open. I have also experimented with using a simple “shell: echo host\_is\_up” command with a retry / do-until loop. However this also fails because Ansible wants the initial SSH connection to be successful, which it will not in this case. So Ansible does not retry ☹

It appears that due to the user not existing until ~3 minutes after it is booted and sshd is listening on port 22, Ansible cannot connect as the initial login user for the CentOS AMI (“centos”). So the SSH port open check is not good enough to detect and wait for the port to be open AND the login user to exist. The simple echo shell command with retry do/until loop also does not work, because the very first SSH connection Ansible tries to make to run the module fails also.

For some detailed debug info, and a playbook to reproduce the issue, please see this Gist: [https://gist.github.com/trinitronx/afd894c89384d413597b](https://gist.github.com/trinitronx/afd894c89384d413597b)

My question is: Has anyone run into a similar issue with EC2 instances being slow to become available causing Ansible to fail to connect, and also found a solution to this?

I realize that a sleep task is one possible solution (and I may be forced to reach for that sledgehammer), but it doesn’t feel like the absolute best solution because we really want to wait for both cloud-init to be finished creating “centos” user on the instance AND SSH to be up. So really, the only other way I can think of is to somehow tell SSH to retry connecting as centos until it succeeds or a surpasses a very long timeout. Is this possible? Are there better ways of handling this?

---

<div class="post-metadata">

**Author:** ![Jared\_Bristow](https://avatars.discourse-cdn.com/v4/letter/j/eada6e/32.png) [@Jared\_Bristow](https://forum.ansible.com/u/Jared_Bristow)\
**Post date:** [May 9, 2016, 7:46pm UTC](https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345/2 "2016-05-09T19:46:21Z")

</div>

I am having this same issue. Did you ever figure out a solution?

I have 3 different images I’m testing against: CentOS6, CentOS7, Sles12. The strange thing is that I only seem to have a problem on CentOS7.

---

<div class="post-metadata">

**Author:** ![Allen\_Sanabria1](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Allen\_Sanabria1](https://forum.ansible.com/u/Allen_Sanabria1)\
**Post date:** [May 9, 2016, 8:20pm UTC](https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345/3 "2016-05-09T20:20:04Z")

</div>

This is what I do, to make sure that SSH comes up, but also wait until the user has been created on my instance.

`

```auto
- set_fact:
    ec2_ip: "{{ ec2_name | get_instance(aws_region, state='running') }}"

- name: Wait for SSH to come up on instance
  wait_for:
    host: "{{ ec2_ip }}"
    port: 22
    delay: 15
    timeout: 320
    state: started

- name: Wait until the ansible user can log into the host.
  local_action: command ssh -oStrictHostKeyChecking=no ansible@{{ ec2_ip }} exit
  register: ssh_output
  until: ssh_output.rc == 0
  retries: 20
  delay: 10

```

`

---

<div class="post-metadata">

**Author:** ![jdelaporte](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jdelaporte/32/366_2.png) [@jdelaporte](https://forum.ansible.com/u/jdelaporte)\
**Post date:** [July 13, 2016, 5:40pm UTC](https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345/4 "2016-07-13T17:40:20Z")

</div>

I just discovered this issue as well, with various random ssh connection or generic authentication/permission failure messages, some occurring after a play successfully passed a task or two. It occurred very consistently with many CentOS 7 t2.nano hosts. A 2-minute pause after waiting for ssh listener resolved it for me. The system logs showed the centos user added about 1 minute into boot time, so I gave it two minutes to be generous:

`

- name: wait for instances to listen on port:22  
wait\_for:  
state: started  
host: “{{ item }}”  
port: 22

- name: wait for boot process to finish  
pause: minutes=2

`

It also helped to make sure I removed old host keys, even though I have strict checking turned off:

`

- name: remove old host keys  
known\_hosts:  
name: “{{item}}”  
state: absent  
with\_items: “{{aws\_eips}}”  
`

Joanna

PS) Some of the errors I saw caused by this:  
“failed to transfer file to /tmp/.ansible”

“Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in "/tmp". Failed command was: ( umask 77 && mkdir -p "` echo /tmp/.ansible/…”

---

<div class="post-metadata">

**Author:** ![j.r.hawkesworth](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@j.r.hawkesworth](https://forum.ansible.com/u/j.r.hawkesworth)\
**Post date:** [July 14, 2016, 8:16pm UTC](https://forum.ansible.com/t/ec2-slow-cloud-init-ansible-ssh-connection-fails-due-to-race-condition-wait-for-is-not-good-enough/20345/5 "2016-07-14T20:16:53Z")

</div>

I’m not an ec2 user but I wonder if it might be possible to adapt the approach used here to wait for a webservice to return:

[https://groups.google.com/forum/#!topic/ansible-project/iLjIbsCASWU](https://groups.google.com/forum/#!topic/ansible-project/iLjIbsCASWU)

In that case he’s using uri with an ‘until’ and using the ‘default’ filter to make sure no results doesn’t mean a task failure.

`- name: run test`  
`uri:`  
`url: "[https://0.0.0.0:3030/api/canary](https://0.0.0.0:3030/api/canary)"`  
`validate_certs: no`  
`register: result`  
`until: result['status']|default(0) == 200`

Obviously you’d have to replace uri with something that tells you the instance is ready to start working, but it might give you a way to get rid of the need for the arbitrary pause.

Jon