# parallel execution

**URL:** <https://forum.ansible.com/t/parallel-execution/13136>\
**Category:** Ansible Project\
**Created:** [July 16, 2013, 2:38pm UTC](https://forum.ansible.com/t/parallel-execution/13136 "2013-07-16T14:38:53Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![i\_iordanov](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@i\_iordanov](https://forum.ansible.com/u/i_iordanov)\
**Post date:** [February 23, 2014, 9:31pm UTC](https://forum.ansible.com/t/parallel-execution/13136/21 "2014-02-23T21:31:32Z")

</div>

I looked at the code in ssh.py, and I don’t understand why you don’t take /etc/ssh/ssh\_known\_hosts2 under consideration when deciding if the host is in known\_hosts. The only file that is taken under consideration is $USER/.ssh/known\_hosts.

Do you think that including standard locations is a good idea?

…  
def not\_in\_host\_file(self, host):  
host\_file = os.path.expanduser(os.path.expandvars(“~${USER}/.ssh/known\_hosts”))  
if not os.path.exists(host\_file):  
print “previous known host file not found”  
return True  
…

Thanks!  
iordan

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [February 24, 2014, 7:21pm UTC](https://forum.ansible.com/t/parallel-execution/13136/22 "2014-02-24T19:21:58Z")

</div>

I’ve got nothing against these additions to look in both locations.

I’d be happy to see pull requests to this effect if you want to pass them along, or if not, you can file a ticket so we don’t forget.

Thanks!

–Michael

---

<div class="post-metadata">

**Author:** ![i\_iordanov](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@i\_iordanov](https://forum.ansible.com/u/i_iordanov)\
**Post date:** [February 24, 2014, 9:26pm UTC](https://forum.ansible.com/t/parallel-execution/13136/23 "2014-02-24T21:26:12Z")

</div>

Hi Michael,

---

<div class="post-metadata">

**Author:** ![i\_iordanov](https://avatars.discourse-cdn.com/v4/letter/i/b3f665/32.png) [@i\_iordanov](https://forum.ansible.com/u/i_iordanov)\
**Post date:** [February 24, 2014, 9:30pm UTC](https://forum.ansible.com/t/parallel-execution/13136/24 "2014-02-24T21:30:57Z")

</div>

Hi again,

I'm not sure how you want to deal with this pull request, so here is a  
link to the actual patch:

[https://github.com/ansible/ansible/pull/6156.patch](https://github.com/ansible/ansible/pull/6156.patch)

for you to have multiple options.

Cheers,  
iordan

---

<div class="post-metadata">

**Author:** ![Trevor\_Hartman](https://avatars.discourse-cdn.com/v4/letter/t/f9ae1b/32.png) [@Trevor\_Hartman](https://forum.ansible.com/u/Trevor_Hartman)\
**Post date:** [March 3, 2014, 7:05pm UTC](https://forum.ansible.com/t/parallel-execution/13136/25 "2014-03-03T19:05:36Z")

</div>

I seem to be experiencing the same or similar issue.

My ansible.cfg:

[si-cluster-settings]  
host\_key\_checking = False  
hostfile = hosts  
forks = 15

When I run a playbook on 10 nodes, they are definitely running serially as I see large delays between results coming back on each node. I also tried setting serial: 10 in the actual playbook.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [March 4, 2014, 3:25pm UTC](https://forum.ansible.com/t/parallel-execution/13136/26 "2014-03-04T15:25:43Z")

</div>

The group you have in inventory called “si-cluster-settings” is the problem.

The Ansible inventory parser doesn’t know anything about that category of settings.

They go in a section called “defaults” like so: [https://github.com/ansible/ansible/blob/devel/examples/ansible.cfg](https://github.com/ansible/ansible/blob/devel/examples/ansible.cfg)

---

<div class="post-metadata">

**Author:** ![Vincent\_Janelle](https://avatars.discourse-cdn.com/v4/letter/v/45deac/32.png) [@Vincent\_Janelle](https://forum.ansible.com/u/Vincent_Janelle)\
**Post date:** [September 29, 2014, 2:35pm UTC](https://forum.ansible.com/t/parallel-execution/13136/27 "2014-09-29T14:35:43Z")

</div>

Just an update at Michael’s request - seeing the exact same situations, with ec2.

Setting this environment variable fixes this.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [September 29, 2014, 2:39pm UTC](https://forum.ansible.com/t/parallel-execution/13136/28 "2014-09-29T14:39:16Z")

</div>

Any chance I can get a copy of your known\_hosts file?

Off list would be preferred.

I’m not sure that’s it, but I suspect it could be.

---

<div class="post-metadata">

**Author:** ![Vincent\_Janelle](https://avatars.discourse-cdn.com/v4/letter/v/45deac/32.png) [@Vincent\_Janelle](https://forum.ansible.com/u/Vincent_Janelle)\
**Post date:** [September 29, 2014, 2:44pm UTC](https://forum.ansible.com/t/parallel-execution/13136/29 "2014-09-29T14:44:43Z")

</div>

Not sure how I’d send you a copy of /dev/null, unless ansible is attempting to parse the contents of ~/.ssh/known\_hosts outside of ssh.

---

<div class="post-metadata">

**Author:** ![James\_Cammarata](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@James\_Cammarata](https://forum.ansible.com/u/James_Cammarata)\
**Post date:** [September 29, 2014, 3:05pm UTC](https://forum.ansible.com/t/parallel-execution/13136/30 "2014-09-29T15:05:54Z")

</div>

Hi Vincent, could you share a sample of the playbook you’re running as well as the results of running it with -f1, -f2 and -f4? That should determine if the playbook is indeed being serialized at some point.

Do note, however, if you’re doing something like this:

- local\_action: ec2 …  
with\_items:
- …
- …
- …

you will see serialized performance. This is caused by the fact that each pass through with\_\* loops must complete on all hosts before the next loop begins, and with local\_action you’d only be executing on a single host (localhost), so this would constrain the playbook to a serial-like performance.

Thanks!

---

<div class="post-metadata">

**Author:** ![Vincent\_Janelle](https://avatars.discourse-cdn.com/v4/letter/v/45deac/32.png) [@Vincent\_Janelle](https://forum.ansible.com/u/Vincent_Janelle)\
**Post date:** [September 29, 2014, 3:37pm UTC](https://forum.ansible.com/t/parallel-execution/13136/31 "2014-09-29T15:37:42Z")

</div>

Exactly like what was described at the start of this thread. ☹ Setting the environment variable produces the desired parallel execution.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [September 29, 2014, 4:28pm UTC](https://forum.ansible.com/t/parallel-execution/13136/32 "2014-09-29T16:28:08Z")

</div>

Ansible does read ~/.ssh/known\_hosts because it needs to know whether to lock itself down to 1 process to ask you the question about adding a new hosts to known\_hosts.

This only happens when it detects a host isn’t already there, because it must detect this before SSH asks.

And this only happens with -c ssh, -c paramiko has it’s own handling (and it’s own issues, I prefer the SSH implementation if folks have a new enough SSH to use ControlPersist).

---

<div class="post-metadata">

**Author:** ![Michael\_Blakeley](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@Michael\_Blakeley](https://forum.ansible.com/u/Michael_Blakeley)\
**Post date:** [September 29, 2014, 4:29pm UTC](https://forum.ansible.com/t/parallel-execution/13136/33 "2014-09-29T16:29:06Z")

</div>

Vincent, I now use a slightly different workaround. Instead of routing known\_hosts to /dev/null I route it to a temp file. This keeps the EC2 noise out of my default known\_hosts file, and seems to play well with ansible.

`  
Host \*.amazonaws.com  
PasswordAuthentication no  
StrictHostKeyChecking no  
UserKnownHostsFile /tmp/ec2\_known\_hosts  
User ec2-user

`

Hope that helps you.

– Mike

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [September 29, 2014, 4:29pm UTC](https://forum.ansible.com/t/parallel-execution/13136/34 "2014-09-29T16:29:20Z")

</div>

Hi James,

Each loop DOES happen within the host loop.

If you have 50 hosts and they are “with\_items”'ing, that still happens 50 hosts at a time.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [September 29, 2014, 4:30pm UTC](https://forum.ansible.com/t/parallel-execution/13136/35 "2014-09-29T16:30:28Z")

</div>

So I’m confused - are you saying you are using known\_hosts that are empty?

This seems to be a completely unrelated question.

The mention of /dev/null above seemed to be based on confusion that we didn’t read it, not that it was actually symlinked to /dev/null.

Can each of you clarify?

---

<div class="post-metadata">

**Author:** ![Michael\_Blakeley](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@Michael\_Blakeley](https://forum.ansible.com/u/Michael_Blakeley)\
**Post date:** [September 29, 2014, 4:45pm UTC](https://forum.ansible.com/t/parallel-execution/13136/36 "2014-09-29T16:45:23Z")

</div>

I took it that Vincent was referring to [my message of 2013-09-12](https://groups.google.com/d/msg/ansible-project/8p3XWlo83ho/Q1SflaZ9dyAJ). In that post I mentioned using /dev/null for the ssh UserKnownHostsFile configuration key, scoped to Host \*.[amazonaws.com](http://amazonaws.com/)

This configuration triggers single-threaded behavior from ansible because ssh never stores any record of connecting to the EC2 hosts: not the first time, not ever. Because known\_hosts is /dev/null.

– Mike

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [September 29, 2014, 4:54pm UTC](https://forum.ansible.com/t/parallel-execution/13136/37 "2014-09-29T16:54:53Z")

</div>

Ansible does not find your known hosts location from ~/.ssh/config on a per host basis and does read your ~/.ssh/known\_hosts.

It does this because it needs to know, in advance of SSH asking, whether it needs to lock.

Assume it’s running at 50/200 forks and needs to ask a question interactively, that’s why it needs to know.

So if you are saying use known\_hosts in a different file, that may be EXACTLY the problem. With host key checking on, and the data going elsewhere, it can’t be found, and ansible is locking pre-emptively.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [September 29, 2014, 4:57pm UTC](https://forum.ansible.com/t/parallel-execution/13136/38 "2014-09-29T16:57:31Z")

</div>

I’m wondering if we can detect configuration of alternative known\_hosts locations in the ~/.ssh/config and issue a warning, which should be able to key people in to turn off the checking feature.

This should close this out, I’d think.

---

<div class="post-metadata">

**Author:** ![Matt\_Jaynes](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@Matt\_Jaynes](https://forum.ansible.com/u/Matt_Jaynes)\
**Post date:** [November 10, 2014, 5:52pm UTC](https://forum.ansible.com/t/parallel-execution/13136/39 "2014-11-10T17:52:43Z")

</div>

Sounds like some great possible solutions.

Either

1. Reading the SSH config to pick up the correct known\_hosts locations (and perhaps setting ‘host\_key\_checking’ to false if the location is ‘/dev/null’ since that’s a common pattern - for instance, Vagrant does this by default, see [https://docs.vagrantup.com/v2/cli/ssh\_config.html](https://docs.vagrantup.com/v2/cli/ssh_config.html) )

or

1. A simple warning message when serialization is triggered due to known\_hosts in order to save folks from some really tough debugging

Just lost a few hours debugging this issue. For several environments, I have a client’s known\_hosts locations set to custom locations in their SSH config, so everything was running serially (a 3 minute process \* 20 servers = 60 minutes!). Persistence and sweat finally lead me to try “host\_key\_checking = False” and it finally ran in parallel - was so nice to finally see since I’d tried just about everything else I could imagine (forks, serial, ssh options, restructuring inventory, removing inventory groups, etc).

Thanks,  
Matt

---

<div class="post-metadata">

**Author:** ![Lorrin\_Nelson](https://avatars.discourse-cdn.com/v4/letter/l/8dc957/32.png) [@Lorrin\_Nelson](https://forum.ansible.com/u/Lorrin_Nelson)\
**Post date:** [June 17, 2015, 9:09pm UTC](https://forum.ansible.com/t/parallel-execution/13136/40 "2015-06-17T21:09:43Z")

</div>

Any updates on this? I took a gander through the GitHub issues but didn’t see one that seemed related.

[Previous page](https://forum.ansible.com/t/parallel-execution/13136.md?page=1)

[Next page](https://forum.ansible.com/t/parallel-execution/13136.md?page=3)
