# Threading vs. forking

**URL:** <https://forum.ansible.com/t/threading-vs-forking/24825>\
**Category:** Ansible Developer\
**Created:** [June 26, 2017, 9:17pm UTC](https://forum.ansible.com/t/threading-vs-forking/24825 "2017-06-26T21:17:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![James\_Cammarata](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@James\_Cammarata](https://forum.ansible.com/u/James_Cammarata)\
**Post date:** [June 26, 2017, 9:17pm UTC](https://forum.ansible.com/t/threading-vs-forking/24825/1 "2017-06-26T21:17:59Z")

</div>

Hi all, I’ve recently been playing around with using threads instead of multiprocessing.Process objects for the workers in Ansible, and I think it’s ready for some real-world testing.

[https://github.com/ansible/ansible/compare/threading\_instead\_of\_forking](https://github.com/ansible/ansible/compare/threading_instead_of_forking)

Here are a few synthetic test results.

First, I have a playbook that does 100 debug calls against localhost. Since debug occurs only as an action plugin, this does not hit the connection plugin layer or remote side in anyway. All of these numbers are the average of five runs:

threading = 1.208s

devel = 2.394s  
stable-2.3 = 1.968s  
stable-1.9 = 0.474s

Next, a playbook that does 100 pings against localhost. This fully involves the connection layer and module execution paths of the executor engine:

threading = 15.44s

devel = 16.53s  
stable-2.3 = 14.84s  
stable-1.9 = 7.62s

Finally, a larger setup in which ~1000 hosts with 19k variables in group\_vars/all. This runs with a limit of `all[0]` (the first host in inventory) and the playbook has a single task, which does `- debug: var=hostvars[inventory_hostname]`:

threading = 5.18s

devel = 5.08s  
stable-2.3 = 4.81s  
stable-1.9 = 2.23s

So, what do these tests tell us? First, 1.x was extremely fast, though we’ve made a lot of strides since 2.1/2.2 in closing the gap from the major overhaul of the engine. Second, it looks like devel has introduced a little bit of slowness from 2.3 (probably because of the inventory rewrite) which may require some tweaking before 2.4 is released.

But what these numbers don’t reflect is memory usage, and that’s the real benefit to using threads. With forking, we’re copying all of the in-memory data to a new process space, and because of the way CPython does ref count updating we’re probably quickly incurring memory copies and not benefiting much from copy-on-write (COW, there’s an Ansible joke in there somewhere…).

With threading, this should not happen, as the memory space is shared and not copied. So if you’ve got a playbook/env. that shows a lot of memory pressure from Ansible, I’d be really interested in hearing if/how much the threading branch improves this.

Eventually, as we move past Python 2.x, we’d look at moving these threading bits to the asyncio framework in Py3.5+, but that would be quite a ways down the road.

Thanks!

James Cammarata

---

<div class="post-metadata">

**Author:** ![Andrew\_Latham](https://avatars.discourse-cdn.com/v4/letter/a/ed8c4c/32.png) [@Andrew\_Latham](https://forum.ansible.com/u/Andrew_Latham)\
**Post date:** [June 26, 2017, 9:40pm UTC](https://forum.ansible.com/t/threading-vs-forking/24825/2 "2017-06-26T21:40:32Z")

</div>

lgtm

Nice work James

---

<div class="post-metadata">

**Author:** ![j.r.hawkesworth](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@j.r.hawkesworth](https://forum.ansible.com/u/j.r.hawkesworth)\
**Post date:** [June 30, 2017, 8:58am UTC](https://forum.ansible.com/t/threading-vs-forking/24825/3 "2017-06-30T08:58:03Z")

</div>

Thanks for this. It certainly looks encouraging, although I found when using the integrated kerberos ticket management I hit problems where the kerberos ticket was not always available when addressing more than one windows host. I guess that part of the code might need a bit of work.

Jon

---

<div class="post-metadata">

**Author:** ![James\_Cammarata](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@James\_Cammarata](https://forum.ansible.com/u/James_Cammarata)\
**Post date:** [July 5, 2017, 2:46pm UTC](https://forum.ansible.com/t/threading-vs-forking/24825/4 "2017-07-05T14:46:23Z")

</div>

That’s odd, considering all of those parts should currently be in the post-fork worker, so not sure why threading would break that. I’ll ping nitz about it.

---

<div class="post-metadata">

**Author:** ![j.r.hawkesworth](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@j.r.hawkesworth](https://forum.ansible.com/u/j.r.hawkesworth)\
**Post date:** [July 6, 2017, 2:23pm UTC](https://forum.ansible.com/t/threading-vs-forking/24825/5 "2017-07-06T14:23:26Z")

</div>

Thank you.

I had a brief look in plugins/connection/winrm.py in \_kerb\_auth

I believe tempfile.NamedTemporaryFile() is threadsafe but I’ve seen posts suggesting subprocess.Popen is not [https://stackoverflow.com/questions/21194380/is-subprocess-popen-not-thread-safe](https://stackoverflow.com/questions/21194380/is-subprocess-popen-not-thread-safe)

let me know if you need more info from me.

Jon