Forks?

Archives Ansible Project
1

I have my forks set to 50 in ~/.Ansible.CTG and the forks setting seems to work for the first set of boxes, but doesn’t seem to work for the rest. I thought the forks setting would keep $forks connections going until all boxes in the inventory have been touched.

Was I thinking wrong? (I’m running an ad hoc command on 16k boxes.)

Mike

2

Mike

There are dedicated docs pages[1][2] for this topic. I must admit there is a fair bit of missing info in your query.

So to recap what we know:

  • ~16k hosts
  • forks set to 50
  • unknown strategy
  • unknown config for async, throttle, serial and the other settings that can affect
  • unknown number of running ansible forks/processes on the control node
  • unknown if still polling for async
  • unknown other things as I am typing this in a hurry
  1. https://docs.ansible.com/ansible/latest/user_guide/playbooks_async.html
  2. https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html
3

I’ll look at the docs you mention. My command is: ansible all -I list -m command -m ‘rpm -qa openssl’. Other than the forks setting and the log setting I’ve not changed the configuration. (forks = 50 and log = /tmp/${USER}-ansible.log

Mike

4

With 16k hosts, it’s possible that you never actually see the max 50 forks in use. We’ve made some pretty big performance improvements to large host counts, but you might only see somewhere around 30 forks in use, maybe less, even though you have specified a higher count.

The forking limitation is caused by having a high fork count, with a CPU (single CPU core) bound process. Effectively, we have a single process that orchestrates forking, and processing results. The more forks, the more CPU bound that process is. The more hosts you have in inventory, the more CPU is used to calculate variables when spawning workers and processing results.

Once the first sets of results start coming in, the process becomes busier, and is unable to process results and spawn new processes to keep the fork count up. So what happens is results take longer to process, which delays spawning new workers, and before you reach the theoretical max of your 50 configured forks, other forks are exiting, and the process never gets back to the max.

So ultimately, what you have to do is baseline your use case, and adjust forks to a more appropriate level. Decrease forks, or split your inventory into smaller chunks, or do both.

5

Great explanation. Thank you.

Mike

6

Installing awx, and ansible-core to support it, gets... a bit
fraught, if you're a bit leery about just using "pip install" willy
nilly for everything.

I've been publishing RPM building tools for ansible-core and ansible
for some time now, at

    https://github.com/nkadel/ansiblerepo

I now publish some to build "awxkit", the package which provides the
"awx" command line tool.

    https://github.com/nkadel/awxkitrepo/

I partly do these because deducing from the ansible installation tools
that you need to install and use python 3.8 or later, instead of the
default python 3.6, is work people should not have to do. Nor should
deployment critical files have to be built up manually on unpredictale
environments with ever-shifting python dependency chain results.

Unfortunately, I'm afrad that mrmeee's old yum repositories for awx
seems to be moribund: this might serve as a useful place to start
from, using the system provided python39 rather than relying on the
more awkward SCLO provided python tools.

Nico Kadel-Garcia