How can I troubleshoot the speed of ansible-playbook runs?

Archives Ansible Project
1

My ansible-playbook runs seem to be going slow. How can I work out where the slowdown is?

One thing I'm wondering is how many tasks are being executed in parallel? I have forks = 20 in ansible.cfg, but I want to confirm this is taking effect. Is there any way to dig into what ansible-playbook is doing?

Cheers,
Alex

2

Hello Alex,

I was under the impression that Ansible uses forks for provisioning different hosts not for parallel execution on one. For the latter it had to calculate dependencies which is does not AFAIK.

Regards
Mirko

3

Indeed. The play is essentially apt update and upgrade. There are about 20 hosts being run, and they seem to complete in pairs, with long waits between groups completing.

When I ssh to individual servers during the run, sometimes no apt command is running on the host when I would expect one to be, since that host hasn't completed yet.

Other directives in ansible.cfg are taking effect, but I'm wondering if the forks directive is. Is there a way for a-p to dump its configuration, so I can see all the settings?

During a run, is there a way to tell how many hosts are being run from ps output? Is there a way for a-p to display or log when it starts acting on individual hosts within a play?

This all happens over a wan, it would be nice to brew able to get some timings/profiling for e.g. logging in, transfer of script, execution time.

Cheers Alex

4

Hello Alex,

I did run lsof -u $USER | grep ssh | grep TCP and saw number of forks sockets. You could try to prepend your ssh command with strace to see what’s really going on (maybe you should only provision two or three hosts at the same time then :slight_smile: ).

Regards
Mirko

5

Sometimes this can happen if the system isn’t adding your host file to a host file ansible is looking for, so Ansible tightens down, executes a lock, and anticipates a question about whether you need to confirm adding a new host to your host file.

Try temporarily disabling host key checking to see if this makes it go away, and if it does, we can persue from there.

Also be sure you are on Ansible 1.8.1 – the latest release version – as well, as we recently made some changes regarding reading more possible host file locations.

6

Indeed it runs much faster with host_key_checking disabled.

I have a dynamic inventory - I written a script to produces the groups, hosts and hostvars.

I’m on ansible 1.7.2+dfsg-2, the version in Debian testing and destined for Debian stable release in the next few months. I’m happy to test with later/latest versions, but once things are working I’ll see if the fix also translates back to 1.7.2 since staying on that would be easier for me.

Cheers,
Alex

7

Please let us know if you observe problems in 1.8.X – we won’t be doing backports to an older series, though I suppose you could if you really wanted to, we can’t support older versions on this list.