ansible cmd run takes over 40s per server

Archives Ansible Project
1

Hi,

I installed a fresh Ansible from git hub on CentOS 6.3 and run a simple command takes over 40s with CPU 100% for Ansible.

This is a line from cProfile(http://pastebin.com/mX4qXsiN).

11523 40.093 0.003 54.893 0.005 hostkeys.py:181(lookup)

It seems to that Ansible is doing a lot of lookups.
The same version works on my box without an issue(Fedora 20).

Command:
ansible/bin/ansible XXX -u root --private-key /root/.ssh/id_rsa -i inventory/ -m shell -a echo test -vvv

I ran Ansible on CentOS 6.3 boxes and I’ve never seen like this before.

Thanks

2

This is pretty unusual.

This is a function inside the paramiko source code that does not (at least obviously does not) do anything like DNS lookups.

I’m at a bit of a loss, perhaps you can debug further?

Fedora 20 has a new enough SSH so the default connection type is “ssh” not paramiko there, curious if you have the same with “-c paramiko” on that box (and using the same key?)

3

ansible XX -c paramiko -u root --private-key .ssh/root_id -i inventory/ -m shell -a echo test

it’s fast on F20.

We have a big /etc/hosts file (1527 hosts), but I copied the host file to my box and it doesn’t make a difference.

4

I deployed on different box (the same version) and it works just fine.
Not sure what is the issue on the first one.

5

nscd turned off? ldap in use?

6

Paramiko’s host file handling has been noted to be a performance problem in some cases where host files get large.

We haven’t exactly tracked down why.

(I really really wish RHEL/CentOS would update their OpenSSH… can’t wait for RHEL 7 – and less and less folks will need paramiko)

7

So the reason is that my known_hosts file has over 5k records.
I think that Paramiko is for looping for each host in known_hosts and /etc/hosts file.
In my case it’s around 800k checks.

8

Paramiko’s processing of known_hosts is rather poor for large host files.

I’d recommend using the SSH transport if you can.