Ansible dynamic inventory pegs CPU and RAM reading in a 54K JSON object

Archives Ansible Project
1

Hi,

I’m trying to feed Ansible 2.0.0.2 dynamic inventory from an Enterprise CMDB system.

The Enterprise CM system already has lots of groups defined, I want to be able reference them through Ansible, so I’m trying to load the Enterprise CM data into Ansible.

The JSON object from the CM system is 1M in size. Ansible 2.0.0.2 pegs the CPU for just over 4 minutes, and grows to gigabytes in RAM, until it exits with “[Errno 12] Cannot allocate memory”.

If I manually trim the JSON object so it only contains the group I want (with 700 hosts), Ansible has no problem reading it in.

Ansible takes 25 seconds to run --list-hosts when the input JSON contains one group with 700 hosts (and I put the group name on the Ansible command line).

When the input JSON contains two groups with 700 hosts each, Ansible takes 39 seconds to run (still selecting the first group only)

If I add a third group with 1 host in it, Ansible takes 5 minutes and 15 seconds to run and uses a gig and a half of memory, pegging CPU the whole time:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6066 root 20 0 772m 519m 4108 R 97.8 13.6 5:08.19 ansible

At this point my JSON object is 54K in size.

Here is an example JSON with three groups Ansible takes over 4 minutes to process:

https://gist.github.com/atsaloli/8ba19f8b4eb4ecbc2e9d

BTW, I used a little shell script to feed my JSON file to Ansible:

time ansible all -i /tmp/di.sh --list-hosts

cat /tmp/di.sh

#!/bin/sh
cat /tmp/list.json

2

Have a look at the documentation here (http://docs.ansible.com/ansible/developing_inventory.html); near the bottom is some information about tuning up large inventories. Basically, you can add “_meta” to your inventory and list the host variables. Just adding the following to your inventory speeds it up on my system from 7 seconds to .3 seconds (with Ansible 1.9).

“_meta”: {

“hostvars”: { }

}

3

Thank you very much, Paul! (Sorry for the delay, somehow I missed your kind reply.)