Monumental slowless in parsing inventory

Archives AWX Project
1

So, we’ve had some issues in the past with Ansible itself in how slow it reads our custom inventory.

But that slowness was in the area of 25 seconds at the worst of times. It improved to 8-ish seconds later.

Now, I wanted to check out AWX for its possibilities and wanted to load our full inventory from the same custom script we are using with ansible-playbook.

It has been processing it for 50 minutes and it is still going.

What is it doing there?

2017-09-11 21:40:16,350 INFO     awx.main.commands.inventory_import Host "10.80.94.193" added
2017-09-11 21:40:16,364 INFO     awx.main.commands.inventory_import Host "10.80.94.194" added
2017-09-11 21:40:16,377 INFO     awx.main.commands.inventory_import Host "10.80.94.195" added
2017-09-11 21:40:16,391 INFO     awx.main.commands.inventory_import Host "10.80.94.196" added
2017-09-11 21:40:16,405 INFO     awx.main.commands.inventory_import Host "10.80.94.197" added
2017-09-11 21:40:16,419 INFO     awx.main.commands.inventory_import Host "10.80.94.198" added
2017-09-11 21:40:16,433 INFO     awx.main.commands.inventory_import Host "10.80.94.199" added
2017-09-11 21:40:16,446 INFO     awx.main.commands.inventory_import Host "10.80.94.200" added
2017-09-11 21:40:16,460 INFO     awx.main.commands.inventory_import Host "10.80.94.204" added
2017-09-11 21:40:16,474 INFO     awx.main.commands.inventory_import Host "10.80.94.208" added
2017-09-11 21:40:16,488 INFO     awx.main.commands.inventory_import Host "10.80.94.209" added
2017-09-11 21:40:16,501 INFO     awx.main.commands.inventory_import Host "10.80.94.215" added
2017-09-11 21:40:16,515 INFO     awx.main.commands.inventory_import Host "10.80.94.216" added
2017-09-11 21:40:16,529 INFO     awx.main.commands.inventory_import Host "10.80.94.220" added




CPU% MEM%   TIME+  Command                                                                                                                    
 0.0  0.0  0:00.17 │  │  ├─ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-time
 0.0  0.0  0:00.29 │  │  ├─ docker-containerd-shim 32f719b3d2a1cf0ec87542591bb4cc760d2bce8a1f6af22c717998d8e7722c1f /var/run/docker/libcontain
 0.0  0.0  0:00.13 │  │  │  ├─ /tini -- /bin/sh -c /usr/bin/launch_awx_task.sh
 0.0  0.0  0:00.00 │  │  │  │  └─ bash /usr/bin/launch_awx_task.sh
 0.7  0.0  0:01.35 │  │  │  │     └─ /usr/bin/python2 /usr/bin/supervisord -c /supervisor_task.conf
 1.3  0.2  0:04.82 │  │  │  │        ├─ python /usr/bin/awx-manage celery beat -l debug --pidfile= -s /var/lib/awx/beat.db
 0.0  0.1  0:03.80 │  │  │  │        ├─ python /usr/bin/awx-manage run_callback_receiver
 0.0  0.1  0:00.24 │  │  │  │        │  ├─ python /usr/bin/awx-manage run_callback_receiver
 0.0  0.1  0:00.23 │  │  │  │        │  ├─ python /usr/bin/awx-manage run_callback_receiver
 0.0  0.1  0:00.26 │  │  │  │        │  ├─ python /usr/bin/awx-manage run_callback_receiver
 0.0  0.1  0:00.26 │  │  │  │        │  └─ python /usr/bin/awx-manage run_callback_receiver
 0.0  0.1  0:04.21 │  │  │  │        ├─ python /usr/bin/awx-manage runworker --only-channels websocket.*
 1.3  0.2  0:07.26 │  │  │  │        └─ python /usr/bin/awx-manage celery worker -l debug --autoscale=4 -Ofair -Q tower_scheduler,tower_broadc
 2.6  0.1  0:00.18 │  │  │  │           ├─ python /usr/bin/awx-manage celery worker -l debug --autoscale=4 -Ofair -Q tower_scheduler,tower_bro
55.4  0.6 25:20.68 │  │  │  │           └─ python /usr/bin/awx-manage celery worker -l debug --autoscale=4 -Ofair -Q tower_scheduler,tower_bro
54.8  0.3  4:06.78 │  │  │  │              └─ python /usr/bin/awx-manage inventory_import --inventory-id 2 --overwrite --overwrite-vars --grou
2

Tobias,

Does your inventory script return a _meta tag? If not, it may be running the script over and over with different --hostname parameters.

Thanks,

-Graham

3

Yes, we have a _meta object with about 4000 hosts and on average 20 hostvars. And then we have over 9000 groups with child host numbers ranging from 1 to a few hundred or a couple thousand members.

Ansible manages to read these in 6-8 seconds. And AWX needed 2 hours:

Inventory import completed for ServerDB inventory in 6520.5s

You can see how I could not wit for completion and had to go to bed …
4

Ok, now I have re-run the custom inventory script and this time it completed significantly quicker.

I’ll have to see if the synchronization will manage to keep everything fresh.