# Developers: Optimizing external inventory scripts **URL:** \ **Category:** Ansible Project\ **Created:** [August 4, 2013, 3:57pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252 "2013-08-04T15:57:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 4, 2013, 3:57pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/1 "2013-08-04T15:57:35Z")
A few folks have pointed out that the inventory script mechanism calls “–host” for every host, that can’t be parallelized, and that’s a bit slow for large numbers of hosts, especially when the inventory script doesn’t do it’s own caching. Previously this resulted in generating a static inventory file being sensible, especially when you have thousands of hosts. Good news! I’ve added something for this. Details in the following commits: [https://github.com/ansible/ansible/commit/bcaa983c2f3ab684dca6c2c2c8d1997742260761](https://github.com/ansible/ansible/commit/bcaa983c2f3ab684dca6c2c2c8d1997742260761) [https://github.com/ansible/ansible/commit/8955ac1eda048eeda7e4126ee0cc26c2a4b06147](https://github.com/ansible/ansible/commit/8955ac1eda048eeda7e4126ee0cc26c2a4b06147) If you’ve written an external inventory script for Ansible (they live in the “plugins/inventory” directory of the checkout, you may wish to update your inventory script to support the above, as it will offer some nice speedups. This is however not required, if you don’t return any “hostvars” in the “\_meta” element, or there is no “\_meta” element, everything will continue to work just as before. ---
**Author:** ![Jesse\_Keating](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jesse\_Keating](https://forum.ansible.com/u/Jesse_Keating)\ **Post date:** [August 4, 2013, 10:09pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/2 "2013-08-04T22:09:44Z")
Rock! Thanks! I'm going to test this out in our environment soon. -jlk ---
**Author:** ![Eugene\_Archibald](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@Eugene\_Archibald](https://forum.ansible.com/u/Eugene_Archibald)\ **Post date:** [August 6, 2013, 8:14pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/3 "2013-08-06T20:14:26Z")
Thank you, Michael! It’s very very much appreciated. ---
**Author:** ![Dan\_Hanks](https://avatars.discourse-cdn.com/v4/letter/d/82dd89/32.png) [@Dan\_Hanks](https://forum.ansible.com/u/Dan_Hanks)\ **Post date:** [August 16, 2013, 4:00pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/4 "2013-08-16T16:00:43Z")
I’m considering using Ansible in an environment with potentially 20,000+ hosts utilizing a centralized CMDB, and I’m concerned about how Ansible manages inventory. From what I read, even with the optimization you’ve made with this change, my external inventory script would still have to fetch the entire hosts/groups list on each invocation before whittling it down with patterns specified on the command-line. What are the recommended strategies for managing inventories of this size? Local caching of the inventory content by the inventory script? We have a dynamic environment with hosts moving in and out of groups at any given time, so local caching could be very problematic. I had hoped I could pass in group names to the inventory script for more targeted fetching of CMDB content. That doesn’t appear to be possible. Suggestions? Thanks, Dan ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 16, 2013, 5:51pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/5 "2013-08-16T17:51:03Z")
The recommendation in this case is in fact to let the inventory script cache to disk and know when to update it in many cases where APIs are slow. The EC2 script, included in plugins/inventory, in fact does the same. However, if your CMDB is in the database, I wouldn’t think it would be that slow. (Have you perhaps considered memcache or a NoSQL store for the exact document you wish to return to Ansible?) You are of course welcome to also carve things up into smaller inventories if you would feel more comfortable with that. ---
**Author:** ![Eugene\_Archibald](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@Eugene\_Archibald](https://forum.ansible.com/u/Eugene_Archibald)\ **Post date:** [August 16, 2013, 6:02pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/6 "2013-08-16T18:02:25Z")
I have been thinking that carving up into smaller inventory scripts is called for if I want to be able to pull down smaller chunks. Prior to the \_meta[“hostnames”] update I was caching things locally because of the individual --host calls, but it caused some potential race conditions that I didn’t like, and I’ll be glad to be rid of it. I’ve been tossing around the idea, along similar lines, of having my inventory script detect what it’s been invoked as and provide different inventories accordingly: ansible-playbook -i /etc/ansible/lab-hosts ansible-playbook -i /etc/ansible/prod-hosts where hosts.py lab-hosts → hosts.py prod-hosts → hosts.py Regards, Eugene ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 16, 2013, 6:06pm UTC](https://forum.ansible.com/t/developers-optimizing-external-inventory-scripts/13252/7 "2013-08-16T18:06:32Z")
yep, interestingly enough our AWX inventory script (that we use behind the scenes in AWX to interface with Ansible) works a lot like that. We pass in the database inventory ID via the environment and the system returns a subset of the overall inventory.