# slow startup with many groups in inventory **URL:** \ **Category:** Ansible Developer\ **Created:** [October 24, 2014, 8:57pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604 "2014-10-24T20:57:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [October 24, 2014, 8:57pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/1 "2014-10-24T20:57:07Z")
Hi everyone, with the update to 1.7 ansible startup times went up to 25 seconds for me. After some debugging I found not really optimized functions in the inventory parsing. With more than 1000 servers and 10.000 groups the current code is not really usable. Problem: get\_group(self, groupname) in lib/ansible/inventory/\_\_init\_\_.py is on O(#groups) and is called #group times. O(#groups^2). Solution: Do not iterate over the list every time, hashtables should reduce the total time to O(#groups \* log(#groups)). With this patch I'm down to 50% startup time: [https://github.com/hrld/ansible/commit/6f349c4d09c5003f970981b247e99ede12ad0d6c](https://github.com/hrld/ansible/commit/6f349c4d09c5003f970981b247e99ede12ad0d6c) Can someone who actually knows python (and ideally knows what the inventory code does) can check if I need to invalidate the cache in more places? Or suggest a better way to implement this? I'd feel better if someone can give feedback before I send a pull request for this... If I'm not the only one who feels ansible could be a bit faster to start and would look at patches I'd appreciate feedback about that. (There are about 120k getcwd/stat64/lstat64 sytem calls during startup in my setup. 95% of that could easily be avoided with some caching.) Thanks! ---
**Author:** ![Jesse\_Keating](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jesse\_Keating](https://forum.ansible.com/u/Jesse_Keating)\ **Post date:** [October 24, 2014, 9:11pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/2 "2014-10-24T21:11:13Z")
Changes to make inventory parsing faster is definitely awesome, but to the point of stats, we use an inventory plugin that provides a \_meta key in the return json. This is where all the host/group facts are stored, and when ansible sees \_meta it’ll avoid doing any stats to read groupvars or hostvars files. This was a HUGE win for us, and I recommend it to anybody that has a large set of hosts or groups. -jlk ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [October 25, 2014, 12:14am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/3 "2014-10-25T00:14:33Z")
Yeah, with what Jesse says, it sounds like your inventory script is forgetting to return “\_meta”, as otherwise I don’t know of any inventory speed issues in 1.7.X. let me know if this is not the case. This is covered in [http://docs.ansible.com/developing\_inventory.html#tuning-the-external-inventory-script](http://docs.ansible.com/developing_inventory.html#tuning-the-external-inventory-script) ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [October 25, 2014, 7:36pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/4 "2014-10-25T19:36:59Z")
Thanks, I missed the \_meta option so far. But my test-setup still calls stat() on everything despite \_meta/hostvars. (1.7.2 and 1.8.) It also uses the hostvars I passed to it. But it does not seem to work for groupvars at all - you use a plugin/patch for that? My main problem is actually NOT the syscalls, it is get\_group(). I suspect that would still be called even if \_meta would work as intended. And with 20k groups and O(number\_of\_groups^2) there is a speed issue. Somehow it was still reasonable in 1.6 (get\_group() is unchanged, so either it was not called as often or something else changed) and is slow since 1.7. The patch [https://github.com/hrld/ansible/commit/6f349c4d09c5003f970981b247e99ede12ad0d6c](https://github.com/hrld/ansible/commit/6f349c4d09c5003f970981b247e99ede12ad0d6c) does nothing about the stat-syscalls but speeds up get\_group() by using a hashtable instead of a list to look for groups. (n vs. log(n) - with my n being ~20.000 that helps a lot...) Assuming that most users will not have group vars and host vars for more than 10% of their hosts and groups (or less than 1000 of each - so it would not matter) reading the relevant directories and keep a hashtable of the existing filenames should make big inventories with hostvars/groupvars in files much faster. Is there any reason not to do that? Best regards, Harald ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [October 25, 2014, 10:42pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/5 "2014-10-25T22:42:38Z")
When you say it calls stat “on everything” do you mean your inventory script, or something else? Which files? Definitely do the “\_meta” thing before proceeding further as the overhead of forking N copies of a dynamic interpreter is going to be quite large. ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [October 26, 2014, 12:34pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/6 "2014-10-26T12:34:03Z")
lstat64 is called on $PATH\_OF\_INVENTORY/host\_vars/$IP.\* and $PATH\_OF\_INVENTORY/group\_vars/$GROUP.\*, about 14k calls with "host\_vars" and 48k with "group\_vars" in it. Example inventory (returned by script): {     "mygroup": {         "hosts": [             "127.0.0.1"         ]     },     "\_meta": {         "hostvars": {             "127.0.0.1" : { "ansible\_ssh\_port": "1234" }         }     } } strace ansible mygroup -m ping 2\>&1 | grep lstat|grep \_vars lstat64("ANSIBLE/inventory/group\_vars/mygroup", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/group\_vars/mygroup.yml", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/group\_vars/mygroup.yaml", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/group\_vars/mygroup.json", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/group\_vars/all", {st\_mode=S\_IFREG|0664, st\_size=71, ...}) = 0 lstat64("ANSIBLE/inventory/group\_vars/all.yml", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/group\_vars/all.yaml", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/group\_vars/all.json", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/host\_vars/127.0.0.1", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/host\_vars/127.0.0.1.yml", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/host\_vars/127.0.0.1.yaml", 0xbfc09f5c) = -1 ENOENT (No such file or directory) lstat64("ANSIBLE/inventory/host\_vars/127.0.0.1.json", 0xbfc09f5c) = -1 ENOENT (No such file or directory) If I understand you and the documentation correctly there should be no attempts to access host\_vars/\*. Ansible is using the ssh-port from the \_meta block, but something must still be wrong here?! ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [October 27, 2014, 1:41pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/7 "2014-10-27T13:41:12Z")
No, if you have host\_vars files, ansible will definitely read them. How many hosts did you have? ---
**Author:** ![Serge\_van\_Ginderacht](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@Serge\_van\_Ginderacht](https://forum.ansible.com/u/Serge_van_Ginderacht)\ **Post date:** [October 27, 2014, 7:52pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/8 "2014-10-27T19:52:33Z")
​Hi Harald,​ > Thanks, I missed the \_meta option so far. But my test-setup still calls > stat() on everything despite \_meta/hostvars. (1.7.2 and 1.8.) ​Tha major difference that happened between 1.6​ and 1.7, is that - until 1.6 group/host\_vars were only read for the hosts in the run (that is with a possibke --limit applied), \*and\* they were only read for new host that weren't touched at the start of each play.    for big evnrionments this was less optimized, as it -re-read every group\_var file for every host that was member of the respective group - starting with 1.7, the whole inventory, and \*all\* host/group\_vars are read at the very start, when initializing the inventory, but eacht group\_vars file now is only read once For very large inventories - like you have - that first initialilsation tends to take a but longer, but the full time of running the whole play is shorter. Basically, from 1.7.2, every host\_vars and group\_var file should only get read once, but they \*all\* are read once. Given your extremely lareg numbers of hosts and groups, I can see this might get less performant than it used to be. But your issue, a I understand, is not about reading lots of files, but about iterating through the groups a host is member of, by name, to retrieve the group object, which is what your patch addresses., Given that a host might me member of a \*very\* large set of groups, I can see your patch being a good optimization. I'd vote you propose a pull request for it.   Serge ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [October 27, 2014, 8:31pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/9 "2014-10-27T20:31:43Z")
Correction: \_meta was already in the original inventory. No surprise that it did not speed things up. If ansible will always try to read all the files there is probably nothing left that can be improved without patching the inventory-parser code. Hosts: 3500 Groups: 12000 Host entries in groups: 60000 lstat() calls that could be optimized away by reading the two directories and caching the list of existing files: (3500+12000)\*4 So if nobody sees a good reason to leave it the way it is I'll also write a patch for that and should get the runtime for a single ansible-ping down from 25 seconds to about 10 seconds in our case. (13 seconds with the current patch.) ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [October 28, 2014, 7:53pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/10 "2014-10-28T19:53:12Z")
[...] > We designed our setup with an inventory of 3000+ hosts so that groups > are made dynamically, rather than pre-made. The reason is that every > permutation of a property (dmz/internal, prod/uat/int/dev/mgt, > virt/phys, os-version, various roles, app-owner, os-version, ...) > quickly inflates the number of groups one may need to have. And not > all our playbooks make actual use of these groups anyway. > > So instead of having the inventory script (in advance) create all the > groups (huge inventory) and have ansible import all this data, we > only have it export the hostvars that can be used to make those > dynamic groups (group\_by) and do it dynamically, on a need-basis. > This works well, although was slow with v1.6 when running for all > 3000+ hosts, but it drastically improved in v1.7 to only a few > seconds. Here it is 3500 hosts and slowed down from ~10 to 25 seconds with 1.7. How many is "a few"? I'd like to compare after patching the other two issues... those might also gain you another 2 seconds as they (partially) depend on the number of hosts. I'll keep the hostvars+group\_by solution in mind if we need more groups in the future. Most of the current groups are useful in many places, so after patching the current bottleneck its probably not worth the extra complexity to first create groups in all playbooks. > Although in the past we seldom ran playbooks on all systems, we now > use ansible for reporting connectivity issues, and that runs on all > systems regularly to identify network/routing/firewall issues. This > is not a typical environment, we have 7x more VLANs than we have > Linux systems (don't ask). > > BTW the dynamic inventory script is not a real-time process, since it > takes time to get all information out of vSphere, Satellite, Infoblox > and DNS. We create a cache every hour through cron, rather then do it > on-the-fly. All our inventory data comes from one database so it is fast and the cache gets created every minute. Even with 2-3 seconds runtime a cache will make sense for everyone using ansible in some interactive way. Thanks for the feedback! Harald ---
**Author:** ![Bob\_Chen](https://avatars.discourse-cdn.com/v4/letter/b/d6d6ee/32.png) [@Bob\_Chen](https://forum.ansible.com/u/Bob_Chen)\ **Post date:** [October 29, 2014, 7:46am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/11 "2014-10-29T07:46:47Z")
Indeed combie\_vars need to be responsible for this, it might take a long time (16s+ on 453 groups and 132 hosts), I think in most situations, vault feature will not be used, by make a test before combie\_vars, we can save a lot of time in most situations: [https://github.com/iambocai/ansible/commit/a09a2a388f9ea743caa8de10c48ac55867ba0c0b](https://github.com/iambocai/ansible/commit/a09a2a388f9ea743caa8de10c48ac55867ba0c0b) 在 2014年10月28日星期二UTC+8上午3时52分55秒，Serge van Ginderachter写道： ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [October 29, 2014, 6:07pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/12 "2014-10-29T18:07:48Z")
This patch probably does not do what you want. With the patch host\_vars and group\_vars files are never touched again. So this does not only affect \_vault\_password usage but at least everyone with variables in files. (Did not test hostvars from \_meta.) ---
**Author:** ![Jeff\_Reter](https://avatars.discourse-cdn.com/v4/letter/j/a88e4f/32.png) [@Jeff\_Reter](https://forum.ansible.com/u/Jeff_Reter)\ **Post date:** [May 22, 2015, 10:04am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/13 "2015-05-22T10:04:13Z")
any news on this? go a big inventory too, and its really slow (patch helped, thanks; but its still to slow) ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [May 28, 2015, 7:15am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/14 "2015-05-28T07:15:42Z")
There are a few things that might be helpful to know when using big inventories. 1. the official solution seems to be to use tower or dynamic (small) inventories (see cloud docs) (both not an option for me for various reasons but either there are no users with many hosts and groups or they are mostly happy with those) 2. in theory you can let ansible merge several inventories (e.g. you have two distinct sources of data). DON'T. It will cost a lot of extra time. (Nobody here wanted to debug that, was not important) 3. it would be possible to reduce the number of lstat() calls (down to less than 1%) with a small patch. After profiling an ansible startup sequence I found it was not my main problem but you might have different results for your setup. If you send me the number of hosts and groups and output of python -m cProfile /usr/bin/ansible $IP -m ping I'd like to compare where time is lost and look into this again. Maybe someone will even pull patches for this after v2 is released 🙂 Best regards, Harald ---
**Author:** ![Dave\_Rawks](https://avatars.discourse-cdn.com/v4/letter/d/b9bd4f/32.png) [@Dave\_Rawks](https://forum.ansible.com/u/Dave_Rawks)\ **Post date:** [December 15, 2015, 11:27pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/15 "2015-12-15T23:27:35Z")
I’m still seeing this behavior in 1.9.4, did you happen to post your patch online anywhere? ---
**Author:** ![Abhijit\_Menon-Sen1](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Abhijit\_Menon-Sen1](https://forum.ansible.com/u/Abhijit_Menon-Sen1)\ **Post date:** [December 16, 2015, 5:00am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/16 "2015-12-16T05:00:42Z")
> I'm still seeing this behavior in 1.9.4, did you happen to post your > patch online anywhere? Hi Dave. Have you tried with devel? There are quite a few patches that should help with large inventories, including one of Harald's: commit 73d6da757faf57e43df52e3332a7a3b9ee67cdba Author: Harald Laabs \ ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [December 16, 2015, 8:03am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/17 "2015-12-16T08:03:16Z")
The PR for this is here: [https://github.com/ansible/ansible/pull/9437](https://github.com/ansible/ansible/pull/9437) It was rebased and merged to 2.0 but if you need it before 2.0 is stable you can just apply it to 1.9.x yourself. If you already have performance issues with 1.9 you probably want to wait and hope for some patches in 2.0 before you try use it. Reading the inventory might be faster there now. Everything else is much slower. Testing with 2.0 is a good idea for anyone with a lot of playbooks and roles. You will find the typos in your playbooks that 1.x silently ignored, the options that should be used in the future (e.g. become, can already be cleaned up now) and changes (e.g. in escaping " with less \\\) you need to roll out the moment you move over to 2.0. I'm not sure if anyone already tried to create a list of changed behaviour from 1.x to 2.0. Might be a bit difficult before the developers decide what they consider a bug and which previous ways to do stuff were never documented and do not need to be supported 🙂 Best regards, Harald ---
**Author:** ![Dave\_Rawks](https://avatars.discourse-cdn.com/v4/letter/d/b9bd4f/32.png) [@Dave\_Rawks](https://forum.ansible.com/u/Dave_Rawks)\ **Post date:** [December 16, 2015, 5:08pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/18 "2015-12-16T17:08:03Z")
Cool, thanks for the patch. It didn’t result in the speed up I was hoping for. I think the big thing for me is the stat per host to check for host var files. Even with the “\_meta” key in my dynamic inventory the startup time is dominated by lstats for non-existent files. Seems like it does 7 lstat calls per host in the inventory even if the parent directory for the constructed path doesn’t exist. I think guarding that call with a stat call on the parent directory followed by an os.listdir could get this down from 72,000 calls to ~20 in my case. -Dave ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [December 16, 2015, 7:29pm UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/19 "2015-12-16T19:29:38Z")
Most of the lstat() can be removed easily (just read the directory and check in the resulting hash if you need to deal with the file) but you might be disappointed by the performance gains there. You can just comment out the whole block to try how much it saves, in my case it was under 20% of the performance gain from the get\_group patch. System calls are not that expensive compared to what ansible is doing elsewhere. (If you find a way to cause real io-operations for each lstat you can probably patch this while one ansible instance is busy reading the inventory. Otherwise use cProfile to check what you really want to patch...) Harald ---
**Author:** ![Harald\_Laabs](https://avatars.discourse-cdn.com/v4/letter/h/9de0a6/32.png) [@Harald\_Laabs](https://forum.ansible.com/u/Harald_Laabs)\ **Post date:** [January 20, 2016, 8:36am UTC](https://forum.ansible.com/t/slow-startup-with-many-groups-in-inventory/16604/20 "2016-01-20T08:36:40Z")
Update regarding this issue in 2.0: Patching the stat-madness saves about 40% of my startup runtime. There is a patch available from Tobias Wolf that just needs to be merged: [https://github.com/ansible/ansible/pull/13957](https://github.com/ansible/ansible/pull/13957) see also: [https://github.com/ansible/ansible/issues/13956](https://github.com/ansible/ansible/issues/13956) Can someone explain why there are several json-libs supported in ansible? Using the fastest one that actually behaves in a sane way (and guarantees unicode strings as return values) seems a no-brainer to me. Otherwise more tests for different behaviour are needed. And installing one more python library seems simple enough. Immediate benefit: the other patch from the ticket saves another 20% startup time in useless unicode-string-checks. If your inventory is small (less than a few thousand hosts and groups) you will not notice any real difference. Harald