# Ansilbe read all variable even they do not need for task (perfomance issue with lookup\_plugins) **URL:** \ **Category:** Ansible Developer\ **Created:** [August 5, 2016, 9:07pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225 "2016-08-05T21:07:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Artyom\_Aleksandrov](https://avatars.discourse-cdn.com/v4/letter/a/2bfe46/32.png) [@Artyom\_Aleksandrov](https://forum.ansible.com/u/Artyom_Aleksandrov)\ **Post date:** [August 5, 2016, 9:07pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/1 "2016-08-05T21:07:39Z")
Hello, I’m using lookup\_plugin which go to external service and get variables via HTTPS protocol. Several time ago I noticed that playbooks had took more time than before. During investigation I found that my lookup\_plugin runs on every tasks even if currently running task do not use variables. For example if I specify _linux\_password_ variable in group _all_ this PluginName will be call on every task on every hostname. [all] linux\_password: “{{ lookup(‘PluginName’, ‘linux\_password’)}}” I’m not sure but seems previously Ansible call lookup\_plugin only in case that is really need this variable. Do you know what have changed and if it possible to turn it back? In the ideal case in my opinoon it would be great if Ansible _can_ cache variable in memory. Best regards ---
**Author:** ![James\_Cammarata](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@James\_Cammarata](https://forum.ansible.com/u/James_Cammarata)\ **Post date:** [August 17, 2016, 3:05pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/2 "2016-08-17T15:05:27Z")
Sorry for the delayed response, but in the case of having it in the `all` group, that var would be referenced on every task due to the fact that it’s being brought in for all hosts. Can you give a different example of that being in a group the current host is not a member of and yet still being referenced? ---
**Author:** ![Artyom\_Aleksandrov](https://avatars.discourse-cdn.com/v4/letter/a/2bfe46/32.png) [@Artyom\_Aleksandrov](https://forum.ansible.com/u/Artyom_Aleksandrov)\ **Post date:** [August 23, 2016, 1:54pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/3 "2016-08-23T13:54:36Z")
Hello, Thank you for answer. It’s bad news because all variables which _can be_ used by host reads on each task. Variables in _all_ group of course provide most reads but problem also applies to other groups and host\_vars. =( > Can you give a different example of that being in a group the current host is not a member of and yet still being referenced? No. There is no problem with this case. Have you thought about changing this behavior or implementing runtime cache for variables? ---
**Author:** ![j.r.hawkesworth](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@j.r.hawkesworth](https://forum.ansible.com/u/j.r.hawkesworth)\ **Post date:** [August 25, 2016, 7:05am UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/4 "2016-08-25T07:05:17Z")
I wonder if you can use uri delegated to your localhost to collect your facts, and use set\_fact to make them available to the play. Also I wonder if a custom setup module might be able to do what you need? If you can get the vars you need into facts you can use one of the existing fact caching mechanisms [http://docs.ansible.com/ansible/playbooks\_variables.html#fact-caching](http://docs.ansible.com/ansible/playbooks_variables.html#fact-caching) Jon ---
**Author:** ![Ben\_Roubicek](https://avatars.discourse-cdn.com/v4/letter/b/13edae/32.png) [@Ben\_Roubicek](https://forum.ansible.com/u/Ben_Roubicek)\ **Post date:** [September 10, 2016, 12:01am UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/5 "2016-09-10T00:01:22Z")
Following up here. The behavior of lookup plugin evaluation has definitely changed post 2.1.0.0. In 2.1.0.0, lookup calls seem to be lazy evaluated so if you perform an HTTP lookup to Hashicorp Vault, for example, that call is only made when the variable is used in the play. Post 2.1.0.0, all variables seem to be evaluated for each task! Thats a huge change of behavior causing considerable performance problems for us since we relay on gathering secrets from an external source specified at the inventory level. Is there an explanation for why that change was made? Can it be changed back to the previous behavior? ---
**Author:** ![Ben\_Roubicek](https://avatars.discourse-cdn.com/v4/letter/b/13edae/32.png) [@Ben\_Roubicek](https://forum.ansible.com/u/Ben_Roubicek)\ **Post date:** [September 10, 2016, 12:31am UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/6 "2016-09-10T00:31:56Z")
Also, I found the commit that changed this behavior. [https://github.com/ansible/ansible/commit/7287effb5ce241ce645d61e55e981edc73fa382a](https://github.com/ansible/ansible/commit/7287effb5ce241ce645d61e55e981edc73fa382a) ---
**Author:** ![Tobias\_Wolf](https://avatars.discourse-cdn.com/v4/letter/t/eada6e/32.png) [@Tobias\_Wolf](https://forum.ansible.com/u/Tobias_Wolf)\ **Post date:** [September 10, 2016, 8:51am UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/7 "2016-09-10T08:51:22Z")
Can you point out how you think that particular commit affects how often lookups are evaluated? I don’t see it. All that is doing is to make a os.listdir (essentially an ls) in the host\_vars and group\_vars directories and then checking group names and host names against this listing instead of checking on the file system causing a storm of lstats. Maybe you were mistaken by the word “lookup”? Lookup refers to `stat` here. regards, Tobias (author of that commit) ---
**Author:** ![Ben\_Roubicek](https://avatars.discourse-cdn.com/v4/letter/b/13edae/32.png) [@Ben\_Roubicek](https://forum.ansible.com/u/Ben_Roubicek)\ **Post date:** [September 12, 2016, 3:55pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/8 "2016-09-12T15:55:20Z")
Tobias, I must have jumped the gun on Friday, apologies. Looks like its actually a different commit causing the issue. Sorry 🙂 Culprit commit: [https://github.com/ansible/ansible/commit/6b286ee0c88f3252a27ff94f39d794566c3f7d6f#diff-76ffc0551f8cf3d6255500316568e60bR412](https://github.com/ansible/ansible/commit/6b286ee0c88f3252a27ff94f39d794566c3f7d6f#diff-76ffc0551f8cf3d6255500316568e60bR412) My scenario is that I have a inventory/foo/group\_vars/all/hashivault.yml file that specifies 100+ secrets that are to be retrieved via a custom lookup plugin. Before this commit, evaluation of the variables would only occur when they are used. With this commit applied, each task is re-evaluating the variables causing thousands of calls during the play. My lookup calls look like: ` db\_password: “{{ lookup(‘vault’, ‘secret/db\_password’, ‘value’) }}” ` Anyway, tinkering with the code a bit… in lib/ansible/executor/task\_executor.py By simply changing ` self.\_connection.set\_host\_overrides(host=self.\_host, hostvars=variables.get(‘hostvars’, {}).get(self.\_host.name, {})) ` back to: ` self.\_connection.set\_host\_overrides(host=self.\_host) ` Reduces the calls back to only those required by the task. I dont believe I understand the code well enough to make a PR, so I would appreciate some advice on how best to proceed with a fix (assuming this was an unintended side-effect of this commit). Thanks -Ben ---
**Author:** ![Ben\_Roubicek](https://avatars.discourse-cdn.com/v4/letter/b/13edae/32.png) [@Ben\_Roubicek](https://forum.ansible.com/u/Ben_Roubicek)\ **Post date:** [September 12, 2016, 8:31pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/9 "2016-09-12T20:31:32Z")
Bug report here: [https://github.com/ansible/ansible/issues/17024](https://github.com/ansible/ansible/issues/17024) ---
**Author:** ![Ben\_Roubicek](https://avatars.discourse-cdn.com/v4/letter/b/13edae/32.png) [@Ben\_Roubicek](https://forum.ansible.com/u/Ben_Roubicek)\ **Post date:** [September 21, 2016, 6:46pm UTC](https://forum.ansible.com/t/ansilbe-read-all-variable-even-they-do-not-need-for-task-perfomance-issue-with-lookup-plugins/22225/10 "2016-09-21T18:46:06Z")
For others that may come across this, the bug was fixed. Comments here [https://github.com/ansible/ansible/issues/17024](https://github.com/ansible/ansible/issues/17024)