# RFC: A form of lazy evaluation **URL:** \ **Category:** Ansible Project\ **Created:** [August 7, 2013, 10:14am UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268 "2013-08-07T10:14:20Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Grzegorz\_Nosek](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@Grzegorz\_Nosek](https://forum.ansible.com/u/Grzegorz_Nosek)\ **Post date:** [August 7, 2013, 10:14am UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/1 "2013-08-07T10:14:20Z")
Hi, Once loaded, a playbook is completely static (includes etc. are evaluated while loading, not while executing). This has its advantages but also makes includes special, e.g. \* they don't handle tags or delegate\_to \* not sure what they do with sudo/sudo\_user from reading the code (silently ignored unless I'm missing something?) \* with\_\* handling is unintuitive (you can't say with\_items: $groups.foo due to very eager evaluation of include/with\_\*) At the very least, we'd like to extend the include directive to push down tags, sudo/sudo\_user, delegate\_to, notify to individual tasks in the file, but for a more flexible approach, how about we introduced some form of lazy evaluation/dynamic playbooks? ("we introduced" meaning "I'll submit a patch once we agree on a design") An action plugin could optionally return a list of dicts describing additional tasks, just like extra facts right now. The tasks would be then fed back into the playbook loop by e.g. PlayBook.\_run\_task calling itself recursively. Note that this has nothing to do with includes and YAML parsing and is very generic. Extra tasks could be loaded from a database, generated on the fly etc. A major issue is security. This feature would open up ways to influence ansible-playbook's actions from outside the playbook YAML file. Given a vulnerable module \_and\_ a compromised host in inventory, the master could be tricked into executing arbitrary modules. I'd suggest this feature be available only to action\_plugins using a class attribute, similar to BYPASS\_HOST\_LOOP ("normal" not being one of these whitelisted plugins). To fully implement our lazy\_includes we'd need some code from Play/PlayBook classes copied or factored out to a helper class but that could remain our in-house mess. We'd also need a way to notify the loader module what tags are white/blacklisted if we want tags in lazy-loaded plays to just work. Yes, it's a major change in ansible's behaviour but right now I think that the static nature of playbooks lead to unexpected behaviour similar to e.g. nginx and its if{} blocks. They look like procedural config (evaluated top-down) but are strictly declarative and evaluated outside in (top level, then ifs, then nested ifs...). This led to endless confusion and "if" is now deprecated in nginx. Or, if letting tasks return more tasks is too drastic (despite being 100% backwards-compatible), maybe pushing the loading process to plugins is a way to go? Best regards,   Grzegorz Nosek ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 7, 2013, 1:38pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/2 "2013-08-07T13:38:56Z")
There are some inaccuracies in what you post here, it talks about lazy evaluation and other things, and then it talks about setting variables on include statements – some of which (things like tags) we can already do. You talk a little bit about use cases but then dive into technical implementations. Sorry, I’m finding the train of thought very hard to follow. I disagree strongly about “unexpected behavior”, what you propose seems to imply lots of unexpected behavior – but I can’t tell, because there aren’t any use cases being discussed. Would it be possible to step back and talk about your use cases first separate from the implementation and code? ---
**Author:** ![Jesse\_Keating](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jesse\_Keating](https://forum.ansible.com/u/Jesse_Keating)\ **Post date:** [August 7, 2013, 2:22pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/3 "2013-08-07T14:22:25Z")
I'm curious what you mean here. I have a number of tasks that are generic and included in multiple play books, and I've set them up in a way that they can be delegated to a host or not, depending on a variable passed at the - include statement: e.g. - name: create facter tree   file: path={{ facterd }} state=directory   delegate\_to: "{{ delegate|default(inventory\_hostname) }}" and in the playbook you can do: - include: tasks/sync-facts.yaml which will run it on the host in the loop, OR - include: tasks/sync-facts.yaml delegate={{ some\_other\_host\_value }} which will cause the task to be delegated to whatever the value of some\_other\_host\_value happens to be. In some of my playbooks, the variable passed to delegate= comes from a fact created using set\_fact earlier in a play for really dynamic optional delegation. -jlk ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 7, 2013, 2:46pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/4 "2013-08-07T14:46:18Z")
Yeah, it’s fine to teach the include construct (like the roles construct) to support understanding to auto-apply a delegate\_to keyword if delegate\_to is set on the include. This already occurs for tags, for instance. Ansible is already a pretty lazy loading system as variables go, for things that are specified in task operations. ---
**Author:** ![Jesse\_Keating](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jesse\_Keating](https://forum.ansible.com/u/Jesse_Keating)\ **Post date:** [August 7, 2013, 3:05pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/5 "2013-08-07T15:05:23Z")
That would be handy! -jlk ---
**Author:** ![Grzegorz\_Nosek](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@Grzegorz\_Nosek](https://forum.ansible.com/u/Grzegorz_Nosek)\ **Post date:** [August 8, 2013, 9:51am UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/6 "2013-08-08T09:51:16Z")
W dniu środa, 7 sierpnia 2013 16:22:25 UTC+2 użytkownik Jesse Keating napisał: > > - they don’t handle tags or delegate\_to > > I’m curious what you mean here. > > I have a number of tasks that are generic and included in multiple play books, and I’ve set them up in a way that they can be delegated to a host or not, depending on a variable passed at the - include statement: > > e.g. > > - name: create facter tree > file: path={{ facterd }} state=directory > delegate\_to: “{{ delegate|default(inventory\_hostname) }}” > > and in the playbook you can do: > > - include: tasks/sync-facts.yaml > > which will run it on the host in the loop, OR > > - include: tasks/sync-facts.yaml delegate={{ some\_other\_host\_value }} > > which will cause the task to be delegated to whatever the value of some\_other\_host\_value happens to be. Sure, this gets the job done. But this now every task in the included file gets bloated and unreadable. Compare: # toplevel.yml - include: included.yml delegate=host sudo=True sudo\_user=whatever # included.yml - action: foo delegate\_to: “{{ delegate|default(inventory\_hostname) }}” sudo: “{{ sudo|default(False) }}” sudo\_user: “{{ sudo\_user|default(“nobody”) }}” - action: bar delegate\_to: “{{ delegate|default(inventory\_hostname) }}” sudo: “{{ sudo|default(False) }}” sudo\_user: “{{ sudo\_user|default(“nobody”) }}” - action: baz delegate\_to: “{{ delegate|default(inventory\_hostname) }}” sudo: “{{ sudo|default(False) }}” sudo\_user: “{{ sudo\_user|default(“nobody”) }}” with: # toplevel.yml - include: included.yml delegate\_to: host sudo: True sudo\_user: whatever # included.yml - action: foo - action: bar - action: baz Best regards, Grzegorz Nosek ---
**Author:** ![Grzegorz\_Nosek](https://avatars.discourse-cdn.com/v4/letter/g/5daacb/32.png) [@Grzegorz\_Nosek](https://forum.ansible.com/u/Grzegorz_Nosek)\ **Post date:** [August 8, 2013, 10:47am UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/7 "2013-08-08T10:47:11Z")
W dniu środa, 7 sierpnia 2013 15:38:56 UTC+2 użytkownik Michael DeHaan napisał: > There are some inaccuracies in what you post here, it talks about lazy evaluation and other things, and then it talks about setting variables on include statements – some of which (things like tags) we can already do. You talk a little bit about use cases but then dive into technical implementations. Sorry, I’m finding the train of thought very hard to follow. > > I disagree strongly about “unexpected behavior”, what you propose seems to imply lots of unexpected behavior – but I can’t tell, because there aren’t any use cases being discussed. > > Would it be possible to step back and talk about your use cases first separate from the implementation and code? Yeah, let’s give it another try. Use case #1: Getting from: - action: foo arg=$item with\_items: ${groups.some\_group} - action: bar arg=$item with\_items: ${groups.some\_group} - action: baz arg=$item with\_items: ${groups.some\_group} to: - include: tasks/foo.yml with\_items: ${groups.some\_group} Yes, I know it is undocumented on purpose, (I think) I understand _why_ it works the way it does etc., it’s just deeply unintuitive that a simple refactoring changes behaviour in an unexpected way, especially given that with\_items: [a, static, list] works the same whether it’s in an include or in a task. Use case #2: - include: tasks/foo.yml delegate\_to: some\_other\_host One major benefit of using Ansible for me is splitting system configuration into manageable chunks, somewhat like Puppet classes. As long as includes behave differently from plain tasks (that _is_ unexpected if you don’t know the internals), I’ll have to either: - bloat every single line of every single task file into 4x the size by adding the delegate\_to/sudo/sudo\_user declarations from elsewhere in the thread - assume the task file can never be delegated to another machine, so e.g. my mail server must always have its database on the same host etc. Use case #3: - include: tasks/foo.yml tags: foo Again, I can either add yet another line to every single action, or always run playbooks in all-or-nothing mode. The simplest fix is indeed to keep patching around the fact that includes (and especially their lookups) are evaluated very early compared to the rest of the play, push down tags, delegate/sudo/sudo\_user, notify directives and keep adding lookup modules (I understand with\_inventory\_hostnames from ver. 1.3 solves my use case #1). However, this leaves the include+with\_\* behaviour subtly different from anything\_else+with\_\* and effectively unfixable. IMHO, for consistency, the “include” statement should be just an action\_plugin executed in-order with the rest of the play, _or_ it should somehow communicate that it’s executed out-of-order. And that’s where the lazy evaluation I described comes in – then includes are consistent with other tasks “for free”. It alse opens up new, exciting and crazy possibilities. Hopefully this clarifies my motivation a little bit. Best regards, Grzegorz Nosek ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 8, 2013, 12:25pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/8 "2013-08-08T12:25:23Z")
A use case would be explaining why you need something in terms of business objectives and what you are automating – with concrete rather than abstract examples, so we could suggest to you how to model it, rather than showing abstract “foo”. I also note you are still using archaic variable syntax with\_items: groups.some\_group is all you need and action: baz arg={{item}} is strongly preferred. ---
**Author:** ![C\_Morgan\_Hamill](https://avatars.discourse-cdn.com/v4/letter/c/ebca7d/32.png) [@C\_Morgan\_Hamill](https://forum.ansible.com/u/C_Morgan_Hamill)\ **Post date:** [August 8, 2013, 2:48pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/9 "2013-08-08T14:48:01Z")
Excerpts from Grzegorz Nosek's message of 2013-08-08 06:47:11 -0400: > - include: tasks/foo.yml >   with\_items: ${groups.some\_group} > > Yes, I know it is undocumented on purpose, (I think) I understand \_why\_ it > works the way it does etc., it's just deeply unintuitive that a simple > refactoring changes behaviour in an unexpected way, especially given that > with\_items: [a, static, list] works the same whether it's in an include > or in a task. As an aside, and having grappled my brain around this once or twice before, I'd add that a very nice \*benefit\* of the current behavior --- which I agree is unintuitive --- is that your task list can be treated as dumb YAML, because the task list must be identical for each host, and so, in theory, one can manipulate the task list as a simple data structure in a script. I imagine, with Michael's "infrastructure as data" thing, that he probably wants very much to avoid destroying that possibility. ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 8, 2013, 4:00pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/10 "2013-08-08T16:00:05Z")
Everyone who thinks it is intuitive is wrong 🙂 ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 8, 2013, 4:00pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/11 "2013-08-08T16:00:17Z")
Unintuitive! Freudian slip! Freudian slip! ---
**Author:** ![C\_Morgan\_Hamill](https://avatars.discourse-cdn.com/v4/letter/c/ebca7d/32.png) [@C\_Morgan\_Hamill](https://forum.ansible.com/u/C_Morgan_Hamill)\ **Post date:** [August 8, 2013, 4:15pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/12 "2013-08-08T16:15:57Z")
Excerpts from Michael DeHaan's message of 2013-08-08 12:00:17 -0400: > Unintuitive! > > Freudian slip! Freudian slip! > > \> Everyone who thinks it is intuitive is wrong 🙂 Haha! I should note that I actually prefer the current behavior --- but I still think it is unintuitive to many, because there's no particular reason, if one is unfamiliar with the implementation details, for this to work:     - debug: msg="{{ item }}"       with\_items: myinventorylist but for this to not:     - include: foo.yaml       with\_items: myinventorylist I strongly prefer the simplicity of implementation and consistency that comes from the current behavior, but it strikes me as perfectly reasonable to expect includes to work with loops. Considering how often this comes up, perhaps it is worth a warning note in the documentation for "include" about this? Regardless of whether it feels intuitive or not to each individual, it's clear that many expect this to work, so for them the current is, by definition, really, unintuitive. ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [August 8, 2013, 4:20pm UTC](https://forum.ansible.com/t/rfc-a-form-of-lazy-evaluation/13268/13 "2013-08-08T16:20:14Z")
That’s why we don’t tell anyone about it. It’s there for people with legacy usage of it only, and it should NOT be shared further. More people asking about it will get it removed sooner.