RFC: A form of lazy evaluation

Archives Ansible Project
1

Hi,

Once loaded, a playbook is completely static (includes etc. are evaluated while loading, not while executing). This has its advantages but also makes includes special, e.g.
* they don't handle tags or delegate_to
* not sure what they do with sudo/sudo_user from reading the code (silently ignored unless I'm missing something?)
* with_* handling is unintuitive (you can't say with_items: $groups.foo due to very eager evaluation of include/with_*)

At the very least, we'd like to extend the include directive to push down tags, sudo/sudo_user, delegate_to, notify to individual tasks in the file, but for a more flexible approach, how about we introduced some form of lazy evaluation/dynamic playbooks? ("we introduced" meaning "I'll submit a patch once we agree on a design")

An action plugin could optionally return a list of dicts describing additional tasks, just like extra facts right now. The tasks would be then fed back into the playbook loop by e.g. PlayBook._run_task calling itself recursively. Note that this has nothing to do with includes and YAML parsing and is very generic. Extra tasks could be loaded from a database, generated on the fly etc.

A major issue is security. This feature would open up ways to influence ansible-playbook's actions from outside the playbook YAML file. Given a vulnerable module _and_ a compromised host in inventory, the master could be tricked into executing arbitrary modules. I'd suggest this feature be available only to action_plugins using a class attribute, similar to BYPASS_HOST_LOOP ("normal" not being one of these whitelisted plugins).

To fully implement our lazy_includes we'd need some code from Play/PlayBook classes copied or factored out to a helper class but that could remain our in-house mess. We'd also need a way to notify the loader module what tags are white/blacklisted if we want tags in lazy-loaded plays to just work.

Yes, it's a major change in ansible's behaviour but right now I think that the static nature of playbooks lead to unexpected behaviour similar to e.g. nginx and its if{} blocks. They look like procedural config (evaluated top-down) but are strictly declarative and evaluated outside in (top level, then ifs, then nested ifs...). This led to endless confusion and "if" is now deprecated in nginx.

Or, if letting tasks return more tasks is too drastic (despite being 100% backwards-compatible), maybe pushing the loading process to plugins is a way to go?

Best regards,
  Grzegorz Nosek

2

There are some inaccuracies in what you post here, it talks about lazy evaluation and other things, and then it talks about setting variables on include statements – some of which (things like tags) we can already do. You talk a little bit about use cases but then dive into technical implementations. Sorry, I’m finding the train of thought very hard to follow.

I disagree strongly about “unexpected behavior”, what you propose seems to imply lots of unexpected behavior – but I can’t tell, because there aren’t any use cases being discussed.

Would it be possible to step back and talk about your use cases first separate from the implementation and code?

3

I'm curious what you mean here.

I have a number of tasks that are generic and included in multiple play books, and I've set them up in a way that they can be delegated to a host or not, depending on a variable passed at the - include statement:

e.g.

- name: create facter tree
  file: path={{ facterd }} state=directory
  delegate_to: "{{ delegate|default(inventory_hostname) }}"

and in the playbook you can do:

- include: tasks/sync-facts.yaml

which will run it on the host in the loop, OR

- include: tasks/sync-facts.yaml delegate={{ some_other_host_value }}

which will cause the task to be delegated to whatever the value of some_other_host_value happens to be.

In some of my playbooks, the variable passed to delegate= comes from a fact created using set_fact earlier in a play for really dynamic optional delegation.

-jlk

4

Yeah, it’s fine to teach the include construct (like the roles construct) to support understanding to auto-apply a delegate_to keyword if delegate_to is set on the include.

This already occurs for tags, for instance.

Ansible is already a pretty lazy loading system as variables go, for things that are specified in task operations.

5

That would be handy!

-jlk

6

W dniu środa, 7 sierpnia 2013 16:22:25 UTC+2 użytkownik Jesse Keating napisał:

  • they don’t handle tags or delegate_to

I’m curious what you mean here.

I have a number of tasks that are generic and included in multiple play books, and I’ve set them up in a way that they can be delegated to a host or not, depending on a variable passed at the - include statement:

e.g.

  • name: create facter tree
    file: path={{ facterd }} state=directory
    delegate_to: “{{ delegate|default(inventory_hostname) }}”

and in the playbook you can do:

  • include: tasks/sync-facts.yaml

which will run it on the host in the loop, OR

  • include: tasks/sync-facts.yaml delegate={{ some_other_host_value }}

which will cause the task to be delegated to whatever the value of some_other_host_value happens to be.

Sure, this gets the job done. But this now every task in the included file gets bloated and unreadable. Compare:

toplevel.yml

  • include: included.yml delegate=host sudo=True sudo_user=whatever

included.yml

  • action: foo
    delegate_to: “{{ delegate|default(inventory_hostname) }}”
    sudo: “{{ sudo|default(False) }}”

sudo_user: “{{ sudo_user|default(“nobody”) }}”

  • action: bar
    delegate_to: “{{ delegate|default(inventory_hostname) }}”
    sudo: “{{ sudo|default(False) }}”

sudo_user: “{{ sudo_user|default(“nobody”) }}”

  • action: baz
    delegate_to: “{{ delegate|default(inventory_hostname) }}”
    sudo: “{{ sudo|default(False) }}”

sudo_user: “{{ sudo_user|default(“nobody”) }}”

with:

toplevel.yml

  • include: included.yml
    delegate_to: host
    sudo: True
    sudo_user: whatever

included.yml

  • action: foo
  • action: bar
  • action: baz

Best regards,
Grzegorz Nosek

7

W dniu środa, 7 sierpnia 2013 15:38:56 UTC+2 użytkownik Michael DeHaan napisał:

There are some inaccuracies in what you post here, it talks about lazy evaluation and other things, and then it talks about setting variables on include statements – some of which (things like tags) we can already do. You talk a little bit about use cases but then dive into technical implementations. Sorry, I’m finding the train of thought very hard to follow.

I disagree strongly about “unexpected behavior”, what you propose seems to imply lots of unexpected behavior – but I can’t tell, because there aren’t any use cases being discussed.

Would it be possible to step back and talk about your use cases first separate from the implementation and code?

Yeah, let’s give it another try.

Use case #1: Getting from:

  • action: foo arg=$item
    with_items: ${groups.some_group}

  • action: bar arg=$item
    with_items: ${groups.some_group}

  • action: baz arg=$item
    with_items: ${groups.some_group}

to:

  • include: tasks/foo.yml
    with_items: ${groups.some_group}

Yes, I know it is undocumented on purpose, (I think) I understand why it works the way it does etc., it’s just deeply unintuitive that a simple refactoring changes behaviour in an unexpected way, especially given that with_items: [ a, static, list ] works the same whether it’s in an include or in a task.

Use case #2:

  • include: tasks/foo.yml
    delegate_to: some_other_host

One major benefit of using Ansible for me is splitting system configuration into manageable chunks, somewhat like Puppet classes. As long as includes behave differently from plain tasks (that is unexpected if you don’t know the internals), I’ll have to either:

  • bloat every single line of every single task file into 4x the size by adding the delegate_to/sudo/sudo_user declarations from elsewhere in the thread
  • assume the task file can never be delegated to another machine, so e.g. my mail server must always have its database on the same host etc.

Use case #3:

  • include: tasks/foo.yml
    tags: foo

Again, I can either add yet another line to every single action, or always run playbooks in all-or-nothing mode.

The simplest fix is indeed to keep patching around the fact that includes (and especially their lookups) are evaluated very early compared to the rest of the play, push down tags, delegate/sudo/sudo_user, notify directives and keep adding lookup modules (I understand with_inventory_hostnames from ver. 1.3 solves my use case #1). However, this leaves the include+with_* behaviour subtly different from anything_else+with_* and effectively unfixable.

IMHO, for consistency, the “include” statement should be just an action_plugin executed in-order with the rest of the play, or it should somehow communicate that it’s executed out-of-order. And that’s where the lazy evaluation I described comes in – then includes are consistent with other tasks “for free”. It alse opens up new, exciting and crazy possibilities.

Hopefully this clarifies my motivation a little bit.

Best regards,
Grzegorz Nosek

8

A use case would be explaining why you need something in terms of business objectives and what you are automating – with concrete rather than abstract examples, so we could suggest to you how to model it, rather than showing abstract “foo”.

I also note you are still using archaic variable syntax

with_items: groups.some_group is all you need

and

action: baz arg={{item}}

is strongly preferred.

9

Excerpts from Grzegorz Nosek's message of 2013-08-08 06:47:11 -0400:

- include: tasks/foo.yml
  with_items: ${groups.some_group}

Yes, I know it is undocumented on purpose, (I think) I understand _why_ it
works the way it does etc., it's just deeply unintuitive that a simple
refactoring changes behaviour in an unexpected way, especially given that
with_items: [ a, static, list ] works the same whether it's in an include
or in a task.

As an aside, and having grappled my brain around this once or twice
before, I'd add that a very nice *benefit* of the current behavior ---
which I agree is unintuitive --- is that your task list can be treated
as dumb YAML, because the task list must be identical for each host, and
so, in theory, one can manipulate the task list as a simple data
structure in a script.

I imagine, with Michael's "infrastructure as data" thing, that he
probably wants very much to avoid destroying that possibility.

10

Everyone who thinks it is intuitive is wrong :slight_smile:

11

Unintuitive!

Freudian slip! Freudian slip!

12

Excerpts from Michael DeHaan's message of 2013-08-08 12:00:17 -0400:

Unintuitive!

Freudian slip! Freudian slip!

> Everyone who thinks it is intuitive is wrong :slight_smile:

Haha! I should note that I actually prefer the current behavior ---
but I still think it is unintuitive to many, because there's no
particular reason, if one is unfamiliar with the implementation details,
for this to work:

    - debug: msg="{{ item }}"
      with_items: myinventorylist

but for this to not:

    - include: foo.yaml
      with_items: myinventorylist

I strongly prefer the simplicity of implementation and consistency that
comes from the current behavior, but it strikes me as perfectly
reasonable to expect includes to work with loops.

Considering how often this comes up, perhaps it is worth a warning note
in the documentation for "include" about this?

Regardless of whether it feels intuitive or not to each individual, it's
clear that many expect this to work, so for them the current is, by
definition, really, unintuitive.

13

That’s why we don’t tell anyone about it. It’s there for people with legacy usage of it only, and it should NOT be shared further.

More people asking about it will get it removed sooner.