looping over group_vars in a task include results in mis-interpolation of $item

Archives Ansible Project
1

Hi,

I’m trying to manage user creation with Ansible, but I am having problems getting variable interpolation, task includes and with_items to play nicely together. This is using a recent devel checkout, but it also seems to be a problem in 1.0 and 0.9.

Below is a stripped down example, in which users just have the pubkey attribute defined. The key idea is that a task file create.yml contains a list of tasks which should be executed for each user, and that the main playbook includes that from within a loop and sets the user attributes differently each time round.

  tasks:
    - include: create_user.yml user="${item}" pubkey="${userdefs.${item}.pubkey}"
      with_items: ${users}

I’ve put a fuller example in a gist here, with two versions of create_user.yml trying different variations of the above: https://gist.github.com/wu-lee/4994831. Output of running it is appended.

The problem I find is, when the variables are defined via group_vars/standard_users, they don’t seem to be interpolated as you’d expect - ${item} becomes “user1,user2”, and not “user1” then “user2” in sequence. This is frustrating since I think the pattern I’m using in createA.yml would be very handy in various other circumstances, when a number of tasks need to be grouped and repeated.

Furthermore, when ${users} is defined in a vars: section within the users.yml playbook, they do result in $item expanding to “user1”, “user2” in sequence. Some subtle interaction between variables, parameterised tasks, and with_items, it seems? However, I don’t want to have to use a vars: section because it means I can’t define a different user list by host/group via host_vars/group_vars.

So although it seems like it ought to work, I can’t get it to. Is this a bug?

And evidently other people must do it differently, since I can’t find much evidence anyone else has this problem. Is there any prior art for creating varying sets of users I can emulate? (For now I’ve worked around it by iterating each task in create_user.yml directly in the parent playbook, but it’s not very DRY).

One final point: in cases when a playbook actually fails to interpolate some variables (rather than interpolating them as something odd), Ansible blindly attempts to continue and, say, create users, only failing because the resulting user name is invalid. Wouldn’t it be better if Ansible refused to execute such commands with failed interpolations? I can’t imagine a situation when you’d actually want Ansible to excecute commands containing the unexpanded string “${item}” (for example). On the contrary, I imagine in some circumstances it might be harmful - if the command executes successfully and does something unexpected. (This sort of non-strict behaviour in Puppet has burnt me in the past.)

Cheers,

Nick

2

So although it seems like it ought to work, I can't get it to. Is this a
bug?

If you can reduce this down to the bare minimum possible gist, I'll take a look.

Right now, I'm failing to see the "include:" line in your gist.

BTW, the only_if stuff can go.

"when_set: $foo" is great.

3

That is approximately the bare minimum *working* gist. Part of the problem is that if I put everything in one self-contained playbook it works, but if I use group_vars, it doesn't.

If you just want to get an idea of the kind of thing I am trying to achieve, see below.

Cheers,

N

--- # users.yml
- name: Create users
user: root
hosts: all
vars: users: # This would be in group_vars - alice - bob userdefs: # This would be a shared list alice: pubkey: sdfjwkfuqwerw # More attributes here.... bob: pubkey: ajdfhUHFWJWRWE
          # More attributes here....
  tasks:
     - include: create_userA.yml user="${item}" pubkey="${userdefs.${item}.pubkey}"
with_items: ${users} --- # create_userA.yml tasks:

- name: Create the "${user}" user
action: user
state=present
name="${user}"
createhome=yes
- name: Insert authorized_keys for user "$user"
action: authorized_key
state=present
user="${user}"
key="${pubkey}"

4

Gah, sorry, something unhelpfully mangled my indentation there.

Try this: https://gist.github.com/wu-lee/5037444

5

Nick wrote:

Hi,

I'm trying to manage user creation with Ansible, but I am having problems
getting variable interpolation, task includes and with_items to play
nicely
together. This is using a recent devel checkout, but it also seems to be a
problem in 1.0 and 0.9.

Below is a stripped down example, in which users just have the pubkey
attribute
defined. The key idea is that a task file create.yml contains a list of
tasks
which should be executed for each user, and that the main playbook
includes that
from within a loop and sets the user attributes differently each time
round.

This is due to how includes are processed. Because they are handled as a
pre-processing directive, setting things as the files are loaded, you can
only use global variables (e.g. vars: or vars_files: without host-specific
variables in them) for with_items on include:.

Daniel

6

Thanks, that makes sense, but is a bit of a shame because using an include inside a loop like this seems such a useful thing to do.

Defining a fixed pool of users and creating a subset on each host/group must be a fairly common use case. Is there a nice way to do it which doesn't run into this limitation I did? I can't see anything very relevant in the examples/best practices docs so far, but it's hard to be sure without combing through every singe file. (I know about LDAP and similar non-Ansible solutions but they're not really appropriate for me.)

- N

ps

I note that my original example tries two ways of doing the creation - and something odd is going on in the second example. create_userB.yml doesn't use parameterised variables, it just uses $item directly in the task file. The output (pasted from my first email) includes this output from a debug action:

TASK: [debug msg="user is '${users}' pubkey is '${userdefs.${item}.pubkey}'"] *********************
ok: [some.host.net] => (item=${users}) => {"item": "${users}", "msg": "user is 'alice,bob' pubkey is '${userdefs.${item}.pubkey}'"}

I find it hard to explain why I can see "alice" and "bob" in there if ${users} isn't even in scope, but I'm guessing this is double-expansion in the debug task's message, where the second time the message is expanded ${users} is in scope and so expands to "alice,bob"?

7

I don't have time to help debug a full playbook stack when you did not
include the complete files. Gist also makes it hard to tell your
structure. I think what you need to do is host a complete *repo* that
illustrates what you are talking about, and
the command you are running that will cause whatever.

Meanwhile, you asked a question about includes and variable scope.

Variables in ansible are late-binding, so you can reference variables
that don't yet expand, but they can't be part of include paths and so
on. If you use one of these variables in a task name, the name won't
be all expanded out, but the action line
will be totally fine.

Think of the task object as something that applies to all hosts in the
group, and then each host decides whether or not to run them, and how
to interpret the variables.

This results in each hosts all having the same total counts of
operations they get to look at, which results in predictable counts as
to how many things they have either skipped (in case of conditionally
decided to run) or executed (pass or failed).

8

I think Daniel HZ already answered my question "why doesn't this work?", and the answer comes with the implication that this is a feature and not a bug.

For a complete repo, see below, but I don't want you to waste your time trying to debug my playbook stack. For the moment let's assume this trick can't be made to work the way things are. Given that, I would prefer to ask:

  - what is a good recipe for creating, on each host/group, a subset of a site-wide predefined list of users?

Which is what I actually want to achieve, and it might be useful for many other readers.

  - what is a good way to loop over a list of N tasks?

This is the more generalised question. I suspect your answer will be "you can't" but maybe there is another way?

Cheers

N

ps. A repo version of that Gist:

     https://github.com/wu-lee/ansible-users-usecase

9

I don't have time to help debug a full playbook stack when you did not
include the complete files. Gist also makes it hard to tell your
structure. I think what you need to do is host a complete *repo* that
illustrates what you are talking about, and
the command you are running that will cause whatever.

I think Daniel HZ already answered my question "why doesn't this work?", and
the answer comes with the implication that this is a feature and not a bug.

His implication is wrong, it's a bug :slight_smile:

There was a time when with_items made task objects, but there is no
reason for this anymore.
In fact, conditional vars_files works fine when based on facts. ex:

vars_files:
    - [ "/tmp/$ansible_domain.yml", "/tmp/default" ]

And that expands fine.

vars_files is fine for instance, so it's just an issue with group_vars
inventory not showing up in this case.

- what is a good recipe for creating, on each host/group, a subset of a
site-wide predefined list of users?

That with_items doesn't expand based on inventory scope is something I
need to fix, and I'll file a bug on this.

- what is a good way to loop over a list of N tasks?

with_items is still that answer.

10

Ticket here:

https://github.com/ansible/ansible/issues/2203

11

Turns out my ticket is incorrectly filed, as my test was wrong.

What you can do is define your list in groupvars/<groupname> or
hostvars/<hostname>

and use with_items: ${alist}

and it will do what you want.

12

Was going to just say "thanks", and that still stands, but now I'm confused.

Turns out my ticket is incorrectly filed, as my test was wrong.

What you can do is define your list in groupvars/<groupname> or
hostvars/<hostname>

and use with_items: ${alist}

and it will do what you want.

In the ticket, dhozac says:

The only case where it doesn't work is for include, which iterates the list and duplicates the tasks/plays for each item in the list

Which is exactly what I was trying to do: bundle some tasks in an include, and repeat it using with_items, expanding $item differently each time. Sounds like Daniel says it won't work, but you (maybe) are saying it will. Can you clarify?

  (We can take this to the ticket comments if you prefer)

N

13

In the ticket, dhozac says:

The only case where it doesn't work is for include, which iterates the
list and duplicates the tasks/plays for each item in the list

Daniel and I were discussing

tasks:
    - yum: name=$item state=present
      with_items: $packages

where $packages comes from group_vars.

It's a little different from your include example.

It's also probably completely workable for what you want to do.

The "include with items" trick has to create task objects, and task
objects need to be created the same for all the hosts in the group, so
they don't have
access to inventory variable information just yet.

Sorry, it's a little confusing, I agree... include+with_items together
is about as far on the 'advanced use case' side as it gets, thankfully
:slight_smile:

14

Timely discussion, just running into this myself… am I correct in understanding that with an external inventory, something like this http://ansible.cc/docs/playbooks2.html#advanced-task-includes isn’t possible? e.g.

playbook.yml

15

Timely discussion, just running into this myself... am I correct in
understanding that with an external inventory, something like this
http://ansible.cc/docs/playbooks2.html#advanced-task-includes isn't
possible? e.g.

# playbook.yml
---
- include: wordpress.yml app=$item
  with_items: $apps

# wordpress.yml
---
- name: check vars
  template: src=checkvars.j2 dest=/tmp/ansible.vars

# checkvars.j2
{{ app }}

$apps is being set via a 'hostfile' script that consumes JSON data from an
API (very similar to ansible commander, e.g. not using hostvars or
group_vars folders/files)... but when I examine the contents of
/tmp/ansible.vars on the remote machine, it has the entire list -
essentially appearing to just dump the contents of $apps into that file...

Correct, you'll have the same thing going on.

What you want to do instead is unroll that include, and reference the
users or packages or other list inside the included file, like I
referenced previously.

tasks:
   - action: whatever something=$item
     with_items: $apps

And that's fine.

Further, I'm not sure you were implying it, but you can't use Jinja2
inside playbooks -- there's a good reason for this -- if you could,
you couldn't parse the text, and they wouldn't be YAML anymore.

It used to technically work, but should not.

I'm kind of leaning on making "include + with_items" undocumented as
confusing as it actually is. We only actually documented it recently
and it's being confusing to people with the limitations it has, and
there are easy workarounds.

--Michael

16

When you say:

"Further, I’m not sure you were implying it, but you can’t use Jinja2
inside playbooks – there’s a good reason for this – if you could,
you couldn’t parse the text, and they wouldn’t be YAML anymore.

It used to technically work, but should not."

Are you saying that something like this wouldn’t or shouldn’t (or someday won’t) work:

17

This works now.

18

Right. Perhaps my misunderstanding on what you meant by “use Jinja2 inside playbooks”… definitely aware and wouldn’t attempt to do something like that, because yes agreed, it should be YAML (and able to be parsed as such)…

Thanks for all your help…

– sean

19

Raises hand - what a coincidence, just experienced this myself and was about to post a message. In my case I was trying to loop over a created fact.