Nested looping with hash/dict so I can override values

tl;dr: is there a way to make with_subelements work with dicts? Or something to that effect?

I’ve been trying to figure out how to run a task over a dict, that also loops over a nested dict in order to run the actual commands.

So far, I haven’t figured out how to use with_items, with_dict, or with_nested to do what I want. with_subelements comes close, but keeps choking on the fact my subelement is not a list.

My playbook (would have pastebined it, but pastebin wouldn’t load for me…):


testing playbook

  • hosts: 192.168.88.2
    vars:
    people:
    johnsmith:
    fname: john
    lname: smith
    locations:
    birthplace:
    state: id
    city: boise
    second:
    state: or
    city: portland
    third:
    state: ha
    city: honolulu
    jilljones:
    fname: jill
    lname: jones
    locations:
    birthplace:
    state: mo
    city: springfield
    second:
    state: mt
    city: fort benton
    third:
    state: id
    city: emmett
    jilljones:
    locations:
    birthplace:
    state: wa
    city: wilbur
    tasks:
  • name: testing dict
    debug: msg=“{{ item.1 }}”
    with_subelements:
  • people
  • locations

The output:

$ ansible-playbook -i hosts -u vagrant --sudo tmp/testing.yml
PLAY [192.168.88.2] ***********************************************************
GATHERING FACTS ***************************************************************
ok: [192.168.88.2]
TASK: [testing dict] **********************************************************
fatal: [192.168.88.2] => the key locations should point to a list, got ‘{‘birthplace’: {‘city’: ‘boise’, ‘state’: ‘id’}, ‘second’: {‘city’: ‘portland’, ‘state’: ‘or’}, ‘third’: {‘city’: ‘honolulu’, ‘state’: ‘ha’}}’
FATAL: all hosts have already failed – aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/home/reagand/testing.retry
192.168.88.2 : ok=1 changed=0 unreachable=1 failed=0

What I want to see is something like:

johnsmith:
birthplace:
state: id
city: boise
second:
state: or
city: portland
third:
state: ha
city: honolulu
jilljones:
birthplace:
state: wa
city: wilbur
second:
state: mt
city: fort benton
third:
state: id
city: emmett

I really need this to be a dict/hash so that hash_behaviour=merge works. For example, if I want jilljones birthplace to default to springfield, mo, in group_vars, but for a specific host it should be seattle, wa, then I would set the following in the specific host_vars file:

people:
jilljones:
locations:
birthplace:
city: seattle
state: wa

Note, I’ve only been using this people hash as something to experiment on. My actual use case is a task that sets configuration settings for a web app. The app config didn’t lend itself to templating. I tried that first. Now I’ve ended up with lininfile regex replacements, each named with their own key. Something like:

configkey:
regex:
value:
configkey2:
regex:
value:

That’s the equivalent of the locations hash in my testing playbook.

I also don’t know how many instances of the app I’ll have per server, so I can’t just do a task per instance.

Ultimately, if I make my subelement a list, http://pastebin.com/mixMTz6H (pastebin worked earlier…) it works, but it overrides all the previously set subelements of the main key. That means I have to copy all the locations from group_vars into host_vars if I want to override something on a specific host. I’d prefer to avoid that. So, is there a with_subelements that would work with a hash like I want? Or another way to get the same effect?

So, I took a look at creating a plugin. I copied subelements.py, and modified it to return a dict. See https://gist.github.com/jerrac/8a16e0c1031df89621da

I think it will do what I want, I’ve only done some basic testing. Anyone have any feedback?

Should I modify the copyright? I’m pretty sure I should, but I’m not sure what to…

“tl;dr: is there a way to make with_subelements work with dicts? Or something to that effect?”

Doesn’t sound like you need the nested loop that subelements provides and you’d be happy with https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/lookup_plugins/dict.py

That is unless you want to loop over each locations for each user, in which case, sure, you’re doing the right thing.

If subelements can be made to do this unobtrusively patches would be accepted.

My pull request for with_nested () allows for all kind of nested loops, including the simpler case that is handled by the ‘with_subelements’ lookup. In David’s example, where the items in the first level of the loop are actually dictionary keys, to loop over the locations for each person, you would write: with_nested: - people - people[item.0].locations Now, for a comparison with the ‘with_subelements’ lookup, consider the following dict: users: - name: alice authorized_keys: - /tmp/alice/onekey.pub - /tmp/alice/twokey.pub - name: bob authorized_keys: - /tmp/bob/id_rsa.pub To construct a loop with “with_subelements” you would write: with_subelements: - users - authorized_keys To construct the same loop with my version of ‘with_nested’ you would write: with_nested: - users - item.0.authorized_keys Of course, the power of my version of ‘with_nested’ is that it allows you to make multiple-level nested loops that cannot be constructed in another way. For example, it allows for the pattern: Loop over the hosts in a specific group, for each host access a list hostvar and run a task for each item in that list. The list hostvar could be as much deep as you want. Just add the loop levels needed to reach it. So, let’s say that you want to create a number of VMs that will be the members of a ‘guests’ group. You would have to connect to your virtualiazation server host, let’s call it ‘vmhost’, and do what ever it is needed to create the VMs. After they have been created, you want to deploy the VMs with Ansible, but to be able to do that you would have first to bootstrap authorized ssh keys for ‘root’ user in order to pass them in the VM bootstrapping process. This is exactly a task for ‘with_nested’. First, you configure your inventory, as needed: basically you create the ‘guests’ group and hostvars files for each host in that group with a ‘users’ variable like the above. Then you construct the loop: At first level, you would loop over each host in the ‘guests’ group. At second level, you would loop over each user in the ‘users’ variable. At third level you would loop over each item in the ‘authorized’ list. Also add a conditional for ‘root’ user and …voila: - hosts: vmhost tasks: - name: Create authorized_keys for bootstrapping vm guests authorized_key: user=“{{ item.1.name }}” key=“{{ lookup(‘file’, item.2) }}” path=/path/to/bootstrap/guests/{{ item.0 }}/root/.ssh/authorized_keys with_nested: - groups[“guests”] - hostvars[item.0].users - item.1.authorized_keys when: item.1.name == “root”

Yeah… I’m confused. I found with_nested to be a bit odd in the first place, much less trying to figure out what Petros is adding.

I ended up with https://github.com/jerrac/aspects/blob/master/ansible_plugins/lookup_plugins/subdict.py used at https://github.com/jerrac/aspects/blob/master/roles/owncloud/tasks/main.yml#L91

If I were to try and make that worth a pull request, I’d want to be able to look at sub dictionaries at any level (item.0, item.1 … item.N). I don’t think my code supports more than two levels right now. I’m not inclined to try that since python is not one of my strong points.

Ignore the rest of the project, I still have a ways to go before I think others should use it. Especially since galaxy.ansible.com now exists.

With my addition, with_nested is extended allowing you to use item.0, item.1, etc… in your expressions. The numbers 0, 1, 2, etc… denote the loop level. 0 is for the first (outer) loop level, 1 is for one level deeper, 2 is for one more level deeper, etc… Nothing more, nothing less, but this addition is quite powerful because it gives you the ability, in each possible loop branch (think of a nested loop as a tree, see below), to reference back to the values that were evaluated in previous loop levels (nodes) in the same branch. Also, have you looked carefully at my other examples? What exactly don’ t you understand in these?