with_recursive: A lookup plugin that chains other lookups

Archives Ansible Developer
1

Hi

I wrote a lookup plugin and I’d like to hear what you think. It allows one to nest loops inside each other. Kind of like with_nested, but with other loop types than ‘items’. The code is here: https://gist.github.com/hkariti/d07695b4b9c5a68d8c02

It allows you to do these:

vars:
some_dict:
key1:
inner_dict:
i_key1: value
key2:
inner_dict:
n_key1: value2
tasks:

Will output ‘value’ and ‘value2’

  • debug: var=item.value
    with_recursive:
  • { name: dict, args: some_dict }
  • { name: dict: args: “{{item.value.inner_dict}}” }

and:

vars:
cmds:

  • cat /etc/passwd
  • ls -l
    tasks:

Will output the stdout of both commands

  • debug: var=item
    with_recursive:
  • { name: items, args: urls }
  • { name: pipe, args: “{{item}}” }

Before callling each lookup plugin, it’s arguments are templated. Each lookup is also provided with the following variables:

  • ‘item’, which is the iterated value from the previous lookup’s results.
  • ‘items’, which is a list of all the items from all previous nesting levels
  • ‘item_idx’, which is the index of ‘item’ in ‘items’

I’d love to hear about whether it’s ugly, cool, unnecessary, etc.

Thanks,
Hagai

2

This is already possible, just with diff format:

with_file:

  • ‘/file/iknow/of’

  • “{{ lookup(‘pipe’, ‘find path/to/dir/with/files/ -type f’}}”

You can even nest lookups {{ lookup(‘pipe’, ‘grep ’ + lookup(‘consul_kv’, ‘special_key’) + ’ path/to/local/files/*’) }}

3

Yes, you can combine lookups, but each lookup needs to be independent in your example. You can’t use the results of the previous lookup. Can you do my first example (with the dict and deep_dict) with lookups?

4

That is what both my examples do, in the 2nd it consul data to return a list of files that file lookup will read and return the file content.

5

The first example doesn’t really work when find returns multiple results, as with_file parses the entire output as a single file. Using the lines lookup instead with the wantlist=True option fails because with_file doesn’t handle nested lists. The second example only works when the consul lookup returns a single value, for the same reason.

6

all lookups take lists as inputs, even when passing a single string it
gets converted into a list, I'll look into the issues you mention, but
this was designed to be chainable, I've also used them this way in the
past.

Most lookups won't do flattening, but 'items' does 1 level and
'flattened' does multiple levels and you can chain them in also.

7

Hello Hagai,

I think this is awesome! It is exactly what I was looking for. I think this is much easier to understand than the lookup option Brian mentions in the continuing thread. Which I completely didn't understand.

Keep up the great work.

Hank