Hii
I have a list of dicts, where I want to inject a number of helper keys.
I can do this with set_facts in a loop. Example playbook (hopefully this displays OK):
Here’s the best I’ve got to offer. I tried pipelining filters etc but couldn’t quite get over the hump. So, back to old school loops.
- set_fact:
backup_objects: |
{% set result = [] %}
{% for obj in all_objects %}
{% set _ = result.append({'basename': obj['Key'] | regex_replace('^' ~ prefix ~ '\d{10}_(.*)\.pgdump', '\1')} | combine(obj)) %}
{% endfor %}{{ result }}
The point of the “set _” line is for the side effect of appending revised objects to the “result” list. Otherwise it’s pretty straightforward.
Cheers,
Create the list of the hashes
bn_regex: '^{{ prefix }}\d{10}_(.*)\.pgdump$'
bn: "{{ all_objects|
map(attribute='Key')|
map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"
gives
bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db
zip the lists and combine the items
backup_object: "{{ all_objects|zip(bn)|map('combine') }}"
There are options. Filter *basename* and get rid of *prefix*
bn_regex: '^\d{10}_(.*)\.pgdump$'
bn: "{{ all_objects|
map(attribute='Key')|
map('basename')|
map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"
Create the hash in *regex_replace* and use *from_yaml* instead
of *community.general.dict_kv*
bn: "{{ all_objects|
map(attribute='Key')|
map('basename')|
map('regex_replace', bn_regex, '{basename: \\1}')|
map('from_yaml') }}"
Instead of *regex_replace* use *splitext* and *split*
bn: "{{ all_objects|
map(attribute='Key')|
map('basename')|
map('splitext')|map('first')|
map('split', '_', 1)|map('last')|
map('community.general.dict_kv', 'basename') }}"
Thanks, Vladimir. I had missed the point of “community.general.dict_kv”. I had gotten as far as this:
- set_fact:
bn: |
{{ query('ansible.builtin.nested', ['basename'], (all_objects
> map(attribute='Key')
> map('regex_replace', '^' ~ prefix ~ '\d{10}_(.*)\.pgdump', ''))) }}
which produces:
bn:
- - basename
- dev_wss_db
- - basename
- dev_wss_db_requests
- - basename
- dev_bss_service_database
- - basename
- dev_bss_frontend_db
- - basename
- dev_mss_db
But I didn’t find a way to map that using “community.general.dict” to create
bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db
This for me is one of the more frustrating things about Jinja pipelines. I keep wishing “map” would take arbitrary expressions rather than the limited set it’s stuck with. So you end up with a fleet of one-off filters like “community.general.dict_kv” which does what “community.general.dict” would do if there were an obvious way to turn this:
- - basename
- dev_wss_db
- - basename
- dev_wss_db_requests
- - basename
- dev_bss_service_database
- - basename
- dev_bss_frontend_db
- - basename
- dev_mss_db
into this:
- - - basename
- dev_wss_db
- - - basename
- dev_wss_db_requests
- - - basename
- dev_bss_service_database
- - - basename
- dev_bss_frontend_db
- - - basename
- dev_mss_db
i.e. a “deepen” counterpart to “flatten”. But that magical incantation has so far eluded me.
bn:
- - basename
- dev_wss_db
- - basename
- dev_wss_db_requests
- - basename
- dev_bss_service_database
- - basename
- dev_bss_frontend_db
- - basename
- dev_mss_dbBut I didn't find a way to map that using "community.general.dict" to create
bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db
You can always use brute-force Jinja as the last resort. For example,
given the list
bn_list:
- dev_wss_db
- dev_wss_db_requests
- dev_bss_service_database
- dev_bss_frontend_db
- dev_mss_db
the below Jinja creates the list of the hashes
bn: |
{% filter from_yaml %}
{% for basename in bn_list %}
- basename: {{ basename }}
{% endfor %}
{% endfilter %}
As a side-note, this is equivalent to
bn: "{{ all_objects|
map(attribute='Key')|
map('regex_replace', bn_regex, '{basename: \\1}')|
map('from_yaml') }}"
This for me is one of the more frustrating things about Jinja
pipelines. I keep wishing "map" would take arbitrary
expressions rather than the limited set it's stuck with.
This is very good point. It would be possible to write such a filter.
However, I'm not sure about the security implications.
see map/select/reject filters .. they are actually loops and normally
much simpler than using jinja command syntax ( {% %} ).
Unfortunately, some filters are not *map* friendly. For example, the
filter *product*
list1|product(list2) .............. works fine
list1|zip(list2)|map('product') ... does not work
Details: Given the list
l1:
- dir: /tmp/test/d1
sub_dir: [a, b]
- dir: /tmp/test/d2
sub_dir: [a, b, c]
the goal is to create the list of products
l2:
- /tmp/test/d1/a
- /tmp/test/d1/b
- /tmp/test/d2/a
- /tmp/test/d2/b
- /tmp/test/d2/c
The iteration (the filter *subelements* not used
to demonstrate the functionality of *product*)
- debug:
msg: "{{ [item.0]|product(item.1) }}"
loop: "{{ dirs|zip(sdirs) }}"
vars:
dirs: "{{ l1|map(attribute='dir') }}"
sdirs: "{{ l1|map(attribute='sub_dir') }}"
works as expected. Gives (abridged)
msg:
- - /tmp/test/d1
- a
- - /tmp/test/d1
- b
msg:
- - /tmp/test/d2
- a
- - /tmp/test/d2
- b
- - /tmp/test/d2
- c
But, the filter *product* doesn't work with *map*
dirs: "{{ l1|map(attribute='dir') }}"
sdirs: "{{ l1|map(attribute='sub_dir') }}"
l3: "{{ dirs|zip(sdirs)|map('product') }}"
gives
l3:
- - - /tmp/test/d1
- - - a
- b
- - - /tmp/test/d2
- - - a
- b
- c
This leaves you with Jinja if you want to avoid the loops in tasks
l3: |
{% filter from_yaml %}
{% for i in l1 %}
{% for s in i.sub_dir %}
- {{ i.dir }}/{{ s }}
{% endfor %}
{% endfor %}
{% endfilter %}
I ended up with the inline jinja for loop.
Also I used a pipe lookup instead of an s3 list task prior to this one.
So now everything is done in a one single task, without noise, and with the same results.
Thanks everyone, it was really useful and I will keep it in mind for future reference.
Dick