# Filtering a complex list of dictionaries into another list of dictionaries in an efficient way **URL:** \ **Category:** Ansible Project\ **Created:** [January 22, 2020, 6:32pm UTC](https://forum.ansible.com/t/filtering-a-complex-list-of-dictionaries-into-another-list-of-dictionaries-in-an-efficient-way/32029 "2020-01-22T18:32:56Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![actionmystique](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/actionmystique/32/1678_2.png) [@actionmystique](https://forum.ansible.com/u/actionmystique)\ **Post date:** [January 22, 2020, 6:32pm UTC](https://forum.ansible.com/t/filtering-a-complex-list-of-dictionaries-into-another-list-of-dictionaries-in-an-efficient-way/32029/1 "2020-01-22T18:32:56Z")
**ansible 2.9.4** Let’s assume the following list of dictionaries: ` list: - key1: - ‘abc’ - ‘def’ key2: ‘ghi’ key3: ‘jkl’ - key1: - ‘mno’ - ‘pqr’ key3: ‘stu’ key4: ‘dfg’ - key1: - ‘vwx’ - ‘yza’ key3: ‘okl’ key4: ‘azel’ ` The goal is threefold: - to extract only the records which match a regex criteria over one of the keys, for instance if “key3” contains a ‘k’. - to keep only a selection of keys, for instance only “key1”, “key3” and “key4”, which may not always be present - to be as efficient as possible with thousands of records In this case, we expect the resulting list: ` result\_list: - key1: - ‘abc’ - ‘def’ key3: ‘jkl’ - key1: - ‘vwx’ - ‘yza’ key3: ‘okl’ key4: ‘azel’ ` The difficulty I’m experiencing concerns “key1” as a list. My solution works without considering “key1”: ` - name: Filtering a complex list of dictionaries into another list of dictionaries vars: “list”: [ { “key1”: [ “abc”, “def” ], “key2”: “ghi”, “key3”: “jkl” }, { “key1”: [ “mno”, “pqr” ], “key3”: “stu”, “key4”: “dfg” }, { “key1”: [ “vwx”, “yza” ], “key3”: “okl”, “key4”: “azel” } ] set\_fact: result\_list: “{{ result\_list|default() + [{‘key3’: item.key3, ‘key4’: item.key4|default(‘’)}] }}” loop: “{{ list }}” when: item.key3 | regex\_search(‘(^._k._$)’) ` leads to: ` TASK [yang : Filtering a complex list of dictionaries into another list of dictionaries] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* task path: test.yml:133 \<172.16.136.116\> attempting to start connection \<172.16.136.116\> using connection plugin network\_cli \<172.16.136.116\> found existing local domain socket, using it! \<172.16.136.116\> updating play\_context for connection \<172.16.136.116\> \<172.16.136.116\> local domain socket path is .ansible/pc/521e859c25 ok: [TEST] =\> (item={‘key1’: [‘abc’, ‘def’], ‘key2’: ‘ghi’, ‘key3’: ‘jkl’}) =\> { “ansible\_facts”: { “result\_list”: [ { “key3”: “jkl”, “key4”: “” } ] }, “ansible\_loop\_var”: “item”, “changed”: false, “item”: { “key1”: [ “abc”, “def” ], “key2”: “ghi”, “key3”: “jkl” } } skipping: [TEST] =\> (item={‘key1’: [‘mno’, ‘pqr’], ‘key3’: ‘stu’, ‘key4’: ‘dfg’}) =\> { “ansible\_loop\_var”: “item”, “changed”: false, “item”: { “key1”: [ “mno”, “pqr” ], “key3”: “stu”, “key4”: “dfg” }, “skip\_reason”: “Conditional result was False” } ok: [TEST] =\> (item={‘key1’: [‘vwx’, ‘yza’], ‘key3’: ‘okl’, ‘key4’: ‘azel’}) =\> { “ansible\_facts”: { “result\_list”: [ { “key3”: “jkl”, “key4”: “” }, { “key3”: “okl”, “key4”: “azel” } ] }, “ansible\_loop\_var”: “item”, “changed”: false, “item”: { “key1”: [ “vwx”, “yza” ], “key3”: “okl”, “key4”: “azel” } } ` How can we insert “ **key1** ” in the picture? Also, when the list contains thousands of records, it may be less compute intensive to use ` **json_query** `, but I don’t know how to use it in this context. ---
**Author:** ![vbotka](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vbotka/32/359_2.png) [@vbotka](https://forum.ansible.com/u/vbotka)\ **Post date:** [January 22, 2020, 8:49pm UTC](https://forum.ansible.com/t/filtering-a-complex-list-of-dictionaries-into-another-list-of-dictionaries-in-an-efficient-way/32029/2 "2020-01-22T20:49:43Z")
The task below does the job     - set\_fact:         result\_list: "{{ result\_list|                          default() + [                          dict(keys|                               zip(keys|                                   map('extract', item)|                                   list))] }}"       vars:         keys: "{{ ['key1', 'key3', 'key4']|                   intersect(item.keys()|list) }}"       loop: "{{ list|                 selectattr('key3', 'regex', '^.\*k.\*$')|                 list }}" json\_query is not much of use here, I think. If you have any large sets' benchmarks I'll be interested to learn. Thank you. HTH,   -vlado ---
**Author:** ![vbotka](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vbotka/32/359_2.png) [@vbotka](https://forum.ansible.com/u/vbotka)\ **Post date:** [January 22, 2020, 10:25pm UTC](https://forum.ansible.com/t/filtering-a-complex-list-of-dictionaries-into-another-list-of-dictionaries-in-an-efficient-way/32029/3 "2020-01-22T22:25:42Z")
Custom filter shall improve the efficiency. For example a filter to select a list of keys from a dictionary   $ cat filter\_plugins/dict\_utils.py   def dict\_select\_list(d, l):       d2 = {}       for k in l:           d2[k] = d[k]       return d2   class FilterModule(object):       def filters(self):           return {               'dict\_select\_list' : dict\_select\_list               } The task below gives the same result     - set\_fact:         result\_list: "{{ result\_list|                          default() + [                          item\>dict\_select\_list(keys)] }}"       vars:         keys: "{{ ['key1', 'key3', 'key4']|                   intersect(item.keys()|list) }}"       loop: "{{ list|                 selectattr('key3', 'regex', '^.\*k.\*$')|                 list }}" HTH,   -vlado ---
**Author:** ![actionmystique](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/actionmystique/32/1678_2.png) [@actionmystique](https://forum.ansible.com/u/actionmystique)\ **Post date:** [January 23, 2020, 6:54pm UTC](https://forum.ansible.com/t/filtering-a-complex-list-of-dictionaries-into-another-list-of-dictionaries-in-an-efficient-way/32029/4 "2020-01-23T18:54:52Z")
fantastic! 🙂 My real use case is little more complex: 1. there are other attributes like “key1” or similar to “list” (within top-level “list”) which must be taken into account 2. the regex filter on “key3” is a little more complex (a list of logical OR) I tried to to implement the 2 points by expanding your solution, and it works **beautifully,** despite the fact that some new keys are themselves list of dictionaries, instead of simple lists or strings. I used something like: ` … vars: keys: “{{ [‘key1’, ‘key3’, ‘key4’, ‘key5’, ‘key6’]| intersect(item.keys()|list) }}” loop: “{{ list| selectattr(‘key3’, ‘regex’, ‘regex1|regex2|regex3’)| list }}” ` Finally, is there an **online documentation** that you would recommend to learn all the necessary tools to be able to perform such great filters (the ansible documentation is very sparse on that subject)? For instance, if I need to add another constraint like another attribute must match another regex alongside “key3” (as an logical AND), I have no clue. ---
**Author:** ![vbotka](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vbotka/32/359_2.png) [@vbotka](https://forum.ansible.com/u/vbotka)\ **Post date:** [January 23, 2020, 7:42pm UTC](https://forum.ansible.com/t/filtering-a-complex-list-of-dictionaries-into-another-list-of-dictionaries-in-an-efficient-way/32029/5 "2020-01-23T19:42:31Z")
It's possible to extend the pipe. For example        loop: "{{ list|                  selectattr('key3', 'regex', 'regex1|regex2|regex3')|                  selectattr('key4', 'defined')|                  list }}" HTH,   -vlado