# How to extract a value of a field within a list of deeply nested dictionaries when some keys match other values **URL:** \ **Category:** Ansible Project\ **Created:** [October 15, 2020, 4:32pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040 "2020-10-15T16:32:04Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![actionmystique](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/actionmystique/32/1678_2.png) [@actionmystique](https://forum.ansible.com/u/actionmystique)\ **Post date:** [October 15, 2020, 4:32pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/1 "2020-10-15T16:32:04Z")
Let’s assume the following fictional dictionary with nested lists, dictionaries, and strings, of arbitrary depth.: _domain\_definition.yml_ ```auto --- domain_definition: - name: server01 cluster: cluster0 port: '8070' - name: server11 cluster: cluster1 port: '8080' workstations: - name: debian1 revision: '2016-06-21' url: http://debian1.example.com:9911/2016-06-21 - name: ubuntu1 revision: '2017-08-16' url: http://ubuntu1.example.com:9912/2017-08-16 - name: server12 cluster: cluster2 port: '8090' workstations: - name: debian2 revision: '2018-04-15' url: http://debian2.example.com:9921/2018-04-15 - name: ubuntu2 revision: '2020-01-12' url: http://ubuntu2.example.com:9922/2020-01-12 ``` The goal is to extract as efficiently as possible (the original file can be quite large) a workstation matching: - a workstation os with its : for instance ubuntu1 ↔ 2017-08-16 - only a workstation os : for instance debian2 As you can see from example above, the key is not always set. A major constraint is to avoid “json\_query” as it cannot deal properly with keys containing a colon (as it may happen in my real context), last time I checked I unsuccessfully tried different tasks, including the following: ```auto - name: Extracting a value of a field deep within a list of complex dictionaries when some keys match other values hosts: - localhost strategy: debug tasks: - name: Including domain_definition include_vars: "domain_definition.yml" - name: Extracting the OS URL when is known vars: - os_name: ubuntu1 - os_revision: 2017-08-16 debug: msg: "os_url: {{ domain_definition | selectattr('workstations') | list | selectattr('name', 'eq', os_name) | list | selectattr('revision', 'eq', os_revision) | map(attribute= 'url') | first }}" ignore_errors: yes - name: Extracting the OS URL when is known vars: - os_name: ubuntu1 - os_revision: 2017-08-16 debug: msg: "os_url: {{ domain_definition | selectattr('workstations.name', 'eq', os_name) | list | selectattr('workstations.revision', 'eq', os_revision) | map(attribute= 'workstations.url') | first }}" ignore_errors: yes - name: Extracting the OS URL when is unknown vars: - os_name: debian2 debug: msg: "os_url: {{ domain_definition | selectattr('workstations') | list | selectattr('name', 'eq', os_name) | map(attribute= 'url') | first }}" ignore_errors: yes - name: Extracting the OS URL when is unknown vars: - os_name: debian2 debug: msg: "os_url: {{ domain_definition | selectattr('workstations.name', 'eq', os_name) | map(attribute= 'workstations.url') | first }}" ignore_errors: yes ``` I always get the error msg: ‘dict object’ has no attribute ‘workstations’. Any suggestion? ---
**Author:** ![vbotka](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vbotka/32/359_2.png) [@vbotka](https://forum.ansible.com/u/vbotka)\ **Post date:** [October 15, 2020, 6:47pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/2 "2020-10-15T18:47:15Z")
Add an empty list if the attribute is missing. For example     - include\_vars:         file: domain\_definition.yml         name: domain\_definition\_raw     - set\_fact:         domain\_definition: "{{ domain\_definition|default() +                                [my\_item] }}"       loop: "{{ domain\_definition\_raw.domain\_definition }}"       vars:         empty\_workstations:           workstations:         my\_item: "{{ (item.workstations is defined)|                      ternary(item, item|                      combine(empty\_workstations)) }}"     - debug:         var: domain\_definition give   domain\_definition:   - cluster: cluster0     name: server01     port: '8070'     workstations:   - cluster: cluster1     name: server11     port: '8080'     workstations:     - name: debian1       revision: '2016-06-21'       url: [http://debian1.example.com:9911/2016-06-21](http://debian1.example.com:9911/2016-06-21)     - name: ubuntu1       revision: '2017-08-16'       url: [http://ubuntu1.example.com:9912/2017-08-16](http://ubuntu1.example.com:9912/2017-08-16)   - cluster: cluster2     name: server12     port: '8090'     workstations:     - name: debian2       revision: '2018-04-15'       url: [http://debian2.example.com:9921/2018-04-15](http://debian2.example.com:9921/2018-04-15)     - name: ubuntu2       revision: '2020-01-12'       url: [http://ubuntu2.example.com:9922/2020-01-12](http://ubuntu2.example.com:9922/2020-01-12) This should solve the problems. ---
**Author:** ![actionmystique](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/actionmystique/32/1678_2.png) [@actionmystique](https://forum.ansible.com/u/actionmystique)\ **Post date:** [October 16, 2020, 2:10pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/3 "2020-10-16T14:10:24Z")
Thanks for your answer Vladimir. Now, there is a ‘workstations’ attribute all the time. However, I still cannot get ‘os\_url’. The following includes your few lines: ```auto - name: Extracting a value of a field deep within a list of complex dictionaries when some keys match other values hosts: - localhost strategy: debug tasks: - include_vars: file: domain_definition.yml name: domain_definition_raw - set_fact: domain_definition: "{{ domain_definition|default([]) + [my_item] }}" loop: "{{ domain_definition_raw.domain_definition }}" vars: empty_workstations: workstations: [] my_item: "{{ (item.workstations is defined)| ternary(item, item|combine(empty_workstations)) }}" - name: (1) Extracting the OS URL when is known vars: - os_name: ubuntu1 - os_revision: 2017-08-16 debug: msg: "os_url: {{ domain_definition | selectattr('workstations') | list | selectattr('name', 'eq', os_name) | list | selectattr('revision', 'eq', os_revision) | map(attribute= 'url') | first }}" ignore_errors: yes ``` leads to: “The task includes an option with an undefined variable. The error was: No first item, sequence was empty…” If I try to list only the workstations dictionaries with: ```auto - name: (0) Listing all workstations debug: msg: "workstations: {{ domain_definition | selectattr('workstations') | list }}" ignore_errors: yes ``` I get: ok: [localhost] =\> { “msg”: “workstations: [{‘name’: ‘server11’, ‘cluster’: ‘cluster1’, ‘port’: ‘8080’, ‘workstations’: [{‘name’: ‘debian1’, ‘revision’: ‘2016-06-21’, ‘url’: ‘[http://debian1.example.com:9911/2016-06-21](http://debian1.example.com:9911/2016-06-21)’}, {‘name’: ‘ubuntu1’, ‘revision’: ‘2017-08-16’, ‘url’: ‘[http://ubuntu1.example.com:9912/2017-08-16](http://ubuntu1.example.com:9912/2017-08-16)’}]}, {‘name’: ‘server12’, ‘cluster’: ‘cluster2’, ‘port’: ‘8090’, ‘workstations’: [{‘name’: ‘debian2’, ‘revision’: ‘2018-04-15’, ‘url’: ‘[http://debian2.example.com:9921/2018-04-15](http://debian2.example.com:9921/2018-04-15)’}, {‘name’: ‘ubuntu2’, ‘revision’: ‘2020-01-12’, ‘url’: ‘[http://ubuntu2.example.com:9922/2020-01-12](http://ubuntu2.example.com:9922/2020-01-12)’}]}]” } On top of that, even if I were able to extract only the ‘workstations’ dictionaries instead of the whole structure, there would also probably be an issue with the empty one(s) since I would not be able to access their non-existent nested attributes (name, version and url). ---
**Author:** ![actionmystique](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/actionmystique/32/1678_2.png) [@actionmystique](https://forum.ansible.com/u/actionmystique)\ **Post date:** [October 16, 2020, 2:15pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/4 "2020-10-16T14:15:03Z")
Regarding my last remark, I probably should rather define the empty workstations as: ```auto empty_workstations: workstations: - name: revision: url: ``` What are your thoughts? ---
**Author:** ![vbotka](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vbotka/32/359_2.png) [@vbotka](https://forum.ansible.com/u/vbotka)\ **Post date:** [October 16, 2020, 2:48pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/5 "2020-10-16T14:48:24Z")
It's up to you. If you need default values put them into the empty\_workstation, or better in this case, into the default\_workstations. For example                  default\_workstations:                    workstations:                      - name: name\_default                        revision: revision\_default                        url: url\_default Otherwise, you might want to simply skip the empty lists. For example     - debug:         msg: "{{ item.name }}:               {{ item.workstations|                  selectattr('name', 'eq', os\_name)|                  selectattr('revision', 'eq', os\_revision)|                  map(attribute= 'url')|                  list }}"       loop: "{{ domain\_definition }}"       when: item.workstations|length \> 0       vars:         os\_name: ubuntu1         os\_revision: '2017-08-16' ---
**Author:** ![actionmystique](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/actionmystique/32/1678_2.png) [@actionmystique](https://forum.ansible.com/u/actionmystique)\ **Post date:** [October 16, 2020, 4:13pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/6 "2020-10-16T16:13:21Z")
There are 2 issues:. 1. The loop can become costly as the number of entries rise: I have files with hundreds (potentially thousands) of servers/workstations. 2. it also results in as many answers as the number of entries, although only one is non empty. I don’t know how to set a variable with the single right answer. ```auto - debug: msg: "{{ item.workstations| selectattr('name', 'eq', os_name)| selectattr('revision', 'eq', os_revision)| map(attribute= 'url')| list }}" loop: "{{ domain_definition }}" when: item.workstations|length > 0 vars: os_name: ubuntu1 os_revision: '2017-08-16' ``` leads to: ```auto ok: [localhost] => (item={'name': 'server01', 'cluster': 'cluster0', 'port': '8070', 'workstations': [{'name': 'name_default', 'revision': 'revision_default', 'url': 'url_default'}]}) => { "msg": [] } ok: [localhost] => (item={'name': 'server11', 'cluster': 'cluster1', 'port': '8080', 'workstations': [{'name': 'debian1', 'revision': '2016-06-21', 'url': 'http://debian1.example.com:9911/2016-06-21'}, {'name': 'ubuntu1', 'revision': '2017-08-16', 'url': 'http://ubuntu1.example.com:9912/2017-08-16'}]}) => { "msg": [ "http://ubuntu1.example.com:9912/2017-08-16" ] } ok: [localhost] => (item={'name': 'server12', 'cluster': 'cluster2', 'port': '8090', 'workstations': [{'name': 'debian2', 'revision': '2018-04-15', 'url': 'http://debian2.example.com:9921/2018-04-15'}, {'name': 'ubuntu2', 'revision': '2020-01-12', 'url': 'http://ubuntu2.example.com:9922/2020-01-12'}]}) => { "msg": [] } ``` Isn’t there a loopless solution? ---
**Author:** ![vbotka](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/vbotka/32/359_2.png) [@vbotka](https://forum.ansible.com/u/vbotka)\ **Post date:** [October 16, 2020, 7:31pm UTC](https://forum.ansible.com/t/how-to-extract-a-value-of-a-field-within-a-list-of-deeply-nested-dictionaries-when-some-keys-match-other-values/34040/7 "2020-10-16T19:31:59Z")
>         - debug: >                 msg: "{{ item.workstations| >                            selectattr('name', 'eq', os\_name)| >                            selectattr('revision', 'eq', os\_revision)| >                            map(attribute= 'url')| >                            list }}" >           loop: "{{ domain\_definition }}" >           when: item.workstations|length \> 0 >           vars: >                 os\_name: ubuntu1 >                 os\_revision: '2017-08-16' > > Isn't there a loopless solution? Next option is a custom filter. For example, > cat filter\_plugins/custom\_select\_url.py def custom\_select\_url(l, os\_name, os\_revision):     l1 =     for i in l:          if 'workstations' in i.keys():              for j in i['workstations']:                  if (j['name'] == os\_name) and \                     (j['revision'] == os\_revision):                      l1.append(j['url'])     return l1 class FilterModule(object):         def filters(self):             return {                 'custom\_select\_url': custom\_select\_url                 } and the playbook > cat pb.yml - hosts: localhost   tasks:     - include\_vars: domain\_definition.yml     - debug:         msg: "{{ domain\_definition|                  custom\_select\_url('ubuntu1', '2017-08-16') }}" give   msg:   - [http://ubuntu1.example.com:9912/2017-08-16](http://ubuntu1.example.com:9912/2017-08-16)