Are there any validated procedures for handling complex JSON structures?

Get Help
,
1

Hello Community :waving_hand:,
the last few days I’ve been working with HTTP requests via ansible.builtin.uri. So far this works as expected. The return value of this API call is a complex JSON structure. It looks something like this:

{
    "link": [{
            "attributes": [{
                    "value": "4711",
                    "name": "id"
                }, {
                    "value": "bambi",
                    "name": "name"
                }, {
                    "name": "description"
                }, {
                    "value": "de.stschnell/bambi",
                    "name": "fqn"
                }, {
                    "value": "0.0.0",
                    "name": "version"
                }
            ],
            "href": "http://localhost/4711/",
            "rel": "yogi"
        }, ...
    ],
    ...
    "total": 1234
}

Here the ID of an attribute must be detected via the fqn. This means the input is de.stschnell/bambi and the output is 4711. I realized this with a two-step approach. In the first step I determine whether the FQN is identical and, if so, the ID is determined in the second step. That also works as expected.

    - ansible.builtin.set_fact:
        attribute: "{{ item }}"
      loop: "{{ dict_actions.link | subelements('attributes') }}"
      when:
        - item.1.name == 'fqn'
        - item.1.value == 'de.stschnell/bambi'

    - ansible.builtin.set_fact:
        action_id: "{{ item.value }}"
      loop: "{{ attribute[0].attributes }}"
      when:
        - item.name == 'id'

Using the search function in this forum, I saw that several questions have already been asked about processing complex JSON structures. I don’t have any questions about this particular case at the moment. But I spent quite some time creating the playbook. Handling complex JSON structures with Ansible seems complicated to me. Perhaps I simply lack experience, but I would like to ask a few questions about this:

  1. Are there any established standard procedures for handling complex JSON structures in Ansible playbooks?
  2. Are there any available resources that describe the handling of complex JSON structures in Ansible playbooks with examples?
  3. I found this great example from Todd [@utoddl] in which two approaches are demonstrated :+1:. Jinja2 is used in one implementation. If I understand correctly, Jinja2 seems to offer a very powerful approach to this requirement. Are there some good resources available for learning how to use Jinja2 in Ansible playbooks?

My Ansible journey has begun. :train:

Thanks for hints and tips.
Best regards
Stefan

1 Like
2

These are my sentiments exactly (about myself, not you). Only, I’ve been doing this now for quite a few years, so I can’t attribute the difficulty to a lack of experience any more. It seems complicated because it is complicated. But experience helps.

<old man story> Here’s a life lesson that was valuable to me. [Feel free to skip ahead; I tend to ramble.] Early-ish in my working career I spent several years commuting to work on a bicycle. There was a significant hill involved, and the last little bit near the top was slightly steeper than the rest. The first few days, I couldn’t make it all. My thighs felt like they were on fire, and I had to walk the rest of the way up. After a few days to my surprise I did make it all the way. And the next day. And the next. And every day thereafter.

It took me over a year to understand this lesson: The trip up the hill never got easier. It was exactly as hard on my first day as it was on my last. What changed was me. I became able to do this hard thing that I couldn’t do before. </old man story>

I think learning to manipulate complex data with Jinja expressions is similar. It’s hard and it’ll never get easier, but with experience you’ll come to be able to achieve your objectives with Jinja pipelines because (a) you get more familiar with the tools available and (b) you’ll become more comfortable in thinking about your data in a Jinja way.

To your specific questions, I’m unaware of a comprehensive tutorial covering the basics up through advanced data manipulation. There are lots of “little” tutorial-style articles that cover various niche parts, and there are extensive reference documents (see ansible-doc -t filter --list), some of which do contain a few examples, but I find the possible spectrum of uses for a lot of them are not particularly obvious. For example, from reading the Jinja2 doc on the sum filter, I wouldn’t expect |sum() to be the go-to solution for joining multiple arrays. But it is.

That “great example” (thanks!) of mine that you referenced earlier was a response to one of lots “interesting data problems” that I’ve seen posted over the years. (That one actually pre-dates this Forum; it was part of a bulk import of content from Ansible oriented Google Groups as those groups were shut down.) Whenever I see something like that - if I have the time - I try to implement a solution, and if I think it’ll be helpful I’ll post it. But more to the point, those problems and solutions all land in my ~/ansible directory. That directory currently has 247 .yml files in it, representing probably 200 different problems. (Some of those were originally my problems.) I didn’t set out to make a library of Jinja tricks, but it’s not uncommon for me to grep through those looking for ways I’ve used various filters. I learned a lot by working through Other People’s Problems. Maybe when I retire I’ll compile a Practical Workbook for the Wannabe Ansible Jinja Ninja.

[scrolls up; blinks; scrolls back down] Wow! This post has a tiny code-to-verbage ratio. Let me tack on one practical trick that should go on somebody’s list. When developing code to process complex data, look for things that could be either a single thing or a list of those things, and make the code flexible so the user can specify either a singleton or a list, whichever is natural for the situation. For example:

pet: rover

or

pet:
  - rover
  - bowser

Don’t write separate code for the singleton case and the list case; make it work with either, thus:

{{ [pet] | flatten }}

Now it’s always a list, even if there’s only one element. (Or zero elements: {{ [pet |default([])] | flatten }}). If you can handle the list case, you get the others for free. Now it works the way tags: work. :coin:

Good luck on your journey!

2 Likes
3

Hello Todd [ @utoddl ],
thank you very much for sharing your experience. I can see that the road ahead on this journey is long and curvy, or better steepy :wink:.
It would be fantastic if you would share your experiences with us - in due course.
Thanks again and best regards
Stefan

4

Here are two informative links: