RFC: jq filter

Archives Ansible Developer
1

Abstract

I would like to contribute a new jq() filter to Ansible, exposing the jq expression language. This would complement the existing json_query() filter, based on jmespath expressions. I wanted to see if such a filter would be welcome, either as PR to Ansible itself, or a third party module through Ansible Galaxy

Background

jq is an expression language used for filtering and transforming JSON data. It is mostly commonly used through the jq command, to filter JSON in a pipeline.

Examples of jq command

$ echo ‘[{“x”:1, “y”:2}, {“x”:11}]’ | jq .[0].x
1

$ echo ‘[{“x”:1, “y”:2}, {“x”:11}]’ | jq ‘[. | .x]’
[
1,
11
]

Motivation
There are some operations expressible with jq that are not expressible with jmespath. The one I have encountered is producing a flat list of objects, from a list of objects each with a nested list of something. e.g.

$ cat databases.json
[
{“db”: “a”, “users”: [{“name”: “alice”}, {“name”: “alex”}]},
{“db”: “b”, “users”: [{“name”: “bob”}, {“name”: “brienne”}]}
]
$ cat databases.json | jq ‘. | map({db, name: .users.name})’
[
{
“db”: “a”,
“name”: “alice”
},
{
“db”: “a”,
“name”: “alex”
},
{
“db”: “b”,
“name”: “bob”
},
{
“db”: “b”,
“name”: “brienne”
}
]

Design
The filter would be invoked in a similar manner to the json_query() filter

Mockup:
$ ansible localhost -mdebug -a’{{ [{“x”:1, “y”:2}, {“x”:11}] | jq(‘[. | .x]’) }}’
localhost | SUCCESS => {
“msg”: [
1,
11
]
}

Initially jq features such as jq arguments/variables would be excluded.

Implementation
The filter would expose one of the existing Python bindings to libjq, the C library on which the jq command is built.

Alternatives
The json_query() filter (based on jmespath) covers much of the same functionality as jq. The few cases where jmespath cannot implement a particular operation alone could be overcome by supplementing with other filters, e.g.

{{ databases | subelements(‘users’)

to_json | from_json
json_query(‘[*].{db: [0].db, name: [1].name}’)
}}

would produce the same result as the proposed {{ databases | jq(‘. | map({db, name: .users.name})’) }}.

References
https://stedolan.github.io/jq/
https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#json-query-filter
https://stackoverflow.com/questions/54160360/jmespath-expression-to-flatten-array-of-objects-each-with-nested-arrays-of-obje

https://jqplay.org/s/5YHjzcspb5
https://pypi.org/project/jq/
https://pypi.org/project/pyjq/

2

My personal opinion would be no. We should not add another filter when json_query or native jinja2 filters can produce the same results.

3

I would say yes. I already implemented such a filter around pyjq for the same reason: there are transformations impossible to express in jmespath. And the pipes-and-filters syntax is cleaner.

Georges

4

Hi

Since there seems to be a new technical requirement in json_query (can
not be done with current ansible), I wonder what it would take extend
json_query to support jq.

(Another minor benefit gain in dependency management would be that there
is no need to install jmespath if jq is already there)

e.g. ... | json_query(query, use=jq)

"use" would be defaulted to "jmespath"

Regards
René

5

Georges, can you release the code for your filter. I would save reinventing the wheel.
With thanks, Alex

6

I confess I would strongly prefer a new filter, than adding a mode argument to json_query(). In my view adding an extra argument would significantly increase the number of test cases to cover, the complexity of the API, and the ease of discovery. This would be more true if/when the jq mode/filter grew support for jq variables, or other jq features that took additional arguments.

Do you have a preference for json_query(…, mode=jq)?

Thanks, Alex

7

Hello, Alex.

You’ll find enclosed my pyjq filter.

Good to know:

  • it lacks even a basic test suite…

  • TBH, until now, I only used the “jq_all” filter and never the “vars” argument.

  • a couple of releases ago, pyjq started to return OrderedDict instead of simple dict. My workaround, where necessary: pipe to a “to_json” filter.

  • for obvious security reasons, I did not implement the “url” argument. I also did not implement the “opener” argument nor the “compile” function.

Thanks George, I started with https://pypi.org/project/jq/, due to the OrderedDict usage. It’s at https://galaxy.ansible.com/moreati/jq_filter. I’ll add some testing and other niceties soon. Agreed that url and opener arguments are more trouble than they’re worth.