# RFC: jq filter

**URL:** <https://forum.ansible.com/t/rfc-jq-filter/29577>\
**Category:** Ansible Developer\
**Tags:** galaxy-ng\
**Created:** [January 30, 2019, 10:23pm UTC](https://forum.ansible.com/t/rfc-jq-filter/29577 "2019-01-30T22:23:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![moreati](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/moreati/32/5303_2.png) [@moreati](https://forum.ansible.com/u/moreati)\
**Post date:** [January 30, 2019, 10:23pm UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/1 "2019-01-30T22:23:49Z")

</div>

**Abstract**

I would like to contribute a new jq() filter to Ansible, exposing the jq expression language. This would complement the existing json\_query() filter, based on jmespath expressions. I wanted to see if such a filter would be welcome, either as PR to Ansible itself, or a third party module through Ansible Galaxy

**Background**

jq is an expression language used for filtering and transforming JSON data. It is mostly commonly used through the jq command, to filter JSON in a pipeline.

Examples of jq command

$ echo ‘[{“x”:1, “y”:2}, {“x”:11}]’ | jq .[0].x  
1

$ echo ‘[{“x”:1, “y”:2}, {“x”:11}]’ | jq ‘[. | .x]’  
[  
1,  
11  
]

**Motivation**  
There are some operations expressible with jq that are not expressible with jmespath. The one I have encountered is producing a flat list of objects, from a list of objects each with a nested list of something. e.g.

$ cat databases.json  
[  
{“db”: “a”, “users”: [{“name”: “alice”}, {“name”: “alex”}]},  
{“db”: “b”, “users”: [{“name”: “bob”}, {“name”: “brienne”}]}  
]  
$ cat databases.json | jq ‘. | map({db, name: .users.name})’  
[  
{  
“db”: “a”,  
“name”: “alice”  
},  
{  
“db”: “a”,  
“name”: “alex”  
},  
{  
“db”: “b”,  
“name”: “bob”  
},  
{  
“db”: “b”,  
“name”: “brienne”  
}  
]

**Design**  
The filter would be invoked in a similar manner to the json\_query() filter

Mockup:  
$ ansible localhost -mdebug -a’{{ [{“x”:1, “y”:2}, {“x”:11}] | jq(‘[. | .x]’) }}’  
localhost | SUCCESS =\> {  
“msg”: [  
1,  
11  
]  
}

Initially jq features such as jq arguments/variables would be excluded.

**Implementation**  
The filter would expose one of the existing Python bindings to libjq, the C library on which the jq command is built.

**Alternatives**  
The json\_query() filter (based on jmespath) covers much of the same functionality as jq. The few cases where jmespath cannot implement a particular operation alone could be overcome by supplementing with other filters, e.g.

{{ databases | subelements(‘users’)

> to\_json | from\_json  
> json\_query(‘[\*].{db: [0].db, name: [1].name}’)  
> }}

would produce the same result as the proposed {{ databases | jq(‘. | map({db, name: .users.name})’) }}.

References  
[https://stedolan.github.io/jq/](https://stedolan.github.io/jq/)  
[https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_filters.html#json-query-filter](https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#json-query-filter)  
[https://stackoverflow.com/questions/54160360/jmespath-expression-to-flatten-array-of-objects-each-with-nested-arrays-of-obje](https://stackoverflow.com/questions/54160360/jmespath-expression-to-flatten-array-of-objects-each-with-nested-arrays-of-obje)

[https://jqplay.org/s/5YHjzcspb5](https://jqplay.org/s/5YHjzcspb5)  
[https://pypi.org/project/jq/](https://pypi.org/project/jq/)  
[https://pypi.org/project/pyjq/](https://pypi.org/project/pyjq/)

---

<div class="post-metadata">

**Author:** ![sivel](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sivel/32/1198_2.png) [@sivel](https://forum.ansible.com/u/sivel)\
**Post date:** [January 30, 2019, 10:32pm UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/2 "2019-01-30T22:32:15Z")

</div>

My personal opinion would be no. We should not add another filter when `json_query` or native jinja2 filters can produce the same results.

---

<div class="post-metadata">

**Author:** ![Georges\_Martin](https://avatars.discourse-cdn.com/v4/letter/g/35a633/32.png) [@Georges\_Martin](https://forum.ansible.com/u/Georges_Martin)\
**Post date:** [January 31, 2019, 7:54am UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/3 "2019-01-31T07:54:24Z")

</div>

I would say yes. I already implemented such a filter around pyjq for the same reason: there are transformations impossible to express in jmespath. And the pipes-and-filters syntax is cleaner.

Georges

---

<div class="post-metadata">

**Author:** ![Rene\_Moser](https://avatars.discourse-cdn.com/v4/letter/r/7bcc69/32.png) [@Rene\_Moser](https://forum.ansible.com/u/Rene_Moser)\
**Post date:** [January 31, 2019, 8:14am UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/4 "2019-01-31T08:14:46Z")

</div>

Hi

Since there seems to be a new technical requirement in json\_query (can  
not be done with current ansible), I wonder what it would take extend  
json\_query to support jq.

(Another minor benefit gain in dependency management would be that there  
is no need to install jmespath if jq is already there)

e.g. ... | json\_query(query, use=jq)

"use" would be defaulted to "jmespath"

Regards  
René

---

<div class="post-metadata">

**Author:** ![moreati](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/moreati/32/5303_2.png) [@moreati](https://forum.ansible.com/u/moreati)\
**Post date:** [February 5, 2019, 7:49pm UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/5 "2019-02-05T19:49:48Z")

</div>

Georges, can you release the code for your filter. I would save reinventing the wheel.  
With thanks, Alex

---

<div class="post-metadata">

**Author:** ![moreati](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/moreati/32/5303_2.png) [@moreati](https://forum.ansible.com/u/moreati)\
**Post date:** [February 5, 2019, 8:06pm UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/6 "2019-02-05T20:06:50Z")

</div>

I confess I would strongly prefer a new filter, than adding a mode argument to json\_query(). In my view adding an extra argument would significantly increase the number of test cases to cover, the complexity of the API, and the ease of discovery. This would be more true if/when the jq mode/filter grew support for jq variables, or other jq features that took additional arguments.

Do you have a preference for json\_query(…, mode=jq)?

Thanks, Alex

---

<div class="post-metadata">

**Author:** ![moreati](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/moreati/32/5303_2.png) [@moreati](https://forum.ansible.com/u/moreati)\
**Post date:** [February 11, 2019, 8:11pm UTC](https://forum.ansible.com/t/rfc-jq-filter/29577/7 "2019-02-11T20:11:39Z")

</div>

> Hello, Alex.
> 
> You’ll find enclosed my pyjq filter.
> 
> Good to know:
> 
> - it lacks even a basic test suite…
> 
> - TBH, until now, I only used the “jq\_all” filter and never the “vars” argument.
> 
> - a couple of releases ago, pyjq started to return OrderedDict instead of simple dict. My workaround, where necessary: pipe to a “to\_json” filter.
> 
> - for obvious security reasons, I did not implement the “url” argument. I also did not implement the “opener” argument nor the “compile” function.

Thanks George, I started with [https://pypi.org/project/jq/](https://pypi.org/project/jq/), due to the OrderedDict usage. It’s at [https://galaxy.ansible.com/moreati/jq\_filter](https://galaxy.ansible.com/moreati/jq_filter). I’ll add some testing and other niceties soon. Agreed that url and opener arguments are more trouble than they’re worth.