# How can I pass a json variable set by set\_fact to jq pipeline in ansible shell task?

**URL:** <https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147>\
**Category:** Ansible Project\
**Created:** [September 27, 2022, 12:09pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147 "2022-09-27T12:09:39Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![TIYA\_JOSE](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TIYA\_JOSE](https://forum.ansible.com/u/TIYA_JOSE)\
**Post date:** [September 27, 2022, 12:09pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/1 "2022-09-27T12:09:39Z")

</div>

Hello,  
I have a ansible playbook that has set a json variable using set\_fact module. I’m trying to use jq command in ansible playbook, to transform that json to a particular format, but got parse error: Invalid numeric literal at line , column

My Playbook:

- name: get data

set\_fact: data: “{{ value.stdout | from\_json }}”

- name: get key value pairs

shell: echo “{{ data }}” | jq ‘to\_entries | map((.key) + “=” + .value)|join(“,”)’

register: key\_value

Error:

TASK [utils : debug] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*

ok: [localhost] =\> {  
“data”: {

“key1”: “keyval1”,  
“key2”: “keyval2”  
}  
}

TASK [utils : get key value pairs] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* FAILED! =\> {“changed”: true, “cmd”: “echo "{‘key1’: ‘keyval1’, ‘key2’: ‘keyval2’}" | jq ‘to\_entries | map((.key) + "=" + .value)|join(",")’”, “stderr”: “parse error: Invalid numeric literal at line 1 , column 22”, “stderr\_lines”: [“parse error: Invalid numeric literal at line 1, column 22”], “stdout”: “”, “stdout\_lines”: }

PS: I had tried using \> YAML construct specified in [How to use jq in ansible shell tasks](https://stackoverflow.com/questions/64045520/how-to-use-jq-in-ansible-shell-tasks/64057402#64057402) but got same error

- name: get key value pairs

shell: \>

echo “{{ data }}”

> jq ‘to\_entries | map((.key) + “=” + .value)|join(“,”)’ register: key\_value

My data variable set using set\_fact module:

“data”: { “key1”: “keyval1”, “key2”: “keyval2” }

Working jq command and expected output: [jqPlay](https://jqplay.org/s/x6hnhPBQUmf)

How can I pass the fact variable set using set\_fact to jq pipeline in ansible and transform my json?

Thank you

---

<div class="post-metadata">

**Author:** ![Walter\_Rowe](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/walter_rowe/32/4506_2.png) [@Walter\_Rowe](https://forum.ansible.com/u/Walter_Rowe)\
**Post date:** [September 27, 2022, 12:38pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/2 "2022-09-27T12:38:29Z")

</div>

You can use ansible itself to transform data. You don’t need to use a shell to jq to achieve this.

[https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_filters.html](https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html)

Walter

---

<div class="post-metadata">

**Author:** ![TIYA\_JOSE](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TIYA\_JOSE](https://forum.ansible.com/u/TIYA_JOSE)\
**Post date:** [September 27, 2022, 12:50pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/3 "2022-09-27T12:50:42Z")

</div>

I want to transform the json obtained from previous ansible task to a key value pair separated by comma.

Json Input:

“data”: {

“key1”: “keyval1”,  
“key2”: “keyval2”,  
“key3”: “keyval3”  
}  
}

Expected output:  
“key1: keyval1,key2: keyval2,key3: keyval3”

Is this transformation of json possible using ansible filters?

---

<div class="post-metadata">

**Author:** ![Walter\_Rowe](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/walter_rowe/32/4506_2.png) [@Walter\_Rowe](https://forum.ansible.com/u/Walter_Rowe)\
**Post date:** [September 27, 2022, 1:14pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/4 "2022-09-27T13:14:33Z")

</div>

Here is my test playbook. You can use jinja2 templates on the left side of assignments. The dict2items creates list

---

<div class="post-metadata">

**Author:** ![Walter\_Rowe](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/walter_rowe/32/4506_2.png) [@Walter\_Rowe](https://forum.ansible.com/u/Walter_Rowe)\
**Post date:** [September 27, 2022, 1:20pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/5 "2022-09-27T13:20:21Z")

</div>

I will add … this is documented well on the Ansible filters page. You need to get comfortable reading their documentation. It is very good.

[https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_filters.html#transforming-dictionaries-into-lists](https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#transforming-dictionaries-into-lists)

Walter

---

<div class="post-metadata">

**Author:** ![TIYA\_JOSE](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TIYA\_JOSE](https://forum.ansible.com/u/TIYA_JOSE)\
**Post date:** [September 27, 2022, 1:38pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/6 "2022-09-27T13:38:13Z")

</div>

Thank you

---

<div class="post-metadata">

**Author:** ![utoddl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/utoddl/32/249_2.png) [@utoddl](https://forum.ansible.com/u/utoddl)\
**Post date:** [September 27, 2022, 4:13pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/7 "2022-09-27T16:13:00Z")

</div>

There’s good advice further up thread about doing data manipulations within Ansible using the tools it provides, and I heartily agree. Still, the original post raises some interesting questions, and it’s instructive to understand why the playbook snippet there doesn’t work.

First, we aren’t shown what `value.stdout` looks like, but since it made it through the `from_json` filter let’s assume it looks something like

```auto
{"app": "myApp", "env": "myEnv"}

```

In the “get key value pairs” step, `jq` is expecting json, but `data` isn’t a json string, so it needs to be filtered through `to_json`. And since that json is going to contain double-quotes, you’ll need to single-quote the whole thing for the shell to handle it properly. (And that’s assuming none of your json values contains single quotes!)

Here’s the closest I could come up with that does what the original post was attempting to do. _Not that is is a good idea_; this is just an exercise in fixing stuff as asked. There are other, better ways to do this.

```auto
---
- name: json to jq
  hosts: localhost
  gather_facts: no
  vars:
    value:
      stdout: '{"app": "myApp", "env": "myEnv"}'
  tasks:
#
    - name: get data
      set_fact:
        data: "{{ value.stdout | from_json }}"
#
    - name: get key value pairs
      shell: |
        echo '{{ data | to_json }}' | jq 'to_entries | map((.key) + "=" + .value)|join(",")'
      register: key_value

```

Cheers,

---

<div class="post-metadata">

**Author:** ![TIYA\_JOSE](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TIYA\_JOSE](https://forum.ansible.com/u/TIYA_JOSE)\
**Post date:** [September 27, 2022, 5:18pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/8 "2022-09-27T17:18:30Z")

</div>

Thanks Todd! That worked!

Error seems to be jq complaining about the JSON passed to it. As mentioned in the question the json printed in error message isn’t valid JSON. Somehow double quotes were replaced by single quote although debug prints valid JSON. Invalid JSON was passed to jq.

Replacing the single quote with double quote also solved the issue.

shell: echo “{{ data }}” | sed “s/'/"/g” | jq ‘to\_entries | map((.key) + “=” + .value)|join(“,”)’

Having said that _this is Not that is is a good idea_; this is just an exercise in fixing stuff as asked. Could you elaborate on other better ways you know and Why this is not a good idea?

---

<div class="post-metadata">

**Author:** ![utoddl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/utoddl/32/249_2.png) [@utoddl](https://forum.ansible.com/u/utoddl)\
**Post date:** [September 27, 2022, 6:56pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/9 "2022-09-27T18:56:28Z")

</div>

We’re feature-creeping here.

Your original ask was, “How can I pass the fact variable set using set\_fact to jq pipeline in ansible and transform my json?”  
The second best answer is `echo "{{ data | to_json }}" | jq …`.  
The first answer is, “No.”

Then it was, “I want to transform the json obtained from previous Ansible task to a key value pair separated by comma.”  
Farther up the thread, Walter gave some excellent suggestions that will require some work on your part to fully appreciate. The most valuable part of that, believe it or not, is the work, not the final result.

This ask is a little shakier than the first, because such a string has no intrinsic value. Presumably you want that format because you’ve got another part of your solution that requires such input. However, I’m suspicious of a process that would consume such a string, and expect that a broader understand of the problem would suggest another approach.

Now we get to, “Why this is not a good idea?” Frankly, this is the deepest question so far, and it gets right at the issue of how to think about data in Ansible. There’s nothing absolutely right or wrong about passing some json through `jq` in a shell script. When you’ve been solving problems that way, it’s natural when learning Ansible to use it as a wrapper around such techniques. But as you gain more experience with Ansible filters, Jinja, etc., you’ll find that sometimes a better solution to such a task is to not put yourself in a situation where you need such a task in the first place. Increased experience gives insight into “the Ansible way” of structuring and manipulating data, to solve data problems within Ansible itself, rather than merely using Ansible as a bridge to a string of external solutions.

This is why I wanted to follow-up to your original post – even though that isn’t how an experienced Ansible user would approach the problem: Because understanding why that playbook didn’t work is fundamental to stepping up to the next level of Ansible proficiency. It was the wrong solution with a broken implementation. Now you have the wrong solution with a working implementation. Whether that matters depends; if this is a one-off, it’s no big deal either way. But if it’s going to run thousands of times a day, it’s reaching out to a remote machine each time to do trivial - perhaps unnecessary - data manipulations that could be done much more efficiently on the Ansible controller itself. It’s easy to choose the less expensive solution when it’s pointed out, but only experience allows one to recognize such traps before falling into them. This is why exploring the techniques Walter pointed to is so valuable. Such exploration is a stepping stone to better solutions on the rest of your Ansible journey.

To step back down a few levels (and then I’ll stop preaching), if you need to pass some json to a tool like `jq`, the solutions are (1) don’t; do something else instead; (2) use “{{ data | to\_json }}”, which is fragile enough but you may get away with it; (3) this other trick (echo “{{ data }}” | sed …) which is even more fragile and pulls in another executable for questionable purposes.

Sorry if I got a little preachy there. Hope you can take it in the spirit intended.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\
**Post date:** [September 27, 2022, 8:39pm UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/10 "2022-09-27T20:39:13Z")

</div>

Or just skip the tasks and assign to variable direclty  
vars:  
&nbsp;&nbsp;myvar: "{{lookup('pipe'. data|to\_json + '|jq "to\_entrie ....

---

<div class="post-metadata">

**Author:** ![TIYA\_JOSE](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TIYA\_JOSE](https://forum.ansible.com/u/TIYA_JOSE)\
**Post date:** [September 28, 2022, 2:24am UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/11 "2022-09-28T02:24:48Z")

</div>

Thank you Todd and Walter for the wonderful explanation to my question. These awesome explanations excites me to gain more expertise in Ansible. I’m a newbie to Ansible and I’m at my 1st week of using Ansible. As a beginner, I was unaware of doing these kind of data transformation in Ansible controller itself. To be Frank, I didn’t even know that this could be done in Ansible itself. I thought it could only be achieved using some json processing tool. I’ve started learning the ansible filters part. As you mentioned, now I know how expensive this operation is going to be.

I’m trying to automate the management of some applications in kubernetes. I’m using Kubernetes collection for ansible and the actual problem statement is finding out certain kubernetes applications based on a match criteria.The initial input I receive from the previous task is json. This json to key-value pair data transformation is done to obtain the match criteria that has to be passed to the next task that has kubectl command to query applications.

---

<div class="post-metadata">

**Author:** ![Walter\_Rowe](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/walter_rowe/32/4506_2.png) [@Walter\_Rowe](https://forum.ansible.com/u/Walter_Rowe)\
**Post date:** [September 28, 2022, 11:10am UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/12 "2022-09-28T11:10:41Z")

</div>

Happy to help. One thing that I came to appreciate and that has served me well is that Ansible is written in Python. Many of the filters and features of data manipulation in Python are exposed in Ansible. When seeking a way to achieve something in Ansible it sometimes can be helpful to think about how it would be done in Python then look for a corresponding filter or method or module in Ansible.

Walter

---

<div class="post-metadata">

**Author:** ![TIYA\_JOSE](https://avatars.discourse-cdn.com/v4/letter/t/bc8723/32.png) [@TIYA\_JOSE](https://forum.ansible.com/u/TIYA_JOSE)\
**Post date:** [September 28, 2022, 11:38am UTC](https://forum.ansible.com/t/how-can-i-pass-a-json-variable-set-by-set-fact-to-jq-pipeline-in-ansible-shell-task/37147/13 "2022-09-28T11:38:13Z")

</div>

Woah, that’s interesting! Thank you