# File lookup plugin behavior concerning quotes @abadger, @bcoca **URL:** \ **Category:** Ansible Developer\ **Created:** [May 5, 2016, 6:14am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358 "2016-05-05T06:14:47Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Rob\_White](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@Rob\_White](https://forum.ansible.com/u/Rob_White)\ **Post date:** [May 5, 2016, 6:14am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/1 "2016-05-05T06:14:47Z")
Hi all, This is mostly aimed at the core team… Can you explain why the file lookup plugin changes the contents of the file as far as quotes are concerned? I.e. it changes all double quotes to single quotes. I am not sure if it used to perform this action but all i know is modules that rely on loading json through a lookup(‘file’,‘my.json’) in to a parameter have recently started to fail and digging about has led me here. Obviously the problem is that single quotes is not valid json. Can we not have the file lookup plugin return exactly what was passed in? Example: Passed in: { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “”, “Effect”: “Allow”, “Principal”: { “Service”: “[ec2.amazonaws.com](http://ec2.amazonaws.com)” }, “Action”: “sts:AssumeRole” } ] } Passed out: {‘Version’: ‘2012-10-17’, ‘Statement’: [{‘Action’: ‘sts:AssumeRole’, ‘Principal’: {‘Service’: ‘[ec2.amazonaws.com](http://ec2.amazonaws.com)’}, ‘Effect’: ‘Allow’, ‘Sid’: ‘’}]} ---
**Author:** ![Adam\_Hamsik](https://avatars.discourse-cdn.com/v4/letter/a/8baadc/32.png) [@Adam\_Hamsik](https://forum.ansible.com/u/Adam_Hamsik)\ **Post date:** [May 5, 2016, 6:34am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/2 "2016-05-05T06:34:23Z")
Hi, > Hi all, > > This is mostly aimed at the core team… > > Can you explain why the file lookup plugin changes the contents of the file as far as quotes are concerned? I.e. it changes all double quotes to single quotes. > > I am not sure if it used to perform this action but all i know is modules that rely on loading json through a lookup(‘file’,‘my.json’) in to a parameter have recently started to fail and digging about has led me here. > > Obviously the problem is that single quotes is not valid json. What happens if you pipe it to to\_json filter. ---
**Author:** ![Rob\_White](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@Rob\_White](https://forum.ansible.com/u/Rob_White)\ **Post date:** [May 5, 2016, 7:14am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/3 "2016-05-05T07:14:59Z")
It’s even worse… “{\n "Version": "2012-10-17",\n "Statement": [\n {\n "Sid": "",\n "Effect": "Allow",\n "Principal": {\n "Service": "ec2.amazonaws.com"\n },\n "Action": "sts:AssumeRole"\n }\n]\n}” ☹ ---
**Author:** ![Rob\_White](https://avatars.discourse-cdn.com/v4/letter/r/71c47a/32.png) [@Rob\_White](https://forum.ansible.com/u/Rob_White)\ **Post date:** [May 5, 2016, 7:36am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/4 "2016-05-05T07:36:20Z")
Here’s the bugs i know of because of this… [https://github.com/ansible/ansible-modules-core/issues/3404](https://github.com/ansible/ansible-modules-core/issues/3404) [https://github.com/ansible/ansible-modules-extras/issues/1813](https://github.com/ansible/ansible-modules-extras/issues/1813) ---
**Author:** ![Brian\_Coca](https://avatars.discourse-cdn.com/v4/letter/b/6de8d8/32.png) [@Brian\_Coca](https://forum.ansible.com/u/Brian_Coca)\ **Post date:** [May 5, 2016, 2:31pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/5 "2016-05-05T14:31:09Z")
This is not a simple problem, aside from an issue with Ansible internals, modules are inconsistent in what they expect when declaring these fields, some expect an actual JSON string, others expect a data structure to be converted (the better ones detect what they got and adjust). The lookup itself does no transformations, the issue is with Jinja2 templating, it returns a string by default, so Ansible tries to detect ‘type’, this conflicts with JSON as any string starting with “[” or “{” is considered a list or dictionary respectively. This is done in an effort of preserving existing types in declared variables and we cannot effectively just ‘turn off’ without creating many other issues. To avoid this ‘typing’ there are a few of things that can be done: - in all versions of Ansible, put in a preceding space " {{lookup… ", this will skip type detection. - in the latest versions a `|to_json` and certain other filters at the end will also bypass the automatic type casting. We are also adding a new feature for modules in 2.2 a type=‘json’ which will accept both JSON strings and/or python data structures which will automatically be transformed to JSON. This will effectively be the same as ‘the better modules’ I mention above and will be ‘transparent’ to users, eventually eliminating the problem and confusion. ---
**Author:** ![constantin.bugneac](https://avatars.discourse-cdn.com/v4/letter/c/b3f665/32.png) [@constantin.bugneac](https://forum.ansible.com/u/constantin.bugneac)\ **Post date:** [May 6, 2016, 9:26am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/6 "2016-05-06T09:26:46Z")
Hi, Tried both with the latest version of devel.: a space in " {{ …}}" and `|to_json` filter but they both return: TASK [Create S3 bucket] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* fatal: [127.0.0.1]: FAILED! =\> {“changed”: false, “failed”: true, “msg”: “Policies must be valid JSON and the first byte must be ‘{’”} ---
**Author:** ![Brian\_Coca](https://avatars.discourse-cdn.com/v4/letter/b/6de8d8/32.png) [@Brian\_Coca](https://forum.ansible.com/u/Brian_Coca)\ **Post date:** [May 6, 2016, 1:56pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/7 "2016-05-06T13:56:07Z")
_sigh_ that is new one … the module is doing a check it shouldn’t, we need to fix that in the module itself. ---
**Author:** ![j.r.hawkesworth](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@j.r.hawkesworth](https://forum.ansible.com/u/j.r.hawkesworth)\ **Post date:** [May 6, 2016, 6:10pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/8 "2016-05-06T18:10:55Z")
Interesting, I’m trying to do something similar with ansible 2.0.0.2 and looking up the json from a file and then posting it to (a solr instance) with uri. The message I’m getting back from solr implies there is something wrong with the json I’m sending it. Can anybody point me at a working example of this? many thanks Jon ---
**Author:** ![Frank\_Van\_Tol](https://avatars.discourse-cdn.com/v4/letter/f/3d9bf3/32.png) [@Frank\_Van\_Tol](https://forum.ansible.com/u/Frank_Van_Tol)\ **Post date:** [May 9, 2016, 12:21pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/9 "2016-05-09T12:21:17Z")
No, the check is correct, the policy string must start with a { One can easily confirm this in the AWS Console. ---
**Author:** ![Toshio\_Kuratomi](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@Toshio\_Kuratomi](https://forum.ansible.com/u/Toshio_Kuratomi)\ **Post date:** [May 9, 2016, 3:29pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/10 "2016-05-09T15:29:26Z")
uri should work and I can help diagnose that. What are you sending to it and what’s the error message? -Toshio ---
**Author:** ![Toshio\_Kuratomi](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@Toshio\_Kuratomi](https://forum.ansible.com/u/Toshio_Kuratomi)\ **Post date:** [May 9, 2016, 3:34pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/11 "2016-05-09T15:34:16Z")
ansible module should be able to take the user input and make it conform to what AWS needs. So, for instance, since one of the workarounds for the mixture of yaml and jinja2 templating doing the wrong thing is to prepend a space to your json string, the ansible module should probably should be stripping leading and trailing whitespace. (that looks fine under the JSON specification to me). -Toshio ---
**Author:** ![Toshio\_Kuratomi](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@Toshio\_Kuratomi](https://forum.ansible.com/u/Toshio_Kuratomi)\ **Post date:** [May 9, 2016, 3:41pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/12 "2016-05-09T15:41:39Z")
>   - in all versions of Ansible, put in a preceding space " {{lookup... ", > this will skip type detection. >   - in the latest versions a `|to_json` and certain other filters at the > end will also bypass the automatic type casting. > > Note on to\_json -- I think that this is a bit confusing as what you'd want to do is pass a dictionary or list to to\_json. But when combining (at least in a single variable) with lookup , what you end up is passing a string to to\_json. That is problematic because you then end up escaping that to make it a single json string. We are also adding a new feature for modules in 2.2 a type='json' which > will accept both JSON strings and/or python data structures which will > automatically be transformed to JSON. This will effectively be the same as > 'the better modules' I mention above and will be 'transparent' to users, > eventually eliminating the problem and confusion. > > I called the new type jsonarg. Making use of it will require modules which have a parameter which is supposed to be a json string to change the parameter's type from "str" to "jsonarg". It is better than the current hodgepodge where each module attempts to deal with this entirely on its own, sometimes leaving out certain cornercases as we're seeing here. -Toshio ---
**Author:** ![Toshio\_Kuratomi](https://avatars.discourse-cdn.com/v4/letter/t/41988e/32.png) [@Toshio\_Kuratomi](https://forum.ansible.com/u/Toshio_Kuratomi)\ **Post date:** [May 9, 2016, 6:45pm UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/13 "2016-05-09T18:45:23Z")
For people experiencing this, can you test out: [https://github.com/ansible/ansible-modules-extras/pull/2126](https://github.com/ansible/ansible-modules-extras/pull/2126) need to know if that’s sufficient or if there’s more code needed to fix this. Also need to know if there’s any regressions introduced by it. -Toshio ---
**Author:** ![j.r.hawkesworth](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@j.r.hawkesworth](https://forum.ansible.com/u/j.r.hawkesworth)\ **Post date:** [May 10, 2016, 11:42am UTC](https://forum.ansible.com/t/file-lookup-plugin-behavior-concerning-quotes-abadger-bcoca/21358/14 "2016-05-10T11:42:33Z")
Thank you. The json file I was sending contained the following (which I have tried on a single line and also on a single line without spaces) ` { “add-field”:{ “name”:“recordNumber”, “type”:“string”, “indexed”:true, “stored”:true, “docValues”:true, “omitNorms”:true, “omitTermFreqAndPositions”:true, “sortMissingLast”:true, “multiValued”:false }, “add-field”:{ “name”:“characterization”, “type”:“text\_general”, “indexed”:true, “stored”:true, “omitNorms”:true, “omitTermFreqAndPositions”:true, “sortMissingLast”:true, “multiValued”:false }, “add-field”:{ “name”:“primaryname”, “type”:“text\_general”, “indexed”:true, “stored”:true, “omitNorms”:true, “omitTermFreqAndPositions”:true, “sortMissingLast”:true, “multiValued”:false }, “add-field”:{ “name”:“originator”, “type”:“text\_general”, “indexed”:true, “stored”:true, “omitNorms”:true, “omitTermFreqAndPositions”:true, “sortMissingLast”:true, “multiValued”:false }, “add-field”:{ “name”:“createdOn”, “type”:“tdate”, “indexed”:true, “stored”:true, “docValues”:true, “omitNorms”:true, “omitTermFreqAndPositions”:true, “multiValued”:false }, “add-field”:{ “name”:“report”, “type”:“text\_en\_splitting”, “indexed”:true, “stored”:true, “omitNorms”:false, “omitTermFreqAndPositions”:false, “multiValued”:false } } ` This is the error I was getting ` ok: [malpdwfftsla001 → localhost] =\> {“changed”: false, “content”: “{\n "responseHeader":{\n "status":0,\n "QTime":0},\n "errors":[{"errorMessages":"Error parsing schema operations :The JSON must be an Object of the form {\"command\": {…},…"}]}\n”, “content\_length”: “185”, “content\_type”: “text/plain;charset=utf-8”, “redirected”: false, “status”: 200} ` I guess it would have been helpful if I had used wireshark or similar to capture what URI was actually POSTing, didn’t think to do that at the time. I tried both with and without the leading space when fetching the json at various points. ` - name: fetch schema json set\_fact: schema: " {{ lookup(‘file’, ‘schema.json’)}}" ` I tried using vars\_files to load the contents but because there are multiple “add-field” keys, I only got the last defined “add-field” command. I played with converting to a list of objects and using with\_items on the uri action (below) but still got the above error for each iteration of the list. ` - name: add the schema uri: url: “http://{{ solr\_node }}:8983/solr/evaluation/schema” method: POST body: “{{ item }}” body\_format: json return\_content: yes with\_items: schema delegate\_to: localhost ` For now I have abandoned modifying solr via the rest api and am pushing the schema files I want to use out directly from ansible.