How to use the JSON part of an HTTP response in the subsequent tasks?

Archives Ansible Project
1

Hi guys,

I am facing the problem of not being able to use the JSON part of the HTTP response in my subsequent tasks in an Ansible playbook. Here are the relevant tasks from the playbook:

  • name: Login to account as the specified user and obtain its id
    local_action: >
    uri url=‘http://{{ account_app }}.{{ target_domain}}/auth/user’
    body=‘{ “email”: “{{ userId }}”, “password”: “{{ sa_password }}” }’
    method=POST
    return_content=yes status_code=200
    HEADER_Content-Type=“application/json”
    register: usp_user
    tags:

  • config-manager

  • bootstrap

  • api

  • name: Archive previous top-level settings config
    local_action: >
    command
    curl -i -f -H “content-type: application/json”
    -H “{{ lookup(‘template’, ‘./user_header.json.j2’) }}”
    -XPUT http://{{ config_manager }}.{{ target_domain }}/archive/settings

uri url=‘http://{{ config_manager }}.{{ target_domain }}/archive/settings’

method=PUT

return_content=yes status_code=200

HEADER_Content-Type=“application/json”

HEADER_user=“{{ usp_user.json }}”

tags:

  • config-manager

  • bootstrap

  • api

  • name: Create new top-level settings config
    local_action: >
    command
    curl -i -f -H “content-type: application/json”
    -H “{{ lookup(‘template’, ‘./user_header.json.j2’) }}”
    -XPOST http://{{ config_manager }}.{{ target_domain }}/config/settings
    -d ‘{{ lookup(‘template’, ‘./stages_request.json.j2’) }}’

uri url=‘http://{{ config_manager }}.{{ target_domain }}/config/settings’

body=‘{{ lookup(‘template’, ‘./stages_request.json.j2’) }}’

method=POST

return_content=yes status_code=200

HEADER_Content-Type=“application/json”

HEADER_user=“{{ usp_user.json }}”

tags:

  • config-manager
  • bootstrap
  • api

The user_header.json.j2 is a simple header template to overcome a YAML parsing problem of having a ‘:’ followed by the curly braces

user: {{ usp_user.json }}

Basically what I am trying to do it to log in as a particular user to an account-managing app, and then use the JSON part of the response that contains required credentials (like the user id and the token) to do some requests to another app. It seems to work if I form the JSON header manually using the individual values from the JSON response, but not when I try to use the JSON as it is by referring to variable.json.
In the commented out code I tried to use the uri module which I’ve become a fan of lately and used curl command to test whether the problem is in the way uri supplies header in the request. Both curl and uri versions fail with the 400 on the “Archive previous top-level settings config” task (where no body is used in the request) and in the verbose output I see that the usp_user.json has ‘u’ prefixes before any of the key/value pairs, like follows:

"user: {u’customer’: u’52afd279fa33dd1f00000004’,…

I suspect this is at least part of the problem if not the problem on itself - please correct me if I am wrong and this is just the weird formatting that is only visible in the verbose output. I tried inserting the task debug var=usp_user.json and it seems to output well-formatted JSON though, without any of the ‘u’ attached.

I would appreciate a guidance on how to use the json part of the response in forming headers for succeeding requests.

With kind regards,
Roman

2

Just one more thing that I forgot to mention: I am able to use the JSON response as it is in subsequent requests using a bash script which is currently in place until this issue with the Ansible playbook is fixed:

Getting the user credentials in JSON format

user_info=“$(curl -f -v -XPOST $account.$target_domain/auth/user -d “$user_login” -H “$content_type_header” | awk ‘$1=$1’ RS= OFS=” " )"

Using them in the header for subsequent requests

curl -i -H “$user_header” -XPUT $config_manager.$target_domain/archive/settings -H “$content_type_header”
…

3

The easiest way to check the response code of a module is

  • debug: var=registered_variable_name

Start there.

The fact that you’re getting back unicode shouldn’t be a problem.

Possible point of confusion – What you print from “debug: var” is actually a Python datastructure, not JSON per se. The ‘json’ structure you get back is the datastructure itself that came from the JSON.

i.e. registered_variable_name.json.some_value_from_your_web_service should be directly accessible as a variable.

4

Hi Michael,

Thanks for your reply.
I think I have already mentioned in my post that I used the debug statements and the structures returned seemed ok to me.

I believe that there is a problem with the way uri module handles the HTTP headers. When I fall back to using curl in my playbook everything works correctly:

  • name: Login to account-ng as the specified user and obtain its id
    local_action: >
    shell curl -v -f -H “content-type: application/json” -d ‘{ “email”: “{{ userId }}”, “password”: “{{ sa_password }}” }’ -XPOST ‘http://{{ account_app }}.{{ target_domain}}/auth/user’ | awk ‘$1=$1’ RS= OFS=" "
    register: usp_user

  • name: Archive previous top-level settings config
    local_action: >
    command curl -i -f -H “content-type: application/json” -H ‘user: {{ usp_user.stdout }}’ -XPUT http://{{ config_manager }}.{{ target_domain }}/archive/settings

However if I try to replace curl in the second task with uri module I get 400 status. Here’s the second task with uri used in place of curl:

  • name: Archive previous top-level settings config
    local_action: >
    uri url=‘http://{{ config_manager }}.{{ target_domain }}/archive/settings’
    method=PUT
    return_content=yes status_code=200
    HEADER_Content-Type=“application/json”
    HEADER_user=“{{ usp_user.stdout }}”

Best regards,
Roman

5

And looks like JSON that is returned from uri module is also bad in some way. Here’s a playbook where I replaced the first call to curl to obtain user credentials with uri and then using usp_user.content in the header of my subsequent curl POST requests, and that works. If I use usp_user.json however, the second request fails with 400 again:

  • name: Login to account-ng as the specified user and obtain its id
    local_action: >
    uri url=‘http://{{ account_app }}.{{ target_domain}}/auth/user’
    body=‘{ “email”: “{{ userId }}”, “password”: “{{ sa_password }}” }’
    method=POST
    return_content=yes status_code=200
    HEADER_Content-Type=“application/json”
    register: usp_user

  • name: Archive previous top-level settings config
    local_action: >
    command curl -i -f -H “content-type: application/json” -H ‘user: {{ usp_user.content | replace(“\n”, “”) }}’ -XPUT http://{{ config_manager }}.{{ target_domain }}/archive/settings

Regards,
Roman

6

Just to provide a summary of my findings, in a nutshell:

Ansible module uri that I started to admire previously seems to be only usable in the cases where one:

  1. does not need to supply custom headers in requests
  2. does not use JSON part of response in its entirety in the subsequent requests (one can use individual key/values to form another JSON structure though)

which renders it to be of very limited use unfortunately.

I would be glad if someone proves that the above statements are wrong

With kind regards,
Roman

7

Here’s a bit late update on how I managed to resolve the issues with the uri module, hopefully that would be useful to someone else:

First of all, I was wrong about the custom headers supplied in the uri module - they do work, as I was able to confirm it both from debugging and looking at the Python code.
What does not work though is using a JSON structure in its entirety when trying to supply it as part of a custom header to uri. The problem is, as I had assumed before and was able to confirm by going into a deep-debugging mode of the uri module (I had to modify the sources of the httplib2 to write out headers of all requests to a file) is the fact that the JSON structure in the response from uri is in the form of the unicode string and in case it is used in the header of a subsequent request to a uri module, it is passed in the unicode form and triggers the 400 errors.

So taking slightly modified above examples, this will not work (400 error in the "Archive … " task):

  • name: Login to account as the specified user and obtain its id
    local_action: >
    uri url=‘http://{{ account_app }}.{{ target_domain}}/auth/user’
    body=‘{ “email”: “{{ userId }}”, “password”: “{{ sa_password }}” }’
    method=POST
    return_content=yes status_code=200
    HEADER_Content-Type=“application/json”
    register: usp_user

  • name: Archive previous top-level settings config
    local_action: >
    command
    uri url=‘http://{{ config_manager }}.{{ target_domain }}/archive/settings’

method=PUT
return_content=yes status_code=200
HEADER_Content-Type=“application/json”
HEADER_user=“{{ usp_user.json }}”

But this one will:

  • name: Archive previous top-level settings config
    local_action: >
    uri url=‘http://{{ config_manager }}.{{ target_domain }}/archive/settings’
    method=PUT
    return_content=yes status_code=200
    HEADER_Content-Type=“application/json”
    HEADER_user=‘{{ usp_user.content | replace(“\n”, “”) }}’

usp_user.content contains the same JSON structure as usp_user.json, only it has got the newlines that I need to remove using the replace Jinja2 filter.

Regards,
Roman