Trouble parsing JSON data returned from URI call - making use of user-data in EC2

Archives Ansible Project
1

I’ve been trying to use the user-data provided by EC2 servers to determine a servers hostname.

So currently new server come up with a hostname of “ec2-123-234-345-456” or similar, and I’d like in my ansible script to rename that server to something like “demonstration”.

To do this, I’ve gone into my EC2 console, and added the following as user-data to my server:

{“hostname”: “demonstration”}

Then, in my ansible playbook, I’ve tried the following:

  • name: Load user data

uri: url=http://169.254.169.254/latest/user-data return_content=yes
register: user_data

  • debug: var=user_data.content

  • name: Set hostname from user data
    command: hostname {{user_data.content.hostname}}
    when: user_data is defined

This produces the output (using the -v flag):

PLAY [ec2] ********************************************************************

GATHERING FACTS ***************************************************************
ok: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com]

TASK: [common | Load user data] ***********************************************
ok: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com] => {“accept_ranges”: “bytes”, “changed”: false, “connection”: “close”, “content”: “{"hostname":"demonstration"}”, “content_length”: “28”, “content_location”: “http://169.254.169.254/latest/user-data”, “content_type”: “application/octet-stream”, “date”: “Fri, 14 Mar 2014 00:30:30 GMT”, “etag”: “"1016948274"”, “item”: “”, “last_modified”: “Thu, 13 Mar 2014 22:02:44 GMT”, “redirected”: false, “server”: “EC2ws”, “status”: 200}

TASK: [common | debug var=user_data.content] **********************************
ok: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com] => {
“item”: “”,
“user_data.content”: {
“hostname”: “demonstration”
}
}

TASK: [common | Set hostname from user data] **********************************
fatal: [ec2-54-253-114-###.ap-southeast-2.compute.amazonaws.com] => One or more undefined variables: ‘unicode object’ has no attribute ‘hostname’

So, whilst it seems that debugging user_data returns a JSON-like return strucutre, and drilling down to user_data.content also returns a JSON-like return structure, the actual JSON content returned by the uri task is unable to be parsed using the Jinja2 dot notation, and instead appears to be an impenetrable object, at least as far as I can determine.

Is there a way of extracting this URI return content using core ansible modules such that I can use it’s JSON payload as ansible variables ?

Regards,
Ben

2

Hi Ben, If you want to access the JSON data it's available in the json
subkey. So instead of:

{{ user_data.content.hostname }}

I believe it would be

{{ user_data.json.hostname }}

or something similar.

Take a look at the {{ user_data.json }} or {{ user_data }} output with
the debug module to get a better idea.

Hope that helps,
Romeo

3

I tried out some more debug statements; the .json suffix doesn’t appear to be present ?

The user-data “json” appears to be coming back as a string…

TASK: [common | debug var=user_data] ******************************************
ok: [ec2-54-253-114-32.ap-southeast-2.compute.amazonaws.com] => {
“item”: “”,
“user_data”: {
“accept_ranges”: “bytes”,
“changed”: false,
“connection”: “close”,
“content”: “{"hostname":"qa"}”,
“content_length”: “17”,
“content_location”: “http://169.254.169.254/latest/user-data”,
“content_type”: “application/octet-stream”,
“date”: “Fri, 14 Mar 2014 02:26:12 GMT”,
“etag”: “"1016948274"”,
“invocation”: {
“module_args”: “url=http://169.254.169.254/latest/user-data return_content=yes”,
“module_name”: “uri”
},
“item”: “”,
“last_modified”: “Thu, 13 Mar 2014 22:02:44 GMT”,
“redirected”: false,
“server”: “EC2ws”,
“status”: 200
}
}

TASK: [common | debug var=user_data.json] *************************************
ok: [ec2-54-253-114-32.ap-southeast-2.compute.amazonaws.com] => {
“item”: “”,
“user_data.json”: “{{ user_data.json }}”
}

TASK: [common | debug var=user_data.content.json] *****************************
ok: [ec2-54-253-114-32.ap-southeast-2.compute.amazonaws.com] => {
“item”: “”,
“user_data.content.json”: “{{ user_data.content.json }}”
}

4

In your Debug output it looks like it came out that way because that was what got stored there.

5

Looks like all “user-data” in AWS is just a “string” at the end of the day.

So there is no way for Ansible to parse that string and pull out values, without a third-party module such as https://github.com/jpmens/ansible-ec2-userdata say ?

Ben

6

Doesn’t the ec2_facts module already query user-data and parse into usable data?

I was pretty sure it did.

7

It would be a very simple improvement to set_fact to allow a string to become a json-object, however the syntax of set_fact was made so that every parameter is a fact name.

However a set_fact_from_json module would be super-easy to implement, so that you can combine it with uri (or other registered variables).

Another option is to extend the uri module with an option like: content_as_json=yes

My prefered solution is to include this in the inventory step (outside of Ansible) so that it can be processed asynchronous (and cached) to Ansible.

8

It is actually possible to use set_fact with json data already, or to use info from json data for pretty much anything. You just need to use the from_json filter.

  • set_fact:
    hostname: “{{ (user_data.content|from_json).hostname }}”

or in the example given:

  • name: Set hostname from user data

command: “hostname {{ (user_data.content|from_json).hostname }}”
when: user_data is defined

Although 2 things to mention:

  1. As mentioned, I do think that the ec2_facts module already grabs user-data <https://github.com/ansible/ansible/blob/devel/library/cloud/ec2_facts#L61>

If I am reading the code correctly, would end up looking like:

{{ hostvars[inventory_hostname][‘ansible_ec2_user-data’][‘hostname’] }}

  1. Instead of issuing a hostname change via the command module, look at the hostname module <http://docs.ansible.com/hostname_module.html>
9

Right, didn't think of doing it like this. Quite powerful indeed.

Thanks :slight_smile:

10

Ok, so the ec2_facts module DOES pull down the EC2 information - thanks for that.

When keys are created, all hypens (-) are converted to underscores (_), so the key is ansible_ec2_user_data

The value of this key still comes down as “a string, which happens to be json”, so it doesn’t work to ask for {{ ansible_ec2_user_data.hostname }}

However, adding in the “from_json” filter to that does work, and got me to this final solution:

  • action: ec2_facts
  • hostname: name={{ (ansible_ec2_user_data|from_json).hostname }}

Thanks for all the tips - putting them together, and going through the source code, got me to this final solution !

Ben