passing nested data to custom modules

Archives Ansible Project
1

Greetings,

I'm working on a AWS cloudformation module that will allow you to
launch predefined cloudformation stacks using ansible.

CloudFormation templates can take a variable number of k/v parameters,
up to 30. I was hoping to solve this issue by doing something like
this in a vars_file:

stacks:
  - stack_name: cluster1
    parameters:
        disk_type: ephemeral
        instance_type: m1.small
        node_count: 3
        availability_zone: a
    region: us-east-1

and in a playbook:

  tasks:
  - cloudformation: >
      stackname=$item.stackname template_parameters=$item.parameters
region=$item.region
      state=$item.state template='$FILE(cfn_templates/basic.json)'
disable_rollback=True
    with_items: $stacks

Presently the module complains "msg: this module requires key=value
arguments" which I'm assuming is because ansible doesn't like sending
the "paramters" dictionary over the line.

Any help on the best way to achieve this would be appreciated.

- James

2

Ok, so yeah. I see the need for this in cases like yours.

I want to do this, and we mainly have to work out a syntax and such --
but also a way that keeps ansible working as it does now, doesn't
break anything, and is also easy to understand. Not hard though.

The module parsing code in AnsibleModuleCommon currently parses a
key=value command line, basically as fed directly from the task line
in Ansible. This is just a flat string now, and for existing modules
would remain so.

However, I have no problem with Ansible being smart and teaching
AnsibleModuleCommon to understand how to parse JSON inputs, with the
understanding that this will be sent only for AnsibleModuleCommon
Python-coded ansible modules that specify a specific flag that they do
allow structured data, because we don't want to break anyone else.
ACCEPTS_STRUCTURED_DATA=True or something is one easy way to do it.
It could even possibly detect that the input was JSON because it
started with a "{". (Then the module could just
deal with it, and then all of the new style modules could just take JSON)

When this is done, it would probably need to look like this to be
acceptable to me syntax wise for passing in a hash:

- task: whatever
   inputs: $data

Where data could be an inline YAML datastructure or a reference to a
Ansible variable.

aka also:

  - tasks: whatever
    inputs:
        - blarg
        - foo: x
          bar: $xyz

As you saw above, the one thing people are going to try though is
structured data that CONTAINS references to Ansible data though, so we
have to walk the hash and Ansible-template all the strings that
contain dollar signs, more or less. This would need to be a function
in utils.

For tasks that support structured data, they would work with the new
"inputs:" but the shorthand of the old key=value way would still work
to send them a basic flat hash.

Finally, the inputs keyword would need to work for older modules too,
in that if a hash was given that was only one level deep it just maps
to key=value entries.

So, that is the technical way forward, it's not even really that
involved (if just a little confusing), and would be accepted.

Let me know if that makes sense :).

3

Michael DeHaan wrote:

Ok, so yeah. I see the need for this in cases like yours.

I want to do this, and we mainly have to work out a syntax and such --
but also a way that keeps ansible working as it does now, doesn't
break anything, and is also easy to understand. Not hard though.

This is something I've been thinking about for a while. My backup system
needs this, and I've previously worked around the inability to do it by
using an action_plugin and running it on the machine.

The module parsing code in AnsibleModuleCommon currently parses a
key=value command line, basically as fed directly from the task line
in Ansible. This is just a flat string now, and for existing modules
would remain so.

However, I have no problem with Ansible being smart and teaching
AnsibleModuleCommon to understand how to parse JSON inputs, with the
understanding that this will be sent only for AnsibleModuleCommon
Python-coded ansible modules that specify a specific flag that they do
allow structured data, because we don't want to break anyone else.
ACCEPTS_STRUCTURED_DATA=True or something is one easy way to do it.
It could even possibly detect that the input was JSON because it
started with a "{". (Then the module could just
deal with it, and then all of the new style modules could just take JSON)

I don't see a reason to do this specifically for Python modules. We could
avoid parsing entirely, and for the module_common case simply put something
like "arguments = %r" % inputs, and for non-module_common provide an
arguments.json file (based on the presence of something like
ACCEPTS_STRUCTURED_DATA).

When this is done, it would probably need to look like this to be
acceptable to me syntax wise for passing in a hash:

- task: whatever
   inputs: $data

Task seems confusing here, as it's usually used to refer to the entirety of
that block. I think keeping action: would be fine, and merge it with
whatever might be specified on the action: line.

Where data could be an inline YAML datastructure or a reference to a
Ansible variable.

aka also:

  - tasks: whatever
    inputs:
        - blarg
        - foo: x
          bar: $xyz

As you saw above, the one thing people are going to try though is
structured data that CONTAINS references to Ansible data though, so we
have to walk the hash and Ansible-template all the strings that
contain dollar signs, more or less. This would need to be a function
in utils.

This is exactly what template_ds does and is for. :wink:

Daniel

4

Michael DeHaan wrote:

Ok, so yeah. I see the need for this in cases like yours.

I want to do this, and we mainly have to work out a syntax and such --
but also a way that keeps ansible working as it does now, doesn't
break anything, and is also easy to understand. Not hard though.

This is something I've been thinking about for a while. My backup system
needs this, and I've previously worked around the inability to do it by
using an action_plugin and running it on the machine.

The module parsing code in AnsibleModuleCommon currently parses a
key=value command line, basically as fed directly from the task line
in Ansible. This is just a flat string now, and for existing modules
would remain so.

However, I have no problem with Ansible being smart and teaching
AnsibleModuleCommon to understand how to parse JSON inputs, with the
understanding that this will be sent only for AnsibleModuleCommon
Python-coded ansible modules that specify a specific flag that they do
allow structured data, because we don't want to break anyone else.
ACCEPTS_STRUCTURED_DATA=True or something is one easy way to do it.
It could even possibly detect that the input was JSON because it
started with a "{". (Then the module could just
deal with it, and then all of the new style modules could just take JSON)

I don't see a reason to do this specifically for Python modules. We could
avoid parsing entirely, and for the module_common case simply put something
like "arguments = %r" % inputs, and for non-module_common provide an
arguments.json file (based on the presence of something like
ACCEPTS_STRUCTURED_DATA).

it could just use a seperate parsing function (or parameter), yes.

details not particularly material, as long as it works, and doesn't
break any existing modules not pulling in module common.

When this is done, it would probably need to look like this to be
acceptable to me syntax wise for passing in a hash:

- task: whatever
   inputs: $data

Task seems confusing here, as it's usually used to refer to the entirety of
that block. I think keeping action: would be fine, and merge it with
whatever might be specified on the action: line.

ENOCAFFEINE.

I mean action here.

Where data could be an inline YAML datastructure or a reference to a
Ansible variable.

aka also:

  - tasks: whatever
    inputs:
        - blarg
        - foo: x
          bar: $xyz

As you saw above, the one thing people are going to try though is
structured data that CONTAINS references to Ansible data though, so we
have to walk the hash and Ansible-template all the strings that
contain dollar signs, more or less. This would need to be a function
in utils.

This is exactly what template_ds does and is for. :wink:

Awesome! Thanks for the reminder.

There is a reason you are Vice President of templating functions
(among several other things!)