New Module: ec2_getall

Archives Ansible Developer
1

We want to use Ansible to create a set of volumes based on an instance’s latest snapshot. Doing this requires a few queries, and I didn’t see anything that provided the functionality I was looking for, so I wrote a new module, named ec2_getall, which I’ve attached here (zipped because Google Groups didn’t allow it directly).

ec2_getall provides a thin wrapper around a boto EC2Connection instance, allowing access to its get_all_* methods. It takes the standard ec2 arguments (e.g., keys and region) as well as a name argument that specifies which (“get_all_” + name) to call. Any other arguments are passed as keyword arguments to the get_all method.

Example usage:

  • hosts: localhost
    vars_files:
  • vars/aws-credentials.yml
    tasks:
  • name: Get all instance Name tags
    ec2_getall:
    name: tags
    region: us-east-1
    aws_access_key: “{{ aws_access_key }}”
    aws_secret_key: “{{ aws_secret_key }}”
    filters:
    key: Name
    resource_type: instance
    register: ec2
  • debug: var=ec2

This is effectively the same as creating an ec2 connection and then calling:

ec2.get_all_tags(filters=dict(key=“Name”, resource_type=“instance”))

The debug output:

“ec2”: {
“changed”: true,
“data”: [
{
“item”: "\n ",
“name”: “Name”,
“res_id”: “i-xxxxxxxx”,
“res_type”: “instance”,
“value”: “XXX”
},
{
“item”: "\n ",
“name”: “Name”,
“res_id”: “i-yyyyyyyy”,
“res_type”: “instance”,
“value”: “YYY”
},
… # remaining tag JSON omitted
],
“invocation”: {
“module_args”: “”,
“module_name”: “ec2_getall”
}
}

If there is general interest in this module, I will be happy to document it, address any concerns, and create a Pull Request.

Best,
Brandon

(attachments)

ec2_getall.zip (837 Bytes)

2

I kind of worry about what kind of programming language Ansible is when people use at as a programming language. That being said I want to understand a little bit more about your use case.

Many things we try to do are to make them idiomatic in the system instead, so extra call/register steps are not required.

For instance, ec2 inventory already creates groups for each tag, so if you want to access machines tagged with a certain thing, they are already in the ec2_tag_TagName_Value group.

(And in recent versions, all things tagged TagName are in ec2_tag_TagName as well).

While we might not be interested in including this in core, but you can bundle a module in a galaxy role by putting it in the “./library” directory of the role.

Can you help me understand a bit more about your use case and what you wish to do with this list of instances?

3

A common task for us is to create new EC2 volumes from a production instance’s snapshots. We can then attach those volumes to a development or staging environment for real world testing.

To do that, we have to:

  1. Fetch the instance tags.
  2. Find the instance id corresponding to the one with the correct Name (i.e…, Web)
  3. Fetch the list of snapshots for that instance.
  4. Find the most recent complete set of snapshots.
  5. Create a new volume for each of those snapshots, attaching them to an indicated instance.

Step 5, once we have the snapshot ids, is already possible with ec2_vol. My module enables the implementation of steps 1 and 3.

I can see how some of this isn’t necessary if you’ve been using Ansible from the beginning, or if there’s no need to work within the confines of existing processes. But I’m trying to push Ansible adoption at my company, and it has to happen incrementally. If we can’t replace existing (useful!) shell scripts with Ansible, it’s hard to make a good use case for it.

At any rate, I’m not super concerned with getting this module accepted if no one else wants it.

I am hoping, though, that the Ansible developers are amenable to my jinja test pull request (https://github.com/ansible/ansible/pull/8217), as that change cannot be packaged up into our configuration repo, and it is pivotal in implementing steps 2 and 4 above.

Thanks for checking out my work!

Cheers,
Brandon

4

The general point, though, is to be able to gather data about an existing EC2 setup. If there is another way for me to do this, I would be happy to use that instead.

5

Hi Brandon,
I have a similar module that I use to look up instances based on tags.
The reasons why inventory doesn't work in our case because I need
other information about the instance (like the instance id) as inputs
into another module.
It was something simple enough where Ansible seemed like the right
tool instead of writing a script using boto directly (certainly easier
to read and maintain this way) but maybe falls outside of the typical
use-case.

This takes tags as arguments and gives you all the information about
the instances that match:
https://github.com/edx/configuration/blob/master/playbooks/library/ec2_lookup

-John

6

Hi John,

Thanks for sharing! I’m glad to hear that I’m not the only one with this problem. It sounds very similar to the problem I was up against, needing an instance ID to pass to another module. This is especially useful when working with existing infrastructure.

If there’s an Ansible-recommended way to do this sort of thing, it sounds like it would benefit us both.

Cheers,

Brandon

7

I’m ok with you hosting a module on Galaxy I think.

I’m still not convinced it’s /quite/ something understandable enough to fit in a core use case, where people have to read over and consider where they might want it.

“The reasons why inventory doesn’t work in our case because I need
other information about the instance (like the instance id) as inputs
into another module.”

I think this is a case for the instance ID to be a variable on the instance.