More configuration options for EC2 dynamic inventories

Archives Ansible Developer
1

All,

I've recently begun using the ec2.py dynamic inventory script and I've
thought of some relatively simple optional enhancements that would be
driven by the ec2.ini configuration file and wanted to see if there
was any interest in having them included.

I'd rather not maintain a company specific fork and I think these
features are generic enough to be useful to others. And since they
would be optional and only take effect if explicitly configured then
they won't impact folks that don't want them.

1) All of the built-in ec2_* facts are lowercase, but because some of
the keys are dynamic from user data in EC2 they can be a mish-mash of
cases. I propose a config setting that would allow them to be either
coerced into the same case or allow aliases (eg, have ec2_tag_Name and
ec2_tag_name point to identical lists).

2) Allow host aliases from other fields. I'm using EC2 inside of VPCs
and the private_ip_address makes sense as the destination_variable and
unique host name. But having a bunch of 10.0.0.* IPs flash across the
screen as tasks are executed isn't ideal since I don't memorize them.
It would be nice to be able to have something like "destination_alias"
(and vpc_destination_alias) that would let me pick some other field
like ec2_tag_Name or ec2_tag_ShortName, etc. This would also make it
easier to target a specific host from the command line with "-l app1"
instead of "-l 10.0.0.123" after having to look up the private IP from
AWS.

3) Group aliases. I'm using a VPC for each environment for production,
staging, etc but local development takes place on local virtual
machines (using vagrant). If my hosts criteria becomes
"ec2_tag_role_web" instead of just "web" then it makes it harder to
manage environments that aren't in EC2. So being able to have a way to
create an alias for a group would be nice. I'm not totally sure of the
right way to present this as configuration so that it's dynamic, but
if there's interest I can come up with a couple of ideas and see what
people think.

Anyways, thanks for reading this far.

2

I’ve thought of another feature that would be helpful. I have several similar, isolated environments running in their own VPC and some random machines running in EC2 classic. It would be helpful (and faster) if the dynamic inventor could be restricted to only machines in the subnet where ansible is running.

So, if I submitted some of these as pull requests would anyone else be interested?

3

All,

I've recently begun using the ec2.py dynamic inventory script and I've
thought of some relatively simple optional enhancements that would be
driven by the ec2.ini configuration file and wanted to see if there
was any interest in having them included.

I'd rather not maintain a company specific fork and I think these
features are generic enough to be useful to others. And since they
would be optional and only take effect if explicitly configured then
they won't impact folks that don't want them.

The inventory plugins are in fact examples so if there become too many
different customization options that are idiomatic, maintaining someone's
own copy is ok.

1) All of the built-in ec2_* facts are lowercase, but because some of
the keys are dynamic from user data in EC2 they can be a mish-mash of
cases. I propose a config setting that would allow them to be either
coerced into the same case or allow aliases (eg, have ec2_tag_Name and
ec2_tag_name point to identical lists).

These are pretty much coming from Boto - though we don't want to create
different groups as that would bloat the number of groups.
An option like downcase_facts=False # default False

would be reasonable

2) Allow host aliases from other fields. I'm using EC2 inside of VPCs
and the private_ip_address makes sense as the destination_variable and
unique host name. But having a bunch of 10.0.0.* IPs flash across the
screen as tasks are executed isn't ideal since I don't memorize them.
It would be nice to be able to have something like "destination_alias"
(and vpc_destination_alias) that would let me pick some other field
like ec2_tag_Name or ec2_tag_ShortName, etc. This would also make it
easier to target a specific host from the command line with "-l app1"
instead of "-l 10.0.0.123" after having to look up the private IP from
AWS.

I'm finding the sprawling of aliases to be somewhat confusing, but it
should also be the case that users should not get attached to EC2 hostnames
(treat machines like cattle, not pets -- don't give them names). I guess
my resistance here is this starts to make it very confusing as to what
someone should set up to get started.

I am perhaps persuadable on this topic but it would require more
documentation explaining in in a really simple easy to understand way with
suggestions.

3) Group aliases. I'm using a VPC for each environment for production,
staging, etc but local development takes place on local virtual
machines (using vagrant). If my hosts criteria becomes
"ec2_tag_role_web" instead of just "web" then it makes it harder to
manage environments that aren't in EC2. So being able to have a way to
create an alias for a group would be nice. I'm not totally sure of the
right way to present this as configuration so that it's dynamic, but
if there's interest I can come up with a couple of ideas and see what
people think.

In the above I think I'm reading that "ec2-tag_role_" starting the tag is
too much typing. I am not sure I believe this makes them harder to manage.

I believe it is important that we continue to indicate which things come
from tags rather than making things automagic, especially when it's not an
EC2 convention. As such, I'd suggest we not do this at this time.

I do understand what you say about your Vagrant groups not being named
ec2_tag_foo, but a way to solve this would be to make ec2_tag_foo a child
group of a group with a different name.

4

The inventory plugins are in fact examples so if there become too many
different customization options that are idiomatic, maintaining someone's
own copy is ok.

While I agree in principle, if I have some feature requests that seem
generic and reusable to others, I usually default to trying to get
those changes into the project itself and not my fork.

These are pretty much coming from Boto - though we don't want to create
different groups as that would bloat the number of groups.
An option like downcase_facts=False # default False

would be reasonable

Yeah, I definitely understand about group bloat since there are
already so many. I'll go with the downcase_facts route.

I'm finding the sprawling of aliases to be somewhat confusing, but it should
also be the case that users should not get attached to EC2 hostnames (treat
machines like cattle, not pets -- don't give them names). I guess my
resistance here is this starts to make it very confusing as to what someone
should set up to get started.

I agree with the cattle/pets analogy, but sometimes it's just useful
to be able to see something like

  ok: [web25]
  failed: [web26]

vs just

  ok: [10.0.0.123]
  failed: [10.0.0.321]

And it shouldn't change what you need to get started. It just might
make it more like static inventories which allow aliases.

I am perhaps persuadable on this topic but it would require more
documentation explaining in in a really simple easy to understand way with
suggestions.

If I add this to my version I'll see how useful it is and see if I can
come up with more real life examples.

In the above I think I'm reading that "ec2-tag_role_" starting the tag is
too much typing. I am not sure I believe this makes them harder to manage.

More typing is just a minor nit.

I believe it is important that we continue to indicate which things come
from tags rather than making things automagic, especially when it's not an
EC2 convention. As such, I'd suggest we not do this at this time.

I do understand what you say about your Vagrant groups not being named
ec2_tag_foo, but a way to solve this would be to make ec2_tag_foo a child
group of a group with a different name.

I'm fine with this too if it could be automated in some way in the
dynamic inventory. When I switched away from a static inventory I had
to edit all kinds of playbooks, roles and templates to change "search"
to "tag_role_search". Not a big deal in itself, but it's not something
that will be easy to "downgrade" for my vagrant setups.

I thought about using multiple group names when selecting hosts in
playbooks (having "hosts: tag_role_web:web") but that only works for
playbook hosts. It doesn't work in templates or tasks where I need to
get specific host vars from certain groups.

Did you have a better way to solve this that I'm missing?

Thanks

5

“Did you have a better way to solve this that I’m missing?”

What I said about putting the tag group in the other group, possibly.

This may require the (not quite yet applied) patch from Serge that allows mixing inventory groups and hosts between dynamic and static sources in directory form, i…e

-i <inventory_dir>
ec2.py
group_info # ini

and in group_info you put the ec2_tag_foo_bar group in a group named foo

6

I've made a pull request for this:
https://github.com/ansible/ansible/pull/7867

7

Yeah, that sounds like a good solution for my problem. I'm guessing
it's this pull request?

https://github.com/ansible/ansible/pull/6379

8

I've created a pull request to take care of this part too. I added a
same_vpc_only boolean option, which if true will only return hosts
that are in the same VPC where the inventory script is running.

https://github.com/ansible/ansible/pull/7916