# Bringing cohesion to ec2\_facts and EC2 inventory plugin **URL:** \ **Category:** Ansible Project\ **Created:** [September 13, 2014, 9:28am UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209 "2014-09-13T09:28:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ches\_Martin](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@Ches\_Martin](https://forum.ansible.com/u/Ches_Martin)\ **Post date:** [September 13, 2014, 9:28am UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/1 "2014-09-13T09:28:42Z")
Hi folks, I dug around the list to see if anyone has raised this point before and didn’t find it addressed directly, excuse me if I missed something. Michael [has stated before](https://groups.google.com/d/msg/ansible-project/ci30yycTbi8/YueCll_vP5oJ) that it would be nice if the ec2\_facts module provided tag data. I concur. And yes, [it’s been pointed out](https://groups.google.com/d/msg/ansible-project/UfDvxM-Rx00/kpKYGX2wkdgJ) that the EC2 inventory plugin provides tags as hostvars—along with other facts—which is very useful, most of the time more convenient than using the ec2\_tag module when you just want to fetch a value. I am reaching an itching point with the disconnect between these two sources of EC2 facts, though. They’re both officially supported but it feels obvious that they originated from different contributions. There is redundant data under different variable names. Different semantics for what might/can be in-memory. My team gets confused about what vars are coming from which source, and when they should use one versus the other. Searching the mailing list seems to suggest that it confuses others too. Usually we prefer info from the inventory plugin when possible (primarily because of the readily available tags, as aforementioned), but it falls apart for provisioning plays, for instance: 1. Someone writes a play that launches EC2 instances. 2. They add a follow-up play that creates DNS records for the instances, using ec2\_public\_dns\_name (a variable from the inventory plugin, which they’ve become accustomed to using). 3. It doesn’t work, because this host variable isn’t available in-memory yet after the instance launch play. 4. Upon a quick googling, they think an ec2\_facts play before the DNS play must be the answer based on what they quickly read. 5. It isn’t. ansible\_ec2\_public\_hostname is the answer, _and_ that ec2\_facts play. This sounds like a lot of incidental complexity to me. And there’s still the matter of tags. Getting tag data into memory for instances launched in the same run [proves to be pretty ugly/cumbersome](http://stackoverflow.com/q/24389053/455009), currently. Wouldn’t it be nice if all you needed was the ec2\_facts refresh play to load new tags? So what can we do about this? Ideally, I’d like to use the EC2 inventory plugin, and have ec2\_facts work to refresh values of the _same_ host var keys in-memory. Inherent in that is that I’d like to have consistent variable names throughout my code base. Of course some people may want to occasionally use ec2\_facts without using the inventory plugin (I guess?). That should be fine, but they could work together seamlessly. I’m willing to take a stab at working on this, but wanted to bring up whether it would be welcomed, what major concerns there may be, etc. Thanks for reading. -Ches ---
**Author:** ![Michael\_Peters](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@Michael\_Peters](https://forum.ansible.com/u/Michael_Peters)\ **Post date:** [September 13, 2014, 7:39pm UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/2 "2014-09-13T19:39:23Z")
This is something I've thought about too. The variable naming mismatch has caused me some headaches as well as the issue of trying to load all of the information that the EC2 dynamic inventory pulls for newly provisioned machines. Would be nice if I could add\_host and ec2\_facts without having to worry about what the vars are called. But renaming the variables in either one would cause problems for existing playbooks, so it would need to keep the old names around for existing uses. This could lead to some confusion as the hostvars will be polluted with duplicate entries and there's no real way to encourage people to use the "right" names for the vars. IMO the ec2.py script should be changed to match the ec2\_facts module as the latter is an official part of ansible and the former is often referred to as a "starting point" for your own inventory scripts. ---
**Author:** ![Ches\_Martin](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@Ches\_Martin](https://forum.ansible.com/u/Ches_Martin)\ **Post date:** [September 13, 2014, 9:06pm UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/3 "2014-09-13T21:06:49Z")
> But renaming the variables in either one would cause problems for > existing playbooks, so it would need to keep the old names around for > existing uses. This could lead to some confusion as the hostvars will > be polluted with duplicate entries and there’s no real way to > encourage people to use the “right” names for the vars. There should be a deprecation period where names to be phased out are still supported, indeed. On the bright side, maybe, is that I imagine most people vendor a copy of ec2.py, it’s awkward to try to symlink it or something from the Ansible distribution on different working environments and production control hosts. So in that case, upgrading the inventory plugin is a very conscious move. > IMO the ec2.py script should be changed to match the ec2\_facts module > as the latter is an official part of ansible and the former is often > referred to as a “starting point” for your own inventory scripts. That’s probably the way to go, given the above assertion about ec2.py being a conscious upgrade decision for most, and also that it doesn’t use the ansible\_ namespace convention expected of official stuff. -Ches ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 14, 2014, 1:35pm UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/4 "2014-09-14T13:35:12Z")
Random things: - I’ve heard from some that the EC2 metadata service doesn’t provide tag data. This seems to be true, which means we can’t really make it provide tag data. - I’m open to expanding this module to use additional APIs. - Of the two modules, ec2\_facts is rarely used, and dynamic inventory is very widely used - It seems best if ec2\_facts conform to inventory, rather than vice versa - We don’t want to break anyone, but this may be unavoidable. As such, I’d suggest a return\_mode=‘v1’, defaulting to something like ‘v2’, and making sure common code from module\_utils is used so that functions from the inventory script return the same data - I don’t understand some statements in the above, like I can’t parse " ansible\_ec2\_public\_hostname is the answer, _and_ that ec2\_facts play." what that means. It might be useful to call out specific examples other than just the tags being missing, such as which variable names you don’t see standardized. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 14, 2014, 1:36pm UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/5 "2014-09-14T13:36:03Z")
> IMO the ec2.py script should be changed to match the ec2\_facts module > as the latter is an official part of ansible and the former is often > referred to as a "starting point" for your own inventory scripts. No, it should not. ec2.py is much more widely used, and both are parts of Ansible. We don't ship inventory scripts in the distro - but we probably should. They are most definitely more widely utilized than the facts module. ---
**Author:** ![Ches\_Martin](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@Ches\_Martin](https://forum.ansible.com/u/Ches_Martin)\ **Post date:** [September 14, 2014, 4:40pm UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/6 "2014-09-14T16:40:15Z")
Thanks Michael, that’s helpful, especially about giving favor to the inventory plugin. I’ll try to drop in IRC when I can start hacking on it (probably a week from now), I’m sure I’ll have some questions about internals like what you’re saying about module\_utils, but a few remarks: > - It seems best if ec2\_facts conform to inventory, rather than vice versa > > It might be useful to call out specific examples other than just the tags being missing, such as which variable names you don’t see standardized. Well… basically all of them: ec2\_facts uses ansible\_ec2\_ as a prefix, the inventory plugin uses ec2\_. It’s not only prefix, the public hostname thing is an example, which I’ll get to in a sec. When I’m sitting at a working environment I’ll gist some censored output from ec2.py --list and ansible -m ec2\_facts for sake of reference in discussion. The inventory plugin has this comment FWIW: > These variables are pulled out of a boto.ec2.instance object. There is a lack of > consistency with variable spellings (camelCase and underscores) since this > just loops through all variables the object exposes. It is preferred to use the > ones with underscores when multiple exist. Setting aside the issue of which names are kept and backwards compat for now (I think it’s pretty clear that we’ll want to preserve it, for awhile at least), would you prefer to standardize on ansible\_ec2\_ as var prefix? > - We don’t want to break anyone, but this may be unavoidable. As such, I’d suggest a return\_mode=‘v1’, defaulting to something like ‘v2’, and making sure common code from module\_utils is used so that functions from the inventory script return the same data I assume the return\_mode refers to a new option for the facts module, that sounds reasonable. What do you think about the inventory plugin? I actually hadn’t realized they weren’t in the distro until you just pointed it out. I have seen, like, one blog post where someone shows an example that git clones ansible, uses the hacking setup, and symlinks ec2.py into their playbooks repo. I tend to think/hope few people do that in “real life”. I have vendored a copy of the script in our repo, so upgrading Ansible wouldn’t break me unexpectedly, but that’s just me. > - I don’t understand some statements in the above, like I can’t parse " ansible\_ec2\_public\_hostname is the answer, _and_ that ec2\_facts play." what that means. Sorry, my anecdote was a little contrived, because the use case of “launch new instances and then create DNS records with their public names” can be solved just by using a register on the launch play, the necessary info is available in the result and ec2\_facts isn’t needed. The point I was trying to make is that the inventory plugin calls it ec2\_public\_dns\_name and there’s no convenient way to get that specific var name associated with newly launched hosts in memory during the run (okay, you could do it manually with add\_host and the values from registered ec2\_result.instances, but I said “convenient”). Imagine a more complex case where you add\_host new instances to an already-existing group, and do more with the group later in the run. So you loop over the group and try to do hostvars[host].ec2\_public\_dns\_name. This will break on the newly launched hosts. What will work is to run ec2\_facts for the group, and then use hostvars[host].ansible\_ec2\_public\_hostname. I hope that makes sense now. My team favors the inventory plugin most of the time but cases like this trip them up. Particularly because we use tags a good deal, we’ve become conditioned to first looking to ec2.py --list to find EC2 vars to use, hence there’s a bunch of ec2\_public\_dns\_name and its ilk used throughout our playbooks. People are confused when they write a provisioning playbook and these fail. I’ve had to explain this discrepancy multiple times. -Ches ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 14, 2014, 4:57pm UTC](https://forum.ansible.com/t/bringing-cohesion-to-ec2-facts-and-ec2-inventory-plugin/16209/7 "2014-09-14T16:57:04Z")
> Thanks Michael, that's helpful, especially about giving favor to the > inventory plugin. I'll try to drop in IRC when I can start hacking on it > (probably a week from now), I'm sure I'll have some questions about > internals like what you're saying about module\_utils, but a few remarks: > > > - It seems best if ec2\_facts conform to inventory, rather than vice versa > > > > It might be useful to call out specific examples other than just the tags > > being missing, such as which variable names you don't see standardized. > > Well... basically all of them: ec2\_facts uses ansible\_ec2\_ as a prefix, > the inventory plugin uses ec2\_. It's not only prefix, the public hostname > thing is an example, which I'll get to in a sec. When I'm sitting at a > working environment I'll gist some censored output from ec2.py --list and ansible > -m ec2\_facts for sake of reference in discussion. The inventory plugin > has this comment FWIW: > > These variables are pulled out of a boto.ec2.instance object. There is a > > > lack of > > consistency with variable spellings (camelCase and underscores) since this > > just loops through all variables the object exposes. It is preferred to > > use the > > ones with underscores when multiple exist. There's a pull request to add an option to standardize the boto facts, that I believe we merged, though I agree it's a little all over the place. Unfortunate result of what upstream is being used - and having a lot of an existing userbase - and not wanting to force a change. I think we need to start considering it though. > Setting aside the issue of which names are kept and backwards compat for > now (I think it's pretty clear that we'll want to preserve it, for awhile > at least), would you prefer to standardize on ansible\_ec2\_ as var prefix? I'd prefer to standardize on what the inventory script does it, since that's more widely used, but to be safe, the easiest thing may be to return \*everything\*. Though I agree that's also ugly, I'm worried about the upgrade case a little. Maybe too much. Need to think about this some. > My team favors the inventory plugin most of the time but cases like this > trip them up. Particularly because we use tags a good deal, we've become > conditioned to first looking to ec2.py --list to find EC2 vars to use, > hence there's a bunch of ec2\_public\_dns\_name and its ilk used throughout > our playbooks. People are confused when they write a provisioning playbook > and these fail. I've had to explain this discrepancy multiple times. \*nod\*. That makes sense to me. Would this be solved short term by just adding "ec2\_public\_dns\_name" as a variable coming out of ec2\_facts and ignore some of the other variable issues for a while? (Serious question...)