# Further EC2 module refactorings **URL:** \ **Category:** Ansible Developer\ **Tags:** aws\ **Created:** [February 13, 2014, 11:00am UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561 "2014-02-13T11:00:59Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![willthames](https://avatars.discourse-cdn.com/v4/letter/w/8dc957/32.png) [@willthames](https://forum.ansible.com/u/willthames)\ **Post date:** [February 13, 2014, 11:00am UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/1 "2014-02-13T11:00:59Z")
I’ve added security\_token and profile\_name support (the latter only works with boto \> 2.24, but that is checked) to the module\_utils/ec2.py, along with an alternative mechanism to get\_ec2\_creds. I think the next step will be a refactor of ec2\_elb and ec2\_elb\_lb to use the common libraries. One thing I’ve tried with some success in my latest update to ec2.py is a common ‘connect\_to\_aws(module, region, params)’ which then just does module.connect\_to\_region(region, params). I’ve also added connect\_to\_aws to module\_utils/ec2.py Such a refactoring might allow all of the AWS modules that currently use connect\_to\_region to use the common code and then instantly get access to the new parameters - those that use their own classes like the elb modules may need more work. We’d then just need a way to maintain common documentation as alluded to in Matt Martz’s other thread. While Michael DeHaan mentions how file does it, would it be ok to say in the docs ‘for other common AWS parameters, see the EC2 module’ - I’m not sure it would be clear with settings would be common and which just for EC2. Anyway, if anyone can help with the testing and/or further refactorings, that would be great. See [https://github.com/ansible/ansible/pull/5989](https://github.com/ansible/ansible/pull/5989) for the pull request. ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [February 13, 2014, 7:10pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/2 "2014-02-13T19:10:17Z")
I have two related pull requests: 1. [https://github.com/ansible/ansible/pull/5917](https://github.com/ansible/ansible/pull/5917) - BUGFIX: fixes a bug introduced by the use of get\_ec2\_creds in the rds module, refactors all use of aws\__key vs. ec2_\_key 2. [https://github.com/ansible/ansible/pull/5928](https://github.com/ansible/ansible/pull/5928) - BUGFIX: Fixes recently introduced UnboundLocalError issue with the ec2 module I also have 7 new ec2 modules for which I have not yet issued pull requests. The modules are cleaned and ready to go but I haven’t had time to actually perform the PRs: 1. ec2\_ami\_description - allows changing AMI descriptions 2. ec2\_ami\_facts - retrieves facts about AMIs by name, description, tag (I lied; this one has a PR: [https://github.com/ansible/ansible/pull/5927](https://github.com/ansible/ansible/pull/5927)) 3. ec2\_instance\_facts - retrieves facts about multiple ec2 instances (not just the current one, as with ec2\_facts) 4. ec2\_vpc\_facts - retrieves facts about configured VPCs 5. ec2\_zone\_facts - retrieves the availability zones that are allowed for a particular account 6. rds\_param\_group - manages RDS parameter groups 7. rds\_subnet\_group - manages RDS subnet groups I will probably have more over the next few months as well. I am in the process of creating a fully Ansible-automated AWS environment and there are a number modules necessary for this task that do not yet exist. I’m also planning on adding NAT setup functionality to the ec2\_vpc module. -scott ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [February 13, 2014, 7:13pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/3 "2014-02-13T19:13:21Z")
“creating a fully Ansible-automated AWS environment” Can you give any additional information about this? ---
**Author:** ![Ed\_Epstein](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@Ed\_Epstein](https://forum.ansible.com/u/Ed_Epstein)\ **Post date:** [February 13, 2014, 8:05pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/4 "2014-02-13T20:05:26Z")
Yes I would love to know more also— I am embarking on a similar project but I’m still feeling out the implementation. Do you plan to open source the other modules? To what extent do you need any help? Also I assume that because you’re creating these modules, there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS. Do you plan to open source that glue? —Ed ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [February 13, 2014, 8:14pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/5 "2014-02-13T20:14:41Z")
“there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS” There is going to be an OSS integration test battery in [github.com/ansible/ansible](http://github.com/ansible/ansible) soon that leverages various modules. As we are retooling the way we do this, and in many cases fixing unit/integration alignment, this will evolve in stages. ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [February 13, 2014, 8:17pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/6 "2014-02-13T20:17:28Z")
My intent is pushbutton creation/management of our entire dev/test/prod AWS environment, one VPC for each, with a set of Ansible scripts using these modules. -scott ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [February 13, 2014, 8:22pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/7 "2014-02-13T20:22:57Z")
> Yes I would love to know more also— I am embarking on a similar project but I’m still feeling out the implementation. Do you plan to open source the other modules? Yes, I just haven’t had time for all of the pull requests, being at a startup myself. 🙂 > To what extent do you need any help? Testing my pull requests would help for sure. I think my main issue is going to be the slow uptake in Ansible master (not bagging on you Michael, I fully understand the limited bandwidth you have) that is going to lag way behind my development speed. > Also I assume that because you’re creating these modules, there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS. Do you plan to open source that glue? The glue will be Ansible scripts. I can certainly post them as examples, although certainly a good deal of it will be specific to my company’s operations. As an example, I have a script that will build an AMI from scratch, instantiating an ec2 maintenance instance if necessary to build the AMI (otherwise just locating and starting the existing maintenance instance), then republishing the AMI with a constant moniker so it can be referred to by scripts to set up load balancers and autoscaling groups. -scott ---
**Author:** ![Ed\_Epstein](https://avatars.discourse-cdn.com/v4/letter/e/58956e/32.png) [@Ed\_Epstein](https://forum.ansible.com/u/Ed_Epstein)\ **Post date:** [February 13, 2014, 9:10pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/8 "2014-02-13T21:10:03Z")
> > Yes I would love to know more also— I am embarking on a similar project but I’m still feeling out the implementation. Do you plan to open source the other modules? > > Yes, I just haven’t had time for all of the pull requests, being at a startup myself. 🙂 Over here we plan to open source a couple of tools in the next few days— a set of Ansible roles that we are using to configure some common services, a set of glue scripts used to generate Ansible playbooks with a focus on vagrant development boxes, and what I’ve been working on lately, which is a very early set of incredibly basic python code used mostly to abstract away all of these tools for users who don’t have time to dive deeply into yet another tool (something that is common in startups). There’s also some design documentation that outlines what we want to achieve (big dreams!). > > To what extent do you need any help? > > Testing my pull requests would help for sure. I think my main issue is going to be the slow uptake in Ansible master (not bagging on you Michael, I fully understand the limited bandwidth you have) that is going to lag way behind my development speed. I’ll certainly keep my eye out for your pull requests! > > Also I assume that because you’re creating these modules, there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS. Do you plan to open source that glue? > > The glue will be Ansible scripts. I can certainly post them as examples, although certainly a good deal of it will be specific to my company’s operations. Have you collected any of your plays into roles? > As an example, I have a script that will build an AMI from scratch, instantiating an ec2 maintenance instance if necessary to build the AMI (otherwise just locating and starting the existing maintenance instance), then republishing the AMI with a constant moniker so it can be referred to by scripts to set up load balancers and autoscaling groups. We are not there yet. One thing that you’ve made me think about is how much of the behaviour we want belongs in a module vs. standalone scripts. So thanks! —Ed > -scott > > > —Ed > > > > > “creating a fully Ansible-automated AWS environment” > > > > > > Can you give any additional information about this? > > > > > > > I have two related pull requests: > > > > > > > > 1. [https://github.com/ansible/ansible/pull/5917](https://github.com/ansible/ansible/pull/5917) - BUGFIX: fixes a bug introduced by the use of get\_ec2\_creds in the rds module, refactors all use of aws\__key vs. ec2_\_key > > > > 2. [https://github.com/ansible/ansible/pull/5928](https://github.com/ansible/ansible/pull/5928) - BUGFIX: Fixes recently introduced UnboundLocalError issue with the ec2 module > > > > > > > > I also have 7 new ec2 modules for which I have not yet issued pull requests. The modules are cleaned and ready to go but I haven’t had time to actually perform the PRs: > > > > > > > > 1. ec2\_ami\_description - allows changing AMI descriptions > > > > 2. ec2\_ami\_facts - retrieves facts about AMIs by name, description, tag (I lied; this one has a PR: [https://github.com/ansible/ansible/pull/5927](https://github.com/ansible/ansible/pull/5927)) > > > > 3. ec2\_instance\_facts - retrieves facts about multiple ec2 instances (not just the current one, as with ec2\_facts) > > > > 4. ec2\_vpc\_facts - retrieves facts about configured VPCs > > > > 5. ec2\_zone\_facts - retrieves the availability zones that are allowed for a particular account > > > > 6. rds\_param\_group - manages RDS parameter groups > > > > 7. rds\_subnet\_group - manages RDS subnet groups > > > > > > > > I will probably have more over the next few months as well. I am in the process of creating a fully Ansible-automated AWS environment and there are a number modules necessary for this task that do not yet exist. > > > > > > > > I’m also planning on adding NAT setup functionality to the ec2\_vpc module. > > > > > > > > -scott > > > > > > > > > I’ve added security\_token and profile\_name support (the latter only works with boto \> 2.24, but that is checked) to the module\_utils/ec2.py, along with an alternative mechanism to get\_ec2\_creds. > > > > > > > > > > I think the next step will be a refactor of ec2\_elb and ec2\_elb\_lb to use the common libraries. > > > > > > > > > > One thing I’ve tried with some success in my latest update to ec2.py is a common ‘connect\_to\_aws(module, region, params)’ which then just does module.connect\_to\_region(region, params). I’ve also added connect\_to\_aws to module\_utils/ec2.py > > > > > > > > > > Such a refactoring might allow all of the AWS modules that currently use connect\_to\_region to use the common code and then instantly get access to the new parameters - those that use their own classes like the elb modules may need more work. > > > > > > > > > > We’d then just need a way to maintain common documentation as alluded to in Matt Martz’s other thread. While Michael DeHaan mentions how file does it, would it be ok to say in the docs ‘for other common AWS parameters, see the EC2 module’ - I’m not sure it would be clear with settings would be common and which just for EC2. > > > > > > > > > > Anyway, if anyone can help with the testing and/or further refactorings, that would be great. > > > > > > > > > > See [https://github.com/ansible/ansible/pull/5989](https://github.com/ansible/ansible/pull/5989) for the pull request. > > > > > > > > – > > > > You received this message because you are subscribed to the Google Groups “Ansible Development” group. > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [ansible-devel+unsubscribe@googlegroups.com](mailto:ansible-devel%2Bunsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > – > > > You received this message because you are subscribed to the Google Groups “Ansible Development” group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [ansible-devel+unsubscribe@googlegroups.com](mailto:ansible-devel+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > Invoke Labs – A business building company that equips new digital ventures for long term success. > > [Twitter](http://www.twitter.com/InvokeLabs) | [Facebook](http://www.facebook.com/InvokeLabs) | [LinkedIn](http://www.linkedin.com/company/invoke-labs) | [Blog](http://www.invokemedia.com/blog/) | > > > > _Check out the latest products from [Invoke Labs](http://www.invokelabs.com/): [Pendo Rent](http://www.pendorent.com/) and [myBestHelper](http://mybesthelper.com/)._ > > > > – > > You received this message because you are subscribed to a topic in the Google Groups “Ansible Development” group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/ansible-devel/1A0qkspMfuc/unsubscribe](https://groups.google.com/d/topic/ansible-devel/1A0qkspMfuc/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [ansible-devel+unsubscribe@googlegroups.com](mailto:ansible-devel+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > – > You received this message because you are subscribed to the Google Groups “Ansible Development” group. > To unsubscribe from this group and stop receiving emails from it, send an email to [ansible-devel+unsubscribe@googlegroups.com](mailto:ansible-devel+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). Invoke Labs – A business building company that equips new digital ventures for long term success. [Twitter](http://www.twitter.com/InvokeLabs) | [Facebook](http://www.facebook.com/InvokeLabs) | [LinkedIn](http://www.linkedin.com/company/invoke-labs) | [Blog](http://www.invokemedia.com/blog/) | _Check out the latest products from [Invoke Labs](http://www.invokelabs.com/): [Pendo Rent](http://www.pendorent.com/) and [myBestHelper](http://mybesthelper.com/)._ ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [February 13, 2014, 11:10pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/9 "2014-02-13T23:10:03Z")
> Over here we plan to open source a couple of tools in the next few days— a set of Ansible roles that we are using to configure some common services, a set of glue scripts used to generate Ansible playbooks with a focus on vagrant development boxes, and what I’ve been working on lately, which is a very early set of incredibly basic python code used mostly to abstract away all of these tools for users who don’t have time to dive deeply into yet another tool (something that is common in startups). That sounds pretty interesting… we have a shared development environment and are starting to explore local environments. We’re entirely Mac, however, so it’s easy for us to do a lot of development on a local machine and have it be very similar to a cloud Linux environment for most purposes. > Have you collected any of your plays into roles? I’m using roles for the image configurations (common, web server, api server, etc — things that are done to actual instances). The rest is all local\_action AWS stuff like VPC configuration, and although I started out with roles, after a while it stopped making sense to do it that way. The includes were starting to get very hairy as well so I refactored the non-image stuff as normal tasks and considerably cut down on the size and complexity of the playbook code base. I may regret that at some point but not so far. > One thing that you’ve made me think about is how much of the behaviour we want belongs in a module vs. standalone scripts. So thanks! Yes, exactly. There’s a simple module I probably won’t be publishing (mainly because it’s trivial and specific to us) that decides how to backup an existing AMI. This was much easier to implement as a custom module instead of twisting a bunch of task defs to do the same. -scott ---
**Author:** ![Matt\_Davis1](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Matt\_Davis1](https://forum.ansible.com/u/Matt_Davis1)\ **Post date:** [October 27, 2014, 11:58pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/10 "2014-10-27T23:58:57Z")
Hey Scott, Any update on NAT instance support for ec2\_vpc? I’ve built something similar that’s working for our single-pass VPC setup playbooks, but I won’t bother polishing it up if you’re planning to PR yours anytime soon. -Matt ---
**Author:** ![erewh0n](https://avatars.discourse-cdn.com/v4/letter/e/ec9cab/32.png) [@erewh0n](https://forum.ansible.com/u/erewh0n)\ **Post date:** [October 28, 2014, 4:28pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/11 "2014-10-28T16:28:02Z")
FWIW I am a couple days away from finishing an idempotent version of ec2\_route\_tables (though due to boto API restrictions still will not be able to modify the main route table). We have an end-to-end AWS VPC deployment playbook and the only two gaps we could find were routing (solved by ec2\_route\_tables) and dynamic bastion hosts (e.g. we would like to specify a bastion host dynamically part way through the execution of a playbook, since we don’t know the bastion IP before it is provisioned). Anyway, my plan is to open a PR for ec2\_route\_tables by the weekend. ---
**Author:** ![Matt\_Davis1](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Matt\_Davis1](https://forum.ansible.com/u/Matt_Davis1)\ **Post date:** [October 28, 2014, 5:09pm UTC](https://forum.ansible.com/t/further-ec2-module-refactorings/14561/12 "2014-10-28T17:09:29Z")
It sounds like we’re running into exactly the same issues in both cases- I’ve proposed a solution for the dynamic bastion host thing (since I’m hacking around that right now too) on the Ansible Project group (it’s currently awaiting moderation, but it’ll be on the end of this thread: [https://groups.google.com/forum/#!topic/ansible-project/PdJmnVMhHn0](https://groups.google.com/forum/#!topic/ansible-project/PdJmnVMhHn0)) -Matt