I’ve added security_token and profile_name support (the latter only works with boto > 2.24, but that is checked) to the module_utils/ec2.py, along with an alternative mechanism to get_ec2_creds.
I think the next step will be a refactor of ec2_elb and ec2_elb_lb to use the common libraries.
One thing I’ve tried with some success in my latest update to ec2.py is a common ‘connect_to_aws(module, region, params)’ which then just does module.connect_to_region(region, params). I’ve also added connect_to_aws to module_utils/ec2.py
Such a refactoring might allow all of the AWS modules that currently use connect_to_region to use the common code and then instantly get access to the new parameters - those that use their own classes like the elb modules may need more work.
We’d then just need a way to maintain common documentation as alluded to in Matt Martz’s other thread. While Michael DeHaan mentions how file does it, would it be ok to say in the docs ‘for other common AWS parameters, see the EC2 module’ - I’m not sure it would be clear with settings would be common and which just for EC2.
Anyway, if anyone can help with the testing and/or further refactorings, that would be great.
I also have 7 new ec2 modules for which I have not yet issued pull requests. The modules are cleaned and ready to go but I haven’t had time to actually perform the PRs:
ec2_ami_description - allows changing AMI descriptions
ec2_instance_facts - retrieves facts about multiple ec2 instances (not just the current one, as with ec2_facts)
ec2_vpc_facts - retrieves facts about configured VPCs
ec2_zone_facts - retrieves the availability zones that are allowed for a particular account
rds_param_group - manages RDS parameter groups
rds_subnet_group - manages RDS subnet groups
I will probably have more over the next few months as well. I am in the process of creating a fully Ansible-automated AWS environment and there are a number modules necessary for this task that do not yet exist.
I’m also planning on adding NAT setup functionality to the ec2_vpc module.
Yes I would love to know more also— I am embarking on a similar project but I’m still feeling out the implementation. Do you plan to open source the other modules? To what extent do you need any help? Also I assume that because you’re creating these modules, there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS. Do you plan to open source that glue?
“there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS”
There is going to be an OSS integration test battery in github.com/ansible/ansible soon that leverages various modules. As we are retooling the way we do this, and in many cases fixing unit/integration alignment, this will evolve in stages.
My intent is pushbutton creation/management of our entire dev/test/prod AWS environment, one VPC for each, with a set of Ansible scripts using these modules.
Yes I would love to know more also— I am embarking on a similar project but I’m still feeling out the implementation. Do you plan to open source the other modules?
Yes, I just haven’t had time for all of the pull requests, being at a startup myself.
To what extent do you need any help?
Testing my pull requests would help for sure. I think my main issue is going to be the slow uptake in Ansible master (not bagging on you Michael, I fully understand the limited bandwidth you have) that is going to lag way behind my development speed.
Also I assume that because you’re creating these modules, there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS. Do you plan to open source that glue?
The glue will be Ansible scripts. I can certainly post them as examples, although certainly a good deal of it will be specific to my company’s operations.
As an example, I have a script that will build an AMI from scratch, instantiating an ec2 maintenance instance if necessary to build the AMI (otherwise just locating and starting the existing maintenance instance), then republishing the AMI with a constant moniker so it can be referred to by scripts to set up load balancers and autoscaling groups.
Yes I would love to know more also— I am embarking on a similar project but I’m still feeling out the implementation. Do you plan to open source the other modules?
Yes, I just haven’t had time for all of the pull requests, being at a startup myself.
Over here we plan to open source a couple of tools in the next few days— a set of Ansible roles that we are using to configure some common services, a set of glue scripts used to generate Ansible playbooks with a focus on vagrant development boxes, and what I’ve been working on lately, which is a very early set of incredibly basic python code used mostly to abstract away all of these tools for users who don’t have time to dive deeply into yet another tool (something that is common in startups). There’s also some design documentation that outlines what we want to achieve (big dreams!).
To what extent do you need any help?
Testing my pull requests would help for sure. I think my main issue is going to be the slow uptake in Ansible master (not bagging on you Michael, I fully understand the limited bandwidth you have) that is going to lag way behind my development speed.
I’ll certainly keep my eye out for your pull requests!
Also I assume that because you’re creating these modules, there is actually some “glue” code/playbooks that uses these individual modules to build whatever system you run in AWS. Do you plan to open source that glue?
The glue will be Ansible scripts. I can certainly post them as examples, although certainly a good deal of it will be specific to my company’s operations.
Have you collected any of your plays into roles?
As an example, I have a script that will build an AMI from scratch, instantiating an ec2 maintenance instance if necessary to build the AMI (otherwise just locating and starting the existing maintenance instance), then republishing the AMI with a constant moniker so it can be referred to by scripts to set up load balancers and autoscaling groups.
We are not there yet.
One thing that you’ve made me think about is how much of the behaviour we want belongs in a module vs. standalone scripts. So thanks!
—Ed
-scott
—Ed
“creating a fully Ansible-automated AWS environment”
Can you give any additional information about this?
I also have 7 new ec2 modules for which I have not yet issued pull requests. The modules are cleaned and ready to go but I haven’t had time to actually perform the PRs:
ec2_ami_description - allows changing AMI descriptions
ec2_instance_facts - retrieves facts about multiple ec2 instances (not just the current one, as with ec2_facts)
ec2_vpc_facts - retrieves facts about configured VPCs
ec2_zone_facts - retrieves the availability zones that are allowed for a particular account
rds_param_group - manages RDS parameter groups
rds_subnet_group - manages RDS subnet groups
I will probably have more over the next few months as well. I am in the process of creating a fully Ansible-automated AWS environment and there are a number modules necessary for this task that do not yet exist.
I’m also planning on adding NAT setup functionality to the ec2_vpc module.
-scott
I’ve added security_token and profile_name support (the latter only works with boto > 2.24, but that is checked) to the module_utils/ec2.py, along with an alternative mechanism to get_ec2_creds.
I think the next step will be a refactor of ec2_elb and ec2_elb_lb to use the common libraries.
One thing I’ve tried with some success in my latest update to ec2.py is a common ‘connect_to_aws(module, region, params)’ which then just does module.connect_to_region(region, params). I’ve also added connect_to_aws to module_utils/ec2.py
Such a refactoring might allow all of the AWS modules that currently use connect_to_region to use the common code and then instantly get access to the new parameters - those that use their own classes like the elb modules may need more work.
We’d then just need a way to maintain common documentation as alluded to in Matt Martz’s other thread. While Michael DeHaan mentions how file does it, would it be ok to say in the docs ‘for other common AWS parameters, see the EC2 module’ - I’m not sure it would be clear with settings would be common and which just for EC2.
Anyway, if anyone can help with the testing and/or further refactorings, that would be great.
Over here we plan to open source a couple of tools in the next few days— a set of Ansible roles that we are using to configure some common services, a set of glue scripts used to generate Ansible playbooks with a focus on vagrant development boxes, and what I’ve been working on lately, which is a very early set of incredibly basic python code used mostly to abstract away all of these tools for users who don’t have time to dive deeply into yet another tool (something that is common in startups).
That sounds pretty interesting… we have a shared development environment and are starting to explore local environments. We’re entirely Mac, however, so it’s easy for us to do a lot of development on a local machine and have it be very similar to a cloud Linux environment for most purposes.
Have you collected any of your plays into roles?
I’m using roles for the image configurations (common, web server, api server, etc — things that are done to actual instances). The rest is all local_action AWS stuff like VPC configuration, and although I started out with roles, after a while it stopped making sense to do it that way. The includes were starting to get very hairy as well so I refactored the non-image stuff as normal tasks and considerably cut down on the size and complexity of the playbook code base. I may regret that at some point but not so far.
One thing that you’ve made me think about is how much of the behaviour we want belongs in a module vs. standalone scripts. So thanks!
Yes, exactly. There’s a simple module I probably won’t be publishing (mainly because it’s trivial and specific to us) that decides how to backup an existing AMI. This was much easier to implement as a custom module instead of twisting a bunch of task defs to do the same.
Any update on NAT instance support for ec2_vpc? I’ve built something similar that’s working for our single-pass VPC setup playbooks, but I won’t bother polishing it up if you’re planning to PR yours anytime soon.
FWIW I am a couple days away from finishing an idempotent version of ec2_route_tables (though due to boto API restrictions still will not be able to modify the main route table).
We have an end-to-end AWS VPC deployment playbook and the only two gaps we could find were routing (solved by ec2_route_tables) and dynamic bastion hosts (e.g. we would like to specify a bastion host dynamically part way through the execution of a playbook, since we don’t know the bastion IP before it is provisioned).
Anyway, my plan is to open a PR for ec2_route_tables by the weekend.
It sounds like we’re running into exactly the same issues in both cases- I’ve proposed a solution for the dynamic bastion host thing (since I’m hacking around that right now too) on the Ansible Project group (it’s currently awaiting moderation, but it’ll be on the end of this thread: https://groups.google.com/forum/#!topic/ansible-project/PdJmnVMhHn0)
