# Powering EC2 instances on/off **URL:** \ **Category:** Ansible Project\ **Tags:** aws\ **Created:** [April 21, 2014, 4:29pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058 "2014-04-21T16:29:03Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![C\_S](https://avatars.discourse-cdn.com/v4/letter/c/ea5d25/32.png) [@C\_S](https://forum.ansible.com/u/C_S)\ **Post date:** [April 21, 2014, 4:29pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/1 "2014-04-21T16:29:03Z")
Hi folks, We’re trying to implement a system where we can power environments on and off AWS when they’re not in use. However the ec2 inventory module excludes instances that are not in a running state. It seems like adding an option to the ec2 module to include stopped instances would work, but then I guess ansible would need a corresponding option to call the module with to include the stopped instances. Which seems a it hacky… Maybe ansible needs a notion of host state? Any thoughts? Thx! -cs ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [April 21, 2014, 5:10pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/2 "2014-04-21T17:10:38Z")
I use this module: [https://github.com/ansible/ansible/pull/6349](https://github.com/ansible/ansible/pull/6349) Full disclosure: Michael believes all inventory should be done via inventory scripts; I respectfully disagree. 🙂 I find ec2.py to be very slow (20 seconds to refresh the cache with a small number of instances, for example) and prefer querying inventory directly in the script itself for many use cases. Regards, -scott ---
**Author:** ![C\_S](https://avatars.discourse-cdn.com/v4/letter/c/ea5d25/32.png) [@C\_S](https://forum.ansible.com/u/C_S)\ **Post date:** [April 21, 2014, 5:39pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/3 "2014-04-21T17:39:20Z")
Thanks! That’s interesting, your module is the same as ec2\_facts just with filtering. And the ec2\_facts module says it may add filtering in the notes. I think I’d agree with Michael’s pov, but it looks like we’ve already gone down facts being outside the inventory module, so maybe a pull request against ec2\_facts with the filters would get accepted. Long run it does seem like hosts and modules need to have some idea of state … ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [April 21, 2014, 5:45pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/4 "2014-04-21T17:45:22Z")
Actually, it’s not the same as ec2\_facts other than it returns facts about an instance. ec2\_facts only works when run on an actual AWS instance (it calls the Amazon ec2 metadata servers) and it only retrieves the facts for that instance alone. ec2\_instance\_facts, on the other hand, can retrieve multiple instance facts at once from anywhere (I use it in a local action). It’s more like ec2.py run for specific instances from within a playbook. Regards, -scott ---
**Author:** ![C\_S](https://avatars.discourse-cdn.com/v4/letter/c/ea5d25/32.png) [@C\_S](https://forum.ansible.com/u/C_S)\ **Post date:** [April 21, 2014, 5:54pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/5 "2014-04-21T17:54:39Z")
Thanks for the clarification, right, the use case and implementation are a bit different. Seems like they could be combined however. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [April 23, 2014, 1:04pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/6 "2014-04-23T13:04:46Z")
"so maybe a pull request against ec2\_facts with the filters would get accepted. Long run it does seem like hosts and modules need to have some idea of state … " Anything applying to more than one host definitely shouldn’t be done by the facts module. ---
**Author:** ![ghexsel](https://avatars.discourse-cdn.com/v4/letter/g/dbc845/32.png) [@ghexsel](https://forum.ansible.com/u/ghexsel)\ **Post date:** [April 25, 2014, 6:31pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/7 "2014-04-25T18:31:59Z")
So, I’m curious, for the case where you want to start “stopped” EC2 instances, what’s the current recommended approach? I’ve kind of ignored this task for now, managing that by hand (it’s just our dev env, but it’s still a couple of dozen instances at least). I’m almost about to pull Scott’s branch in locally since it looks so much better than manual management. ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [April 25, 2014, 6:59pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/8 "2014-04-25T18:59:23Z")
Here’s an example in case you do use ec2\_instance\_facts. This example creates maintenance instances for updating AMIs. Notes: - This is part of a set of scripts that will create an entire load balanced application environment (including DNS, VPC, centralized logging, and RDS) in a bare AWS account in about 20-30 minutes. - app\_environment is dev, test, stage, or prod. The scripts will create the same setup in each environment with some differences such as RDS size, domain name, and so forth. - I use a naming convention for AWS resources of ‘---’, eg. foo-stage-ec2-logging or foo-prod-ami-web. # The base image is created from a standard Ubuntu LTS instance. Then, packages common to all # of the images (eg. security, ansible, boto, etc.) are installed and configured. # There’s a separate pull request (also rejected, hi Michael… 😉 for the ec2\_ami\_facts module. - name: Obtain list of existing AMIs local\_action: module: ec2\_ami\_facts description: “{{ ami\_image\_name }}” tags: environment: “{{ app\_environment }}” region: “{{ vpc\_region }}” aws\_access\_key: “{{ aws\_access\_key }}” aws\_secret\_key: “{{ aws\_secret\_key }}” register: ami\_facts ignore\_errors: yes # If a version of the AMI exists, record this. Otherwise use the base Ubuntu image. - set\_fact: environment\_base\_image\_id: “{{ ami\_facts.images[0].id }}” when: ami\_facts.images|count \> 0 - set\_fact: environment\_base\_image\_id: “{{ ami\_base\_image\_id }}” when: ami\_facts.images|count == 0 # See if the maintenance image for this image type for this environment is running. - name: Obtain list of existing instances local\_action: module: ec2\_instance\_facts name: "{{ ami\_maint\_instance\_name }}” # Everything but terminated states: - pending - running - shutting-down - stopped - stopping tags: environment: “{{ app\_environment }}” region: “{{ vpc\_region }}” aws\_access\_key: “{{ aws\_access\_key }}” aws\_secret\_key: “{{ aws\_secret\_key }}” register: instance\_facts ignore\_errors: yes - set\_fact: environment\_maint\_instance: “{{ instance\_facts.instances\_by\_name.get(ami\_maint\_instance\_name) }}” when: instance\_facts.instances|count \> 0 # If there is no such instance, create one. - name: Create an instance for managing the AMI creation local\_action: module: ec2 state: present image: “{{ environment\_base\_image\_id }}” instance\_type: t1.micro group: “{{ environment\_public\_ssh\_security\_group }}” instance\_tags: Name: “{{ ami\_maint\_instance\_name }}” environment: “{{ app\_environment }}” key\_name: “{{ environment\_public\_ssh\_key\_name }}” vpc\_subnet\_id: “{{ environment\_vpc\_public\_subnet\_az1\_id }}” assign\_public\_ip: yes wait: yes wait\_timeout: 600 region: “{{ vpc\_region }}” aws\_access\_key: “{{ aws\_access\_key }}” aws\_secret\_key: “{{ aws\_secret\_key }}” register: maint\_instance when: environment\_maint\_instance is not defined - set\_fact: environment\_maint\_instance: “{{ maint\_instance.instances[0] }}” when: maint\_instance is defined and maint\_instance.instances|count \> 0 - name: Ensure instance is running local\_action: module: ec2 state: running instance\_ids: “{{ environment\_maint\_instance.id }}” wait: yes wait\_timeout: 600 region: “{{ vpc\_region }}” aws\_access\_key: “{{ aws\_access\_key }}” aws\_secret\_key: “{{ aws\_secret\_key }}” register: maint\_instance when: environment\_maint\_instance is defined # If we had to start the instance then the public IP will not have been defined when # we gathered facts above, so get it again. - name: Obtain public IP of newly running instance local\_action: module: ec2\_instance\_facts name: “{{ ami\_maint\_instance\_name }}” states: - running tags: environment: “{{ app\_environment }}” region: “{{ vpc\_region }}” aws\_access\_key: “{{ aws\_access\_key }}” aws\_secret\_key: “{{ aws\_secret\_key }}” register: instance\_facts when: maint\_instance|changed - set\_fact: environment\_maint\_instance: “{{ instance\_facts.instances\_by\_name.get(ami\_maint\_instance\_name) }}” when: maint\_instance|changed # Pass the collected facts on the new maintenance image host for configuration by role. - name: Add new maintentance instance to host group local\_action: module: add\_host hostname: “{{ environment\_maint\_instance.public\_ip }}” groupname: maint\_instance app\_environment: “{{ app\_environment }}” # This passes the new/existing private key file to ansible for use in contacting the hosts. Better way to do this? ansible\_ssh\_private\_key\_file: “{{ environment\_public\_ssh\_private\_key\_file }}” environment\_maint\_instance: “{{ environment\_maint\_instance }}” - name: Wait for SSH on maintenance host local\_action: module: wait\_for host: “{{ environment\_maint\_instance.public\_ip }}” port: 22 # This is annoying as Hades. Sometimes the delay works, sometimes it’s not enough. # The check fails if the port is open but the ssh daemon isn’t yet ready to accept # actual traffic, right after the maintenance instance is started. #delay: 10 timeout: 320 state: started # TODO fix the hardcoded user too - name: Really wait for SSH on maintenance host local\_action: command ssh -o StrictHostKeyChecking=no -i {{ environment\_public\_ssh\_private\_key\_file }} ubuntu@{{ environment\_maint\_instance.public\_ip }} echo Rhubarb register: result until: result.rc == 0 retries: 20 delay: 10 Regards, -scott ---
**Author:** ![James\_Carroll](https://avatars.discourse-cdn.com/v4/letter/j/e36b37/32.png) [@James\_Carroll](https://forum.ansible.com/u/James_Carroll)\ **Post date:** [April 25, 2014, 7:08pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/9 "2014-04-25T19:08:07Z")
I’m fairly new to Ansible. How do I get your code into my Ansible install so I can use it? I run from source. Thanks! James ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [April 25, 2014, 7:10pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/10 "2014-04-25T19:10:07Z")
I keep all of my new/modified modules in a library directory under where my play books are. Ansible will find the libraries there and use them over the ones in the Ansible install. Regards, -scott ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [April 25, 2014, 8:57pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/11 "2014-04-25T20:57:54Z")
Using local ./library content is fine, but please don’t run a fork with extra packages added if you are going to ask questions about them – or at least identify that you are when you do. It can make Q&A very confusing when people ask about things that aren’t merged. ---
**Author:** ![ghexsel](https://avatars.discourse-cdn.com/v4/letter/g/dbc845/32.png) [@ghexsel](https://forum.ansible.com/u/ghexsel)\ **Post date:** [April 25, 2014, 11:17pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/12 "2014-04-25T23:17:16Z")
Just as a side-note, I was able to get the wait\_for mode to work for ssh with a bit of fiddling (so you don’t have to wait with 2 tasks): - hosts: 127.0.0.1 connection: local gather\_facts: false vars\_files: - env.yaml tasks: - name: Wait for SSH to come up after the reboot wait\_for: host={{item}} port=22 delay=60 timeout=90 state=started with\_items: groups.tag\_env\_{{pod}}\_ ignore\_errors: yes register: result until: result.failed is not defined retries: 5 This seems to work for me all the time, but maybe I just got lucky. I create groups based on tags. Groups are tag-based “class\_database”: “”, “class\_monitoring”: “”, “env\_qa1”: “”, which I register using add\_host. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [April 26, 2014, 12:36pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/13 "2014-04-26T12:36:00Z")
I’m always a bit wary when so many keywords come together. It’s usually the sign something can be simplified and is not “Ansible-like” enough. - name: Wait for SSH to come up after the reboot wait\_for: host={{item}} port=22 delay=60 timeout=90 state=started with\_items: groups.tag\_env\_{{pod}}\_ ignore\_errors: yes register: result until: result.failed is not defined retries: 5 Can likely be simplified to: - hosts: localhost tasks: - ec2: # provisioning step here with add\_host… - hosts: groups.tag\_env\_{{ pod }}\_ tasks: - name: Wait for SSH to come up after the reboot local\_action: wait\_for host={{item}} port=22 delay=60 timeout=90 A few key concepts: (A) Using the host loop is clearer than doing a “with\_items” across the group (B) You should only need to do one wait\_for. Consider increasing the timeout rather than looping over a retry (C) You should not need to register the result of the retry since there is no loop (D) You won’t need to ignore errors because we’re running wait\_for off localhost, which we know we can connect to. ---
**Author:** ![ghexsel](https://avatars.discourse-cdn.com/v4/letter/g/dbc845/32.png) [@ghexsel](https://forum.ansible.com/u/ghexsel)\ **Post date:** [April 26, 2014, 5:32pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/14 "2014-04-26T17:32:44Z")
Then I can consider this a bug report. Without retries, wait\_for fails for every EC2 AMI I tried (admitedly, they’re all variations of CentOS). Things I’ve seen: - it reports port open, then refuses to connect - it reports times out even though I was able to manually log in prior to the timeout - it fails with ssh errors while checking the port (this one is a bit rare) This combination is less than ideal, but it seemed to work for all my cases. Also, a minor thing, you have an ec2 task then you start using the groups.tag\_xxx, is it implied you have an add\_host there? Cause my ec2 instances won’t appear unless I add that. ---
**Author:** ![ghexsel](https://avatars.discourse-cdn.com/v4/letter/g/dbc845/32.png) [@ghexsel](https://forum.ansible.com/u/ghexsel)\ **Post date:** [April 26, 2014, 5:33pm UTC](https://forum.ansible.com/t/powering-ec2-instances-on-off/15058/15 "2014-04-26T17:33:44Z")
Nvm, saw the add\_host in the comment.