# Behavior of ec2\_elb module **URL:** \ **Category:** Ansible Project\ **Created:** [February 13, 2014, 9:26pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565 "2014-02-13T21:26:12Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Bruce\_Pennypacker](https://avatars.discourse-cdn.com/v4/letter/b/0ea827/32.png) [@Bruce\_Pennypacker](https://forum.ansible.com/u/Bruce_Pennypacker)\ **Post date:** [February 13, 2014, 9:26pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/1 "2014-02-13T21:26:12Z")
I occasionally run into an issue with the ec2\_elb module that I think needs to be addressed. This module is used to add and remove Amazon EC2 instances to/from an Amazon load balancer. The API into Amazon is asynchronous, so when you use it to add an instance to a load balancer it issues a “insert” command then it goes into a loop where it sleeps for a second, checks the status, and continues to loop until the instance reaches an “In Service” state in the load balancer. There’s also a check to see if the instance enters an error state, and if it does then the module immediately returns an error. Here’s a bit of pseudocode to demonstrate the behavior: while True: get\_instance\_state if InService: return success; else if instance\_error: return error; sleep 1 So when an instance is added to the load balancer, the module waits until it reaches the InService state. If at any point along that way it enters an error state then it immediately fails. The problem I have is that it’s not terribly uncommon for an instance to enter a transient unhealthy state for a couple of seconds prior to being successfully put into service. I have on a number of occasions had my Ansible playbook fail because the ec2\_elb module throws an error and yet the EC2 instance is successfully put into service in the load balancer. If the module had simply waited a few more seconds to check on the health of the instance then my playbook would have run successfully. I would like to propose making a change to the ec2\_elb module to address these sorts of transient errors. There really should be a timeout associated with this while loop in the module. It should only fail if the instance is not put into service during that period of time, and any errors that occur within that time period should be ignored. To maintain the current state it shouldn’t be too difficult to add an optional timeout parameter that changes the behavior only if it is set. So if a timeout parameter is added then the above loop woudl look something like this: while not timeout\_exceeded: get\_instance\_state if InService: return success; else if instance\_error AND timeout\_exceeded: return error; sleep 1 return timeout Any comments/suggestions about this, especially from other folks using the ec2\_elb module? -Bruce ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [February 13, 2014, 10:07pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/2 "2014-02-13T22:07:06Z")
I’m ok with the above. ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [February 14, 2014, 5:35pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/3 "2014-02-14T17:35:27Z")
Are there other modules that behave in this fashion as well? If so a generic retry loop that takes a callable for checking state might be useful as a bit of refactoring. -scott ---
**Author:** ![Alexander\_Popov](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Alexander\_Popov](https://forum.ansible.com/u/Alexander_Popov)\ **Post date:** [February 20, 2014, 6:18pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/4 "2014-02-20T18:18:59Z")
I observed similar issue with ec2\_group and ec2\_vpc (while creating subnets, route tables, associations, etc.) ---
**Author:** ![C\_S](https://avatars.discourse-cdn.com/v4/letter/c/ea5d25/32.png) [@C\_S](https://forum.ansible.com/u/C_S)\ **Post date:** [March 10, 2014, 3:55am UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/5 "2014-03-10T03:55:29Z")
We’ve had issues with this module too (1.4.x)… The main issue being that it de-registeres the instance from the ELB and doesn’t store which ELB is removed it from anywhere. So if there is a failure before the re-register, you have to manually add it back to the ELB pool (and it breaks autoscale in the mean time). I wonder if the module should update an EC2 tag on the instance first, so it knows how to put it back on subsequent runs if it couldn’t re-register it in a previous run. Our workaround was to de-register and immediately re-register the instance with the ELB so that just the healthcheck gets reset while we update an instance’s app. Additionally we had add retries to the module, since we occasionally will get throttled by the AWS APIs. Here is the algorithm that AWS recommends for their APIs: [http://docs.aws.amazon.com/general/latest/gr/api-retries.html](http://docs.aws.amazon.com/general/latest/gr/api-retries.html) ---
**Author:** ![Scott\_Anderson](https://avatars.discourse-cdn.com/v4/letter/s/ed655f/32.png) [@Scott\_Anderson](https://forum.ansible.com/u/Scott_Anderson)\ **Post date:** [March 10, 2014, 11:26am UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/6 "2014-03-10T11:26:26Z")
We’re going to use a different means of updating our instances: 1. Update the application AMI offline 2. Change the autoscaler to use the new AMI 3. Gracefully down and kill the old AMI instances in the ELB 4. The ELB will start up new AMI instances as the old ones die 5. Profit! -scott ---
**Author:** ![C\_S](https://avatars.discourse-cdn.com/v4/letter/c/ea5d25/32.png) [@C\_S](https://forum.ansible.com/u/C_S)\ **Post date:** [March 15, 2014, 8:26am UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/7 "2014-03-15T08:26:30Z")
Cool. Do you have the whole process automated with Ansible? ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [March 15, 2014, 1:27pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/8 "2014-03-15T13:27:31Z")
This seems fine to me, +1 ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [March 15, 2014, 1:27pm UTC](https://forum.ansible.com/t/behavior-of-ec2-elb-module/14565/9 "2014-03-15T13:27:58Z")
Rarrgh, replies to old thread, please disregard.