Add retry with exponential backoff when we receive throttling error code from cloudformation module

Archives Ansible Developer
1

I have just submitted PR https://github.com/ansible/ansible-modules-core/pull/224

Occasionally when polling for completion of the creation or deletion of a CloudFormation stack, AWS will return an exception indicating that a rate limit has been exceeded and throttling is occurring. It looks like this:

boto.exception.BotoServerError: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>
  <RequestId>2ab5db0d-5bca-11e4-9592-272cff50ba2d</RequestId>
</ErrorResponse>

When this happens, it causes our environment creation process to fail. It can be avoided by retrying the operation repeatedly with an exponential backoff up to a max number of retries.

This PR implements such a retry for all describe_stacks operations within the cloudformation module.

Retry is only invoked when the error code received indicates throttling is occurring, otherwise the exception is re-raised to maintain prior functionality.

It occurs to me that Cloudformation is not the only AWS module where throttling may occur. For now this enhancement is constrained only to the Cloudformation module.

Any feedback welcome!!

Best,

  • Mick
2

This appears to have been stuck with a merge conflict, so I’ve created a new PR that overlays the same changes atop the current state of the devel branch.

https://github.com/ansible/ansible-modules-core/pull/569