EC2 Idempotency: client-token vs ec2 tags

Archives Ansible Project
1

Based on discussions in the IRC channel today with hazmat and jarv, I’ve got some comments and suggestions about how the ec2 module currently handles idempotency.

By default, the ec2 module is not idempotent. If I write a playbook in which count=N ec2 instances are provisioned, then every time I run that playbook N new instances are provisioned.

However in version 1.2 it became possible to make this provisioning operation idempotent. This pull request (which I believe made it into 1.2) allows one to specify “id” parameter, which is used by boto to specify the EC2 client-token. In a way, this is a sensible way to add idempotency, because according to the AWS documentation, the client-token is indented to ensure idempotent provisioning of ec2 instances. However, that same documentation points out an annoying limitation:

The client token is valid for at least 24 hours after the termination of the instance. You should not reuse a client token for another call later on.

Thus, imagine I want to provision a cluster of ec2 instances using the client-token “webservers”. The first time I provision these (using id=“webservers”), all is well and the idempotency functions as expected. Now lets say I spin these machines down, and then two weeks later I want to spin them back up. The AWS documentation says I should not do this, because I should not recycle the client-id. I’m not sure what kind of problems this actually creates, but if the AWS documentations says not to do it, then ansible should probably avoid it.

An alternative approach would be to allow one to provision ec2 instances idempotently using ec2 tags. We could add an option called something like tag_idempotency which would be set to false by default to preserve backward compatability. If set to true, then before provisioning new instances, the ec2 module first checks to see how many instances with the specified tag(s) already exist, and only provisions extra instances if necessary. This could be implemented exactly as the current client-token based idempotency is implemented, except line 305 would change from:

filter_dict = {‘client-token’:id, ‘instance-state-name’ : ‘running’}

to something like

filter_dict = dict((“tag:” + tn, tv) for tn, tv in module.from_json(instance_tags).items())

(I haven’t tested this code yet, it’s just a sketch)

According to this thread, we should also note in the documentation that tag names should not contain underscores.

Any thoughts on this proposal?

2

Thus, imagine I want to provision a cluster of ec2 instances using the client-token “webservers”. The first time I provision these (using id=“webservers”), all is well and the idempotency functions as expected. Now lets say I spin these machines down, and then two weeks later I want to spin them back up. The AWS documentation says I should not do this, because I should not recycle the client-id. I’m not sure what kind of problems this actually creates, but if the AWS documentations says not to do it, then ansible should probably avoid it.

I might be misunderstanding you here but why would you be interested in idempotency if you had already terminated the instances? The idempotency is to stop you provisioning more if you make the same request whilst there are instances still running. Is this purely so that you don’t have to edit the playbook when building the environment from scratch again AND whilst you are continually running the provisioning task to ensure the instances are present? If this were the case you could probably automate the creation of the client token based on some condition(s) or external factors, or by just running the playbook and supplying an external var (ansible-playbook -e) when you want a rebuild.

An alternative approach would be to allow one to provision ec2 instances idempotently using ec2 tags. We could add an option called something like tag_idempotency which would be set to false by default to preserve backward compatability. If set to true, then before provisioning new instances, the ec2 module first checks to see how many instances with the specified tag(s) already exist, and only provisions extra instances if necessary.

It’s not so much about preserving backward compatibility, it should absolutely be enabled with some additional parameter such as your suggestion of tag_idempotency. The reason for this is due to the other attributes instances can have. If we structured the module purely to base idempotency off tags by default, it would be slightly misleading. What about parameters such as image, keypair, group etc. ? These might all change in a new action but the tags remain the same and thus the module would lead me to think I have “full” idempotency. If it were to replace client token I’d argue that for each request we should check tags and all other parameters.

Still, this would be good to implement and should be shared with other AWS service modules. I think this tag idempotency method should be offered as an alternative to client token, not replace it.

3

It's not so much about preserving backward compatibility, it should
absolutely be enabled with some additional parameter such as your
suggestion of tag_idempotency.

Ok, fair enough. I can see why the idempotent behavior shouldn't be the
default (even forgetting about backwards compatibility).

The reason for this is due to the other attributes instances can have. If
we structured the module purely to base idempotency off tags by default, it
would be slightly misleading. What about parameters such as image,
keypair, group etc. ? These might all change in a new action but the tags
remain the same and thus the module would lead me to think I have "full"
idempotency. If it were to replace client token I'd argue that for each
request we should check tags and all other parameters.

You make a good point. The basic functionality we need is first to select
instances based on some attribute, and then ensure that there are count=N
of them up. EC2 tags are just one possible attribute one might use.

Still, this would be good to implement and should be shared with other AWS

service modules. I think this tag idempotency method should be offered as
an *alternative* to client token, not replace it.

I've added this functionality in a way that should both be backwards
compatible, and allow one to choose which attribute should be used for
selecting the set of instances. This currently supports tags, security
group, security group id, and image id. I've only tested this code a little
using an example based on tags. You can find the pull request
here<https://github.com/ansible/ansible/pull/3249&gt;
.

4

Given the feedback from my pull request, I’ve given this feature request another go. You can see my changes here.

There are a couple of issues to discuss before I make another pull request

  • If the ec2 module is used idempotently, then upon exiting it should sometimes report that nothing was changed. I have tried to implement this in my version (see changes around line 425 and 455), but I’m not sure I did this correctly, because I’m not sure of the proper way that ansible modules exit and pass this information onwards.
  • If a user specifies that there should be count=10 ec2 instances with some attribute, and in fact there are more than ten, this module does not terminate the excess instances. That’s probably a good thing (terminating excess instances seems dangerous), but on the other hand it means that we don’t guarantee the state that there will be count=N instances with the specified attribute. We should probably note this in the documentation.
  • Based on the other options, I’ve added documentation and an example. Can someone make sure I’ve done this in the standard way?