Specifically, an elasticsearch cluster, but doing this the right way would also apply to other kinds of clusters, like RabbitMQ or Redis.
I’m using https://github.com/LaneCommunityCollege/aspects_elasticsearch to manage my elasticsearch cluster. Currently, if I modify the configuration settings, Ansible would issue a restart to all the nodes in the cluster.
The obvious answer is to only run the playbook on one node at a time. But there are situations where that isn’t convenient. If I run Ansible like a puppet agent then setting the configuration setting in the group_vars file will apply it on all the nodes, thus restarting them all. Or if I have a large number of nodes. Or simply need to apply the change, but don’t have time to do it one or two nodes at a time.
The other solutions I’ve thought of involve custom scripts that would not be part of the Ansible playbook. And likely be application specific.
Is there a way to tell ansible to set a service restart task sometime in the future? Say, right now for node1, 5 minutes later for node2, 10 minutes later for node3, and so on.
Maybe the at module? How would it know to set it 10 minutes in the future instead of 5 for the third node?
So, yeah, is there a good Ansible specific method for this? Or do I need to look outside of Ansible?
“Is there a way to tell ansible to set a service restart task sometime in the future? Say, right now for node1, 5 minutes later for node2, 10 minutes later for node3, and so on.”
There is a module for “at” which could be used to do this.
at runs some commands at scheduled times in the future, without setting up a crontab.
You could also set a simple play like this, which would probably be cleaner:
- hosts: blarg
serial: 1
tasks:
- pause: seconds={{ n }}
- service: name=foo state=restarted
Which will walk across your infra 1 host at a time, with a lag of n seconds between, and restart everything. The serial: 1 could likely be increased beyond 1 depending on the application context.
Is there a way to do something like this as a handler? I think that's
the main problem here is that a configuration change should trigger a
restart handler, but right now there's no way to throttle/serialize
the execution of that handler. Unless I'm missing something...
A possible workaround would be to make a somewhat non-idiomatic version of ‘handlers’ by putting the tasks that restart the services in a separate play in the end of your playbook with serial: 1. This will cause these tasks to run one after the other. You would also have ‘when’ conditionals on these tasks checking whether any other tasks that should cause restart of services have actually made changes. Last, if you need to add a delay between the restarts, you can use the ‘wait_for’ directive on the tasks restarting the services.
On the template task, register a variable that says to restart the service, then at the end of the play I could use the at module to set a service restart for sometime in the future.
So, handlers seem like they are just special tasks that you can call with the “notify” option. If that’s the case, couldn’t I just make a handler that uses the at module instead of the service module? Actually, I think I’ve done that already in a differen role… Hmm…
What about adjusting how far into the future the at module should run restart the service? As in, for the first node done, it would be right away, for the next 5 minutes from now, for the next 10 minutes from now, and so on. Can I register a variable and then add to it?
You could also set a simple play like this, which would probably be cleaner:
- hosts: blarg
serial: 1
tasks:
- pause: seconds={{ n }}
- service: name=foo state=restarted
I did think of something like that, but then my playbook would end up waiting that long to be finished, and I’m not sure how to get the rolling restarts I want.
Hmm… Maybe waiting to end the playbook is actually a good thing. I’ll have to think about it.
The problem with at is that you have to manually figure out how long
something should take and hope it never takes over that and then try
to randomize how quickly how many servers restart.
Or you leave a lot of time between which makes it inefficient.
Just saying it would be nice if we could already leverage all of the
power of serial (including all of the niceties like
max_fail_percentage).
Heh, yeah, guessing how long it takes to restart is not ideal. It’s just what I was thinking of as a work around that properly tweaked would avoid clobbering the cluster.
The more I think about it, the more I’m not sure there’s a good way to do this in Ansible. For Elasticsearch, I’d want to issue a command like ‘service elasticsearch safe-restart’ that waits for the status of the cluster to be green before restarting. For RabbitMQ, I’d want something similar, though I haven’t worked with it enough to know exaction what.
I think I’ll head off to the elasticsearch group and Google to see how others have avoided clobbering their clusters when updating config. 
Thanks for all the responses so far.
Right now handlers are just about notifying things in the context of one host.
There’s not (yet) a concept of a global handler, nor really a “global: True” kind of variant on set_fact. If there were, it would be easy to set a global flag True if a particular host hit a state, and then use that state to trigger actions on another set, without having to rely on the result of just one particular host, which you can already do with hostvars.
Sorry if this doesn’t entirely make sense.
There are ways you could do it that are a little bit hackish, like creating a signal file with a handler that delegated to localhost, but mass delegation to localhost with 500 systems is usually going to make your control server unhappy.
I think it’s about time we do have a “set_global” that a handler could call, and you could use that in another play. Seems like a good idea now.
Have you thought about managing global cluster state with something that can keep track of node health? You could use Ansible to poll this before restarting individual nodes, waiting for all nodes to register as healthy, then acquiring a global lock before proceeding with the restart. With this approach, you can attempt restarts on all nodes in parallel, but be assured you will only be restarting one at a time (or as many locks as you configure). Each parallel Ansible run will just sit polling the agent until it gets a chance to acquire the lock.
I do not think using Ansible to track global cluster state will work out well, but Ansible can work really nicely in tandem with such a cluster monitor.
Consul is a nice tool for this: a distributed CP service discovery, health checking, and key-value store system. It is quite general-purpose and does not require you to modify your services in any way to make Consul aware of them. Just hook the service registration into the service’s systemd unit/init script, set up a health check that tells you when the node is well and truly a happy member of the cluster, and then poll Consul from anywhere to see what the overall cluster health is.
This doc discusses using Consul to build distributed locks:
http://www.consul.io/docs/internals/sessions.html
It’s a very nice system, I should have some Ansible/Consul examples published soon! With Docker in the mix, we are calling the collection of microservices/clustering platform patterns using this trio “marina”. “marina” is just the conceptual glue to make these things work together in a coherent way, you won’t be able to clone “marina”! But I think the patterns contained in the playbooks may help answer coordination problems like this.
You wouldn’t need consul for registering services and finding out what was where if you knew what service was in each group, which you have in the grouping system for ansible.
If using clouds, tag each machine with it’s purpose, and each also becomes a group.