# What is the best way of restarting services that are part of a cluster without taking the cluster down? **URL:** \ **Category:** Ansible Project\ **Created:** [July 23, 2014, 11:16pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751 "2014-07-23T23:16:50Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![David\_Reagan](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@David\_Reagan](https://forum.ansible.com/u/David_Reagan)\ **Post date:** [July 23, 2014, 11:16pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/1 "2014-07-23T23:16:50Z")
Specifically, an elasticsearch cluster, but doing this the right way would also apply to other kinds of clusters, like RabbitMQ or Redis. I’m using [https://github.com/LaneCommunityCollege/aspects\_elasticsearch](https://github.com/LaneCommunityCollege/aspects_elasticsearch) to manage my elasticsearch cluster. Currently, if I modify the configuration settings, Ansible would issue a restart to all the nodes in the cluster. The obvious answer is to only run the playbook on one node at a time. But there are situations where that isn’t convenient. If I run Ansible like a puppet agent then setting the configuration setting in the group\_vars file will apply it on all the nodes, thus restarting them all. Or if I have a large number of nodes. Or simply need to apply the change, but don’t have time to do it one or two nodes at a time. The other solutions I’ve thought of involve custom scripts that would not be part of the Ansible playbook. And likely be application specific. Is there a way to tell ansible to set a service restart task sometime in the future? Say, right now for node1, 5 minutes later for node2, 10 minutes later for node3, and so on. Maybe the at module? How would it know to set it 10 minutes in the future instead of 5 for the third node? So, yeah, is there a good Ansible specific method for this? Or do I need to look outside of Ansible? ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 24, 2014, 11:00am UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/2 "2014-07-24T11:00:56Z")
“Is there a way to tell ansible to set a service restart task sometime in the future? Say, right now for node1, 5 minutes later for node2, 10 minutes later for node3, and so on.” There is a module for “at” which could be used to do this. at runs some commands at scheduled times in the future, without setting up a crontab. You could also set a simple play like this, which would probably be cleaner: - hosts: blarg serial: 1 tasks: - pause: seconds={{ n }} - service: name=foo state=restarted Which will walk across your infra 1 host at a time, with a lag of n seconds between, and restart everything. The serial: 1 could likely be increased beyond 1 depending on the application context. ---
**Author:** ![Michael\_Peters](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@Michael\_Peters](https://forum.ansible.com/u/Michael_Peters)\ **Post date:** [July 24, 2014, 1:38pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/3 "2014-07-24T13:38:17Z")
Is there a way to do something like this as a handler? I think that's the main problem here is that a configuration change should trigger a restart handler, but right now there's no way to throttle/serialize the execution of that handler. Unless I'm missing something... ---
**Author:** ![David\_Reagan](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@David\_Reagan](https://forum.ansible.com/u/David_Reagan)\ **Post date:** [July 24, 2014, 3:54pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/4 "2014-07-24T15:54:24Z")
> A possible workaround would be to make a somewhat non-idiomatic version of ‘handlers’ by putting the tasks that restart the services in a separate play in the end of your playbook with serial: 1. This will cause these tasks to run one after the other. You would also have ‘when’ conditionals on these tasks checking whether any other tasks that should cause restart of services have actually made changes. Last, if you need to add a delay between the restarts, you can use the ‘wait\_for’ directive on the tasks restarting the services. On the template task, register a variable that says to restart the service, then at the end of the play I could use the at module to set a service restart for sometime in the future. So, handlers seem like they are just special tasks that you can call with the “notify” option. If that’s the case, couldn’t I just make a handler that uses the at module instead of the service module? Actually, I think I’ve done that already in a differen role… Hmm… What about adjusting how far into the future the at module should run restart the service? As in, for the first node done, it would be right away, for the next 5 minutes from now, for the next 10 minutes from now, and so on. Can I register a variable and then add to it? > You could also set a simple play like this, which would probably be cleaner: > > - hosts: blarg > serial: 1 > tasks: > - pause: seconds={{ n }} > - service: name=foo state=restarted I did think of something like that, but then my playbook would end up waiting that long to be finished, and I’m not sure how to get the rolling restarts I want. Hmm… Maybe waiting to end the playbook is actually a good thing. I’ll have to think about it. ---
**Author:** ![Michael\_Peters](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@Michael\_Peters](https://forum.ansible.com/u/Michael_Peters)\ **Post date:** [July 24, 2014, 6:05pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/5 "2014-07-24T18:05:01Z")
The problem with at is that you have to manually figure out how long something should take and hope it never takes over that and then try to randomize how quickly how many servers restart. Or you leave a lot of time between which makes it inefficient. Just saying it would be nice if we could already leverage all of the power of serial (including all of the niceties like max\_fail\_percentage). ---
**Author:** ![David\_Reagan](https://avatars.discourse-cdn.com/v4/letter/d/71e660/32.png) [@David\_Reagan](https://forum.ansible.com/u/David_Reagan)\ **Post date:** [July 24, 2014, 7:14pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/6 "2014-07-24T19:14:56Z")
Heh, yeah, guessing how long it takes to restart is not ideal. It’s just what I was thinking of as a work around that properly tweaked would avoid clobbering the cluster. The more I think about it, the more I’m not sure there’s a good way to do this in Ansible. For Elasticsearch, I’d want to issue a command like ‘service elasticsearch safe-restart’ that waits for the status of the cluster to be green before restarting. For RabbitMQ, I’d want something similar, though I haven’t worked with it enough to know exaction what. I think I’ll head off to the elasticsearch group and Google to see how others have avoided clobbering their clusters when updating config. 🙂 Thanks for all the responses so far. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 24, 2014, 10:51pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/7 "2014-07-24T22:51:39Z")
Right now handlers are just about notifying things in the context of one host. There’s not (yet) a concept of a global handler, nor really a “global: True” kind of variant on set\_fact. If there were, it would be easy to set a global flag True if a particular host hit a state, and then use that state to trigger actions on another set, without having to rely on the result of just one particular host, which you can already do with hostvars. Sorry if this doesn’t entirely make sense. There are ways you could do it that are a little bit hackish, like creating a signal file with a handler that delegated to localhost, but mass delegation to localhost with 500 systems is usually going to make your control server unhappy. I think it’s about time we do have a “set\_global” that a handler could call, and you could use that in another play. Seems like a good idea now. ---
**Author:** ![Steffen\_Prince](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@Steffen\_Prince](https://forum.ansible.com/u/Steffen_Prince)\ **Post date:** [July 25, 2014, 8:54am UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/8 "2014-07-25T08:54:36Z")
Have you thought about managing global cluster state with something that can keep track of node health? You could use Ansible to poll this before restarting individual nodes, waiting for all nodes to register as healthy, then acquiring a global lock before proceeding with the restart. With this approach, you can attempt restarts on all nodes in parallel, but be assured you will only be restarting one at a time (or as many locks as you configure). Each parallel Ansible run will just sit polling the agent until it gets a chance to acquire the lock. I do not think using Ansible to track global cluster state will work out well, but Ansible can work really nicely in tandem with such a cluster monitor. [Consul](http://www.consul.io/) is a nice tool for this: a distributed CP service discovery, health checking, and key-value store system. It is quite general-purpose and does not require you to modify your services in any way to make Consul aware of them. Just hook the service registration into the service’s systemd unit/init script, set up a health check that tells you when the node is well and truly a happy member of the cluster, and then poll Consul from anywhere to see what the overall cluster health is. This doc discusses using Consul to build distributed locks: [http://www.consul.io/docs/internals/sessions.html](http://www.consul.io/docs/internals/sessions.html) It’s a very nice system, I should have some Ansible/Consul examples published soon! With Docker in the mix, we are calling the collection of microservices/clustering platform patterns using this trio “marina”. “marina” is just the conceptual glue to make these things work together in a coherent way, you won’t be able to clone “marina”! But I think the patterns contained in the playbooks may help answer coordination problems like this. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 25, 2014, 7:19pm UTC](https://forum.ansible.com/t/what-is-the-best-way-of-restarting-services-that-are-part-of-a-cluster-without-taking-the-cluster-down/15751/9 "2014-07-25T19:19:03Z")
You wouldn’t need consul for registering services and finding out what was where if you knew what service was in each group, which you have in the grouping system for ansible. If using clouds, tag each machine with it’s purpose, and each also becomes a group.