# Rolling Restart an Elasticsearch Cluster with Ansible **URL:** \ **Category:** Ansible Project\ **Created:** [September 17, 2014, 7:40pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261 "2014-09-17T19:40:52Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Lance\_A\_Brown](https://avatars.discourse-cdn.com/v4/letter/l/f6c823/32.png) [@Lance\_A\_Brown](https://forum.ansible.com/u/Lance_A_Brown)\ **Post date:** [September 17, 2014, 7:40pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/1 "2014-09-17T19:40:52Z")
I've come up with what I think is a safe way to rolling restart an Elasticsearch cluster using Ansible handlers. Why is this needed? Even if you use a serial setting to limit the number of nodes processed at one time, Ansible will restart elasticsearch nodes and continue processing as soon as the elasticsearch service restart reports itself complete. This pushes the cluster into a red state due to muliple data nodes being restarted at once, and can cause performance problems. Solution: Perform a rolling restart of each elasticsearch node and wait for the cluster to stabilize before continuing processing nodes. This set of chained handlers restarts each node while keeping the cluster from thrashing on reallocating shards during the process. A gist of the handlers/main.yml file for my Elasticsearch role is at [https://gist.github.com/labrown/5341ebec47bfba6dd7d4](https://gist.github.com/labrown/5341ebec47bfba6dd7d4) I welcome any comments and/or suggestions. --[Lance] ---
**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\ **Post date:** [September 18, 2014, 2:22am UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/2 "2014-09-18T02:22:38Z")
check the wait\_for module, it was created for these situations. ---
**Author:** ![Lance\_A\_Brown](https://avatars.discourse-cdn.com/v4/letter/l/f6c823/32.png) [@Lance\_A\_Brown](https://forum.ansible.com/u/Lance_A_Brown)\ **Post date:** [September 18, 2014, 2:42am UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/3 "2014-09-18T02:42:50Z")
Brian Coca said the following on 9/17/2014 10:22 PM: > check the wait\_for module, it was created for these situations. How do you make wait\_for wait on an URL? I know you can have it wait on a socket to come up and look for data on the port connection, but I need more than that to ensure the cluster is in the correct state before proceeding. --[Lance] ---
**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\ **Post date:** [September 18, 2014, 3:06am UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/4 "2014-09-18T03:06:04Z")
well, in that case I would use wait\_for to let the server come up and then get\_url inside a do-until loop to get the 'green light'. ---
**Author:** ![Ludovic\_Petetin](https://avatars.discourse-cdn.com/v4/letter/l/f14d63/32.png) [@Ludovic\_Petetin](https://forum.ansible.com/u/Ludovic_Petetin)\ **Post date:** [September 18, 2014, 12:47pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/5 "2014-09-18T12:47:13Z")
Hi, I’m operating a Solr Cluster of 50 servers and I have been confronted to the same question than you. The warmup of the Solr index can take up to 30 seconds and wait\_for on the port was not enough. At first, I wrote a new module to launch a query every X seconds and expect a result (wait\_for\_result command=‘wget -O - “http://{{ ansible\_fqdn }}:{{ active\_country\_port[country].port }}/searchsolrnode{{ country }}/{{ country}}/select?q=_:_&rows=0”’ result=“numFound” delay=10 timeout=600) until Ansible came up with it’s do…until loop which is perfect in this case : - shell: wget -O - “http://{{ ansible\_fqdn }}:{{ active\_country\_port[country].port }}/searchsolrnode{{ country }}/{{ country}}/select?q=_:_&rows=0” register: result until: result.stdout.find(“numFound”) != -1 retries: 20 delay: 5 ---
**Author:** ![Lance\_A\_Brown](https://avatars.discourse-cdn.com/v4/letter/l/f6c823/32.png) [@Lance\_A\_Brown](https://forum.ansible.com/u/Lance_A_Brown)\ **Post date:** [September 18, 2014, 1:21pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/6 "2014-09-18T13:21:08Z")
I had that configuration initially after I realized wait\_for would not do what I needed. Then I realized setting a timeout on the curl call (-m 2) gets me effectively the same thing. If the node isn't back up yet, the curl call times out and the grep fails. No need to run two tasks. --[Lance] - name: wait node {{ansible\_hostname}}-{{service}}    local\_action: "shell curl -s -m 2 'localhost:9200/\_cat/nodes?h=name' | tr -d ' ' | grep -E '^{{ansible\_hostname}}-{{service}}$' "    register: result    until: result.rc == 0    retries: 200    delay: 3    notify:    - cluster routing all {{ansible\_hostname}}-{{service}} ---
**Author:** ![James\_Martin1](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@James\_Martin1](https://forum.ansible.com/u/James_Martin1)\ **Post date:** [September 18, 2014, 2:05pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/7 "2014-09-18T14:05:26Z")
I think a more “ansiblish” approach would be: - name: wait node {{ansible\_hostname}}-{{service}} uri: url=[http://localhost:9200/\_cat/nodes?h=name](http://localhost:9200/_cat/nodes?h=name) timeout=120 return\_content=yes register: result until: (ansible\_hostname + ‘-’ + service) in result.content retries: 200 delay: 3 ---
**Author:** ![Lance\_A\_Brown](https://avatars.discourse-cdn.com/v4/letter/l/f6c823/32.png) [@Lance\_A\_Brown](https://forum.ansible.com/u/Lance_A_Brown)\ **Post date:** [September 18, 2014, 2:17pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/8 "2014-09-18T14:17:40Z")
Oh! I like that. Thanks! --[Lance] ---
**Author:** ![Sam\_Doran](https://avatars.discourse-cdn.com/v4/letter/s/6a8cbe/32.png) [@Sam\_Doran](https://forum.ansible.com/u/Sam_Doran)\ **Post date:** [September 22, 2014, 4:23pm UTC](https://forum.ansible.com/t/rolling-restart-an-elasticsearch-cluster-with-ansible/16261/9 "2014-09-22T16:23:52Z")
I’ve written two rolling [Elasticsearch playbooks](https://gist.github.com/samdoran/bd996dc0a3018ff6ba8e): one for restarting the cluster, the other for updating the cluster. I use the feature of Ansible that will intelligently take JSON output in a register variable and make it addressable (not sure if I’m using the right terminology). Because Elasticsearch returns JSON when you query the API, you can use the registered output of the uri module and a [do-until loop](http://docs.ansible.com/playbooks_loops.html#do-until-loops) to wait for a certain condition before continuing. So instead of using result.stdout or result.content with a search, you can use result.json.[field] to be very precise. ` ` ` - name: Wait for cluster health to return to yellow uri: url=[http://localhost](http://localhost):{{ es\_http\_port }}/\_cluster/health method=GET register: response until: “response.json.status == ‘yellow’” retries: 5 delay: 30 ` ` ` The contents of response.json are: ` `ok: [node01.acme.com] => { "response": { "changed": false, "content_length": "227", "content_location": "http://localhost:9500/_cluster/health", "content_type": "application/json; charset=UTF-8", "invocation": { "module_args": "url=http://localhost:9500/_cluster/health method=GET", "module_name": "uri" }, "json": { "active_primary_shards": 405, "active_shards": 810, "cluster_name": "elasticsearch", "initializing_shards": 0, "number_of_data_nodes": 3, "number_of_nodes": 4, "relocating_shards": 0, "status": "green", "timed_out": false, "unassigned_shards": 0 }, "redirected": false, "status": 200 } }` ` I’m not sure of the value of using pre and post tasks with a serial of one in my playbooks. It could all most likely be one list of tasks. I based the playbooks on the Elasticsearch documentation for [updating 1.0 and later](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#_1_0_and_later) and [rolling restart](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-shutdown.html#_rolling_restart_of_nodes_full_cluster_restart).