Best practice for cluster nodes

Archives Ansible Project
1

I have several clusters containing several nodes. I want to run configuration jobs on all cluster nodes. The configuration jobs involve a small down time of the cluster node. I have to take care, that at least a minimum number of nodes in each cluster is still running. What is the best practice to achieve this?

I found an option in Ansible to limit the total number of simultaneous connections. But this does not help, because I need simultaneous connections per cluster.

Another way is to put all nodes of each cluster in groups: First the group of all first cluster members, second the group of all second cluster members and so on. This would make it possible to apply the configuration role to each group. But this is still not exactly what I am looking for, because this serializes the cluster nodes more than necessary. If I have 10 nodes per cluster and I want to keep at least 50% of all nodes running, it would be possible for Ansible to process 5 nodes in parallel.

What is the best practice for Ansible to handle cluster nodes?

2

Have you seen the serial: argument to plays?

    http://docs.ansible.com/ansible/latest/playbooks_delegation.html#rolling-update-batch-size

That's intended for working on a 'chunk' of N servers in a given group at once.

3

No. Thanks for the link!

4

I had a similar issue with a cassandra cluster recently. Serial wasn’t really a workable option for me but I found an alternative solution using delegate_to. I did a write up of it here; http://www.youdidwhatwithtsql.com/staged-service-restart-ansible/2337/

Rhys