I have a deployment of multiple clusters of 4 nodes each with their own load balancer, and I want to update all the clusters in parallel, but apply rolling updates to the hosts in those clusters. This would allow me to update all clusters in a short service window without downtime.
The number of clusters and the host IPs change over time, so we will generate an inventory before each playbook run.
This must be a common problem, but I can’t find any examples of rolling updates to a dynamic number of clusters!
My first thought was to define each cluster as a group, and execute a play against each group with serial set to 1, but I have learned that I can’t execute the same play in parallel against different groups, only against different hosts.
I considered looping through a big set of variables which define the clusters, and having the inventory only include localhost, but that seems to be missing the point of ansible working against an inventory of hosts and I’m not sure it’d work at all.
How can I do this? Is it possible to use AWX to run the playbook in parallel on the controller against a separate inventory for every cluster?
I have a deployment of multiple clusters of 4 nodes each with their own
load balancer, and I want to update all the clusters in parallel, but apply
rolling updates to the hosts in those clusters. This would allow me to
update all clusters in a short service window without downtime.
The number of clusters and the host IPs change over time, so we will
generate an inventory before each playbook run.
This must be a common problem, but I can't find any examples of rolling
updates to a dynamic number of clusters!
Ansible doesn't have that feature.
But you have all the tools in the Linux realm at your disposal so you can run as many ansible-playbook in parallel as you need.
My first thought was to define each cluster as a group, and execute a play
against each group with serial set to 1, but I have learned that I can't
execute the same play in parallel against different groups, only against
different hosts.
Use xargs parallel feature or GNU Parallel to run multiple ansible-playbook's in parallel.
Is that the best practice or the “standard” way people solve this issue? Is AWX/Tower not usable with this approach?
Thanks
Is that the best practice or the "standard" way people solve this issue?
Best is a vague term IMHO, best in what? It all comes down to the requirement.
Both works great and you should be up an running in no time. And since Ansible doesn't support it you need some external tools.
My requirement was to run as many playbook in parallel as possible to reduce total run time, but only run one playbook against the same host at any one time.
To solve that I create a Python script that "parsed" all the playbook to identified which host they run against and the created python threads that executed as many ansible-playbook in parallel as possible.
The script is not very sophisticated, it was the first time I used thread in Python, developer using Python would probably frown at the code and it could be more clever to run even more playbooks in parallel, but it solved my requirement so I'm happy.
Is AWX/Tower not usable with this approach?
You should probably ask the AWX list that question.