# Example of multi-datacenter deployments using Ansible

**URL:** <https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083>\
**Category:** Ansible Project\
**Created:** [December 6, 2013, 5:17am UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083 "2013-12-06T05:17:04Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![joel2](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@joel2](https://forum.ansible.com/u/joel2)\
**Post date:** [December 6, 2013, 5:17am UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/1 "2013-12-06T05:17:04Z")

</div>

Hello,

We _currently_ have three datacenters, and they’re all three basically the same with some very minor differences. Up until now, what I’ve done is created a playbook for each, and a hosts file for each datacenter. This was primarily because the ethernet interfaces slightly differed and we didn’t want to have to remember to provide forgettable variables on the command line.

Now, we’re finally migrating our systems where all datacenters will be exactly the same, so we can use the same playbook. To keep it simple, lets say that each datacenter has one load balancer, and two web servers.

**DC1**

> [webservers]  
> 1.1.1.20  
> 1.1.1.30
> 
> [lbservers]  
> 1.1.1.10

**DC2**

> [webservers]
> 
> 1.1.2.20
> 
> 1.1.2.30
> 
> [lbservers]
> 
> 1.1.2.10

**DC3**

> [webservers]
> 
> 1.1.3.20
> 
> 1.1.3.30
> 
> [lbservers]
> 
> 1.1.3.10

Typically, in our previous setup where each datacenter had its own hosts file and playbook, we’d do the following to deploy all the installation tasks:

> ansible-playbook -i /ansible/dc1/hosts.txt /ansible/dc1/all\_full\_deploy.yml

Since our datacenters will basically be the same, and the playbook can now be the same, I understand that I could use just one playbook, and change out the hosts file and it’ll work to isolate deployments to that datacenter. The issue, and why I’m writing this is that I’d also be able to, for example, deploy our website to all [webservers] regardless of which datacenter it’s in—and it has to perform a few tasks on the respective load balancer when it does it (taking it out and adding it back).

So, what is the recommended way to have a multi-datacenter hosts file so that we can work with a single datacenter, or all of them ideally using the same hosts file and the same playbook?

Thank you guys in advance for any advice you can provide.

Sincerely,  
Joel Strellner

---

<div class="post-metadata">

**Author:** ![James\_Martin2](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@James\_Martin2](https://forum.ansible.com/u/James_Martin2)\
**Post date:** [December 6, 2013, 5:19pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/2 "2013-12-06T17:19:51Z")

</div>

Joel,

This is untested, but might work (depending on what your playbooks are  
doing). Try using multiple groups and parent groups with children.

Assuming your playbooks ran against the webservers or lbserver goup:

if you wanted updated only dc1 :

ansible-playbook myplabook.yml --limit dc1

if you wanted to run against both datacenters:

ansible-playbook myplaybook.yml

hosts file:

[dc1:children]  
dc1\_webservers  
dc1\_lbservers

[dc1\_webservers]  
1.1.1.20  
1.1.1.30

[dc1\_lbservers]  
1.1.1.10

[dc2:children]  
dc2\_webservers  
dc2\_lbservers

[dc1\_webservers]  
1.1.2.20  
1.1.2.30

[dc1\_lbservers]  
1.1.2.10

[webservers:children]  
dc1\_webservers  
dc2\_webservers

[lbservers:children]  
dc1\_lbservers  
dc2\_lbservers

- James

---

<div class="post-metadata">

**Author:** ![joel2](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@joel2](https://forum.ansible.com/u/joel2)\
**Post date:** [December 6, 2013, 5:32pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/3 "2013-12-06T17:32:06Z")

</div>

Hi James,

First, thanks for the response.

Would the example you provided still work when the .yml files declare there hosts as:

> - hosts: lbservers

Or would it have to be changed to say:

> - hosts: dc1\_lbservers

We have specific tasks that would need to be run on the lbservers vs the web servers.

Thanks,

-Joel

---

<div class="post-metadata">

**Author:** ![James\_Martin2](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@James\_Martin2](https://forum.ansible.com/u/James_Martin2)\
**Post date:** [December 6, 2013, 5:38pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/4 "2013-12-06T17:38:47Z")

</div>

Joel,

It would be:

- hosts: lbservers

In theory, the --limit dc1 would limit the playbook to the lbservers  
in dc1. Just make sure you don't forget to limit!

- James

---

<div class="post-metadata">

**Author:** ![joel2](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@joel2](https://forum.ansible.com/u/joel2)\
**Post date:** [December 6, 2013, 5:44pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/5 "2013-12-06T17:44:54Z")

</div>

Hi James,

If I didn’t limit, would it run the tasks for **each** lbservers for **each** webserver, or would it properly know that dc1’s webservers should only interact with dc1’s load balancer?

I am trying to make it so that I could run a playbook for a specific datacenter, or I could run it for all at the same time (like a code deploy). The sticking point seems to be that the lbserver in dc1 should only know about the webservers in dc1.

---

<div class="post-metadata">

**Author:** ![James\_Martin2](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@James\_Martin2](https://forum.ansible.com/u/James_Martin2)\
**Post date:** [December 6, 2013, 5:50pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/6 "2013-12-06T17:50:38Z")

</div>

Joel,

I think I understand what you mean. It would help to see the  
playbook(s). Perhaps you need an inventory variable for the  
load balancers that define what group they are in charge of?  
I don't think ansible is capable of automatically inferring it on its own.  
You might be able to write some logic around it in your  
playbook/template as well. Declaring it might be faster and easier to  
understand.

- James

---

<div class="post-metadata">

**Author:** ![joel2](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@joel2](https://forum.ansible.com/u/joel2)\
**Post date:** [December 6, 2013, 6:06pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/7 "2013-12-06T18:06:02Z")

</div>

Hi James,

They’re pretty complicated, and we have several depending on the scenario. The most common that we run though, and simplest, is a code deploy, and it looks like this:

> # gather facts from monitoring nodes for iptables rules
> 
> - hosts: webservers  
> serial: 1  
> accelerate: true
> 
> # These are the tasks to run before applying updates:
> 
> pre\_tasks:
> 
> - name: disable the staging server in haproxy  
> shell: echo “disable server web\_staging/{{ ansible\_hostname }}” | socat stdio /var/lib/haproxy/stats  
> delegate\_to: “{{ item }}”  
> with\_items: groups.lbservers
> - name: disable the production server in haproxy  
> shell: echo “disable server web\_production/{{ ansible\_hostname }}” | socat stdio /var/lib/haproxy/stats  
> delegate\_to: “{{ item }}”  
> with\_items: groups.lbservers  
> roles:
> - deploy\_web
> 
> # These tasks run after the roles:
> 
> post\_tasks:
> 
> - name: Enable the staging server in haproxy  
> shell: echo “enable server web\_staging/{{ ansible\_hostname }}” | socat stdio /var/lib/haproxy/stats  
> delegate\_to: “{{ item }}”  
> with\_items: groups.lbservers
> - name: Enable the production server in haproxy  
> shell: echo “enable server web\_production/{{ ansible\_hostname }}” | socat stdio /var/lib/haproxy/stats  
> delegate\_to: “{{ item }}”  
> with\_items: groups.lbservers

In short, it removes the web server from HA Proxy, then does a code update (deploy\_web role is basically a git pull), then it adds the server back to HA Proxy.

I have not considered an inventory variable. I’ll look into it to see if it can be done in a way that makes sense to me.

I’m starting to think the cleanest way might just be three hosts files, and then just running the playbook three times. ☹

-Joel

---

<div class="post-metadata">

**Author:** ![James\_Martin2](https://avatars.discourse-cdn.com/v4/letter/j/b9e5f3/32.png) [@James\_Martin2](https://forum.ansible.com/u/James_Martin2)\
**Post date:** [December 6, 2013, 6:31pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/8 "2013-12-06T18:31:13Z")

</div>

Thanks for the playbook.

So based on what I was saying before.. Make a group\_var for the  
webservers that tell them what lbserver group owns them. Maybe call  
it {{ lb\_server\_group }}

then your delegate operations could look like:

&nbsp;&nbsp;&nbsp;delegate\_to: "{{ item }}"  
&nbsp;&nbsp;&nbsp;&nbsp;with\_items: groups.{{ lb\_server\_group }}

It does seem cleaner to do it with different hosts files.

- James

---

<div class="post-metadata">

**Author:** ![joel2](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@joel2](https://forum.ansible.com/u/joel2)\
**Post date:** [December 6, 2013, 6:40pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/9 "2013-12-06T18:40:04Z")

</div>

Thanks James,

I’ll play with doing it that way, and I’ll probably also do the multiple hosts file. Whatever ends up working the best and is least likely to allow errors to happen is what I’ll go with.

Thanks again.

-Joel

---

<div class="post-metadata">

**Author:** ![Davide1](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Davide1](https://forum.ansible.com/u/Davide1)\
**Post date:** [May 24, 2015, 2:48pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/10 "2015-05-24T14:48:54Z")

</div>

Hello Joel! I ran into the same identical problem (cfr. [http://serverfault.com/questions/693469/multi-datacenter-ansible-load-balancer-template](http://serverfault.com/questions/693469/multi-datacenter-ansible-load-balancer-template)) and was wondering if you could share whatever solution you ended up using?

Thank you so much!  
D

---

<div class="post-metadata">

**Author:** ![joel2](https://avatars.discourse-cdn.com/v4/letter/j/c6cbf5/32.png) [@joel2](https://forum.ansible.com/u/joel2)\
**Post date:** [May 26, 2015, 10:32am UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/11 "2015-05-26T10:32:32Z")

</div>

Hi Davide,

We ended up going with multiple host files, one for each datacenter, each containing only the relevant information for that datacenters information. The added benefit of going this route is that we can deploy to a single datacenter, verify that all is well, and then roll out to the others (like upgrading or installing new packages). To get around the annoyance for certain playbooks that I know can always be run in all datacenters—like most of our code deploys—on our ansible server we have alias commands in our bash profile that can run the ansible command on all datacenters.

I wish there was a better way, but I just couldn’t find one.

---

<div class="post-metadata">

**Author:** ![Ravi\_Bhure](https://avatars.discourse-cdn.com/v4/letter/r/a88e57/32.png) [@Ravi\_Bhure](https://forum.ansible.com/u/Ravi_Bhure)\
**Post date:** [May 26, 2015, 5:43pm UTC](https://forum.ansible.com/t/example-of-multi-datacenter-deployments-using-ansible/14083/12 "2015-05-26T17:43:48Z")

</div>

Surprising !  
There is still better way available to do code deploy using Ansible ..

Task should looks like

1) remove webserver from lb - use module haproxy (wait till connection drain)  
2) silent monitoring for web health check - use nagios/zabbix or any another monitoring module  
3) deploy code  
4) do manual web health check or use nagios socket  
5) add back in lb  
6) unshut web health check in monitoring

If step 4 fails revert to previous version of deployment and continue the steps 5 and 6

We had the multi dc and multi env setup where 5 dc and 3 env used

For that we have used group\_vars to keep all our dc specific folders and the env as yml var file, just to use the extra-vars dc=cool env=prod solved the issues.  
In these env var files we have added all our info, just like if we have prod env running at cool dc

lb: lbserver  
web: "web1, web2"  
watchdog: nagioserver

In task header just to add the following and each task delegate to the addressed server.

host:{{target}}  
var\_files: {{dc}}

cmd:  
ansible-playbook deploy.yml -e "dc=colo env=prod"