# multiple instance for AWX instalation (HA)

**URL:** <https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405>\
**Category:** AWX Project\
**Tags:** awx\
**Created:** [January 3, 2018, 8:39am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405 "2018-01-03T08:39:03Z")\
**Posts on this page:** 16\
**Page:** 2

<div class="post-metadata">

**Author:** ![Philipp\_Wiesner](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@Philipp\_Wiesner](https://forum.ansible.com/u/Philipp_Wiesner)\
**Post date:** [March 8, 2018, 6:36pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/21 "2018-03-08T18:36:04Z")

</div>

Hi Dnc92301,

you should see both your nodes below the AWX instance group.

Here is a link how Ansible Tower is setup as a cluster: [http://docs.ansible.com/ansible-tower/latest/html/administration/clustering.html#job-runtime-behavior](http://docs.ansible.com/ansible-tower/latest/html/administration/clustering.html#job-runtime-behavior)

The image shows, that on each node RabbitMQ is running, which are providing the cluster functionality. With AWX you have on each node then docker containers running with the awx\_web, awx\_task and memcache images. For each node the settings.py is in this way configured, that the rabbitmq\_host points to it one machine. Meaning that when you have two nodes: node1 and node2, that on node1 the settings.py rabbitmq\_host points to node1 and on node2 the settings.py rabbitmq\_host points to node2. The same goes for the celery worker. Therefore we have added this cluster\_node inventory variable. We have the installation directory on each seperate node, set the cluster\_node to the machine name and run the installation playbook on each node.

The cluster work in this way, that when the capacity on one node is full, the next triggered job will be started on the next node of the cluster with free capacity. A failover of a job like mentioned in your example will not work, as the playbook run is triggered by one worker process on one node. When you restart this node, the worker process is lost.

The page I provided to you gives you a good overview, of what the cluster functionality of Ansible Tower/AWX provides and how it works. For our setup I have basically reenginered this setup into AWX.

---

<div class="post-metadata">

**Author:** ![matburt](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/matburt/32/3612_2.png) [@matburt](https://forum.ansible.com/u/matburt)\
**Post date:** [March 8, 2018, 7:26pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/22 "2018-03-08T19:26:26Z")

</div>

It’s probably worth pointing out at this point that clustered AWX is now supported on Openshift and Kubernetes without needing to hand-roll your own solution.

---

<div class="post-metadata">

**Author:** ![Dong\_Yang](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@Dong\_Yang](https://forum.ansible.com/u/Dong_Yang)\
**Post date:** [March 8, 2018, 8:01pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/23 "2018-03-08T20:01:52Z")

</div>

I did take your suggested recommendation by updating all config files - settings.py and others .

I’m looking at tower HA configuration . API/2/ping it shows both nodes within the instance group . I’m AWX it lists awx within instance\_group , ps ef |grep celery - shows celery@awx (on both nodes ) .I think a working setup should be showing as celery@hostname

Thanks for all your help ..

---

<div class="post-metadata">

**Author:** ![Philipp\_Wiesner](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@Philipp\_Wiesner](https://forum.ansible.com/u/Philipp_Wiesner)\
**Post date:** [March 8, 2018, 8:20pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/24 "2018-03-08T20:20:40Z")

</div>

I Think you are on the right path, when the API already shows both nodes within the instance group.

There is this file:

`image_build/files/supervisor_task.conf`

In the installation image\_build role. If you adjust the celery worker command to:  
`command = /var/lib/awx/venv/awx/bin/celery worker -A awx -l DEBUG --autoscale=``50``,``4` `-Ofair -Q`  
`tower_scheduler,tower_broadcast_all,tower,%(ENV_CLUSTER_NODE)s -n celery@%(ENV_CLUSTER_NODE)s`

ps ef | grep celery should also show the correct output. Please be aware of the ENV\_ in front of the environment variable, this is needed by supervisord for reading environment variables.

---

<div class="post-metadata">

**Author:** ![Dong\_Yang](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@Dong\_Yang](https://forum.ansible.com/u/Dong_Yang)\
**Post date:** [March 8, 2018, 8:38pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/25 "2018-03-08T20:38:11Z")

</div>

It seems the variable is not being passed corrected in the supervisor\_task.conf file . ...tower\_broadcast\_all,tower,%(ENV\_CLUSTER\_NODE)s - n celery@%(ENV\_CLUSTER\_NODE)s .

As celeryd starts up as ,tower\_broadcast\_all,tower,awx -n celery@localhost .

In tower , it is \*tower\_broadcast\_all,tower,hostname -n celery@hostname

Within inventory i tried with cluster\_node , wig and without FQDN and also with cluster\_node and CLUSTER\_NODE.

It’s odd

---

<div class="post-metadata">

**Author:** ![Philipp\_Wiesner](https://avatars.discourse-cdn.com/v4/letter/p/839c29/32.png) [@Philipp\_Wiesner](https://forum.ansible.com/u/Philipp_Wiesner)\
**Post date:** [March 9, 2018, 10:30am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/26 "2018-03-09T10:30:36Z")

</div>

Are you also setting the CLUSTER\_NODE variable in your env?

Here an except from our local\_docker/tasks/main.yml file:

env:  
CLUSTER\_NODE: “{{ cluster\_node | default(‘localhost’) }}”

This was set for the awx task and web container, providing the cluster\_node variable to the environment. So supervisor\_task.conf file can read this inside the container.

---

<div class="post-metadata">

**Author:** ![Dong\_Yang](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@Dong\_Yang](https://forum.ansible.com/u/Dong_Yang)\
**Post date:** [March 17, 2018, 12:52pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/27 "2018-03-17T12:52:08Z")

</div>

Hi Phillipp - can you tell me what version /tag you used for your setup ? I wanted to see if I can reproduce it . I had done exactly the changes you had described by separating out Rabbitmq (2-nodes) and having containers(awx/task/memcache) installed (with the modified configurations) on the same hosts running rabbitmq. PostgreSQL on a different node . So in all a 3 server configuration. Thanks

---

<div class="post-metadata">

**Author:** ![Abhishek.J\_Gowda](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Abhishek.J\_Gowda](https://forum.ansible.com/u/Abhishek.J_Gowda)\
**Post date:** [June 7, 2018, 7:13am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/28 "2018-06-07T07:13:25Z")

</div>

Has anyone successfuly setup the high available AWX set up which can scale?, need some suggestions, thanks

---

<div class="post-metadata">

**Author:** ![Dong\_Yang](https://avatars.discourse-cdn.com/v4/letter/d/59ef9b/32.png) [@Dong\_Yang](https://forum.ansible.com/u/Dong_Yang)\
**Post date:** [June 15, 2018, 2:23am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/29 "2018-06-15T02:23:26Z")

</div>

Docker swarm might be the way forward ..

---

<div class="post-metadata">

**Author:** ![Abhishek.J\_Gowda](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Abhishek.J\_Gowda](https://forum.ansible.com/u/Abhishek.J_Gowda)\
**Post date:** [June 22, 2018, 3:52am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/30 "2018-06-22T03:52:23Z")

</div>

hi Matt Jones,

can you pls point me to the documentation for clustered AWX in openshift,

thanks

---

<div class="post-metadata">

**Author:** ![matburt](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/matburt/32/3612_2.png) [@matburt](https://forum.ansible.com/u/matburt)\
**Post date:** [June 27, 2018, 6:23pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/31 "2018-06-27T18:23:21Z")

</div>

Everything you need is here: [https://github.com/ansible/awx/blob/devel/INSTALL.md](https://github.com/ansible/awx/blob/devel/INSTALL.md)

---

<div class="post-metadata">

**Author:** ![The\_Vandyy\_Vines](https://avatars.discourse-cdn.com/v4/letter/t/7993a0/32.png) [@The\_Vandyy\_Vines](https://forum.ansible.com/u/The_Vandyy_Vines)\
**Post date:** [September 3, 2018, 6:46am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/32 "2018-09-03T06:46:43Z")

</div>

Hello, Did you manage to find solution for this issue?

Thanks.

---

<div class="post-metadata">

**Author:** ![The\_Vandyy\_Vines](https://avatars.discourse-cdn.com/v4/letter/t/7993a0/32.png) [@The\_Vandyy\_Vines](https://forum.ansible.com/u/The_Vandyy_Vines)\
**Post date:** [September 5, 2018, 3:22am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/33 "2018-09-05T03:22:28Z")

</div>

Hello,

Will this help?

# Allow Containers to be installed to a stack in Docker Swarm #1287https://github.com/ansible/awx/issues/1287

BR//  
Vandana

---

<div class="post-metadata">

**Author:** ![CV1](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@CV1](https://forum.ansible.com/u/CV1)\
**Post date:** [February 16, 2019, 8:48pm UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/34 "2019-02-16T20:48:39Z")

</div>

Hi Phil,

I am having trouble finding the parts of the file you edited to accomplish a multiple instance (HA) installation. Current version is 3.0.1

which command did you replace in supervisor\_task.conf with "`command = /var/lib/awx/venv/awx/bin/celery worker -A awx -l ERROR --autoscale=``50``,``4` `-Ofair -Q`  
`tower_scheduler,tower_broadcast_all,tower,%(ENV_CLUSTER_NODE)s -n celery@%(ENV_CLUSTER_NODE)s"?`

`In addition, I do not see any references to rabbitmg in main.yml or `set\_image.yml (the only task imported in main.yml) in local\_docker/tasks/ .

---

<div class="post-metadata">

**Author:** ![Sujith\_A\_R1](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@Sujith\_A\_R1](https://forum.ansible.com/u/Sujith_A_R1)\
**Post date:** [March 8, 2019, 12:58am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/35 "2019-03-08T00:58:37Z")

</div>

This particular change realted with celery isn’t needed for latest AWX version 3.0.1 as it auto picks up and schedule jobs under what instance group we defined over there.

Check this out as well : [https://groups.google.com/forum/m/?utm\_medium=email&utm\_source=footer#!msg/awx-project/E6mlcdfED2s/MR3dtETLBAAJ](https://groups.google.com/forum/m/?utm_medium=email&utm_source=footer#!msg/awx-project/E6mlcdfED2s/MR3dtETLBAAJ)

---

<div class="post-metadata">

**Author:** ![Sujith\_A\_R1](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@Sujith\_A\_R1](https://forum.ansible.com/u/Sujith_A_R1)\
**Post date:** [April 6, 2019, 4:25am UTC](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405/36 "2019-04-06T04:25:45Z")

</div>

Update towards enabling HA in Docker StandAlone method: [https://github.com/ansible/awx/issues/3627](https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fansible%2Fawx%2Fissues%2F3627&sa=D&sntz=1&usg=AFQjCNEYwqaBKyzmHo6BRbd0dFdf6xaPFw)

[Previous page](https://forum.ansible.com/t/multiple-instance-for-awx-instalation-ha/26405.md?page=1)
