# Rabbitmq configuration in AWX deployment on K8s Cluster

**URL:** <https://forum.ansible.com/t/rabbitmq-configuration-in-awx-deployment-on-k8s-cluster/31929>\
**Category:** AWX Project\
**Tags:** awx, kubernetes\
**Created:** [January 8, 2020, 12:34pm UTC](https://forum.ansible.com/t/rabbitmq-configuration-in-awx-deployment-on-k8s-cluster/31929 "2020-01-08T12:34:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vibin](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vibin](https://forum.ansible.com/u/Vibin)\
**Post date:** [January 8, 2020, 12:34pm UTC](https://forum.ansible.com/t/rabbitmq-configuration-in-awx-deployment-on-k8s-cluster/31929/1 "2020-01-08T12:34:04Z")

</div>

Hello Team,

Looking for some solutions/clarity on the rabbitmq configuration of AWX deployed on Kubernetes cluster.

With the default rabbitmq configuration, the rabbitmq container fails to deploy. But if I comment out “cluster\_formation.peer\_discovery\_backend = rabbit\_peer\_discovery\_k8s”, the container gets deployed but the rabbitmq cluster is not formed.

So my question is, will the rabbitmq cluster get configured if we have multiple instances of awx (scaled) or will they be standalone containers. If it is standalone, then what is the use/purpose of above line.

apiVersion: v1  
kind: ConfigMap  
metadata:  
name: rabbitmq-config  
namespace: {{ kubernetes\_namespace }}  
data:  
enabled\_plugins: |  
[rabbitmq\_management,rabbitmq\_peer\_discovery\_k8s].  
rabbitmq\_definitions.json: |  
{  
“users”:[{“name”: “{{ rabbitmq\_user }}”, “password”: “{{ rabbitmq\_password }}”, “tags”: “administrator”}],  
“permissions”:[  
{“user”:“{{ rabbitmq\_user }}”,“vhost”:“awx”,“configure”:“._“,“write”:”._”,“read”:“._“}  
],  
“vhosts”:[{“name”:“awx”}],  
“policies”:[  
{“vhost”:“awx”,“name”:“ha-all”,“pattern”:”._”,“definition”:{“ha-mode”:“all”,“ha-sync-mode”:“automatic”}}  
]  
}  
rabbitmq.conf: |

## Clustering

management.load\_definitions = /etc/rabbitmq/rabbitmq\_definitions.json  
cluster\_formation.peer\_discovery\_backend = rabbit\_peer\_discovery\_k8s  
cluster\_formation.k8s.host = kubernetes.default.svc  
cluster\_formation.k8s.address\_type = ip

cluster\_formation.node\_cleanup.interval = 10  
cluster\_formation.node\_cleanup.only\_log\_warning = false  
cluster\_partition\_handling = autoheal

## queue master locator

queue\_master\_locator=min-masters

## enable guest user

loopback\_users.guest = false  
log.file.level = debug

# kubectl logs awx-0 -n awx awx-rabbit

## 

## ## RabbitMQ 3.7.15. Copyright (C) 2007-2019 Pivotal Software, Inc.

########## Licensed under the MPL. See [https://www.rabbitmq.com/](https://www.rabbitmq.com/)

###### 

########## Logs:

Starting broker…  
2020-01-08 12:20:05.474 [info] \<0.221.0\>  
Starting RabbitMQ 3.7.15 on Erlang 22.0.5  
Copyright (C) 2007-2019 Pivotal Software, Inc.  
Licensed under the MPL. See [https://www.rabbitmq.com/](https://www.rabbitmq.com/)  
2020-01-08 12:20:05.480 [info] \<0.221.0\>  
node : [rabbit@10.140.3.115](mailto:rabbit@10.140.3.115)  
home dir : /var/lib/rabbitmq  
config file(s) : /etc/rabbitmq/rabbitmq.conf  
cookie hash : at619UOZzsenF44tSK3ulA==  
log(s) :   
database dir : /var/lib/rabbitmq/mnesia/rabbit@10.140.3.115  
2020-01-08 12:20:07.381 [info] \<0.221.0\> Running boot step pre\_boot defined by app rabbit  
2020-01-08 12:20:07.381 [info] \<0.221.0\> Running boot step rabbit\_core\_metrics defined by app rabbit  
2020-01-08 12:20:07.382 [info] \<0.221.0\> Running boot step rabbit\_alarm defined by app rabbit  
2020-01-08 12:20:07.386 [info] \<0.229.0\> Memory high watermark set to 3128 MiB (3280291430 bytes) of 7820 MiB (8200728576 bytes) total  
2020-01-08 12:20:07.390 [info] \<0.231.0\> Enabling free disk space monitoring  
2020-01-08 12:20:07.390 [info] \<0.231.0\> Disk free limit set to 50MB  
2020-01-08 12:20:07.393 [info] \<0.221.0\> Running boot step code\_server\_cache defined by app rabbit  
2020-01-08 12:20:07.393 [info] \<0.221.0\> Running boot step file\_handle\_cache defined by app rabbit  
2020-01-08 12:20:07.393 [info] \<0.234.0\> Limiting to approx 1048476 file handles (943626 sockets)  
2020-01-08 12:20:07.393 [info] \<0.235.0\> FHC read buffering: OFF  
2020-01-08 12:20:07.393 [info] \<0.235.0\> FHC write buffering: ON  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Running boot step worker\_pool defined by app rabbit  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Running boot step database defined by app rabbit  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@10.140.3.115 is empty. Assuming we need to join an existing cluster or initialise from scratch…  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Configured peer discovery backend: rabbit\_peer\_discovery\_k8s  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Will try to lock with peer discovery backend rabbit\_peer\_discovery\_k8s  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Peer discovery backend does not support locking, falling back to randomized delay  
2020-01-08 12:20:07.394 [info] \<0.221.0\> Peer discovery backend rabbit\_peer\_discovery\_k8s does not support registration, skipping randomized startup delay.  
2020-01-08 12:20:07.395 [error] \<0.220.0\> CRASH REPORT Process \<0.220.0\> with 0 neighbours exited with reason: no match of right hand value {error,eacces} in rabbit\_peer\_discovery\_k8s:make\_request/0 line 109 in application\_master:init/4 line 138  
2020-01-08 12:20:07.396 [info] \<0.43.0\> Application rabbit exited with reason: no match of right hand value {error,eacces} in rabbit\_peer\_discovery\_k8s:make\_request/0 line 109  
{“Kernel pid terminated”,application\_controller,“{application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{badmatch,{error,eacces}},[{rabbit\_peer\_discovery\_k8s,make\_request,0,[{file,"src/rabbit\_peer\_discovery\_k8s.erl"},{line,109}]},{rabbit\_peer\_discovery\_k8s,list\_nodes,0,[{file,"src/rabbit\_peer\_discovery\_k8s.erl"},{line,55}]},{rabbit\_peer\_discovery,discover\_cluster\_nodes,0,[{file,"src/rabbit\_peer\_discovery.erl"},{line,120}]},{rabbit\_mnesia,init\_from\_config,0,[{file,"src/rabbit\_mnesia.erl"},{line,164}]},{rabbit\_mnesia,init\_with\_lock,3,[{file,"src/rabbit\_mnesia.erl"},{line,144}]},{rabbit\_mnesia,init,0,[{file,"src/rabbit\_mnesia.erl"},{line,111}]},{rabbit\_boot\_steps,‘-run\_step/2-lc$^1/1-1-’,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,55}]},{rabbit\_boot\_steps,run\_step,2,[{file,"src/rabbit\_boot\_steps.erl"},{line,52}]}]}}}}}”}  
Kernel pid terminated (application\_controller) ({application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{badmatch,{error,eacces}},[{rabbit\_peer\_discovery\_k8s,make\_request,0,

Crash dump is being written to: /var/log/rabbitmq/erl\_crash.dump…done

---

<div class="post-metadata">

**Author:** ![Christopher\_Meyers](https://avatars.discourse-cdn.com/v4/letter/c/3e96dc/32.png) [@Christopher\_Meyers](https://forum.ansible.com/u/Christopher_Meyers)\
**Post date:** [January 8, 2020, 2:13pm UTC](https://forum.ansible.com/t/rabbitmq-configuration-in-awx-deployment-on-k8s-cluster/31929/2 "2020-01-08T14:13:04Z")

</div>

The rabbitmq containers will cluster, yes.

What do you mean by “the rabbitmq container fails to deploy”. I presume the error message you pasted its the case when you remove the peer discovery backend.

-Chris

---

<div class="post-metadata">

**Author:** ![Vibin](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vibin](https://forum.ansible.com/u/Vibin)\
**Post date:** [January 8, 2020, 3:27pm UTC](https://forum.ansible.com/t/rabbitmq-configuration-in-awx-deployment-on-k8s-cluster/31929/3 "2020-01-08T15:27:21Z")

</div>

Thanks for the response, Chris.

With the default configuration (peer discovery enabled), the rabbitmq container fails to start/run and goes in to CrashLoopBackOff.

**# grep cluster roles/kubernetes/templates/deployment.yml.j2**  
cluster\_formation.peer\_discovery\_backend = rabbit\_peer\_discovery\_k8s  
cluster\_formation.k8s.host = kubernetes.default.svc  
cluster\_formation.k8s.address\_type = ip  
cluster\_formation.node\_cleanup.interval = 10  
cluster\_formation.node\_cleanup.only\_log\_warning = false  
cluster\_partition\_handling = autoheal

**# kubectl get pods -n awx**  
NAME READY STATUS RESTARTS AGE  
awx-0 3/4 CrashLoopBackOff 6 7m2s

**# kubectl logs awx-0 -n awx awx-rabbit**

## 

## ## RabbitMQ 3.7.15. Copyright (C) 2007-2019 Pivotal Software, Inc.

########## Licensed under the MPL. See [https://www.rabbitmq.com/](https://www.rabbitmq.com/)

###### 

########## Logs:

Starting broker…  
2020-01-08 15:19:04.045 [info] \<0.221.0\>  
Starting RabbitMQ 3.7.15 on Erlang 22.0.5  
Copyright (C) 2007-2019 Pivotal Software, Inc.  
Licensed under the MPL. See [https://www.rabbitmq.com/](https://www.rabbitmq.com/)  
2020-01-08 15:19:04.065 [info] \<0.221.0\>  
node : [rabbit@10.140.3.121](mailto:rabbit@10.140.3.121)  
home dir : /var/lib/rabbitmq  
config file(s) : /etc/rabbitmq/rabbitmq.conf  
cookie hash : at619UOZzsenF44tSK3ulA==  
log(s) :   
database dir : /var/lib/rabbitmq/mnesia/[rabbit@10.140.3.121](mailto:rabbit@10.140.3.121)  
2020-01-08 15:19:05.757 [info] \<0.221.0\> Running boot step pre\_boot defined by app rabbit  
2020-01-08 15:19:05.757 [info] \<0.221.0\> Running boot step rabbit\_core\_metrics defined by app rabbit  
2020-01-08 15:19:05.758 [info] \<0.221.0\> Running boot step rabbit\_alarm defined by app rabbit  
2020-01-08 15:19:05.768 [info] \<0.229.0\> Memory high watermark set to 3128 MiB (3280291430 bytes) of 7820 MiB (8200728576 bytes) total  
2020-01-08 15:19:05.773 [info] \<0.231.0\> Enabling free disk space monitoring  
2020-01-08 15:19:05.773 [info] \<0.231.0\> Disk free limit set to 50MB  
2020-01-08 15:19:05.776 [info] \<0.221.0\> Running boot step code\_server\_cache defined by app rabbit  
2020-01-08 15:19:05.776 [info] \<0.221.0\> Running boot step file\_handle\_cache defined by app rabbit  
2020-01-08 15:19:05.776 [info] \<0.234.0\> Limiting to approx 1048476 file handles (943626 sockets)  
2020-01-08 15:19:05.776 [info] \<0.235.0\> FHC read buffering: OFF  
2020-01-08 15:19:05.776 [info] \<0.235.0\> FHC write buffering: ON  
2020-01-08 15:19:05.777 [info] \<0.221.0\> Running boot step worker\_pool defined by app rabbit  
2020-01-08 15:19:05.777 [info] \<0.221.0\> Running boot step database defined by app rabbit  
2020-01-08 15:19:05.777 [info] \<0.221.0\> Node database directory at /var/lib/rabbitmq/mnesia/[rabbit@10.140.3.121](mailto:rabbit@10.140.3.121) is empty. Assuming we need to join an existing cluster or initialise from scratch…  
2020-01-08 15:19:05.777 [info] \<0.221.0\> Configured peer discovery backend: rabbit\_peer\_discovery\_k8s  
2020-01-08 15:19:05.777 [info] \<0.221.0\> Will try to lock with peer discovery backend rabbit\_peer\_discovery\_k8s  
2020-01-08 15:19:05.777 [info] \<0.221.0\> Peer discovery backend does not support locking, falling back to randomized delay  
2020-01-08 15:19:05.778 [info] \<0.221.0\> Peer discovery backend rabbit\_peer\_discovery\_k8s does not support registration, skipping randomized startup delay.  
2020-01-08 15:19:05.779 [error] \<0.220.0\> CRASH REPORT Process \<0.220.0\> with 0 neighbours exited with reason: no match of right hand value {error,eacces} in rabbit\_peer\_discovery\_k8s:make\_request/0 line 109 in application\_master:init/4 line 138  
2020-01-08 15:19:05.779 [info] \<0.43.0\> Application rabbit exited with reason: no match of right hand value {error,eacces} in rabbit\_peer\_discovery\_k8s:make\_request/0 line 109  
{“Kernel pid terminated”,application\_controller,“{application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{badmatch,{error,eacces}},[{rabbit\_peer\_discovery\_k8s,make\_request,0,[{file,"src/rabbit\_peer\_discovery\_k8s.erl"},{line,109}]},{rabbit\_peer\_discovery\_k8s,list\_nodes,0,[{file,"src/rabbit\_peer\_discovery\_k8s.erl"},{line,55}]},{rabbit\_peer\_discovery,discover\_cluster\_nodes,0,[{file,"src/rabbit\_peer\_discovery.erl"},{line,120}]},{rabbit\_mnesia,init\_from\_config,0,[{file,"src/rabbit\_mnesia.erl"},{line,164}]},{rabbit\_mnesia,init\_with\_lock,3,[{file,"src/rabbit\_mnesia.erl"},{line,144}]},{rabbit\_mnesia,init,0,[{file,"src/rabbit\_mnesia.erl"},{line,111}]},{rabbit\_boot\_steps,‘-run\_step/2-lc$^1/1-1-’,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,55}]},{rabbit\_boot\_steps,run\_step,2,[{file,"src/rabbit\_boot\_steps.erl"},{line,52}]}]}}}}}”}  
Kernel pid terminated (application\_controller) ({application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{badmatch,{error,eacces}},[{rabbit\_peer\_discovery\_k8s,make\_request,0,

Crash dump is being written to: /var/log/rabbitmq/erl\_crash.dump…done

**Inventory file:**

# grep rabbit inventory

# These are the request and limit values for a pod’s container for task/web/rabbitmq/memcached/management.

rabbitmq\_cpu\_limit=1500  
rabbitmq\_mem\_limit=4  
rabbitmq\_user=awx  
rabbitmq\_password=\*\*\*\*\*\*\*\*\*\*\*\*  
rabbitmq\_erlang\_cookie=cookiemonster

Am I doing something wrong, could you please help in identifying/fixing the issue. Are there any other logs/details which might help in diagnosing the problem.

Appreciate your response.

Regards,Vibin

---

<div class="post-metadata">

**Author:** ![Vibin](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vibin](https://forum.ansible.com/u/Vibin)\
**Post date:** [January 10, 2020, 9:51am UTC](https://forum.ansible.com/t/rabbitmq-configuration-in-awx-deployment-on-k8s-cluster/31929/4 "2020-01-10T09:51:12Z")

</div>

Hello Chris,

This issue has been resolved.

We have SELINUX enabled in our environment and that was causing the cluster formation to fail due to the restrictions on the container.

Removed the restrictions on the container by making it privileged fixed the issue. Now everything looks and works fine.

- name: {{ kubernetes\_deployment\_name }}-rabbit  
securityContext:  
privileged: true

Regards,  
Vibin