# Error in rabbitmq container of awx pod when trying to install awx v9.2.0 on kubernetes cluster **URL:** \ **Category:** AWX Project\ **Tags:** ubuntu, awx, kubernetes\ **Created:** [March 28, 2020, 2:40pm UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551 "2020-03-28T14:40:07Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![SANCHIT\_KHANDELWAL](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@SANCHIT\_KHANDELWAL](https://forum.ansible.com/u/SANCHIT_KHANDELWAL)\ **Post date:** [March 28, 2020, 2:40pm UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/1 "2020-03-28T14:40:07Z")
Hi All, Hope you are doing well. I have tried installing awx 9.2.0 version on kubernetes on single node(ubuntu machine) and got success. Now when I are trying to install the same version on multi node(1-master 2=worker), the awx-o pod goes into crashloopbackoff state. It gets fails due to awx-rabbit container. Below are the logs for rabbitmq container in the awx pod: 2020-03-28 14:21:15.306 [info] \<0.235.0\> Running boot step pre\_boot defined by app rabbit 2020-03-28 14:21:15.306 [info] \<0.235.0\> Running boot step rabbit\_core\_metrics defined by app rabbit 2020-03-28 14:21:15.307 [info] \<0.235.0\> Running boot step rabbit\_alarm defined by app rabbit 2020-03-28 14:21:15.310 [info] \<0.241.0\> Memory high watermark set to 6418 MiB (6730439065 bytes) of 16046 MiB (16826097664 bytes) total 2020-03-28 14:21:15.313 [info] \<0.243.0\> Enabling free disk space monitoring 2020-03-28 14:21:15.313 [info] \<0.243.0\> Disk free limit set to 50MB 2020-03-28 14:21:15.316 [info] \<0.235.0\> Running boot step code\_server\_cache defined by app rabbit 2020-03-28 14:21:15.316 [info] \<0.235.0\> Running boot step file\_handle\_cache defined by app rabbit 2020-03-28 14:21:15.316 [info] \<0.246.0\> Limiting to approx 1048476 file handles (943626 sockets) 2020-03-28 14:21:15.316 [info] \<0.247.0\> FHC read buffering: OFF 2020-03-28 14:21:15.316 [info] \<0.247.0\> FHC write buffering: ON 2020-03-28 14:21:15.316 [info] \<0.235.0\> Running boot step worker\_pool defined by app rabbit 2020-03-28 14:21:15.316 [info] \<0.236.0\> Will use 8 processes for default worker pool 2020-03-28 14:21:15.316 [info] \<0.236.0\> Starting worker pool ‘worker\_pool’ with 8 processes in it 2020-03-28 14:21:15.317 [info] \<0.235.0\> Running boot step database defined by app rabbit 2020-03-28 14:21:15.317 [info] \<0.235.0\> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@192.168.184.11 is empty. Assuming we need to join an existing cluster or initialise from scratch… 2020-03-28 14:21:15.317 [info] \<0.235.0\> Configured peer discovery backend: rabbit\_peer\_discovery\_k8s 2020-03-28 14:21:15.317 [info] \<0.235.0\> Will try to lock with peer discovery backend rabbit\_peer\_discovery\_k8s 2020-03-28 14:21:15.317 [info] \<0.235.0\> Peer discovery backend does not support locking, falling back to randomized delay 2020-03-28 14:21:15.317 [info] \<0.235.0\> Peer discovery backend rabbit\_peer\_discovery\_k8s does not support registration, skipping randomized startup delay. 2020-03-28 14:21:17.569 [info] \<0.235.0\> Failed to get nodes from k8s - {failed\_connect,[{to\_address,{“10.96.0.1”,443}},{inet,[inet],timeout}]} 2020-03-28 14:21:17.570 [error] \<0.234.0\> CRASH REPORT Process \<0.234.0\> with 0 neighbours exited with reason: no case clause matching {error,“{failed\_connect,[{to\_address,{"10.96.0.1",443}},{inet,[inet],timeout}]}”} in rabbit\_mnesia:init\_from\_config/0 line 167 in application\_master:init/4 line 138 2020-03-28 14:21:17.570 [info] \<0.43.0\> Application rabbit exited with reason: no case clause matching {**error,“{failed\_connect,[{to\_address,{kubernetes.default.svc”,443}},{inet,[inet],timeout}]}"} in rabbit\_mnesia:init\_from\_config/0 line 167** {“Kernel pid terminated”,application\_controller,“{application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{case\_clause,{error,"{failed\_connect,[{to\_address,{\“10.96.0.1\”,443}},{inet,[inet],timeout}]}"}},[{rabbit\_mnesia,init\_from\_config,0,[{file,"src/rabbit\_mnesia.erl"},{line,167}]},{rabbit\_mnesia,init\_with\_lock,3,[{file,"src/rabbit\_mnesia.erl"},{line,147}]},{rabbit\_mnesia,init,0,[{file,"src/rabbit\_mnesia.erl"},{line,114}]},{rabbit\_boot\_steps,‘-run\_step/2-lc$^1/1-1-’,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,55}]},{rabbit\_boot\_steps,run\_step,2,[{file,"src/rabbit\_boot\_steps.erl"},{line,59}]},{rabbit\_boot\_steps,‘-run\_boot\_steps/1-lc$^0/1-0-’,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,28}]},{rabbit\_boot\_steps,run\_boot\_steps,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,29}]},{rabbit,start,2,[{file,"src/rabbit.erl"},{line,937}]}]}}}}}”} Kernel pid terminated (application\_controller) ({application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{case\_clause,{error,"{failed\_connect,[{to\_address,{"10.96.0.1",443} Crash dump is being written to: /var/log/rabbitmq/erl\_crash.dump…done Kindly assist in getting this error resolved. Thanks, Sanchit ---
**Author:** ![Selvam\_Elangovan](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@Selvam\_Elangovan](https://forum.ansible.com/u/Selvam_Elangovan)\ **Post date:** [March 28, 2020, 2:59pm UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/2 "2020-03-28T14:59:30Z")
Can you please post the log from K8. Kubectl events Kubectl describe pod Kubectl logs Thanks. ---
**Author:** ![SANCHIT\_KHANDELWAL](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@SANCHIT\_KHANDELWAL](https://forum.ansible.com/u/SANCHIT_KHANDELWAL)\ **Post date:** [March 28, 2020, 5:40pm UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/3 "2020-03-28T17:40:40Z")
kubectl get events :: LAST SEEN TYPE REASON OBJECT MESSAGE 47m Normal Scheduled pod/ansible-tower-management Successfully assigned awx/ansible-tower-management to ubuntu-worker 47m Normal Pulled pod/ansible-tower-management Container image “ansible/awx\_task:9.2.0” already present on machine 47m Normal Created pod/ansible-tower-management Created container ansible-tower-management 47m Normal Started pod/ansible-tower-management Started container ansible-tower-management 46m Normal Killing pod/ansible-tower-management Stopping container ansible-tower-management 36m Normal Scheduled pod/ansible-tower-management Successfully assigned awx/ansible-tower-management to ubuntu-worker 36m Normal Pulled pod/ansible-tower-management Container image “ansible/awx\_task:9.2.0” already present on machine 36m Normal Created pod/ansible-tower-management Created container ansible-tower-management 36m Normal Started pod/ansible-tower-management Started container ansible-tower-management 36m Normal Killing pod/ansible-tower-management Stopping container ansible-tower-management 47m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 47m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 47m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 47m Normal Created pod/awx-0 Created container awx-web 47m Normal Started pod/awx-0 Started container awx-web 47m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 47m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 47m Normal Created pod/awx-0 Created container awx-celery 47m Normal Started pod/awx-0 Started container awx-celery 47m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 46m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 46m Normal Created pod/awx-0 Created container awx-rabbit 46m Normal Started pod/awx-0 Started container awx-rabbit 47m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 47m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 47m Normal Created pod/awx-0 Created container awx-memcached 47m Normal Started pod/awx-0 Started container awx-memcached 47m Warning Unhealthy pod/awx-0 Readiness probe errored: rpc error: code = Unknown desc = container not running (b2f64e5a0339a1dca0ffed37d9b9dca9e0781851c5ed19c85be862a87c808004) 46m Normal Killing pod/awx-0 Stopping container awx-web 46m Normal Killing pod/awx-0 Stopping container awx-rabbit 46m Normal Killing pod/awx-0 Stopping container awx-memcached 46m Normal Killing pod/awx-0 Stopping container awx-celery 46m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 46m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 46m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 46m Normal Created pod/awx-0 Created container awx-web 46m Normal Started pod/awx-0 Started container awx-web 46m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 46m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 46m Normal Created pod/awx-0 Created container awx-celery 46m Normal Started pod/awx-0 Started container awx-celery 45m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 45m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 45m Normal Created pod/awx-0 Created container awx-rabbit 46m Normal Started pod/awx-0 Started container awx-rabbit 46m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 46m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 46m Normal Created pod/awx-0 Created container awx-memcached 46m Normal Started pod/awx-0 Started container awx-memcached 41m Warning BackOff pod/awx-0 Back-off restarting failed container 36m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 36m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 36m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 36m Normal Created pod/awx-0 Created container awx-web 36m Normal Started pod/awx-0 Started container awx-web 36m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 36m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 36m Normal Created pod/awx-0 Created container awx-celery 36m Normal Started pod/awx-0 Started container awx-celery 36m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 36m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 36m Normal Created pod/awx-0 Created container awx-rabbit 36m Normal Started pod/awx-0 Started container awx-rabbit 36m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 36m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 36m Normal Created pod/awx-0 Created container awx-memcached 36m Normal Started pod/awx-0 Started container awx-memcached 35m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 35m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 35m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 35m Normal Created pod/awx-0 Created container awx-web 35m Normal Started pod/awx-0 Started container awx-web 35m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 35m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 35m Normal Created pod/awx-0 Created container awx-celery 35m Normal Started pod/awx-0 Started container awx-celery 35m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 34m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 34m Normal Created pod/awx-0 Created container awx-rabbit 34m Normal Started pod/awx-0 Started container awx-rabbit 35m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 35m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 35m Normal Created pod/awx-0 Created container awx-memcached 35m Normal Started pod/awx-0 Started container awx-memcached 30m Warning BackOff pod/awx-0 Back-off restarting failed container 30m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 30m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 30m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 30m Normal Created pod/awx-0 Created container awx-web 30m Normal Started pod/awx-0 Started container awx-web 30m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 30m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 30m Normal Created pod/awx-0 Created container awx-celery 30m Normal Started pod/awx-0 Started container awx-celery 29m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 29m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 29m Normal Created pod/awx-0 Created container awx-rabbit 29m Normal Started pod/awx-0 Started container awx-rabbit 29m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 29m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 29m Normal Created pod/awx-0 Created container awx-memcached 29m Normal Started pod/awx-0 Started container awx-memcached 25m Warning BackOff pod/awx-0 Back-off restarting failed container 19m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 19m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 19m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 19m Normal Created pod/awx-0 Created container awx-web 19m Normal Started pod/awx-0 Started container awx-web 19m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 19m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 19m Normal Created pod/awx-0 Created container awx-celery 19m Normal Started pod/awx-0 Started container awx-celery 19m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 19m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 19m Normal Created pod/awx-0 Created container awx-rabbit 19m Normal Started pod/awx-0 Started container awx-rabbit 19m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 19m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 19m Normal Created pod/awx-0 Created container awx-memcached 19m Normal Started pod/awx-0 Started container awx-memcached 19m Warning BackOff pod/awx-0 Back-off restarting failed container 16m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 16m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 15m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 15m Normal Created pod/awx-0 Created container awx-web 15m Normal Started pod/awx-0 Started container awx-web 15m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 15m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 15m Normal Created pod/awx-0 Created container awx-celery 15m Normal Started pod/awx-0 Started container awx-celery 15m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 15m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 15m Normal Created pod/awx-0 Created container awx-rabbit 15m Normal Started pod/awx-0 Started container awx-rabbit 15m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 15m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 15m Normal Created pod/awx-0 Created container awx-memcached 15m Normal Started pod/awx-0 Started container awx-memcached 15m Warning BackOff pod/awx-0 Back-off restarting failed container 10m Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 10m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 10m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 10m Normal Created pod/awx-0 Created container awx-web 10m Normal Started pod/awx-0 Started container awx-web 10m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 10m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 10m Normal Created pod/awx-0 Created container awx-celery 10m Normal Started pod/awx-0 Started container awx-celery 10m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 10m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 10m Normal Created pod/awx-0 Created container awx-rabbit 10m Normal Started pod/awx-0 Started container awx-rabbit 10m Normal Pulling pod/awx-0 Pulling image “memcached:latest” 10m Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 10m Normal Created pod/awx-0 Created container awx-memcached 10m Normal Started pod/awx-0 Started container awx-memcached 10m Warning BackOff pod/awx-0 Back-off restarting failed container 9m4s Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 9m3s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 9m Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 9m Normal Created pod/awx-0 Created container awx-web 9m Normal Started pod/awx-0 Started container awx-web 9m Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 8m57s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 8m57s Normal Created pod/awx-0 Created container awx-celery 8m57s Normal Started pod/awx-0 Started container awx-celery 8m18s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 8m15s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 8m15s Normal Created pod/awx-0 Created container awx-rabbit 8m43s Normal Started pod/awx-0 Started container awx-rabbit 8m54s Normal Pulling pod/awx-0 Pulling image “memcached:latest” 8m50s Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 8m50s Normal Created pod/awx-0 Created container awx-memcached 8m50s Normal Started pod/awx-0 Started container awx-memcached 8m32s Warning BackOff pod/awx-0 Back-off restarting failed container 4m16s Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 4m15s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 4m12s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 4m12s Normal Created pod/awx-0 Created container awx-web 4m12s Normal Started pod/awx-0 Started container awx-web 4m12s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 4m8s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 4m8s Normal Created pod/awx-0 Created container awx-celery 4m8s Normal Started pod/awx-0 Started container awx-celery 3m36s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 3m33s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 3m33s Normal Created pod/awx-0 Created container awx-rabbit 3m33s Normal Started pod/awx-0 Started container awx-rabbit 4m4s Normal Pulling pod/awx-0 Pulling image “memcached:latest” 4m1s Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 4m1s Normal Created pod/awx-0 Created container awx-memcached 4m Normal Started pod/awx-0 Started container awx-memcached 3m45s Warning BackOff pod/awx-0 Back-off restarting failed container 3m14s Normal Scheduled pod/awx-0 Successfully assigned awx/awx-0 to ubuntu-worker 3m13s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_web:9.2.0” 3m8s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_web:9.2.0” 3m8s Normal Created pod/awx-0 Created container awx-web 3m8s Normal Started pod/awx-0 Started container awx-web 3m8s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_task:9.2.0” 3m5s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_task:9.2.0” 3m5s Normal Created pod/awx-0 Created container awx-celery 3m5s Normal Started pod/awx-0 Started container awx-celery 2m31s Normal Pulling pod/awx-0 Pulling image “ansible/awx\_rabbitmq:3.7.21” 2m28s Normal Pulled pod/awx-0 Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” 2m28s Normal Created pod/awx-0 Created container awx-rabbit 2m51s Normal Started pod/awx-0 Started container awx-rabbit 3m1s Normal Pulling pod/awx-0 Pulling image “memcached:latest” 2m57s Normal Pulled pod/awx-0 Successfully pulled image “memcached:latest” 2m57s Normal Created pod/awx-0 Created container awx-memcached 2m57s Normal Started pod/awx-0 Started container awx-memcached 2m43s Warning BackOff pod/awx-0 Back-off restarting failed container 48m Warning FailedScheduling pod/awx-postgresql-postgresql-0 error while running “VolumeBinding” filter plugin for pod “awx-postgresql-postgresql-0”: pod has unbound immediate PersistentVolumeClaims 48m Normal Scheduled pod/awx-postgresql-postgresql-0 Successfully assigned awx/awx-postgresql-postgresql-0 to ubuntu-worker 48m Normal Pulling pod/awx-postgresql-postgresql-0 Pulling image “[docker.io/bitnami/minideb:stretch](http://docker.io/bitnami/minideb:stretch)” 48m Normal Pulled pod/awx-postgresql-postgresql-0 Successfully pulled image “[docker.io/bitnami/minideb:stretch](http://docker.io/bitnami/minideb:stretch)” 48m Normal Created pod/awx-postgresql-postgresql-0 Created container init-chmod-data 48m Normal Started pod/awx-postgresql-postgresql-0 Started container init-chmod-data 48m Normal Pulled pod/awx-postgresql-postgresql-0 Container image “[docker.io/bitnami/postgresql:11.6.0-debian-10-r5](http://docker.io/bitnami/postgresql:11.6.0-debian-10-r5)” already present on machine 48m Normal Created pod/awx-postgresql-postgresql-0 Created container awx-postgresql 48m Normal Started pod/awx-postgresql-postgresql-0 Started container awx-postgresql 48m Normal SuccessfulCreate statefulset/awx-postgresql-postgresql create Claim data-awx-postgresql-postgresql-0 Pod awx-postgresql-postgresql-0 in StatefulSet awx-postgresql-postgresql success 48m Normal SuccessfulCreate statefulset/awx-postgresql-postgresql create Pod awx-postgresql-postgresql-0 in StatefulSet awx-postgresql-postgresql successful 46m Normal SuccessfulCreate statefulset/awx create Pod awx-0 in StatefulSet awx successful 46m Normal SuccessfulDelete statefulset/awx delete Pod awx-0 in StatefulSet awx successful 36m Warning FailedCreate statefulset/awx create Pod awx-0 in StatefulSet awx failed error: The POST operation against Pod could not be completed at this time, please try again. 3m14s Normal SuccessfulCreate statefulset/awx create Pod awx-0 in StatefulSet awx successful 36m Normal SuccessfulDelete statefulset/awx delete Pod awx-0 in StatefulSet awx successful kubectl describe pod awx-0 -n awx :: Name: awx-0 Namespace: awx Priority: 0 Node: ubuntu-worker/192.168.101.245 Start Time: Sun, 29 Mar 2020 04:31:14 +1100 Labels: app=awx controller-revision-hash=awx-7cd4d674b8 name=awx-web-deploy service=django [statefulset.kubernetes.io/pod-name=awx-0](http://statefulset.kubernetes.io/pod-name=awx-0) Annotations: [cni.projectcalico.org/podIP:](http://cni.projectcalico.org/podIP:) 192.168.184.15/32 Status: Running IP: 192.168.184.15 IPs: IP: 192.168.184.15 Controlled By: StatefulSet/awx Containers: awx-web: Container ID: docker://0a03d868b5d9c724f2c1dac4db5b351f4d1a7d84c95633f5333a89d51c503ef6 Image: ansible/awx\_web:9.2.0 Image ID: docker-pullable://ansible/awx\_web@sha256:57232e6820eb1bfad3c6910bd993e1a8c0d644c0c0fcfb75a25a0b51b3d6fdec Port: 8052/TCP Host Port: 0/TCP State: Running Started: Sun, 29 Mar 2020 04:31:20 +1100 Ready: True Restart Count: 0 Requests: cpu: 500m memory: 1Gi Environment: Mounts: /etc/nginx/nginx.conf from awx-nginx-config (ro,path=“nginx.conf”) /etc/tower/SECRET\_KEY from awx-secret-key (ro,path=“SECRET\_KEY”) /etc/tower/conf.d/ from awx-application-credentials (ro) /etc/tower/settings.py from awx-application-config (ro,path=“settings.py”) /var/run/secrets/kubernetes.io/serviceaccount from awx-token-hj4g4 (ro) awx-celery: Container ID: docker://c260b524f32bb4de0df21934ce19a4c6ee669b4bab5a620cbd8b234024f39947 Image: ansible/awx\_task:9.2.0 Image ID: docker-pullable://ansible/awx\_task@sha256:6dd8b36faecd4a522ee8dbff8c24cefff4826ff7c91da2653c5bb28bab8c1e66 Port: Host Port: Command: /usr/bin/launch\_awx\_task.sh State: Running Started: Sun, 29 Mar 2020 04:31:23 +1100 Ready: True Restart Count: 0 Requests: cpu: 1500m memory: 2Gi Environment: AWX\_SKIP\_MIGRATIONS: 1 Mounts: /etc/tower/SECRET\_KEY from awx-secret-key (ro,path=“SECRET\_KEY”) /etc/tower/conf.d/ from awx-application-credentials (ro) /etc/tower/settings.py from awx-application-config (ro,path=“settings.py”) /var/run/secrets/kubernetes.io/serviceaccount from awx-token-hj4g4 (ro) awx-rabbit: Container ID: docker://419fb0e3cef70683216e8c1e704953aec8b2b288c0008ef091255514c60cee75 Image: ansible/awx\_rabbitmq:3.7.21 Image ID: docker-pullable://ansible/awx\_rabbitmq@sha256:fdc1c4cde8c7de4192ef981aef57908363ca2f6b8bf7e7aec24873024c972d03 Ports: 15672/TCP, 5672/TCP Host Ports: 0/TCP, 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 29 Mar 2020 04:35:02 +1100 Finished: Sun, 29 Mar 2020 04:35:09 +1100 Ready: False Restart Count: 5 Requests: cpu: 500m memory: 2Gi Liveness: exec [/usr/local/bin/healthchecks/rabbit\_health\_node.py] delay=30s timeout=10s period=10s #success=1 #failure=3 Readiness: exec [/usr/local/bin/healthchecks/rabbit\_health\_node.py] delay=10s timeout=10s period=10s #success=1 #failure=3 Environment: MY\_POD\_IP: (v1:status.podIP) RABBITMQ\_USE\_LONGNAME: true RABBITMQ\_NODENAME: rabbit@$(MY\_POD\_IP) RABBITMQ\_ERLANG\_COOKIE: \ Optional: false K8S\_SERVICE\_NAME: rabbitmq RABBITMQ\_USER: awx RABBITMQ\_PASSWORD: \ Optional: false Mounts: /etc/rabbitmq from rabbitmq-config (rw) /usr/local/bin/healthchecks from rabbitmq-healthchecks (rw) /var/run/secrets/kubernetes.io/serviceaccount from awx-token-hj4g4 (ro) awx-memcached: Container ID: docker://88eb3ed91dce786a16d64ade1504f513f7c230f48af6a36c8e5c9be47946ecc6 Image: memcached:latest Image ID: docker-pullable://memcached@sha256:6ce1e76815f51d9be35f86b3015604134ee781ce26acb3d695ce4388123a422e Port: Host Port: State: Running Started: Sun, 29 Mar 2020 04:31:31 +1100 Ready: True Restart Count: 0 Requests: cpu: 500m memory: 1Gi Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from awx-token-hj4g4 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: awx-application-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: awx-config Optional: false awx-nginx-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: awx-config Optional: false awx-application-credentials: Type: Secret (a volume populated by a Secret) SecretName: awx-secrets Optional: false awx-secret-key: Type: Secret (a volume populated by a Secret) SecretName: awx-secrets Optional: false rabbitmq-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: rabbitmq-config Optional: false rabbitmq-healthchecks: Type: ConfigMap (a volume populated by a ConfigMap) Name: awx-healthchecks Optional: false awx-token-hj4g4: Type: Secret (a volume populated by a Secret) SecretName: awx-token-hj4g4 Optional: false QoS Class: Burstable Node-Selectors: Tolerations: [node.kubernetes.io/not-ready:NoExecute](http://node.kubernetes.io/not-ready:NoExecute) for 300s [node.kubernetes.io/unreachable:NoExecute](http://node.kubernetes.io/unreachable:NoExecute) for 300s Events: Type Reason Age From Message * * * Normal Scheduled 5m22s default-scheduler Successfully assigned awx/awx-0 to ubuntu-worker Normal Pulling 5m21s kubelet, ubuntu-worker Pulling image “ansible/awx\_web:9.2.0” Normal Pulled 5m16s kubelet, ubuntu-worker Successfully pulled image “ansible/awx\_web:9.2.0” Normal Created 5m16s kubelet, ubuntu-worker Created container awx-web Normal Started 5m16s kubelet, ubuntu-worker Started container awx-web Normal Pulling 5m16s kubelet, ubuntu-worker Pulling image “ansible/awx\_task:9.2.0” Normal Pulled 5m13s kubelet, ubuntu-worker Successfully pulled image “ansible/awx\_task:9.2.0” Normal Created 5m13s kubelet, ubuntu-worker Created container awx-celery Normal Started 5m13s kubelet, ubuntu-worker Started container awx-celery Normal Pulling 5m9s kubelet, ubuntu-worker Pulling image “memcached:latest” Normal Started 5m5s kubelet, ubuntu-worker Started container awx-memcached Normal Created 5m5s kubelet, ubuntu-worker Created container awx-memcached Normal Pulled 5m5s kubelet, ubuntu-worker Successfully pulled image “memcached:latest” Normal Started 4m59s (x2 over 5m9s) kubelet, ubuntu-worker Started container awx-rabbit Normal Pulling 4m39s (x3 over 5m13s) kubelet, ubuntu-worker Pulling image “ansible/awx\_rabbitmq:3.7.21” Normal Created 4m36s (x3 over 5m9s) kubelet, ubuntu-worker Created container awx-rabbit Normal Pulled 4m36s (x3 over 5m10s) kubelet, ubuntu-worker Successfully pulled image “ansible/awx\_rabbitmq:3.7.21” Warning BackOff 7s (x24 over 4m52s) kubelet, ubuntu-worker Back-off restarting failed container kubectl logs awx-0 awx-rabbit :: 2020-03-28 17:38:08.297 [info] \<0.8.0\> Feature flags: list of feature flags found: 2020-03-28 17:38:08.297 [info] \<0.8.0\> Feature flags: feature flag states written to disk: yes 2020-03-28 17:38:08.327 [info] \<0.235.0\> Starting RabbitMQ 3.7.21 on Erlang 22.1.7 Copyright (C) 2007-2019 Pivotal Software, Inc. Licensed under the MPL. See [https://www.rabbitmq.com/](https://www.rabbitmq.com/) ## ## ## RabbitMQ 3.7.21. Copyright (C) 2007-2019 Pivotal Software, Inc. ########## Licensed under the MPL. See [https://www.rabbitmq.com/](https://www.rabbitmq.com/) ###### ########## Logs: Starting broker… 2020-03-28 17:38:08.327 [info] \<0.235.0\> node : [rabbit@192.168.184.15](mailto:rabbit@192.168.184.15) home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.conf cookie hash : at619UOZzsenF44tSK3ulA== log(s) : database dir : /var/lib/rabbitmq/mnesia/rabbit@192.168.184.15 2020-03-28 14:21:15.306 [info] \<0.235.0\> Running boot step pre\_boot defined by app rabbit 2020-03-28 14:21:15.306 [info] \<0.235.0\> Running boot step rabbit\_core\_metrics defined by app rabbit 2020-03-28 14:21:15.307 [info] \<0.235.0\> Running boot step rabbit\_alarm defined by app rabbit 2020-03-28 14:21:15.310 [info] \<0.241.0\> Memory high watermark set to 6418 MiB (6730439065 bytes) of 16046 MiB (16826097664 bytes) total 2020-03-28 14:21:15.313 [info] \<0.243.0\> Enabling free disk space monitoring 2020-03-28 14:21:15.313 [info] \<0.243.0\> Disk free limit set to 50MB 2020-03-28 14:21:15.316 [info] \<0.235.0\> Running boot step code\_server\_cache defined by app rabbit 2020-03-28 14:21:15.316 [info] \<0.235.0\> Running boot step file\_handle\_cache defined by app rabbit 2020-03-28 14:21:15.316 [info] \<0.246.0\> Limiting to approx 1048476 file handles (943626 sockets) 2020-03-28 14:21:15.316 [info] \<0.247.0\> FHC read buffering: OFF 2020-03-28 14:21:15.316 [info] \<0.247.0\> FHC write buffering: ON 2020-03-28 14:21:15.316 [info] \<0.235.0\> Running boot step worker\_pool defined by app rabbit 2020-03-28 14:21:15.316 [info] \<0.236.0\> Will use 8 processes for default worker pool 2020-03-28 14:21:15.316 [info] \<0.236.0\> Starting worker pool ‘worker\_pool’ with 8 processes in it 2020-03-28 14:21:15.317 [info] \<0.235.0\> Running boot step database defined by app rabbit 2020-03-28 14:21:15.317 [info] \<0.235.0\> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@192.168.184.11 is empty. Assuming we need to join an existing cluster or initialise from scratch… 2020-03-28 14:21:15.317 [info] \<0.235.0\> Configured peer discovery backend: rabbit\_peer\_discovery\_k8s 2020-03-28 14:21:15.317 [info] \<0.235.0\> Will try to lock with peer discovery backend rabbit\_peer\_discovery\_k8s 2020-03-28 14:21:15.317 [info] \<0.235.0\> Peer discovery backend does not support locking, falling back to randomized delay 2020-03-28 14:21:15.317 [info] \<0.235.0\> Peer discovery backend rabbit\_peer\_discovery\_k8s does not support registration, skipping randomized startup delay. 2020-03-28 14:21:17.569 [info] \<0.235.0\> Failed to get nodes from k8s - {failed\_connect,[{to\_address,{“10.96.0.1”,443}},{inet,[inet],timeout}]} 2020-03-28 14:21:17.570 [error] \<0.234.0\> CRASH REPORT Process \<0.234.0\> with 0 neighbours exited with reason: no case clause matching {error,“{failed\_connect,[{to\_address,{"10.96.0.1",443}},{inet,[inet],timeout}]}”} in rabbit\_mnesia:init\_from\_config/0 line 167 in application\_master:init/4 line 138 2020-03-28 14:21:17.570 [info] \<0.43.0\> Application rabbit exited with reason: no case clause matching {**error,“{failed\_connect,[{to\_address,{kubernetes.default.svc”,443}},{inet,[inet],timeout}]}"} in rabbit\_mnesia:init\_from\_config/0 line 167** {“Kernel pid terminated”,application\_controller,“{application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{case\_clause,{error,"{failed\_connect,[{to\_address,{\“10.96.0.1\”,443}},{inet,[inet],timeout}]}"}},[{rabbit\_mnesia,init\_from\_config,0,[{file,"src/rabbit\_mnesia.erl"},{line,167}]},{rabbit\_mnesia,init\_with\_lock,3,[{file,"src/rabbit\_mnesia.erl"},{line,147}]},{rabbit\_mnesia,init,0,[{file,"src/rabbit\_mnesia.erl"},{line,114}]},{rabbit\_boot\_steps,‘-run\_step/2-lc$^1/1-1-’,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,55}]},{rabbit\_boot\_steps,run\_step,2,[{file,"src/rabbit\_boot\_steps.erl"},{line,59}]},{rabbit\_boot\_steps,‘-run\_boot\_steps/1-lc$^0/1-0-’,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,28}]},{rabbit\_boot\_steps,run\_boot\_steps,1,[{file,"src/rabbit\_boot\_steps.erl"},{line,29}]},{rabbit,start,2,[{file,"src/rabbit.erl"},{line,937}]}]}}}}}”} Kernel pid terminated (application\_controller) ({application\_start\_failure,rabbit,{bad\_return,{{rabbit,start,[normal,]},{‘EXIT’,{{case\_clause,{error,"{failed\_connect,[{to\_address,{"10.96.0.1",443} Crash dump is being written to: /var/log/rabbitmq/erl\_crash.dump…done ---
**Author:** ![Selvam\_Elangovan](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@Selvam\_Elangovan](https://forum.ansible.com/u/Selvam_Elangovan)\ **Post date:** [March 28, 2020, 7:50pm UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/4 "2020-03-28T19:50:25Z")
Kindly share the below output kubectl get all -n awx kubectl -n awx get pods -o wide I think you are using pgsql within the kubernetes instead of external. Did you follow the steps for installing pgsql in kubernetes in the github install page? [https://github.com/ansible/awx/blob/devel/INSTALL.md](https://github.com/ansible/awx/blob/devel/INSTALL.md) also please share: cat inventory |grep -i -v “^#” ---
**Author:** ![SANCHIT\_KHANDELWAL](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@SANCHIT\_KHANDELWAL](https://forum.ansible.com/u/SANCHIT_KHANDELWAL)\ **Post date:** [March 29, 2020, 6:53am UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/5 "2020-03-29T06:53:48Z")
Hi Selvam, The problem is not related to postgresql…postgresql pod is already in running state. The problem is in the rabbit container(awx-rabbit) of awx-0 pod which is terminating again n again thereby causing the pod to go into CrashLoopBackOff state. kubectl get all -n awx :: NAME READY STATUS RESTARTS AGE pod/awx-0 3/4 CrashLoopBackOff 3 2m2s pod/awx-postgresql-postgresql-0 1/1 Running 0 4m8s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-postgresql ClusterIP 10.104.27.52 5432/TCP 4m8s service/awx-postgresql-headless ClusterIP None 5432/TCP 4m8s service/awx-rmq-mgmt ClusterIP 10.96.131.35 15672/TCP 3m5s service/awx-web-svc NodePort 10.111.218.39 80:32299/TCP 3m5s service/rabbitmq NodePort 10.98.186.33 15672:31962/TCP,5672:31225/TCP 3m5s NAME READY AGE statefulset.apps/awx 0/1 3m5s statefulset.apps/awx-postgresql-postgresql 1/1 4m8s kubectl -n awx get pods -o wide:: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES awx-0 3/4 CrashLoopBackOff 3 2m45s 192.168.184.19 ubuntu-worker awx-postgresql-postgresql-0 1/1 Running 0 4m51s 192.168.184.17 ubuntu-worker cat inventory |grep -i -v “^#” :: dockerhub\_base=ansible kubernetes\_context=kubeawx kubernetes\_namespace=awx pg\_persistence\_storageClass=standard awx\_task\_hostname=awx awx\_web\_hostname=awxweb postgres\_data\_dir=“~/.awx/pgdocker” host\_port=80 host\_port\_ssl=443 docker\_compose\_dir=“~/.awx/awxcompose” pg\_username=awx pg\_password=awxpass pg\_database=awx pg\_port=5432 rabbitmq\_password=awxpass rabbitmq\_erlang\_cookie=cookiemonster admin\_user=admin admin\_password=password create\_preload\_data=True secret\_key=awxsecret Thanks, Sanchit ---
**Author:** ![Selvam\_Elangovan](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@Selvam\_Elangovan](https://forum.ansible.com/u/Selvam_Elangovan)\ **Post date:** [March 29, 2020, 7:22am UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/6 "2020-03-29T07:22:25Z")
Thanks. Please remove the below or put # front of the line then try to install it again. host\_port\_ssl=443 I do see some connection error while trying to reach 443. Hence please put # docker\_compose\_dir=“~/.awx/awxcompose” – This is not required in kubernetes install. put # remove all the awx pods and services then try it again with rerunning ansible install playbook. Thanks. ---
**Author:** ![SANCHIT\_KHANDELWAL](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@SANCHIT\_KHANDELWAL](https://forum.ansible.com/u/SANCHIT_KHANDELWAL)\ **Post date:** [March 29, 2020, 1:56pm UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/7 "2020-03-29T13:56:21Z")
Hi Selvam, We have tried implementing the below changes/suggestions but no success. Thanks, Sanchit ---
**Author:** ![Ajitkumar\_Tanwade](https://avatars.discourse-cdn.com/v4/letter/a/bbce88/32.png) [@Ajitkumar\_Tanwade](https://forum.ansible.com/u/Ajitkumar_Tanwade)\ **Post date:** [March 30, 2020, 3:46am UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/8 "2020-03-30T03:46:44Z")
Hi All, We are also not able to install AWX on Kubernetes (multinode). Tried with version 9.3.0 first, however, there are some issues with it. Like not able to sync the Git repo. Considering it is a major release, went with 9.1.0 however it got stuck at Migrate database task. Then tried to install 9.2.0, it is stuck at RabbitMQ.Giving below error **Failed to get nodes from k8s - {failed\_connect,[{to\_address,{“kubernetes.default”,443}}, {inet,[inet],timeout}]}** Can you please help? It looks like the issue with cluster\_formation.peer\_discovery\_backend settings, something is missing here. Has anyone installed 9.1.0 or 9.2.0 on Kubernetes (multi-node)? Regards, Ajit ---
**Author:** ![Selvam\_Elangovan](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@Selvam\_Elangovan](https://forum.ansible.com/u/Selvam_Elangovan)\ **Post date:** [March 30, 2020, 3:53am UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/9 "2020-03-30T03:53:42Z")
I did but with external postgres. ---
**Author:** ![Selvam\_Elangovan](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@Selvam\_Elangovan](https://forum.ansible.com/u/Selvam_Elangovan)\ **Post date:** [March 30, 2020, 3:56am UTC](https://forum.ansible.com/t/error-in-rabbitmq-container-of-awx-pod-when-trying-to-install-awx-v9-2-0-on-kubernetes-cluster/32551/10 "2020-03-30T03:56:34Z")
Is it giving same error. Can you please send me the logs.