awx deployment failed in k3s cluster

Archives AWX Project
, ,
1

Hi Team

When I try to deploy the awx in my k3s cluster, awx pod not started and failed to start and got “CrashLoopBackOff”. when I describe the pod I got the event as below.

Events:
Type Reason Age From Message

Normal Scheduled 39m default-scheduler Successfully assigned awx/awx-78ccbd4b9b-r7lsc to xl8phxc360awxtst
Normal Pulled 39m kubelet Container image “quay.io/ansible/awx-ee:latest” already present on machine
Normal Created 39m kubelet Created container init
Normal Started 39m kubelet Started container init
Normal Pulled 39m kubelet Container image “docker.io/redis:7” already present on machine
Normal Created 39m kubelet Created container redis
Normal Started 39m kubelet Started container redis
Normal Pulled 39m kubelet Container image “quay.io/ansible/awx:21.8.0” already present on machine
Normal Created 39m kubelet Created container awx-task
Normal Started 39m kubelet Started container awx-task
Normal Pulled 39m kubelet Container image “quay.io/ansible/awx-ee:latest” already present on machine
Normal Created 39m kubelet Created container awx-ee
Normal Started 39m kubelet Started container awx-ee
Normal Created 38m (x3 over 39m) kubelet Created container awx-web
Normal Started 38m (x3 over 39m) kubelet Started container awx-web
Normal Pulled 37m (x4 over 39m) kubelet Container image “quay.io/ansible/awx:21.8.0” already present on machine
Warning BackOff 4m40s (x139 over 38m) kubelet Back-off restarting failed container

My awx status as below!!!

[kubeadmin@xl8phxc360awxtst ~]$ kubectl -n awx get awx,all,ingress,secrets
NAME AGE
awx.awx.ansible.com/awx 81m

NAME READY STATUS RESTARTS AGE
pod/awx-operator-controller-manager-585df5d96d-hk2xs 2/2 Running 0 94m
pod/awx-postgres-13-0 1/1 Running 0 47m
pod/awx-78ccbd4b9b-r7lsc 3/4 CrashLoopBackOff 12 (4m55s ago) 47m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-operator-controller-manager-metrics-service ClusterIP xx.xx.xx.xx 8443/TCP 94m
service/awx-postgres-13 ClusterIP None 5432/TCP 80m
service/awx-service ClusterIP xx.xx.xx.xx 80/TCP 78m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/awx-operator-controller-manager 1/1 1 1 94m
deployment.apps/awx 0/1 1 0 78m

NAME DESIRED CURRENT READY AGE
replicaset.apps/awx-operator-controller-manager-585df5d96d 1 1 1 94m
replicaset.apps/awx-78ccbd4b9b 1 1 0 78m

NAME READY AGE
statefulset.apps/awx-postgres-13 1/1 80m

NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/awx-ingress traefik xl8phxc360awxtst.ipengine360s1.net xx.xx.xx.xx 80, 443 78m

NAME TYPE DATA AGE
secret/awx-admin-password Opaque 1 81m
secret/awx-postgres-configuration Opaque 6 81m
secret/awx-secret-tls kubernetes.io/tls 2 81m
secret/redhat-operators-pull-secret Opaque 1 81m
secret/awx-receptor-ca kubernetes.io/tls 2 78m
secret/awx-receptor-work-signing Opaque 2 78m
secret/awx-app-credentials Opaque 3 78m
secret/awx-secret-key Opaque 1 80m
secret/awx-broadcast-websocket Opaque 1 80m
[kubeadmin@xl8phxc360awxtst ~]$

Please suggest and hel;p me to resolve the issue

2

sounds like it could be due to insufficient resources to start the container. Does bumping available resources help get the pod started?

take a look at this guide https://komodor.com/learn/how-to-fix-crashloopbackoff-kubernetes-error/

AWX Team

3

Hi Team

We have 12GB free RAM and our server having 8vcpu and we have enough free space in all filesystem including root. Not sure where/what we bump in resources.

4

Please describe the pod so we can see further errors

5

Hi Yen

Here attached output of pod describe and container status

(attachments)

pods_containers_details.txt (15.2 KB)

6

Hi were you able to resolve this issue? If possible can you provide logging details emitted from the awx-web container (if it starts at all)? you can use tool like stern to aggregate all logs emitted from all pods in the awx namespace. With additional info we can probably help figure out the root cause to the crash backoff.

Thanks!

AWX Team