Install AWX on k8s cluster

Get Help
, , ,
1

I am facing the issue to install awx 23.0.0 with operator 2.10. I am getting below error in init-projects container of AWX. however awx-operator pod is running fine.

error:
chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted
chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

awx instance yaml file. also please not that I am using external postgres database.

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awxprod
namespace: awx
spec:
image_version: 23.6.0
service_type: ClusterIP
ingress_type: ingress
hostname: awx.k8s.demo.com
projects_persistence: true
projects_storage_class: isilon
postgres_configuration_secret: awxprod-postgres-configuration

2

It looks like this PR may be related.

3

I am still facing same issue

init-projects:
Container ID: docker://11b5655e36a06ba93ae1bb1ede7a3dabf23760ef17d20d15495024e5b68c0a5c
Image: Quay
Image ID: docker-pullable://quay.io/centos/centos@sha256:bf1c623981405eba655b9b633bdf17f2230ff4e9d93ccf59ad148e031976dde0
Port:
Host Port:
Command:
/bin/sh
-c
chmod 775 /var/lib/awx/projects
chgrp 0 /var/lib/awx/projects

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 18 Jan 2024 05:01:17 +0000
  Finished:     Thu, 18 Jan 2024 05:01:17 +0000
Ready:          False
Restart Count:  5
Requests:
  cpu:     100m
  memory:  128Mi
Environment:
  MY_POD_NAME:  awxprod-web-6bdb8bccd7-599ct (v1:metadata.name)
Mounts:
  /var/lib/awx/projects from awxprod-projects (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9g6lx (ro)

Normal Started 3m52s (x5 over 5m15s) kubelet Started container init-projects
Warning BackOff 12s (x25 over 5m14s) kubelet Back-off restarting failed container init-projects in pod awxprod-web-6bdb8bccd7-599ct_awx(635e6c57-b523-4549-b802-99

$ kubectl26 logs awxprod-web-6bdb8bccd7-599ct -n awx -c init-projects
chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted
chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

4 
projects_storage_class: isilon

If this is NFS based storage, NFS service on Isilon has root squash enabled by default, so chmod or chgrp may be failed for root user. Also check rootClientEnabled in StorageClass definition.

I’m not familiar with detailed configuration on Isilon, so I don’t know how to give minimal privileges/permissions only to certain PVs, but anyway, I think it’s a combination issue with the storage configuration and implementation in operator.

1 Like
5

below is the setting in storageclass

RootClientEnabled=false

6

Hi,

thanks for your help, I was able to fix the issue on storage side and now AWX instance deployed without any issue.

Thanks,
Dinesh K.

1 Like
7

Hello @dinesh6478 ,
How do you fix. Because we are getting crash loopback for web containers. Remaining postgres and task pods created and running. Thsnks

8

Hello @kurokobo ,
good day. We needyour help. We are deploying awx 2.10.0 in k8s cluster 1.26 with nfs. Our awx task and postgres created. But we pods getting crash loopback errors. Getting superwatcher terminated errors. Dig the logs found that thesee was some db connection issue. Getting bad name or connection unknown

9

Hi, if your issue seems to have different cause from this topic, please create a new topic with detailed logs and output of the command to investigate that you’ve made.

1 Like
10

Hi @Veeralakumar

I was getting below specific error which I resolved by fixing permission on storage side. what errors you are getting in your installation ?

chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted
chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

11

Installation completed. Now i am getting error /var/lib/awx permission denied. Error no 13. I hope i need to do some settings for the absolute path. My existing awx cluster /var/lib/awx/projects folder is read writable using ansible playbooks.

With thanks
Veera

12

like @kurokobo mentioned your issue seems to be different from @dinesh6478’s problem

would u mind opening a different post to discuss your issue?

also in your new post please use kubectl describe pod to show which container is crashing and than use kubectl log -p to get the log from the crashed container