# Install AWX on k8s cluster

**URL:** <https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351>\
**Category:** Get Help\
**Tags:** kubernetes, resolved, collections, awx\
**Created:** [January 17, 2024, 2:49pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351 "2024-01-17T14:49:04Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dinesh6478](https://avatars.discourse-cdn.com/v4/letter/d/46a35a/32.png) [@dinesh6478](https://forum.ansible.com/u/dinesh6478)\
**Post date:** [January 17, 2024, 2:49pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/1 "2024-01-17T14:49:04Z")

</div>

I am facing the issue to install awx 23.0.0 with operator 2.10. I am getting below error in init-projects container of AWX. however awx-operator pod is running fine.

**error:**  
chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted  
chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

awx instance yaml file. also please not that I am using external postgres database.

* * *

apiVersion: [awx.ansible.com/v1beta1](http://awx.ansible.com/v1beta1)  
kind: AWX  
metadata:  
name: awxprod  
namespace: awx  
spec:  
image\_version: 23.6.0  
service\_type: ClusterIP  
ingress\_type: ingress  
hostname: [awx.k8s.demo.com](http://awx.k8s.demo.com)  
projects\_persistence: true  
projects\_storage\_class: isilon  
postgres\_configuration\_secret: awxprod-postgres-configuration

---

<div class="post-metadata">

**Author:** ![rooftopcellist](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rooftopcellist/32/97_2.png) [@rooftopcellist](https://forum.ansible.com/u/rooftopcellist)\
**Post date:** [January 17, 2024, 6:30pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/2 "2024-01-17T18:30:04Z")

</div>

It looks like this PR may be related.

- [FIx permissions /var/lib/awx/projects when projects\_persistence is true. by paytroff · Pull Request #1654 · ansible/awx-operator · GitHub](https://github.com/ansible/awx-operator/pull/1654)

---

<div class="post-metadata">

**Author:** ![dinesh6478](https://avatars.discourse-cdn.com/v4/letter/d/46a35a/32.png) [@dinesh6478](https://forum.ansible.com/u/dinesh6478)\
**Post date:** [January 18, 2024, 5:07am UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/3 "2024-01-18T05:07:17Z")

</div>

I am still facing same issue

init-projects:  
Container ID: docker://11b5655e36a06ba93ae1bb1ede7a3dabf23760ef17d20d15495024e5b68c0a5c  
Image: [Quay](http://quay.io/centos/centos:stream9)  
Image ID: docker-pullable://quay.io/centos/centos@sha256:bf1c623981405eba655b9b633bdf17f2230ff4e9d93ccf59ad148e031976dde0  
Port:   
Host Port:   
Command:  
/bin/sh  
-c  
chmod 775 /var/lib/awx/projects  
chgrp 0 /var/lib/awx/projects

```
State: Waiting
  Reason: CrashLoopBackOff
Last State: Terminated
  Reason: Error
  Exit Code: 1
  Started: Thu, 18 Jan 2024 05:01:17 +0000
  Finished: Thu, 18 Jan 2024 05:01:17 +0000
Ready: False
Restart Count: 5
Requests:
  cpu: 100m
  memory: 128Mi
Environment:
  MY_POD_NAME: awxprod-web-6bdb8bccd7-599ct (v1:metadata.name)
Mounts:
  /var/lib/awx/projects from awxprod-projects (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9g6lx (ro)

```

Normal Started 3m52s (x5 over 5m15s) kubelet Started container init-projects  
Warning BackOff 12s (x25 over 5m14s) kubelet Back-off restarting failed container init-projects in pod awxprod-web-6bdb8bccd7-599ct\_awx(635e6c57-b523-4549-b802-99

$ kubectl26 logs awxprod-web-6bdb8bccd7-599ct -n awx -c init-projects  
chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted  
chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [January 18, 2024, 5:51am UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/4 "2024-01-18T05:51:18Z")

</div>

> ```yaml
> projects_storage_class: isilon
> 
> ```

If this is NFS based storage, NFS service on Isilon has root squash enabled by default, so `chmod` or `chgrp` may be failed for root user. Also check `rootClientEnabled` in StorageClass definition.

I’m not familiar with detailed configuration on Isilon, so I don’t know how to give minimal privileges/permissions only to certain PVs, but anyway, I think it’s a combination issue with the storage configuration and implementation in operator.

---

<div class="post-metadata">

**Author:** ![dinesh6478](https://avatars.discourse-cdn.com/v4/letter/d/46a35a/32.png) [@dinesh6478](https://forum.ansible.com/u/dinesh6478)\
**Post date:** [January 18, 2024, 6:04am UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/5 "2024-01-18T06:04:24Z")

</div>

below is the setting in storageclass

RootClientEnabled=false

---

<div class="post-metadata">

**Author:** ![dinesh6478](https://avatars.discourse-cdn.com/v4/letter/d/46a35a/32.png) [@dinesh6478](https://forum.ansible.com/u/dinesh6478)\
**Post date:** [January 18, 2024, 9:04am UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/6 "2024-01-18T09:04:20Z")

</div>

Hi,

thanks for your help, I was able to fix the issue on storage side and now AWX instance deployed without any issue.

Thanks,  
Dinesh K.

---

<div class="post-metadata">

**Author:** ![Veeralakumar](https://avatars.discourse-cdn.com/v4/letter/v/d78d45/32.png) [@Veeralakumar](https://forum.ansible.com/u/Veeralakumar)\
**Post date:** [February 14, 2024, 9:29pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/7 "2024-02-14T21:29:27Z")

</div>

Hello @dinesh6478 ,  
How do you fix. Because we are getting crash loopback for web containers. Remaining postgres and task pods created and running. Thsnks

---

<div class="post-metadata">

**Author:** ![Veeralakumar](https://avatars.discourse-cdn.com/v4/letter/v/d78d45/32.png) [@Veeralakumar](https://forum.ansible.com/u/Veeralakumar)\
**Post date:** [February 14, 2024, 9:55pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/8 "2024-02-14T21:55:07Z")

</div>

Hello @kurokobo ,  
good day. We needyour help. We are deploying awx 2.10.0 in k8s cluster 1.26 with nfs. Our awx task and postgres created. But we pods getting crash loopback errors. Getting superwatcher terminated errors. Dig the logs found that thesee was some db connection issue. Getting bad name or connection unknown

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [February 15, 2024, 2:03pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/9 "2024-02-15T14:03:13Z")

</div>

Hi, if your issue seems to have different cause from this topic, please create a new topic with detailed logs and output of the command to investigate that you’ve made.

---

<div class="post-metadata">

**Author:** ![dinesh6478](https://avatars.discourse-cdn.com/v4/letter/d/46a35a/32.png) [@dinesh6478](https://forum.ansible.com/u/dinesh6478)\
**Post date:** [February 15, 2024, 3:02pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/10 "2024-02-15T15:02:51Z")

</div>

Hi @Veeralakumar

I was getting below specific error which I resolved by fixing permission on storage side. what errors you are getting in your installation ?

chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted  
chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

---

<div class="post-metadata">

**Author:** ![Veeralakumar](https://avatars.discourse-cdn.com/v4/letter/v/d78d45/32.png) [@Veeralakumar](https://forum.ansible.com/u/Veeralakumar)\
**Post date:** [February 21, 2024, 3:23pm UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/11 "2024-02-21T15:23:06Z")

</div>

Installation completed. Now i am getting error /var/lib/awx permission denied. Error no 13. I hope i need to do some settings for the absolute path. My existing awx cluster /var/lib/awx/projects folder is read writable using ansible playbooks.

With thanks  
Veera

---

<div class="post-metadata">

**Author:** ![TheRealHaoLiu](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/therealhaoliu/32/100_2.png) [@TheRealHaoLiu](https://forum.ansible.com/u/TheRealHaoLiu)\
**Post date:** [February 27, 2024, 3:30am UTC](https://forum.ansible.com/t/install-awx-on-k8s-cluster/3351/12 "2024-02-27T03:30:20Z")

</div>

like @kurokobo mentioned your issue seems to be different from @dinesh6478’s problem

would u mind opening a different post to discuss your issue?

also in your new post please use `kubectl describe pod` to show which container is crashing and than use `kubectl log -p` to get the log from the crashed container
