# awx-on-k3s changing permissions of '/var/lib/awx/projects': Operation not permitted **URL:** \ **Category:** AWX Project\ **Tags:** awx\ **Created:** [September 19, 2022, 12:37pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124 "2022-09-19T12:37:43Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![HG1](https://avatars.discourse-cdn.com/v4/letter/h/b782af/32.png) [@HG1](https://forum.ansible.com/u/HG1)\ **Post date:** [September 19, 2022, 12:37pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/1 "2022-09-19T12:37:43Z")
Hello all, I am following [https://github.com/kurokobo/awx-on-k3s](https://github.com/kurokobo/awx-on-k3s). This time with version 0.29 of the operator and 0.29 awx-on-k3s. AWX does not startup correctly: awx awx-operator-controller-manager-5cb6fc8f9-x2vn6 2/2 Running 0 19m awx awx-postgres-13-0 1/1 Running 0 13m awx awx-84f598c555-lvbss 0/4 Init:CrashLoopBackOff 7 (58s ago) 13m In the awx\_init logfile the following error appears: 2022-09-19T14:04:08.60017162+02:00 stderr F chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted 2022-09-19T14:04:08.602818388+02:00 stderr F chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted I have specified the following however for the project pv. ---
**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\ **Post date:** [September 19, 2022, 7:41pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/2 "2022-09-19T19:41:52Z")
Hi, I have the same experience along with some other members of the community. The issue seems to be only on K3s [https://github.com/ansible/awx-operator/issues/1055](https://github.com/ansible/awx-operator/issues/1055) Regards, Antuelle78 ---
**Author:** ![kurokobo1](https://avatars.discourse-cdn.com/v4/letter/k/f0a364/32.png) [@kurokobo1](https://forum.ansible.com/u/kurokobo1)\ **Post date:** [September 19, 2022, 8:13pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/3 "2022-09-19T20:13:48Z")
Hi, Thanks for using my repo. Ensure the permission of “/u01/data/projects” on your K3s host. My guide includes “sudo chmod 1000:0 /data/projects” to change permission for the project directory before attempting deployment to make chmod and chgrp possible by init container. Try “sudo chmod 1000:0 /u01/data/projects” before “kubectl apply -k base”. Regards, ---
**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\ **Post date:** [September 19, 2022, 8:27pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/4 "2022-09-19T20:27:08Z")
Hi kurokobo, Thanks for the heads up. I am also looking at your repo. I plan to write an ansible role to simplify the process. Something similar for Ubuntu is available here: [https://github.com/antuelle78/deploy-awx-k3s-ubuntu](https://github.com/antuelle78/deploy-awx-k3s-ubuntu) Regards, Antuelle78 ---
**Author:** ![kurokobo1](https://avatars.discourse-cdn.com/v4/letter/k/f0a364/32.png) [@kurokobo1](https://forum.ansible.com/u/kurokobo1)\ **Post date:** [September 19, 2022, 9:18pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/5 "2022-09-19T21:18:53Z")
Hi Antuelle78, Thanks for your interest in my repo 🙂 My workaround in my previous mail maybe not work for you since it is for HG and your situation seems to be a bit differ from HG's one. Will reply on the issue ansible/awx-operator#1055 Regards, ---
**Author:** ![HG1](https://avatars.discourse-cdn.com/v4/letter/h/b782af/32.png) [@HG1](https://forum.ansible.com/u/HG1)\ **Post date:** [September 20, 2022, 7:59am UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/6 "2022-09-20T07:59:40Z")
Hi, This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25. But when I use 0.29 it fails. awx-84f598c555-c4hn6 0/4 Pending 0 0s awx-84f598c555-c4hn6 0/4 Pending 0 0s awx-84f598c555-c4hn6 0/4 Init:0/1 0 1s awx-84f598c555-c4hn6 0/4 Init:0/1 0 77s awx-84f598c555-c4hn6 0/4 Init:Error 0 79s awx-84f598c555-c4hn6 0/4 Init:Error 1 (3s ago) 80s awx-84f598c555-c4hn6 0/4 Init:CrashLoopBackOff 1 (12s ago) 91s /u01/data/projects has: drwxrwxrwx. 2 root root 6 Sep 20 09:44 projects The sudo chmod and chgrp are in both version: **0.29** ./roles/installer/templates/deployments/deployment.yaml.j2: chmod 775 /var/lib/awx/projects ./roles/installer/templates/deployments/deployment.yaml.j2: chgrp 1000 /var/lib/awx/projects **0.25** ./roles/installer/templates/deployment.yaml.j2: chmod 775 /var/lib/awx/projects ./roles/installer/templates/deployment.yaml.j2: chgrp 1000 /var/lib/awx/projects So it is unclear to me what changed. Regards Hans ---
**Author:** ![HG1](https://avatars.discourse-cdn.com/v4/letter/h/b782af/32.png) [@HG1](https://forum.ansible.com/u/HG1)\ **Post date:** [September 20, 2022, 8:16am UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/7 "2022-09-20T08:16:43Z")
The whole install starts failing with version 0.26 (awx-operator awx-on-k3s) ---
**Author:** ![kurokobo1](https://avatars.discourse-cdn.com/v4/letter/k/f0a364/32.png) [@kurokobo1](https://forum.ansible.com/u/kurokobo1)\ **Post date:** [September 20, 2022, 10:58am UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/8 "2022-09-20T10:58:45Z")
Hi HG, > This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25. Yes, correct. This issue came with 0.29.0. For technical details, refer my comment on [https://github.com/ansible/awx-operator/issues/1055#issuecomment-1251613435](https://github.com/ansible/awx-operator/issues/1055#issuecomment-1251613435) In short, in 0.28.0 or earlier, chmod/chgrp are invoked by root, but in 0.29.0, chmod/chgrp are invoked by UID:1000. So if your /u01/data/projects is owned by root:root, UID:1000 can’t be invoke chmod/chgrp and this causes “operation not permitted”. For 0.29.0, sadly, it seems that allowing UID:1000 to chmod/chgrp for /var/lib/awx/projects _before_ deployment of awx pod by changing owner/perm for the actual directory on your K3s host is required in some situation. This can be achieved by following workaround in my previous mail. > Try “sudo chmod 1000:0 /u01/data/projects” before “kubectl apply -k base”. I think this issue should be fixed in upstream, but in 0.29.0, this workaround is required anyway. Regards, ---
**Author:** ![m.nelson](https://avatars.discourse-cdn.com/v4/letter/m/c37758/32.png) [@m.nelson](https://forum.ansible.com/u/m.nelson)\ **Post date:** [September 20, 2022, 11:45am UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/9 "2022-09-20T11:45:11Z")
Hi kurokobo, Your work around works as expected. I tested with molecule using vagrant platform on Ubuntu 20.04/22.04 and CentOS Stream 8. K3s: v1.25.0+k3s1 Operator: 0.29.0 AWX: 21.6.0 I even test upgrading from: K3s: v1.21.9+k3s1 Operator: 0.28.0 AWX: 21.5.0 And had no issues. The code is available here: [https://github.com/antuelle78/awx-install-on-k3s](https://github.com/antuelle78/awx-install-on-k3s) Regards, Antuelle78 ---
**Author:** ![HG1](https://avatars.discourse-cdn.com/v4/letter/h/b782af/32.png) [@HG1](https://forum.ansible.com/u/HG1)\ **Post date:** [September 20, 2022, 12:00pm UTC](https://forum.ansible.com/t/awx-on-k3s-changing-permissions-of-var-lib-awx-projects-operation-not-permitted/37124/10 "2022-09-20T12:00:27Z")
So the userid 1000 = runner at the host level should match the userid 1000 inside the containers.