awx-on-k3s changing permissions of '/var/lib/awx/projects': Operation not permitted

Hello all,
I am following https://github.com/kurokobo/awx-on-k3s.
This time with version 0.29 of the operator and 0.29 awx-on-k3s.

AWX does not startup correctly:
awx awx-operator-controller-manager-5cb6fc8f9-x2vn6 2/2 Running 0 19m
awx awx-postgres-13-0 1/1 Running 0 13m
awx awx-84f598c555-lvbss 0/4 Init:CrashLoopBackOff 7 (58s ago) 13m

In the awx_init logfile the following error appears:

2022-09-19T14:04:08.60017162+02:00 stderr F chmod: changing permissions of ‘/var/lib/awx/projects’: Operation not permitted
2022-09-19T14:04:08.602818388+02:00 stderr F chgrp: changing group of ‘/var/lib/awx/projects’: Operation not permitted

I have specified the following however for the project pv.

Hi,
I have the same experience along with some other members of the community.
The issue seems to be only on K3s https://github.com/ansible/awx-operator/issues/1055

Regards,
Antuelle78

Hi,

Thanks for using my repo.

Ensure the permission of “/u01/data/projects” on your K3s host.

My guide includes “sudo chmod 1000:0 /data/projects” to change permission for the project directory before attempting deployment to make chmod and chgrp possible by init container.

Try “sudo chmod 1000:0 /u01/data/projects” before “kubectl apply -k base”.

Regards,

Hi kurokobo,

Thanks for the heads up. I am also looking at your repo.
I plan to write an ansible role to simplify the process.
Something similar for Ubuntu is available here: https://github.com/antuelle78/deploy-awx-k3s-ubuntu

Regards,
Antuelle78

Hi Antuelle78,

Thanks for your interest in my repo :slight_smile:
My workaround in my previous mail maybe not work for you since it is for HG and your situation seems to be a bit differ from HG's one.

Will reply on the issue ansible/awx-operator#1055

Regards,

Hi,

This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25.

But when I use 0.29 it fails.

awx-84f598c555-c4hn6 0/4 Pending 0 0s
awx-84f598c555-c4hn6 0/4 Pending 0 0s
awx-84f598c555-c4hn6 0/4 Init:0/1 0 1s
awx-84f598c555-c4hn6 0/4 Init:0/1 0 77s
awx-84f598c555-c4hn6 0/4 Init:Error 0 79s
awx-84f598c555-c4hn6 0/4 Init:Error 1 (3s ago) 80s
awx-84f598c555-c4hn6 0/4 Init:CrashLoopBackOff 1 (12s ago) 91s

/u01/data/projects has:

drwxrwxrwx. 2 root root 6 Sep 20 09:44 projects

The sudo chmod and chgrp are in both version:

0.29

./roles/installer/templates/deployments/deployment.yaml.j2: chmod 775 /var/lib/awx/projects
./roles/installer/templates/deployments/deployment.yaml.j2: chgrp 1000 /var/lib/awx/projects
0.25

./roles/installer/templates/deployment.yaml.j2: chmod 775 /var/lib/awx/projects
./roles/installer/templates/deployment.yaml.j2: chgrp 1000 /var/lib/awx/projects

So it is unclear to me what changed.

Regards Hans

The whole install starts failing with version 0.26 (awx-operator awx-on-k3s)

Hi HG,

This issue does not appear with awx-operator 0.25 and awx-on-k3s 0.25.

Yes, correct. This issue came with 0.29.0.
For technical details, refer my comment on https://github.com/ansible/awx-operator/issues/1055#issuecomment-1251613435
In short, in 0.28.0 or earlier, chmod/chgrp are invoked by root, but in 0.29.0, chmod/chgrp are invoked by UID:1000.
So if your /u01/data/projects is owned by root:root, UID:1000 can’t be invoke chmod/chgrp and this causes “operation not permitted”.

For 0.29.0, sadly, it seems that allowing UID:1000 to chmod/chgrp for /var/lib/awx/projects before deployment of awx pod by changing owner/perm for the actual directory on your K3s host is required in some situation.
This can be achieved by following workaround in my previous mail.

Try “sudo chmod 1000:0 /u01/data/projects” before “kubectl apply -k base”.

I think this issue should be fixed in upstream, but in 0.29.0, this workaround is required anyway.

Regards,

Hi kurokobo,

Your work around works as expected.
I tested with molecule using vagrant platform on Ubuntu 20.04/22.04 and CentOS Stream 8.

K3s: v1.25.0+k3s1
Operator: 0.29.0
AWX: 21.6.0

I even test upgrading from:

K3s: v1.21.9+k3s1
Operator: 0.28.0
AWX: 21.5.0

And had no issues.

The code is available here: https://github.com/antuelle78/awx-install-on-k3s

Regards,
Antuelle78

So the userid 1000 = runner at the host level should match the userid 1000 inside the containers.