Hi awx awx-operator folks - Posting here to hopefully get some assistance with an issue I’m having with a new AWX deployment on K3s.
Summary
With a fresh deployment of the AWX Operator, I’m unable to access the AWX web interface with the following error when logging in with admin:
Inspecting the awx-web container logs inside awx-pod, I see permission related errors:
kubectl -n awx logs awx-web-7c7df8c745-mstx8 awx-web
# Output
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/projects.py", line 67, in get_local_path_choices
for x in os.listdir(settings.PROJECTS_ROOT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects'
Versions
- Kubernetes:
K3s - v1.29.4 - AWX Operator:
2.18.0 - AWX:
24.5.0
Cluster
- 3 x K3s server nodes running RHEL 9.4.
- Rook CSI using
CephFileSystemCRD.
AWX Operator Deployment
# kustomization.yml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Specify a custom namespace in which to install AWX
namespace: awx
generatorOptions:
disableNameSuffixHash: true
# Set the image tags to match the git version from above
images:
- name: quay.io/ansible/awx-operator
newTag: 2.18.0
# Generate secrets for AWX deployment
secretGenerator:
- name: awx-admin-password
literals:
- password=<Password>
- name: awx-postgres-configuration
type: Opaque
literals:
- host=postgres.example.com
- port=5432
- database=awx
- username=awx
- password=<Password>
- sslmode=prefer
- target_session_attrs=read-write
- type=unmanaged
- name: awx-custom-certs
files:
- bundle-ca.crt=./ca-bundle.pem
resources:
- ./repo/config/default
- ./awx.yml
# awx.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
spec:
# Disable IPv6
ipv6_disabled: true
# Define secret containing external PostgreSQL connection details
postgres_configuration_secret: awx-postgres-configuration
# Define secret containing the AWX admin password
admin_password_secret: awx-admin-password
# Define secret containing a custom CA bundle
# Must contain the data field bundle-ca.crt
bundle_cacert_secret: awx-custom-certs
# Project persistence configuration
projects_persistence: true
projects_storage_class: rook-cephfs
projects_storage_size: 10Gi
projects_storage_access_mode: ReadWriteMany
Additional Information
I took a look at the /etc/passwd and /etc/group files inside the containers (awx-web & awx-task) and noticed the group ID 1000 is not referenced in /etc/group. Assume this group is awx based on entrypoint.sh.
Furthermore, I noticed that in awx-task, the running awx user has group 1000 assigned, but in awx-web it is not:
kubectl -n awx exec -it awx-web-f647c5455-6d8m7 -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 37 Jul 3 15:40 .
drwxr-xr-x. 1 root root 17 Jun 4 19:40 ..
prw-------. 1 awx root 0 Jul 3 15:40 awxfifo
drwxr-xr-x. 3 root root 19 Jun 4 19:40 .local
drwxrwsr-x. 2 root 1000 0 Jul 3 15:39 projects
drwxr-xr-x. 3 root root 20 Jun 4 19:40 public
drwxrwxr-x. 1 root root 40 Jun 4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun 4 19:36 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
ls: cannot open directory '/var/lib/awx/projects/': Permission denied
bash-5.1$ sestatus
SELinux status: disabled
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ cat /etc/passwd | grep '1000'
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash
bash-5.1$ cat /etc/group | grep '1000'
kubectl -n awx exec -it awx-task-599dd95b8c-7vl8w -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 22 Jul 3 15:40 .
drwxr-xr-x. 1 root root 17 Jun 4 19:40 ..
drwxr-xr-x. 3 root root 19 Jun 4 19:40 .local
drwxrwsr-x. 2 root 1000 0 Jul 3 15:39 projects
drwxr-xr-x. 3 root root 20 Jun 4 19:40 public
drwxrwxr-x. 1 root root 40 Jun 4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun 4 19:36 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000 0 Jul 3 15:39 .
drwxrwxr-x. 1 root root 43 Jul 3 15:44 ..
bash-5.1$ sestatus
SELinux status: disabled
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ cat /etc/passwd | grep '1000'
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash
bash-5.1$ cat /etc/group | grep '1000'
In an attempt to resolve this, I tried building my own custom image (below), but the user inside awx-web still doesn’t have group 1000 assigned 
# docker build -t quay.io/ansible/awx:24.5.0-fix1 .
FROM quay.io/ansible/awx:24.5.0
USER root
RUN echo "awx:x:1000:0:,,,:/var/lib/awx:/bin/bash" >> /etc/passwd
RUN echo "awx:x:1000:awx" >> /etc/group
# We are doing the same as above but just to be sure!
RUN usermod -aG awx awx
USER awx
Amended awx.yml:
# awx.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
spec:
# Same as previous
image: reg.example.com/ansible/awx
image_version: 24.5.0-fix1
image_pull_policy: Always
kubectl -n awx describe po awx-web-6d9684d79b-brg2z
awx-web:
...
Image: reg.example.com/ansible/awx:24.5.0-fix1
Args:
/usr/bin/launch_awx_web.sh
State: Running
kubectl -n awx exec -it awx-web-6d9684d79b-brg2z -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ cat /etc/group | grep '1000'
awx:x:1000:awx
I also tried with the latest versions of the AWX-Operator and AWX and I still have this issue:
- Kubernetes:
K3s - v1.29.4 - AWX Operator:
2.19.1 - AWX:
24.6.1
I’ve looked at the issues below but haven’t found a solution yet:
- no /var/lib/awx/projects · Issue #1429 · ansible/awx-operator (github.com)
- Permission denied: /var/lib/awx/projects/ · Issue #1644 · ansible/awx-operator (github.com)
- “Permission denied: '/var/lib/awx/projects” when running two replicas · Issue #1176 · ansible/awx-operator (github.com)
I did see the following PR from sometime ago FIx permissions /var/lib/awx/projects when projects_persistence is true. by paytroff · Pull Request #1654 · ansible/awx-operator (github.com) where someone seems to describe the same problem. But I think the solution may be to get the awx user in awx-web to be in group 1000 rather than amending the AWX Operator deployment manifests?
Any help is greatly appreciated and hope you all have a great day 
Funny how OCP works in my favor sometimes…
Apologies for that oversight.
Thanks 