Awx-web: PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects'

Hi awx awx-operator folks - Posting here to hopefully get some assistance with an issue I’m having with a new AWX deployment on K3s.

Summary

With a fresh deployment of the AWX Operator, I’m unable to access the AWX web interface with the following error when logging in with admin:

Inspecting the awx-web container logs inside awx-pod, I see permission related errors:

kubectl -n awx logs awx-web-7c7df8c745-mstx8 awx-web
# Output
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/projects.py", line 67, in get_local_path_choices
    for x in os.listdir(settings.PROJECTS_ROOT)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects'

Versions

  • Kubernetes: K3s - v1.29.4
  • AWX Operator: 2.18.0
  • AWX: 24.5.0

Cluster

  • 3 x K3s server nodes running RHEL 9.4.
  • Rook CSI using CephFileSystem CRD.

AWX Operator Deployment

# kustomization.yml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Specify a custom namespace in which to install AWX
namespace: awx

generatorOptions:
  disableNameSuffixHash: true

# Set the image tags to match the git version from above
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.18.0

# Generate secrets for AWX deployment
secretGenerator:
  - name: awx-admin-password
    literals:
      - password=<Password>

  - name: awx-postgres-configuration
    type: Opaque
    literals:
      - host=postgres.example.com
      - port=5432
      - database=awx
      - username=awx
      - password=<Password>
      - sslmode=prefer
      - target_session_attrs=read-write
      - type=unmanaged

  - name: awx-custom-certs
    files:
      - bundle-ca.crt=./ca-bundle.pem

resources:
  - ./repo/config/default
  - ./awx.yml
# awx.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Disable IPv6
  ipv6_disabled: true
  # Define secret containing external PostgreSQL connection details
  postgres_configuration_secret: awx-postgres-configuration
  # Define secret containing the AWX admin password
  admin_password_secret: awx-admin-password
  # Define secret containing a custom CA bundle
  # Must contain the data field bundle-ca.crt
  bundle_cacert_secret: awx-custom-certs
  # Project persistence configuration
  projects_persistence: true
  projects_storage_class: rook-cephfs
  projects_storage_size: 10Gi
  projects_storage_access_mode: ReadWriteMany

Additional Information

I took a look at the /etc/passwd and /etc/group files inside the containers (awx-web & awx-task) and noticed the group ID 1000 is not referenced in /etc/group. Assume this group is awx based on entrypoint.sh.

Furthermore, I noticed that in awx-task, the running awx user has group 1000 assigned, but in awx-web it is not:

kubectl -n awx exec -it awx-web-f647c5455-6d8m7 -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 37 Jul  3 15:40 .
drwxr-xr-x. 1 root root 17 Jun  4 19:40 ..
prw-------. 1 awx  root  0 Jul  3 15:40 awxfifo
drwxr-xr-x. 3 root root 19 Jun  4 19:40 .local
drwxrwsr-x. 2 root 1000  0 Jul  3 15:39 projects
drwxr-xr-x. 3 root root 20 Jun  4 19:40 public
drwxrwxr-x. 1 root root 40 Jun  4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun  4 19:36 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
ls: cannot open directory '/var/lib/awx/projects/': Permission denied
bash-5.1$ sestatus
SELinux status:                 disabled
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ cat /etc/passwd | grep '1000'
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash
bash-5.1$ cat /etc/group | grep '1000'

kubectl -n awx exec -it awx-task-599dd95b8c-7vl8w -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 22 Jul  3 15:40 .
drwxr-xr-x. 1 root root 17 Jun  4 19:40 ..
drwxr-xr-x. 3 root root 19 Jun  4 19:40 .local
drwxrwsr-x. 2 root 1000  0 Jul  3 15:39 projects
drwxr-xr-x. 3 root root 20 Jun  4 19:40 public
drwxrwxr-x. 1 root root 40 Jun  4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun  4 19:36 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000  0 Jul  3 15:39 .
drwxrwxr-x. 1 root root 43 Jul  3 15:44 ..
bash-5.1$ sestatus
SELinux status:                 disabled
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ cat /etc/passwd | grep '1000'
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash
bash-5.1$ cat /etc/group | grep '1000'

In an attempt to resolve this, I tried building my own custom image (below), but the user inside awx-web still doesn’t have group 1000 assigned :disappointed:

# docker build -t quay.io/ansible/awx:24.5.0-fix1 .
FROM quay.io/ansible/awx:24.5.0
USER root
RUN echo "awx:x:1000:0:,,,:/var/lib/awx:/bin/bash" >> /etc/passwd
RUN echo "awx:x:1000:awx" >> /etc/group
# We are doing the same as above but just to be sure!
RUN usermod -aG awx awx
USER awx

Amended awx.yml:

# awx.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Same as previous
  image: reg.example.com/ansible/awx
  image_version: 24.5.0-fix1
  image_pull_policy: Always
kubectl -n awx describe po awx-web-6d9684d79b-brg2z
awx-web:
...
  Image:         reg.example.com/ansible/awx:24.5.0-fix1
  Args:
    /usr/bin/launch_awx_web.sh
  State:          Running

kubectl -n awx exec -it awx-web-6d9684d79b-brg2z -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ cat /etc/group | grep '1000'
awx:x:1000:awx

I also tried with the latest versions of the AWX-Operator and AWX and I still have this issue:

  • Kubernetes: K3s - v1.29.4
  • AWX Operator: 2.19.1
  • AWX: 24.6.1

I’ve looked at the issues below but haven’t found a solution yet:

I did see the following PR from sometime ago FIx permissions /var/lib/awx/projects when projects_persistence is true. by paytroff · Pull Request #1654 · ansible/awx-operator (github.com) where someone seems to describe the same problem. But I think the solution may be to get the awx user in awx-web to be in group 1000 rather than amending the AWX Operator deployment manifests?

Any help is greatly appreciated and hope you all have a great day :slight_smile:

In the deployment manifest for awx-task the securityContext is configured with fsGroup:1000 awx-operator/roles/installer/templates/deployments/task.yaml.j2 at devel · ansible/awx-operator (github.com)

This isn’t applied in awx-web deployment manifest awx-operator/roles/installer/templates/deployments/web.yaml.j2 at devel · ansible/awx-operator (github.com)

Testing using security_context_settings to apply fsGroup: 1000 to the awx-web pod:

security_context_settings:
   fsGroup: 1000

The awx user in the awx-web pod now has group with ID 1000 but there is still permission denied:

kubectl edit po -n awx awx-web-5dfd86cf94-jgdf4
...
securityContext:
    fsGroup: 1000
...

$ kubectl -n awx exec -it awx-web-5dfd86cf94-jgdf4 -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ ls -lah /var/lib/awx/projects/
ls: cannot open directory '/var/lib/awx/projects/': Permission denied
bash-5.1$ ls -lah /var/lib/awx/
total 4.0K
drwxrwxr-x. 1 root root 58 Jul  5 11:16 .
drwxr-xr-x. 1 root root 17 Jul  2 20:18 ..
prw-------. 1 awx  root  0 Jul  5 11:05 awxfifo
-rw-------. 1 awx  root 64 Jul  5 11:16 .bash_history
drwxr-xr-x. 3 root root 19 Jul  2 20:18 .local
drwxrwsr-x. 2 root 1000  0 Jul  5 11:05 projects
drwxr-xr-x. 3 root root 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root 17 Jul  2 20:14 venv
bash-5.1$ sestatus
SELinux status:                 disabled

Sorry for not responding to the mention on the GitHub.
Have you confirmed that the PV created by Rook CSI with Ceph can be written to from a simple test Pod, independently of AWX?
Considering that a lot of users are using AWX and AWX Operator as they are without any issues, I don’t think it’s necessarily required to add it to GID 1000.

Hi @kurokobo

Thanks for your response :slight_smile: AFAIK, it’s the same PVC mounted on both the awx-task and awx-web pods, and awx-task is able to write to the directory just fine:

$ kubectl -n awx exec -it awx-task-6d4c9d4588-76kq5 -- bash
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000  0 Jul  5 11:24 .
drwxrwxr-x. 1 root root 22 Jul  5 11:24 ..
bash-5.1$ mkdir -pv /var/lib/awx/projects/testing
mkdir: created directory '/var/lib/awx/projects/testing'
bash-5.1$ date
Fri Jul  5 01:22:53 PM UTC 2024
bash-5.1$ date >> /var/lib/awx/projects/testing/date
bash-5.1$ cat /var/lib/awx/projects/testing/date
Fri Jul  5 01:23:04 PM UTC 2024

I’ve created a test deployment independent from AWX as you suggest, mounting the PVC awx-projects-claim:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: awx-troubleshoot
  namespace: awx
  labels:
    app: awx-troubleshoot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: awx-troubleshoot
  template:
    metadata:
      labels:
        app: awx-troubleshoot
    spec:
      containers:
        - name: awx-troubleshoot
          image: quay.io/centos/centos:stream9
          imagePullPolicy: Always
          command:
            - /bin/sh
            - -c
            - |
              sleep 3600
          resources:
            limits:
              memory: 100Mi
          volumeMounts:
              - mountPath: /var/lib/awx/projects
                name: awx-projects
      volumes:
        - name: awx-projects
          persistentVolumeClaim:
            claimName: awx-projects-claim

It’s able to write just fine, which makes sense as this container user is root and has u=rwx permissions on /var/lib/awx/projects/.

$ kubectl -n awx exec -it awx-troubleshoot-7cd98b747f-fjjxc -- /bin/sh
sh-5.1# ls -lah /var/lib/awx/
total 0
drwxr-xr-x. 3 root root 22 Jul  5 15:34 .
drwxr-xr-x. 1 root root 17 Jul  5 15:34 ..
drwxrwsr-x. 3 root 1000  1 Jul  5 13:22 projects
sh-5.1# ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 3 root 1000  1 Jul  5 13:22 .
drwxr-xr-x. 3 root root 22 Jul  5 15:34 ..
drwxr-sr-x. 2 1000 1000  1 Jul  5 13:23 testing
sh-5.1# cat /var/lib/awx/projects/testing/date
Fri Jul  5 01:23:04 PM UTC 2024
sh-5.1# id
uid=0(root) gid=0(root) groups=0(root)
sh-5.1# date > /var/lib/awx/projects/troubleshoot
sh-5.1# cat /var/lib/awx/projects/troubleshoot
Fri Jul  5 15:35:20 UTC 2024

@kurokobo Looks like group permissions is missing execution permissions, as that stickybit should be uppercased.

Should be drwxrwSr-x. 2 root 1000 0 Jul 5 11:05 projects if the stickybit is needed, but that GitHub thread shows examples of no stickybit being set at all. So it used to be drwxrwxr-x. 2 root 1000 0 Jul 5 11:05 projects.

Edit: Nvm, s is the executable stickybit, decided to fact check myself. I’m still curious if the stickybit is the culprit though, since that doesn’t show up in @stuntguy3000’s output, for e.g.

Edit2: I’m seeing the stickybit on my deployment and having no issues there, so nevermind me on this.

Edit3: Wait, my web pod is in all the same groups as my task pod, so it has identical permissions. :man_shrugging: Funny how OCP works in my favor sometimes…

Hi @Denney-tech

Thanks for taking a look at this :slightly_smiling_face:

I also looked at the SGID but as you’ve stated, in your deployment it’s working for you. What version of AWX and the operator do you have deployed?

I’ll have to continue looking at this on Monday.

@kurokobo @Denney-tech Do either of you know where/what is setting the SGID?

I don’t know.

I do know that, in OpenShift 4, both of my pods have identical uid/gid primary and supplimental group assignments. This is partially an OpenShift quirk, but might explain why my pods don’t have permission issues. (The reason I say partially is that I also have to custom build AWX to support the random uid/gid assignment from OpenShift).

As a result, I’m not using uid/gid 1000 anywhere.

So, the sticky bit might not impact my deployment due to lucky circumstances. I will try adjusting the security contexts to see if I can get a more apples to apples comparison.

1 Like

Thanks @Denney-tech :slightly_smiling_face:

What versions are you running of AWX and the Operator, maybe I can try the version you’ve deployed and see if I have the same issues as I have currently :thinking:

@kurokobo - Do you have any other suggestions?

I’m using the latest version, same as you.

1 Like

@dbrennand @Denney-tech

Sorry for the delayed response. Interesting, I don’t know the cause immediately, but I suspect it’s probably an issue on the PV side.

As you mentioned, here are some facts:

  • If projects_persistence is set to true, only the task pod has fsGroup: 1000 as securityContext
  • This is why the awx user in the task pod belongs to sub-group 1000
  • I believe the SGID you’ve mentioned is also due to this fsGroup

In my environment, the pods was in the following state. I can’t reproduce your issue here.

Task pod:

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ ls -l /var/lib/awx/
total 0
drwxrwsr-x. 2 root 1000  6 Jul 10 15:31 projects
drwxr-xr-x. 3 root root 20 Jun 21 01:24 public
drwxrwxr-x. 1 root root 40 Jun 21 01:24 rsyslog
drwxr-xr-x. 3 root root 17 Jun 21 01:20 venv
bash-5.1$ ls -l /var/lib/awx/projects
total 0

Web pod:

$ kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$  ls -l /var/lib/awx/
total 0
prw-------. 1 awx  root  0 Jul 10 15:32 awxfifo
drwxrwsr-x. 2 root 1000  6 Jul 10 15:31 projects
drwxr-xr-x. 3 root root 20 Jun 21 01:24 public
drwxrwxr-x. 1 root root 40 Jun 21 01:24 rsyslog
drwxr-xr-x. 3 root root 17 Jun 21 01:20 venv
bash-5.1$  ls -l /var/lib/awx/projects/
total 0

Do you get any suspicious output with ls -lZ or getfacl ? Here is my case:

bash-5.1$ ls -lZ /var/lib/awx/
total 0
prw-------. 1 awx  root system_u:object_r:container_file_t:s0  0 Jul 10 15:32 awxfifo
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0  6 Jul 10 15:31 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0 20 Jun 21 01:24 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0 40 Jun 21 01:24 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0 17 Jun 21 01:20 venv
bash-5.1$ getfacl /var/lib/awx/projects/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/awx/projects/
# owner: root
# group: 1000
# flags: -s-
user::rwx
group::rwx
other::r-x

Additionally, since it seems you are creating the PV with ReadWriteMany, is there a possibility that access to the PV changes depending on the Node where the Pod is located?
How about running both the Web and Task pods on the same Node?

I’m not sure if it will reveal anything, but could you show me the configurations of the two pods using kubectl -n awx describe pod ?

2 Likes

Hi @kurokobo

Thank you for your response :slight_smile:

I checked and the awx-task and awx-web pods are running on the same node:

NAME                                               READY   STATUS      RESTARTS   AGE   IP            NODE                           NOMINATED NODE   READINESS GATES
awx-migration-24.6.1-mf75r                         0/1     Completed   0          14m   10.42.1.83    node2.example.com   <none>           <none>
awx-operator-controller-manager-745b55d94b-bhztl   2/2     Running     0          42h   10.42.2.128   node3.example.com   <none>           <none>
awx-task-8456d44c4f-rf9t8                          4/4     Running     0          14m   10.42.2.140   node3.example.com   <none>           <none>
awx-web-bc6b797fc-ggvbq                            3/3     Running     0          14m   10.42.2.139   node3.example.com   <none>           <none>

Task Pod:

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ ls -l /var/lib/awx/
total 0
drwxrwsr-x. 2 root 1000  0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root 17 Jul  2 20:14 venv
bash-5.1$ ls -l /var/lib/awx/projects/
total 0

Web Pod:

$ kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ ls -l /var/lib/awx/
total 0
prw-------. 1 awx  root  0 Jul 11 07:48 awxfifo
drwxrwsr-x. 2 root 1000  0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root 17 Jul  2 20:14 venv

Web Pod - SELinux Labels and getfacl:

bash-5.1$ ls -lZ /var/lib/awx/
total 0
prw-------. 1 awx  root system_u:object_r:container_file_t:s0:c277,c610  0 Jul 11 07:48 awxfifo
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974  0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c277,c610 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c277,c610 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c277,c610 17 Jul  2 20:14 venv
bash-5.1$ getfacl /var/lib/awx/projects/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/awx/projects/
# owner: root
# group: 1000
# flags: -s-
user::rwx
group::rwx
other::r-x

Task Pod - SELinux Labels and getfacl:

$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ ls -lahZ /var/lib/awx/
total 4.0K
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 43 Jul 11 07:50 .
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul  2 20:18 ..
-rw-------. 1 awx  root system_u:object_r:container_file_t:s0:c924,c974 57 Jul 11 07:50 .bash_history
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 19 Jul  2 20:18 .local
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974  0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul  2 20:14 venv
bash-5.1$ getfacl /var/lib/awx/projects/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/awx/projects/
# owner: root
# group: 1000
# flags: -s-
user::rwx
group::rwx
other::r-x

Nothing seems a miss with getfacl. However, when comparing my output to yours I see all the directories in /var/lib/awx have additional SELinux categories applied (e.g., c924,c974); not just the sensitivity label.

In the awx-task pod, they all seem to be the same (c924,c974) whereas, in the awx-web pod, I see /var/lib/awx/projects is different to all the others being c924,c974 - same as in the awx-task pod. I wonder if the awx-task pod is started after the awx-web pod and the categories are being re-applied, and because this PVC is mounted across both pods, the awx-task pod categories are overriding on /var/lib/awx/projects.

I’m not very familiar with the Multi-Category Security concept in SELinux, but I do think this is playing a factor here.

Also, it looks like I failed to mention that SELinux is enabled on the underlying K3s nodes in my initial post :slightly_frowning_face: Apologies for that oversight.

Fix

I’ve taken a look at Configure a Security Context for a Pod or Container | Kubernetes and I think setting the same SELinux label and categories for both pods may help:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # ...
  # Security context configuration
  security_context_settings:
    seLinuxOptions:
      level: "s0:c924,c974"

Redeployed with this change and it has fixed the permission denied issues in the awx-web pod and I can now access the AWX GUI :slightly_smiling_face: :tada: :partying_face:

kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ ls -lahZ /var/lib/awx/
total 0
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 22 Jul 11 10:01 .
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul  2 20:18 ..
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 19 Jul  2 20:18 .local
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974  0 Jul 11 10:01 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul  2 20:14 venv
bash-5.1$ exit
exit
kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$ ls -lahZ /var/lib/awx/
total 0
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 37 Jul 11 10:01 .
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul  2 20:18 ..
prw-------. 1 awx  root system_u:object_r:container_file_t:s0:c924,c974  0 Jul 11 10:01 awxfifo
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 19 Jul  2 20:18 .local
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974  0 Jul 11 10:01 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 20 Jul  2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 40 Jul  2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul  2 20:14 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000  0 Jul 11 10:01 .
drwxrwxr-x. 1 root root 37 Jul 11 10:01 ..

Final AWX spec for reference:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Disable IPv6
  ipv6_disabled: true
  # Define secret containing external PostgreSQL connection details
  postgres_configuration_secret: awx-postgres-configuration
  # Define secret containing the AWX admin password
  admin_password_secret: awx-admin-password
  # Define secret containing a custom CA bundle
  # Must contain the data field bundle-ca.crt
  bundle_cacert_secret: awx-custom-certs
  image: reg.example.com/ansible/awx
  image_version: 24.6.1
  image_pull_policy: Always
  # Security context configuration
  security_context_settings:
    seLinuxOptions:
      level: "s0:c924,c974"
  # Project persistance configuration
  projects_persistence: true
  projects_storage_class: rook-cephfs
  projects_storage_size: 10Gi
  projects_storage_access_mode: ReadWriteMany

Versions

  • Kubernetes: v1.29.4+k3s1
  • AWX Operator: 2.19.1
  • AWX: 24.6.1

Well… that was a fun adventure :laughing: Thanks @kurokobo @Denney-tech for your responses on this :slight_smile:

2 Likes

Congratulations!
I’m glad to hear that you were able to resolve the issue :smiley:

This topic has also been very valuable knowledge for me. Thank you for diligently updating me on the situation!

2 Likes

That’s definitely an unusual problem, but now we’re all better equipped to help the next person with this particular problem! Looking at my pods, the directories all have s0:c10,c26, both task and web. So, it’s curious why yours was inconsistent between pods (or even directories).

@kurokobo good call on checking the security contexts, and good job @dbrennand finding a solution!

1 Like