# Awx-web: PermissionError: \[Errno 13\] Permission denied: '/var/lib/awx/projects'

**URL:** <https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074>\
**Category:** Get Help\
**Tags:** awx-operator, rhel, awx, kubernetes\
**Created:** [July 5, 2024, 9:39am UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074 "2024-07-05T09:39:33Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 5, 2024, 9:39am UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/1 "2024-07-05T09:39:33Z")

</div>

Hi #awx #awx-operator folks - Posting here to hopefully get some assistance with an issue I’m having with a new AWX deployment on K3s.

## Summary

With a fresh deployment of the AWX Operator, I’m unable to access the AWX web interface with the following error when logging in with `admin`:

 ![image](https://us1.discourse-cdn.com/ansible/original/2X/d/d1e70dc58fa3938d08f1b959d9a51910b23e9c16.jpeg)

Inspecting the `awx-web` container logs inside `awx-pod`, I see permission related errors:

```auto
kubectl -n awx logs awx-web-7c7df8c745-mstx8 awx-web
# Output
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/projects.py", line 67, in get_local_path_choices
    for x in os.listdir(settings.PROJECTS_ROOT)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/var/lib/awx/projects'

```

## Versions

- Kubernetes: `K3s - v1.29.4`
- AWX Operator: `2.18.0`
- AWX: `24.5.0`

## Cluster

- 3 x K3s server nodes running RHEL 9.4.
- Rook CSI using `CephFileSystem` CRD.

### AWX Operator Deployment

```yaml
# kustomization.yml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Specify a custom namespace in which to install AWX
namespace: awx

generatorOptions:
  disableNameSuffixHash: true

# Set the image tags to match the git version from above
images:
  - name: quay.io/ansible/awx-operator
    newTag: 2.18.0

# Generate secrets for AWX deployment
secretGenerator:
  - name: awx-admin-password
    literals:
      - password=<Password>

  - name: awx-postgres-configuration
    type: Opaque
    literals:
      - host=postgres.example.com
      - port=5432
      - database=awx
      - username=awx
      - password=<Password>
      - sslmode=prefer
      - target_session_attrs=read-write
      - type=unmanaged

  - name: awx-custom-certs
    files:
      - bundle-ca.crt=./ca-bundle.pem

resources:
  - ./repo/config/default
  - ./awx.yml

```

```yaml
# awx.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Disable IPv6
  ipv6_disabled: true
  # Define secret containing external PostgreSQL connection details
  postgres_configuration_secret: awx-postgres-configuration
  # Define secret containing the AWX admin password
  admin_password_secret: awx-admin-password
  # Define secret containing a custom CA bundle
  # Must contain the data field bundle-ca.crt
  bundle_cacert_secret: awx-custom-certs
  # Project persistence configuration
  projects_persistence: true
  projects_storage_class: rook-cephfs
  projects_storage_size: 10Gi
  projects_storage_access_mode: ReadWriteMany

```

## Additional Information

I took a look at the `/etc/passwd` and `/etc/group` files inside the containers (`awx-web` & `awx-task`) and noticed the group ID 1000 is not referenced in `/etc/group`. Assume this group is `awx` based on [`entrypoint.sh`](https://github.com/ansible/awx/blob/devel/tools/docker-compose/entrypoint.sh#L11).

Furthermore, I noticed that in `awx-task`, the running `awx` user has group `1000` assigned, but in `awx-web` it is not:

```auto
kubectl -n awx exec -it awx-web-f647c5455-6d8m7 -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 37 Jul 3 15:40 .
drwxr-xr-x. 1 root root 17 Jun 4 19:40 ..
prw-------. 1 awx root 0 Jul 3 15:40 awxfifo
drwxr-xr-x. 3 root root 19 Jun 4 19:40 .local
drwxrwsr-x. 2 root 1000 0 Jul 3 15:39 projects
drwxr-xr-x. 3 root root 20 Jun 4 19:40 public
drwxrwxr-x. 1 root root 40 Jun 4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun 4 19:36 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
ls: cannot open directory '/var/lib/awx/projects/': Permission denied
bash-5.1$ sestatus
SELinux status: disabled
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ cat /etc/passwd | grep '1000'
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash
bash-5.1$ cat /etc/group | grep '1000'

kubectl -n awx exec -it awx-task-599dd95b8c-7vl8w -- bash
bash-5.1$ ls -lah /var/lib/awx/
total 0
drwxrwxr-x. 1 root root 22 Jul 3 15:40 .
drwxr-xr-x. 1 root root 17 Jun 4 19:40 ..
drwxr-xr-x. 3 root root 19 Jun 4 19:40 .local
drwxrwsr-x. 2 root 1000 0 Jul 3 15:39 projects
drwxr-xr-x. 3 root root 20 Jun 4 19:40 public
drwxrwxr-x. 1 root root 40 Jun 4 19:40 rsyslog
drwxr-xr-x. 3 root root 17 Jun 4 19:36 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000 0 Jul 3 15:39 .
drwxrwxr-x. 1 root root 43 Jul 3 15:44 ..
bash-5.1$ sestatus
SELinux status: disabled
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ cat /etc/passwd | grep '1000'
awx:x:1000:0:,,,:/var/lib/awx:/bin/bash
bash-5.1$ cat /etc/group | grep '1000'

```

In an attempt to resolve this, I tried building my own custom image (below), but the user inside `awx-web` still doesn’t have group `1000` assigned 😞

```Dockerfile
# docker build -t quay.io/ansible/awx:24.5.0-fix1 .
FROM quay.io/ansible/awx:24.5.0
USER root
RUN echo "awx:x:1000:0:,,,:/var/lib/awx:/bin/bash" >> /etc/passwd
RUN echo "awx:x:1000:awx" >> /etc/group
# We are doing the same as above but just to be sure!
RUN usermod -aG awx awx
USER awx

```

Amended `awx.yml`:

```yaml
# awx.yml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Same as previous
  image: reg.example.com/ansible/awx
  image_version: 24.5.0-fix1
  image_pull_policy: Always

```

```auto
kubectl -n awx describe po awx-web-6d9684d79b-brg2z
awx-web:
...
  Image: reg.example.com/ansible/awx:24.5.0-fix1
  Args:
    /usr/bin/launch_awx_web.sh
  State: Running

kubectl -n awx exec -it awx-web-6d9684d79b-brg2z -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ cat /etc/group | grep '1000'
awx:x:1000:awx

```

I also tried with the latest versions of the AWX-Operator and AWX and I still have this issue:

- Kubernetes: `K3s - v1.29.4`
- AWX Operator: `2.19.1`
- AWX: `24.6.1`

I’ve looked at the issues below but haven’t found a solution yet:

- [no /var/lib/awx/projects · Issue #1429 · ansible/awx-operator (github.com)](https://github.com/ansible/awx-operator/issues/1429)
- [Permission denied: /var/lib/awx/projects/ · Issue #1644 · ansible/awx-operator (github.com)](https://github.com/ansible/awx-operator/issues/1644)
- [“Permission denied: '/var/lib/awx/projects” when running two replicas · Issue #1176 · ansible/awx-operator (github.com)](https://github.com/ansible/awx-operator/issues/1176)

I did see the following PR from sometime ago [FIx permissions /var/lib/awx/projects when projects\_persistence is true. by paytroff · Pull Request #1654 · ansible/awx-operator (github.com)](https://github.com/ansible/awx-operator/pull/1654) where someone seems to describe the same problem. But I think the solution may be to get the `awx` user in `awx-web` to be in group `1000` rather than amending the AWX Operator deployment manifests?

Any help is greatly appreciated and hope you all have a great day 🙂

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 5, 2024, 11:26am UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/2 "2024-07-05T11:26:34Z")

</div>

In the deployment manifest for `awx-task` the `securityContext` is configured with `fsGroup:1000` [awx-operator/roles/installer/templates/deployments/task.yaml.j2 at devel · ansible/awx-operator (github.com)](https://github.com/ansible/awx-operator/blob/devel/roles/installer/templates/deployments/task.yaml.j2#L506)

This isn’t applied in `awx-web` deployment manifest [awx-operator/roles/installer/templates/deployments/web.yaml.j2 at devel · ansible/awx-operator (github.com)](https://github.com/ansible/awx-operator/blob/devel/roles/installer/templates/deployments/web.yaml.j2#L371C7-L371C32)

Testing using `security_context_settings` to apply `fsGroup: 1000` to the `awx-web` pod:

```yaml
security_context_settings:
   fsGroup: 1000

```

The `awx` user in the `awx-web` pod now has group with ID 1000 but there is still permission denied:

```auto
kubectl edit po -n awx awx-web-5dfd86cf94-jgdf4
...
securityContext:
    fsGroup: 1000
...

$ kubectl -n awx exec -it awx-web-5dfd86cf94-jgdf4 -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ ls -lah /var/lib/awx/projects/
ls: cannot open directory '/var/lib/awx/projects/': Permission denied
bash-5.1$ ls -lah /var/lib/awx/
total 4.0K
drwxrwxr-x. 1 root root 58 Jul 5 11:16 .
drwxr-xr-x. 1 root root 17 Jul 2 20:18 ..
prw-------. 1 awx root 0 Jul 5 11:05 awxfifo
-rw-------. 1 awx root 64 Jul 5 11:16 .bash_history
drwxr-xr-x. 3 root root 19 Jul 2 20:18 .local
drwxrwsr-x. 2 root 1000 0 Jul 5 11:05 projects
drwxr-xr-x. 3 root root 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root 17 Jul 2 20:14 venv
bash-5.1$ sestatus
SELinux status: disabled

```

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [July 5, 2024, 12:29pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/3 "2024-07-05T12:29:19Z")

</div>

Sorry for not responding to the mention on [the GitHub](https://github.com/ansible/awx-operator/issues/1429).  
Have you confirmed that the PV created by Rook CSI with Ceph can be written to from a simple test Pod, independently of AWX?  
Considering that a lot of users are using AWX and AWX Operator as they are without any issues, I don’t think it’s necessarily required to add it to GID 1000.

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 5, 2024, 3:43pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/4 "2024-07-05T15:43:39Z")

</div>

Hi @kurokobo

Thanks for your response 🙂 AFAIK, it’s the same PVC mounted on both the `awx-task` and `awx-web` pods, and `awx-task` is able to write to the directory just fine:

```auto
$ kubectl -n awx exec -it awx-task-6d4c9d4588-76kq5 -- bash
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000 0 Jul 5 11:24 .
drwxrwxr-x. 1 root root 22 Jul 5 11:24 ..
bash-5.1$ mkdir -pv /var/lib/awx/projects/testing
mkdir: created directory '/var/lib/awx/projects/testing'
bash-5.1$ date
Fri Jul 5 01:22:53 PM UTC 2024
bash-5.1$ date >> /var/lib/awx/projects/testing/date
bash-5.1$ cat /var/lib/awx/projects/testing/date
Fri Jul 5 01:23:04 PM UTC 2024

```

I’ve created a test deployment independent from AWX as you suggest, mounting the PVC `awx-projects-claim`:

```yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: awx-troubleshoot
  namespace: awx
  labels:
    app: awx-troubleshoot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: awx-troubleshoot
  template:
    metadata:
      labels:
        app: awx-troubleshoot
    spec:
      containers:
        - name: awx-troubleshoot
          image: quay.io/centos/centos:stream9
          imagePullPolicy: Always
          command:
            - /bin/sh
            - -c
            - |
              sleep 3600
          resources:
            limits:
              memory: 100Mi
          volumeMounts:
              - mountPath: /var/lib/awx/projects
                name: awx-projects
      volumes:
        - name: awx-projects
          persistentVolumeClaim:
            claimName: awx-projects-claim

```

It’s able to write just fine, which makes sense as this container user is `root` and has `u=rwx` permissions on `/var/lib/awx/projects/`.

```auto
$ kubectl -n awx exec -it awx-troubleshoot-7cd98b747f-fjjxc -- /bin/sh
sh-5.1# ls -lah /var/lib/awx/
total 0
drwxr-xr-x. 3 root root 22 Jul 5 15:34 .
drwxr-xr-x. 1 root root 17 Jul 5 15:34 ..
drwxrwsr-x. 3 root 1000 1 Jul 5 13:22 projects
sh-5.1# ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 3 root 1000 1 Jul 5 13:22 .
drwxr-xr-x. 3 root root 22 Jul 5 15:34 ..
drwxr-sr-x. 2 1000 1000 1 Jul 5 13:23 testing
sh-5.1# cat /var/lib/awx/projects/testing/date
Fri Jul 5 01:23:04 PM UTC 2024
sh-5.1# id
uid=0(root) gid=0(root) groups=0(root)
sh-5.1# date > /var/lib/awx/projects/troubleshoot
sh-5.1# cat /var/lib/awx/projects/troubleshoot
Fri Jul 5 15:35:20 UTC 2024

```

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [July 5, 2024, 5:18pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/5 "2024-07-05T17:18:56Z")

</div>

@kurokobo ~~Looks like group permissions is missing execution permissions, as that stickybit should be uppercased.~~

> [@dbrennand](#):
>
> `drwxrwsr-x. 2 root 1000 0 Jul 5 11:05 projects`

~~Should be `drwxrwSr-x. 2 root 1000 0 Jul 5 11:05 projects`~~ if the stickybit is needed, but that GitHub thread shows examples of no stickybit being set at all. So it used to be `drwxrwxr-x. 2 root 1000 0 Jul 5 11:05 projects`.

Edit: Nvm, `s` is the executable stickybit, decided to fact check myself. I’m still curious if the stickybit is the culprit though, since that doesn’t show up in @stuntguy3000’s output, for e.g.

Edit2: I’m seeing the stickybit on my deployment and having no issues there, so nevermind me on this.

Edit3: Wait, my web pod is in all the same groups as my task pod, so it has identical permissions. 🤷‍♂️ Funny how OCP works in my favor sometimes…

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 6, 2024, 9:52am UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/6 "2024-07-06T09:52:45Z")

</div>

Hi @Denney-tech

Thanks for taking a look at this 🙂

I also looked at the SGID but as you’ve stated, in your deployment it’s working for you. What version of AWX and the operator do you have deployed?

I’ll have to continue looking at this on Monday.

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 9, 2024, 1:29pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/7 "2024-07-09T13:29:18Z")

</div>

@kurokobo @Denney-tech Do either of you know where/what is setting the SGID?

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [July 9, 2024, 2:37pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/8 "2024-07-09T14:37:17Z")

</div>

I don’t know.

I do know that, in OpenShift 4, both of my pods have identical uid/gid primary and supplimental group assignments. This is partially an OpenShift quirk, but might explain why my pods don’t have permission issues. (The reason I say partially is that I also have to custom build AWX to support the random uid/gid assignment from OpenShift).

As a result, I’m not using uid/gid 1000 anywhere.

So, the sticky bit might not impact my deployment due to lucky circumstances. I will try adjusting the security contexts to see if I can get a more apples to apples comparison.

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 10, 2024, 8:04am UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/9 "2024-07-10T08:04:41Z")

</div>

Thanks @Denney-tech 🙂

What versions are you running of AWX and the Operator, maybe I can try the version you’ve deployed and see if I have the same issues as I have currently 🤔

@kurokobo - Do you have any other suggestions?

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [July 10, 2024, 1:06pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/10 "2024-07-10T13:06:19Z")

</div>

I’m using the latest version, same as you.

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [July 10, 2024, 4:22pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/11 "2024-07-10T16:22:38Z")

</div>

@dbrennand @Denney-tech

Sorry for the delayed response. Interesting, I don’t know the cause immediately, but I suspect it’s probably an issue on the PV side.

As you mentioned, here are some facts:

- If `projects_persistence` is set to `true`, only the task pod has `fsGroup: 1000` as `securityContext`
- This is why the `awx` user in the task pod belongs to sub-group `1000`
- I believe the SGID you’ve mentioned is also due to this `fsGroup`

In my environment, the pods was in the following state. I can’t reproduce your issue here.

**Task pod:**

```bash
$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ ls -l /var/lib/awx/
total 0
drwxrwsr-x. 2 root 1000 6 Jul 10 15:31 projects
drwxr-xr-x. 3 root root 20 Jun 21 01:24 public
drwxrwxr-x. 1 root root 40 Jun 21 01:24 rsyslog
drwxr-xr-x. 3 root root 17 Jun 21 01:20 venv
bash-5.1$ ls -l /var/lib/awx/projects
total 0

```

**Web pod:**

```bash
$ kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ ls -l /var/lib/awx/
total 0
prw-------. 1 awx root 0 Jul 10 15:32 awxfifo
drwxrwsr-x. 2 root 1000 6 Jul 10 15:31 projects
drwxr-xr-x. 3 root root 20 Jun 21 01:24 public
drwxrwxr-x. 1 root root 40 Jun 21 01:24 rsyslog
drwxr-xr-x. 3 root root 17 Jun 21 01:20 venv
bash-5.1$ ls -l /var/lib/awx/projects/
total 0

```

Do you get any suspicious output with `ls -lZ` or `getfacl` ? Here is my case:

```bash
bash-5.1$ ls -lZ /var/lib/awx/
total 0
prw-------. 1 awx root system_u:object_r:container_file_t:s0 0 Jul 10 15:32 awxfifo
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0 6 Jul 10 15:31 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0 20 Jun 21 01:24 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0 40 Jun 21 01:24 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0 17 Jun 21 01:20 venv

```

```bash
bash-5.1$ getfacl /var/lib/awx/projects/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/awx/projects/
# owner: root
# group: 1000
# flags: -s-
user::rwx
group::rwx
other::r-x

```

Additionally, since it seems you are creating the PV with ReadWriteMany, is there a possibility that access to the PV changes depending on the Node where the Pod is located?  
How about running both the Web and Task pods on the same Node?

I’m not sure if it will reveal anything, but could you show me the configurations of the two pods using `kubectl -n awx describe pod` ?

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [July 11, 2024, 10:28am UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/12 "2024-07-11T10:28:26Z")

</div>

Hi @kurokobo

Thank you for your response 🙂

I checked and the `awx-task` and `awx-web` pods are running on the same node:

```auto
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
awx-migration-24.6.1-mf75r 0/1 Completed 0 14m 10.42.1.83 node2.example.com <none> <none>
awx-operator-controller-manager-745b55d94b-bhztl 2/2 Running 0 42h 10.42.2.128 node3.example.com <none> <none>
awx-task-8456d44c4f-rf9t8 4/4 Running 0 14m 10.42.2.140 node3.example.com <none> <none>
awx-web-bc6b797fc-ggvbq 3/3 Running 0 14m 10.42.2.139 node3.example.com <none> <none>

```

**Task Pod** :

```auto
$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root),1000
bash-5.1$ ls -l /var/lib/awx/
total 0
drwxrwsr-x. 2 root 1000 0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root 17 Jul 2 20:14 venv
bash-5.1$ ls -l /var/lib/awx/projects/
total 0

```

**Web Pod:**

```auto
$ kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$ id
uid=1000(awx) gid=0(root) groups=0(root)
bash-5.1$ ls -l /var/lib/awx/
total 0
prw-------. 1 awx root 0 Jul 11 07:48 awxfifo
drwxrwsr-x. 2 root 1000 0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root 17 Jul 2 20:14 venv

```

**Web Pod - SELinux Labels and getfacl** :

```auto
bash-5.1$ ls -lZ /var/lib/awx/
total 0
prw-------. 1 awx root system_u:object_r:container_file_t:s0:c277,c610 0 Jul 11 07:48 awxfifo
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974 0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c277,c610 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c277,c610 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c277,c610 17 Jul 2 20:14 venv
bash-5.1$ getfacl /var/lib/awx/projects/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/awx/projects/
# owner: root
# group: 1000
# flags: -s-
user::rwx
group::rwx
other::r-x

```

**Task Pod - SELinux Labels and getfacl** :

```auto
$ kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ ls -lahZ /var/lib/awx/
total 4.0K
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 43 Jul 11 07:50 .
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul 2 20:18 ..
-rw-------. 1 awx root system_u:object_r:container_file_t:s0:c924,c974 57 Jul 11 07:50 .bash_history
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 19 Jul 2 20:18 .local
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974 0 Jul 11 07:48 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul 2 20:14 venv
bash-5.1$ getfacl /var/lib/awx/projects/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/awx/projects/
# owner: root
# group: 1000
# flags: -s-
user::rwx
group::rwx
other::r-x

```

Nothing seems a miss with `getfacl`. However, when comparing my output to yours I see all the directories in `/var/lib/awx` have additional SELinux categories applied (e.g., `c924,c974`); not just the sensitivity label.

In the `awx-task` pod, they all seem to be the same (`c924,c974`) whereas, in the `awx-web` pod, I see `/var/lib/awx/projects` is different to all the others being `c924,c974` - same as in the `awx-task` pod. I wonder if the `awx-task` pod is started after the `awx-web` pod and the categories are being re-applied, and because this PVC is mounted across both pods, the `awx-task` pod categories are overriding on `/var/lib/awx/projects`.

I’m not very familiar with the Multi-Category Security concept in SELinux, but I do think this is playing a factor here.

Also, it looks like I failed to mention that SELinux is enabled on the underlying K3s nodes in my initial post 🙁 Apologies for that oversight.

## Fix

I’ve taken a look at [Configure a Security Context for a Pod or Container | Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#assign-selinux-labels-to-a-container) and I think setting the same SELinux label and categories for both pods may help:

```auto
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # ...
  # Security context configuration
  security_context_settings:
    seLinuxOptions:
      level: "s0:c924,c974"

```

Redeployed with this change and it has fixed the permission denied issues in the `awx-web` pod and I can now access the AWX GUI 🙂 🎉 🥳

```auto
kubectl -n awx exec -it deployment/awx-task -c awx-task -- bash
bash-5.1$ ls -lahZ /var/lib/awx/
total 0
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 22 Jul 11 10:01 .
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul 2 20:18 ..
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 19 Jul 2 20:18 .local
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974 0 Jul 11 10:01 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul 2 20:14 venv
bash-5.1$ exit
exit
kubectl -n awx exec -it deployment/awx-web -c awx-web -- bash
bash-5.1$ ls -lahZ /var/lib/awx/
total 0
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 37 Jul 11 10:01 .
drwxr-xr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul 2 20:18 ..
prw-------. 1 awx root system_u:object_r:container_file_t:s0:c924,c974 0 Jul 11 10:01 awxfifo
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 19 Jul 2 20:18 .local
drwxrwsr-x. 2 root 1000 system_u:object_r:container_file_t:s0:c924,c974 0 Jul 11 10:01 projects
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 20 Jul 2 20:18 public
drwxrwxr-x. 1 root root system_u:object_r:container_file_t:s0:c924,c974 40 Jul 2 20:18 rsyslog
drwxr-xr-x. 3 root root system_u:object_r:container_file_t:s0:c924,c974 17 Jul 2 20:14 venv
bash-5.1$ ls -lah /var/lib/awx/projects/
total 0
drwxrwsr-x. 2 root 1000 0 Jul 11 10:01 .
drwxrwxr-x. 1 root root 37 Jul 11 10:01 ..

```

**Final AWX spec for reference:**

```yaml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx
spec:
  # Disable IPv6
  ipv6_disabled: true
  # Define secret containing external PostgreSQL connection details
  postgres_configuration_secret: awx-postgres-configuration
  # Define secret containing the AWX admin password
  admin_password_secret: awx-admin-password
  # Define secret containing a custom CA bundle
  # Must contain the data field bundle-ca.crt
  bundle_cacert_secret: awx-custom-certs
  image: reg.example.com/ansible/awx
  image_version: 24.6.1
  image_pull_policy: Always
  # Security context configuration
  security_context_settings:
    seLinuxOptions:
      level: "s0:c924,c974"
  # Project persistance configuration
  projects_persistence: true
  projects_storage_class: rook-cephfs
  projects_storage_size: 10Gi
  projects_storage_access_mode: ReadWriteMany

```

### Versions

- Kubernetes: `v1.29.4+k3s1`
- AWX Operator: `2.19.1`
- AWX: `24.6.1`

Well… that was a fun adventure 😆 Thanks @kurokobo @Denney-tech for your responses on this 🙂

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [July 11, 2024, 12:10pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/13 "2024-07-11T12:10:58Z")

</div>

Congratulations!  
I’m glad to hear that you were able to resolve the issue 😃

This topic has also been very valuable knowledge for me. Thank you for diligently updating me on the situation!

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [July 11, 2024, 1:38pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/14 "2024-07-11T13:38:17Z")

</div>

That’s definitely an unusual problem, but now we’re all better equipped to help the next person with this particular problem! Looking at my pods, the directories all have `s0:c10,c26`, both task and web. So, it’s curious why yours was inconsistent between pods (or even directories).

@kurokobo good call on checking the security contexts, and good job @dbrennand finding a solution!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\
**Post date:** [August 10, 2024, 1:38pm UTC](https://forum.ansible.com/t/awx-web-permissionerror-errno-13-permission-denied-var-lib-awx-projects/7074/15 "2024-08-10T13:38:52Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.