# Permission denied for .git/refs/heads/awx\_internal **URL:** \ **Category:** AWX Project\ **Tags:** awx, ee\ **Created:** [July 28, 2022, 10:53am UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914 "2022-07-28T10:53:15Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![andybourges](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@andybourges](https://forum.ansible.com/u/andybourges)\ **Post date:** [July 28, 2022, 10:53am UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/1 "2022-07-28T10:53:15Z")
Hi, …I ran into a really strange problem. I have 6 templates in a project, that are triggered via webhooks - everything working fine with them. Yesterday I added another Project/template that should be run periodically from AWX (no webhook). But whenever I try to launch the template, it tails with the following error: PermissionError: [Errno 13] Permission denied: ‘/var/lib/awx/projects/\_24\_\_awx\_vlan\_deployment\_061407147\_pm/.git/refs/heads/awx\_internal’ I tried different EE’s - even with the default EE:latest is gives the same error. Any hint on how to troubleshoot this problem? Thanks, Andreas ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [August 1, 2022, 12:24pm UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/2 "2022-08-01T12:24:37Z")
This should be a file on the web container. Can you log into it and check the permissions on the file? -The AWX Team ---
**Author:** ![andybourges](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@andybourges](https://forum.ansible.com/u/andybourges)\ **Post date:** [August 1, 2022, 12:43pm UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/3 "2022-08-01T12:43:12Z")
Hi, …thanks a lot for your input – web-container looks like this: root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – /bin/ls -l /var/lib/awx/projects total 20 drwxr-xr-x 5 awx root 4096 Jul 29 12:01 \_10\_\_awx\_hello\_world -rwxr-xr-x 1 awx root 0 Jul 29 12:01 \_10\_\_awx\_hello\_world.lock drwxr-xr-x 6 awx root 4096 Jul 19 08:24 \_12\_\_awx\_vlan\_deployment -rwxr-xr-x 1 awx root 0 Jul 14 14:00 \_12\_\_awx\_vlan\_deployment.lock drwxr-xr-x 5 awx root 4096 Jul 27 14:33 \_21\_\_device\_backup\_check -rwxr-xr-x 1 awx root 0 Jul 27 14:33 \_21\_\_device\_backup\_check.lock drwxr-xr-x 5 awx root 4096 Jul 27 16:15 \_24\_\_awx\_vlan\_deployment\_061407147\_pm -rwxr-xr-x 1 awx root 0 Jul 27 16:14 \_24\_\_awx\_vlan\_deployment\_061407147\_pm.lock drwxr-xr-x 4 awx root 4096 Jul 29 12:03 \_8\_\_awx\_inventory -rwxr-xr-x 1 awx root 0 Jul 14 14:00 \_8\_\_awx\_inventory.lock …and the project’s .git is root-owned – which is probably the problem: root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – /bin/ls -la /var/lib/awx/projects/\_10\_\_awx\_hello\_world/.git/refs/heads total 12 drwxr-xr-x 2 root root 4096 Jul 29 12:20 . drwxr-xr-x 5 root root 4096 Jul 29 12:01 … -rw-r–r-- 1 root root 41 Jul 29 12:01 main However, I’m not allowed to change the ownership within the container: root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – /bin/chown -R awx.root /var/lib/awx/projects/\_10\_\_awx\_hello\_world chown: changing ownership of ‘/var/lib/awx/projects/\_10\_\_awx\_hello\_world/.git/config’: Operation not permitted chown: changing ownership of ‘/var/lib/awx/projects/\_10\_\_awx\_hello\_world/.git/logs/refs/remotes/origin/HEAD’: Operation not permitted chown: changing ownership of ‘/var/lib/awx/projects/\_10\_\_awx\_hello\_world/.git/logs/refs/remotes/origin’: Operation not permitted chown: changing ownership of ‘/var/lib/awx/projects/\_10\_\_awx\_hello\_world/.git/logs/refs/remotes’: Operation not permitted chown: changing ownership of ‘/var/lib/awx/projects/\_10\_\_awx\_hello\_world/.git/logs/refs/heads/main’: Operation not permitted I would suspect, that the git checkout is performed using the wrong (root) user – is there a place, where this can be configured? Thanks, Andreas ---
**Author:** ![andybourges](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@andybourges](https://forum.ansible.com/u/andybourges)\ **Post date:** [August 1, 2022, 2:16pm UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/4 "2022-08-01T14:16:28Z")
…seems my mail-response got lost - so I answer via web-interface: ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [August 1, 2022, 2:58pm UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/5 "2022-08-01T14:58:12Z")
The awx user should be in the root group so, from a quick look, that should actually be ok as the files/dirs look rx by the root group. Can you confirm that the awx user is in the root group in both the web and task containers? Also, are you using the projects\_persistence setting? -The AWX Team ---
**Author:** ![andybourges](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@andybourges](https://forum.ansible.com/u/andybourges)\ **Post date:** [August 1, 2022, 3:17pm UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/6 "2022-08-01T15:17:51Z")
Hi, …web-container - no: root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – cat /etc/group root:x:0: bin:x:1: daemon:x:2: sys:x:3: […] task-container - no: root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-task – cat /etc/group root:x:0: bin:x:1: daemon:x:2: sys:x:3: […] and yes - project\_persistence is true: root@net04-mgmt-vm:~/awx-operator# grep projects\_persistence awx-deploy.yml projects\_persistence: true root@net04-mgmt-vm:~/awx-operator# Thanks, andreas ---
**Author:** ![andybourges](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@andybourges](https://forum.ansible.com/u/andybourges)\ **Post date:** [August 2, 2022, 4:09am UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/7 "2022-08-02T04:09:08Z")
Hi, …just double-checked and I can tell that the awx-user **_is_** in the root-group (sorry for the wrong information above): root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-web – /bin/id uid=1000(awx) gid=0(root) groups=0(root),1000 root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-task – /bin/id uid=1000(awx) gid=0(root) groups=0(root),1000 root@net04-mgmt-vm:~/awx-operator# but when looking at the directories, they’re not group-writable: root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-web – /bin/ls -ld /var/lib/awx/projects/\_21\_\_device\_backup\_check/.git/refs/heads/ drwxr-xr-x 2 root root 4096 Jul 27 14:33 /var/lib/awx/projects/\_21\_\_device\_backup\_check/.git/refs/heads/ root@net04-mgmt-vm:~/awx-operator# And that’s why the “mkdir()” fails: File “/usr/lib64/python3.9/os.py”, line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: ‘/var/lib/awx/projects/\_24\_\_awx\_vlan\_deployment\_061407147\_pm/.git/refs/heads/awx\_internal’ Thanks, Andreas ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [August 3, 2022, 3:03pm UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/8 "2022-08-03T15:03:53Z")
This problem sounds similar to this opened issue [https://github.com/ansible/awx-operator/issues/931](https://github.com/ansible/awx-operator/issues/931) If not, this might be a good candidate to open a new issue for AWX Team ---
**Author:** ![andybourges](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@andybourges](https://forum.ansible.com/u/andybourges)\ **Post date:** [August 10, 2022, 7:30am UTC](https://forum.ansible.com/t/permission-denied-for-git-refs-heads-awx-internal/36914/9 "2022-08-10T07:30:25Z")
OK - just verified. It boils down to the control-plane EE - if I use the default from [quay.io/ansible/awx-ee:latest](http://quay.io/ansible/awx-ee:latest), everything works fine. Have to have a look at the custom image - which should be based on the [quay.io](http://quay.io) image ☹