Permission denied for .git/refs/heads/awx_internal

Archives AWX Project
,
1

Hi,

…I ran into a really strange problem. I have 6 templates in a project, that are triggered via webhooks - everything working fine with them. Yesterday I added another Project/template that should be run periodically from AWX (no webhook). But whenever I try to launch the template, it tails with the following error:

PermissionError: [Errno 13] Permission denied: ‘/var/lib/awx/projects/_24__awx_vlan_deployment_061407147_pm/.git/refs/heads/awx_internal’

I tried different EE’s - even with the default EE:latest is gives the same error.

Any hint on how to troubleshoot this problem?

Thanks,

Andreas

2

This should be a file on the web container. Can you log into it and check the permissions on the file?

-The AWX Team

3

Hi,

…thanks a lot for your input – web-container looks like this:

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – /bin/ls -l /var/lib/awx/projects

total 20

drwxr-xr-x 5 awx root 4096 Jul 29 12:01 _10__awx_hello_world

-rwxr-xr-x 1 awx root 0 Jul 29 12:01 _10__awx_hello_world.lock

drwxr-xr-x 6 awx root 4096 Jul 19 08:24 _12__awx_vlan_deployment

-rwxr-xr-x 1 awx root 0 Jul 14 14:00 _12__awx_vlan_deployment.lock

drwxr-xr-x 5 awx root 4096 Jul 27 14:33 _21__device_backup_check

-rwxr-xr-x 1 awx root 0 Jul 27 14:33 _21__device_backup_check.lock

drwxr-xr-x 5 awx root 4096 Jul 27 16:15 _24__awx_vlan_deployment_061407147_pm

-rwxr-xr-x 1 awx root 0 Jul 27 16:14 _24__awx_vlan_deployment_061407147_pm.lock

drwxr-xr-x 4 awx root 4096 Jul 29 12:03 _8__awx_inventory

-rwxr-xr-x 1 awx root 0 Jul 14 14:00 _8__awx_inventory.lock

…and the project’s .git is root-owned – which is probably the problem:

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – /bin/ls -la /var/lib/awx/projects/_10__awx_hello_world/.git/refs/heads

total 12

drwxr-xr-x 2 root root 4096 Jul 29 12:20 .

drwxr-xr-x 5 root root 4096 Jul 29 12:01 …

-rw-r–r-- 1 root root 41 Jul 29 12:01 main

However, I’m not allowed to change the ownership within the container:

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – /bin/chown -R awx.root /var/lib/awx/projects/_10__awx_hello_world

chown: changing ownership of ‘/var/lib/awx/projects/_10__awx_hello_world/.git/config’: Operation not permitted

chown: changing ownership of ‘/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/remotes/origin/HEAD’: Operation not permitted

chown: changing ownership of ‘/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/remotes/origin’: Operation not permitted

chown: changing ownership of ‘/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/remotes’: Operation not permitted

chown: changing ownership of ‘/var/lib/awx/projects/_10__awx_hello_world/.git/logs/refs/heads/main’: Operation not permitted

I would suspect, that the git checkout is performed using the wrong (root) user – is there a place, where this can be configured?

Thanks,

Andreas

4

…seems my mail-response got lost - so I answer via web-interface:

5

The awx user should be in the root group so, from a quick look, that should actually be ok as the files/dirs look rx by the root group.
Can you confirm that the awx user is in the root group in both the web and task containers?
Also, are you using the projects_persistence setting?

-The AWX Team

6

Hi,

…web-container - no:

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-web – cat /etc/group
root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:
[…]

task-container - no:

root@net04-mgmt-vm:~# kubectl exec awx-7dd9bf96c7-ml7zf -c awx-task – cat /etc/group
root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:

[…]

and yes - project_persistence is true:

root@net04-mgmt-vm:~/awx-operator# grep projects_persistence awx-deploy.yml
projects_persistence: true
root@net04-mgmt-vm:~/awx-operator#

Thanks,

andreas

7

Hi,

…just double-checked and I can tell that the awx-user is in the root-group (sorry for the wrong information above):

root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-web – /bin/id
uid=1000(awx) gid=0(root) groups=0(root),1000
root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-task – /bin/id
uid=1000(awx) gid=0(root) groups=0(root),1000
root@net04-mgmt-vm:~/awx-operator#

but when looking at the directories, they’re not group-writable:

root@net04-mgmt-vm:~/awx-operator# kubectl exec awx-7dd9bf96c7-rn25c -c awx-web – /bin/ls -ld /var/lib/awx/projects/_21__device_backup_check/.git/refs/heads/
drwxr-xr-x 2 root root 4096 Jul 27 14:33 /var/lib/awx/projects/_21__device_backup_check/.git/refs/heads/
root@net04-mgmt-vm:~/awx-operator#

And that’s why the “mkdir()” fails:

File “/usr/lib64/python3.9/os.py”, line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: ‘/var/lib/awx/projects/_24__awx_vlan_deployment_061407147_pm/.git/refs/heads/awx_internal’

Thanks,

Andreas

8

This problem sounds similar to this opened issue https://github.com/ansible/awx-operator/issues/931

If not, this might be a good candidate to open a new issue for

AWX Team

9

OK - just verified. It boils down to the control-plane EE - if I use the default from quay.io/ansible/awx-ee:latest, everything works fine. Have to have a look at the custom image - which should be based on the quay.io image :frowning: