Jobs Failing with git.exc.GitCommandError: Cmd('git') failed due to: exit code(-9)

Archives AWX Project
,
1

Hi AWX team,

We have k8s deployment of AWX on EKS cluster. Issue I’m facing is Jobs fail intermittently with below error. Next second I run same job it runs fine. I tried google around but could not find any solution.

Is this a known issue? What is fix/workaround of this issue?

Stdout of failed job:

Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py”, line 457, in run
self.pre_run_hook(self.instance, private_data_dir)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py”, line 970, in pre_run_hook
RunProjectUpdate.make_local_copy(job.project, private_data_dir, scm_revision=job_revision)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py”, line 1374, in make_local_copy
git.Repo.clone_from(
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/repo/base.py”, line 1019, in clone_from
return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/repo/base.py”, line 960, in _clone
finalize_process(proc, stderr=stderr)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/util.py”, line 328, in finalize_process
proc.wait(**kwargs)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/git/cmd.py”, line 408, in wait
raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd(‘git’) failed due to: exit code(-9)
cmdline: git clone --branch=awx_internal/de9a0c84-3209-4c3b-8222-182440e579d4 --depth=1 --single-branch -v file:///var/lib/awx/projects/_493__kc1_azure_automations /tmp/awx_34605_50jrcqp5/project
stderr: 'Cloning into ‘/tmp/awx_34605_50jrcqp5/project’…
Updating files: 4% (173/3704)
Updating files: 5% (186/3704)
Updating files: 6% (223/3704)
Updating files: 7% (260/3704)
Updating files: 8% (297/3704)
Updating files: 9% (334/3704)
Updating files: 10% (371/3704)
Updating files: 11% (408/3704)
Updating files: 12% (445/3704)
Updating files: 13% (482/3704)
Updating files: 14% (519/3704)
Updating files: 15% (556/3704)
Updating files: 16% (593/3704)
Updating files: 17% (630/3704)
Updating files: 18% (667/3704)
Updating files: 19% (704/3704)
Updating files: 20% (741/3704)
Updating files: 21% (778/3704)
Updating files: 22% (815/3704)
Updating files: 23% (852/3704)
Updating files: 24% (889/3704)
Updating files: 25% (926/3704)
Updating files: 26% (964/3704)
Updating files: 27% (1001/3704)
Updating files: 28% (1038/3704)
Updating files: 29% (1075/3704)
Updating files: 30% (1112/3704)
Updating files: 31% (1149/3704)
Updating files: 32% (1186/3704)
Updating files: 33% (1223/3704)
Updating files: 34% (1260/3704)
Updating files: 35% (1297/3704)
Updating files: 36% (1334/3704)
Updating files: 37% (1371/3704)
Updating files: 38% (1408/3704)
Updating files: 39% (1445/3704)
Updating files: 40% (1482/3704)
Updating files: 41% (1519/3704)
Updating files: 42% (1556/3704)
Updating files: 43% (1593/3704)
Updating files: 44% (1630/3704)
Updating files: 45% (1667/3704)
Updating files: 46% (1704/3704)
Updating files: 47% (1741/3704)
Updating files: 48% (1778/3704)
Updating files: 48% (1796/3704)
Updating files: 49% (1815/3704)

’

2

Could be a bug in the gitpython module, lots of hits on google around this error e.g https://github.com/gitpython-developers/GitPython/discussions/1432

3

Is there any workaround/fix for this?

Thanks & Regards,
Ahmed

4

Hi,

Is there any workaround for this issue?

Thanks in advance,
Ahmed

5

Hi,

Wondering if this error is specific to the EKS cluster deployment, have you tried testing AWX on something like Kind, k3s, or minikube to see if the problem exists there? might be a useful exercise to narrow down the problem. Let us know what you find

AWX Team

6

Also, which version of AWX are you on? we addressed an issue around this in 21.5.0 https://github.com/ansible/awx/releases/tag/21.5.0

particularly this PR https://github.com/ansible/awx/pull/12356