Project update failures between different projects

Archives AWX Project
,
1

Hi group,

We are running AWX 15.0.1 in production. It is a 10-node cluster running in Kubernetes.

Recently we started encountering failures running some jobs (and seemingly more apparent when running multiple instances of the same job in parallel), with an error about the project update having failed.

[…]
File “/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/tasks.py”, line 1542, in run raise AwxTaskError.TaskError(self.instance, rc) Exception: project_update 586629 (failed) encountered an error (rc=2), please see task stdout for details.

I took a look at the AWX task logs and found:

fatal: [localhost]: FAILED! => {“changed”: false, “cmd”: [“/usr/bin/git”, “fetch”, “–tags”, “origin”, “refs/heads/:refs/remotes/origin/”], “msg”: “Failed to download remote objects and refs: From https:///scm/\n ! [rejected] → origin/ (non-fast-forward)\n”}

In my experience this error usually occurs when someone does a push force to the repo in question and the AWX project does not have the option to delete before cloning.

Where it gets interesting is that the project update that failed is using the master branch, whereas the failure reported therein, from BitBucket, references the development branch. There is however another project in AWX pointed to the development branch.

Is it possible that these two projects are conflicting with each other somehow?

The project for our master branch has “update revision on launch” disabled, but “clean” and “delete on update” enabled.

The other project, using the development branch, has “update on launch enabled” and delete on update" disabled. I turned on the “delete on update” option for this branch just in case it might help, but we haven’t had much time to judge whether this has helped or not.

Thanks in advance for any help.

2

I’m seeing about one job a day failing with this error. That’s on AWX 14.1.0 / Ansible 2.9.11, also using BitBucket. Hoping someone can provide some tips.

We override the branch to launch a template with a commit id. The branch it rejects to fetch is often not even the one that contains the commit.
I’ve seen the fetch failure when trying to launch a job template with the commit id of the prod branch, where the non-fast-forward branch is just a feature branch (that like you said, I suspect had a force push). This probably makes a lot of sense to someone versed with fetching refs? :slight_smile:

In my case it’s a single project, and I have enabled all four options of Clean, Delete on update, Update revision on launch and Allow Branch override. Also experimented with a cache timeout of 0 vs 600s - seeing the issue with both.

I was wondering if this could be related - would a git fetch --force be able to get past this issue, and is there a way to set that flag in AWX…
https://github.com/ansible/ansible/issues/67972

Cheers.
D

3

Since I prefixed the default refspec with a plus in my projects I haven’t seen this job failure reoccur: “+refs/heads/:refs/remotes/origin/”. (Also reset the cache timeout back to 0 but left the other four scm update options enabled).