AWX Cleanup Job Details process failed

Get Help
1

Hello,
I have a problem with AWX (v. 23.3.0) installed on AKS cluster.

The maintenance job “Cleanup Job Details” fails with this error and not clean the job history on AWX:

Traceback (most recent call last):
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/base.py”, line 313, in _commit
return self.connection.commit()
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 884, in commit
self.wait(self._commit_gen())
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 957, in wait
return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/waiting.py”, line 283, in wait_epoll
s = gen.send(ready)
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 532, in _commit_gen
yield from self._exec_command(b"COMMIT")
File “/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/connection.py”, line 466, in exec_command
raise e.error_from_result(result, encoding=pgconn_encoding(self.pgconn))
psycopg.errors.InternalError: could not find trigger 38872

In top of stdout log I read this message:

Table _unpartitioned_main_jobevent has no events in it
Dropping table _unpartitioned_main_jobevent since no records are newer than 2024-01-06 13:17:04.163481+00:00
jobs: 317 deleted, 12 skipped.

but no jobs was really deleted, I see the same total in jobs view.

Any idea about this error? It’s an issue due to something not clean in AWX db?

Thank you very much.
Elia

2

For now I solved running this manual command in awx-task container in bash shell:

awx-manage shell_plus

and then:

from awx.main.models import Job
for record_id in range(1, 1000):
    try:
        db_object = Job.objects.get(id=record_id)
        db_object.delete()
    except:
        pass

Now I can run the job “Cleanup Job Details” from AWX without errors.
I don’t know if a temporarily solved the issue and it will happens again.

Elia

3

hey @epi82 thanks for pointing this out, if you hit this again, can you let us know here and also open an issue on GitHub - ansible/awx: AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.

for future triage, AAP-16926 Delete unpartitioned tables in a separate transaction by AlanCoding · Pull Request #14572 · ansible/awx · GitHub might be of interest.