# AWX Jobs stuck pending after upgrade to 2.19.1 / v24.6.1

**URL:** <https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191>\
**Category:** Get Help\
**Tags:** awx-operator, awx\
**Created:** [October 11, 2024, 4:41pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191 "2024-10-11T16:41:15Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![trippinnik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/trippinnik/32/4308_2.png) [@trippinnik](https://forum.ansible.com/u/trippinnik)\
**Post date:** [October 11, 2024, 4:41pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/1 "2024-10-11T16:41:15Z")

</div>

I updated by the usual process with awx-operator. Seemed to complete but jobs do not start. Just stay pending. I see this in the logs on the task container.

```auto
django.db.utils.ProgrammingError: unrecognized configuration parameter "idle_session_timeout"
2024-10-11 16:35:06,664 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch.periodic scheduler found send_subsystem_metrics to run, 0.01080179214477539 seconds after target
2024-10-11 16:35:06,665 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch.periodic Scheduler next run is metrics_gather in 7.988242149353027 seconds
2024-10-11 16:35:06,665 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch task 2616f50c-c5e3-4f84-8326-e2b7104fb047 starting awx.main.analytics.analytics_tasks.send_subsystem_metrics(*[])
2024-10-11 16:35:06,811 INFO [-] awx.main.wsrelay Producer 10.51.194.142-job_events-25636 has no subscribers, shutting down.
2024-10-11 16:35:06,813 INFO [-] awx.main.wsrelay Producer 10.51.194.142-jobs-summary has no subscribers, shutting down.
2024-10-11 16:35:14,506 DEBUG [-] awx.main.wsrelay Web host awx-demo-web-c4d6f5ff-jp88b (10.51.194.142) online heartbeat received.
2024-10-11 16:35:14,667 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch.periodic scheduler found metrics_gather to run, 0.013670206069946289 seconds after target
2024-10-11 16:35:14,671 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch.periodic Scheduler next run is tower_scheduler in 1.9822404384613037 seconds
2024-10-11 16:35:16,660 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch.periodic scheduler found tower_scheduler to run, 0.007393598556518555 seconds after target
2024-10-11 16:35:16,661 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch.periodic Scheduler next run is cluster_heartbeat in 0.9922580718994141 seconds
2024-10-11 16:35:16,662 DEBUG [d949237159a54888b7c5ab8450725256] awx.main.dispatch task 1eb4d8e3-fd9b-4290-b89a-5f0fe4433a7e starting awx.main.tasks.system.awx_periodic_scheduler(*[])
2024-10-11 16:35:16,683 ERROR [d949237159a54888b7c5ab8450725256] awx.main.dispatch Worker failed to run task awx.main.tasks.system.awx_periodic_scheduler(*[], **{}
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.UndefinedObject: unrecognized configuration parameter "idle_session_timeout"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 103, in perform_work
    result = self.run_callable(body)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 78, in run_callable
    return _call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/system.py", line 719, in awx_periodic_scheduler
    with advisory_lock('awx_periodic_scheduler_lock', lock_session_timeout_milliseconds=lock_session_timeout_milliseconds, wait=False) as acquired:
  File "/usr/lib64/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/pglock.py", line 19, in advisory_lock
    idle_session_timeout = cur.execute('SHOW idle_session_timeout').fetchone()[0]
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)

```

---

<div class="post-metadata">

**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\
**Post date:** [October 14, 2024, 9:05am UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/2 "2024-10-14T09:05:19Z")

</div>

Hi @trippinnik

EDIT: Sorry I see you’re using `24.6.1` from the title.

I think based on line:

```auto
 File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/pglock.py", line 19, in advisory_lock
    idle_session_timeout = cur.execute('SHOW idle_session_timeout').fetchone()[0]

```

This looks to be related to: [Add TASK\_MANAGER\_LOCK\_TIMEOUT by TheRealHaoLiu · Pull Request #15300 · ansible/awx (github.com)](https://github.com/ansible/awx/pull/15300)? 🤔

cc @TheRealHaoLiu

---

<div class="post-metadata">

**Author:** ![trippinnik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/trippinnik/32/4308_2.png) [@trippinnik](https://forum.ansible.com/u/trippinnik)\
**Post date:** [October 14, 2024, 11:56am UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/3 "2024-10-14T11:56:19Z")

</div>

So it’s not included yet? How do i get it working?

---

<div class="post-metadata">

**Author:** ![ildjarn](https://avatars.discourse-cdn.com/v4/letter/i/8491ac/32.png) [@ildjarn](https://forum.ansible.com/u/ildjarn)\
**Post date:** [October 16, 2024, 6:26am UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/4 "2024-10-16T06:26:51Z")

</div>

Based on ‘SHOW idle\_session\_timeout’ this is a postgres query. This parameter was added to Postgres 14, so if you’re still on 13, you probably need to upgrade Postgres.

---

<div class="post-metadata">

**Author:** ![trippinnik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/trippinnik/32/4308_2.png) [@trippinnik](https://forum.ansible.com/u/trippinnik)\
**Post date:** [October 18, 2024, 7:28pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/5 "2024-10-18T19:28:52Z")

</div>

I upgraded Postgres to 15.8 and now I get:

```auto
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
psycopg.errors.SyntaxError: trailing junk after numeric literal at or near "1d"
LINE 1: SET idle_in_transaction_session_timeout = 1d
                                                  ^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 103, in perform_work
    result = self.run_callable(body)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 78, in run_callable
    return _call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/system.py", line 719, in awx_periodic_scheduler
    with advisory_lock('awx_periodic_scheduler_lock', lock_session_timeout_milliseconds=lock_session_timeout_milliseconds, wait=False) as acquired:
  File "/usr/lib64/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/pglock.py", line 26, in advisory_lock
    cur.execute(f"SET idle_in_transaction_session_timeout = {idle_in_transaction_session_timeout}")
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
    raise ex.with_traceback(None)
django.db.utils.ProgrammingError: trailing junk after numeric literal at or near "1d"
LINE 1: SET idle_in_transaction_session_timeout = 1d

```

---

<div class="post-metadata">

**Author:** ![trippinnik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/trippinnik/32/4308_2.png) [@trippinnik](https://forum.ansible.com/u/trippinnik)\
**Post date:** [October 18, 2024, 7:30pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/6 "2024-10-18T19:30:22Z")

</div>

Also I don’t see anything in release notes about change in Postgres version requirements.

---

<div class="post-metadata">

**Author:** ![ildjarn](https://avatars.discourse-cdn.com/v4/letter/i/8491ac/32.png) [@ildjarn](https://forum.ansible.com/u/ildjarn)\
**Post date:** [October 18, 2024, 7:52pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/7 "2024-10-18T19:52:01Z")

</div>

Sorry can’t help you with that error.  
But see [Database Configuration - Ansible AWX Operator Documentation](https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/database-configuration.html) for the database requirement.

---

<div class="post-metadata">

**Author:** ![mbutton77](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mbutton77/32/2432_2.png) [@mbutton77](https://forum.ansible.com/u/mbutton77)\
**Post date:** [December 19, 2024, 1:56pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/8 "2024-12-19T13:56:16Z")

</div>

Hi Nik !

Just got the same stacktrace, while upgrading AWX to 24.6.1 on PG 15.10 : did you find a solution ? 🤞

---

<div class="post-metadata">

**Author:** ![mbutton77](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mbutton77/32/2432_2.png) [@mbutton77](https://forum.ansible.com/u/mbutton77)\
**Post date:** [December 19, 2024, 2:52pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/9 "2024-12-19T14:52:14Z")

</div>

Some explanations are available in the following issue :

[AWX task error: django.db.utils.ProgrammingError: trailing junk after numeric literal at or near 10m, LINE 1: SET idle\_in\_transaction\_session\_timeout = 500s · Issue #15406 · ansible/awx](https://github.com/ansible/awx/issues/15406)

---

<div class="post-metadata">

**Author:** ![mbutton77](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mbutton77/32/2432_2.png) [@mbutton77](https://forum.ansible.com/u/mbutton77)\
**Post date:** [December 19, 2024, 3:11pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/10 "2024-12-19T15:11:45Z")

</div>

Ok, will be fixed in the upcoming release.

More info :

> <https://github.com/ansible/awx/issues/15406#issuecomment-2554515918>
>
> \### Please confirm the following
> 
> \- \[X\] I agree to follow this project's \[code… of conduct\](https://docs.ansible.com/ansible/latest/community/code\_of\_conduct.html).
> \- \[X\] I have checked the \[current issues\](https://github.com/ansible/awx/issues) for duplicates.
> \- \[X\] I understand that AWX is open source software provided for free and that I might not receive a timely response.
> \- \[X\] I am \*\*NOT\*\* reporting a (potential) security vulnerability. (These should be emailed to \`security@ansible.com\` instead.)
> 
> \### Bug Summary
> 
> awx task error "django.db.utils.ProgrammingError: trailing junk after numeric literal at or near..." when pg config "idle\_in\_transaction\_session\_timeout"
> 
> \### AWX version
> 
> 24.6.1
> 
> \### Select the relevant components
> 
> \- \[\] UI
> \- \[\] UI (tech preview)
> \- \[X\] API
> \- \[\] Docs
> \- \[\] Collection
> \- \[\] CLI
> \- \[\] Other
> 
> \### Installation method
> 
> openshift
> 
> \### Modifications
> 
> no
> 
> \### Ansible version
> 
> \_No response\_
> 
> \### Operating system
> 
> \_No response\_
> 
> \### Web browser
> 
> Chrome
> 
> \### Steps to reproduce
> 
> 1. openshift use awx operator 2.19.1
> 2. use default config install awxoperator and awx instance
> 3. exec pg container change pc config, set global idle\_in\_transaction\_session\_timeout to 600000 or other value, pg store will change the unit, restart the pg process
> 4. check the awx task log, find the error
> 
> \### Expected results
> 
> no error log
> 
> \### Actual results
> 
> many error log similar to " django.db.utils.ProgrammingError: trailing junk after numeric literal at or near "500s"
> LINE 1: SET idle\_in\_transaction\_session\_timeout = 500s"
> 
> \### Additional information
> 
> \`\`\`
> 2024-07-29 05:25:58,176 ERROR \[-\] awx.main.dispatch Worker failed to run task awx.main.tasks.system.awx\_periodic\_scheduler(\*\[\], \*\*{}
> Traceback (most recent call last):
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in \_execute
> return self.cursor.execute(sql)
> ^^^^^^^^^^^^^^^^^^^^^^^^
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
> raise ex.with\_traceback(None)
> psycopg.errors.SyntaxError: trailing junk after numeric literal at or near "500s"
> LINE 1: SET idle\_in\_transaction\_session\_timeout = 500s
> ^
> The above exception was the direct cause of the following exception:
> Traceback (most recent call last):
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 103, in perform\_work
> result = self.run\_callable(body)
> ^^^^^^^^^^^^^^^^^^^^^^^
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 78, in run\_callable
> return \_call(\*args, \*\*kwargs)
> ^^^^^^^^^^^^^^^^^^^^^^
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/system.py", line 719, in awx\_periodic\_scheduler
> with advisory\_lock('awx\_periodic\_scheduler\_lock', lock\_session\_timeout\_milliseconds=lock\_session\_timeout\_milliseconds, wait=False) as acquired:
> File "/usr/lib64/python3.11/contextlib.py", line 144, in exit
> next(self.gen)
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/pglock.py", line 26, in advisory\_lock
> cur.execute(f"SET idle\_in\_transaction\_session\_timeout = {idle\_in\_transaction\_session\_timeout}")
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
> return self.\_execute\_with\_wrappers(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 80, in \_execute\_with\_wrappers
> return executor(sql, params, many, context)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 84, in \_execute
> with self.db.wrap\_database\_errors:
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/utils.py", line 91, in exit
> raise dj\_exc\_value.with\_traceback(traceback) from exc\_value
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/backends/utils.py", line 87, in \_execute
> return self.cursor.execute(sql)
> ^^^^^^^^^^^^^^^^^^^^^^^^
> File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/psycopg/cursor.py", line 732, in execute
> raise ex.with\_traceback(None)
> django.db.utils.ProgrammingError: trailing junk after numeric literal at or near "500s"
> LINE 1: SET idle\_in\_transaction\_session\_timeout = 500s
> \`\`\`

---

<div class="post-metadata">

**Author:** ![onefourfive](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/onefourfive/32/1775_2.png) [@onefourfive](https://forum.ansible.com/u/onefourfive)\
**Post date:** [December 20, 2024, 6:58pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/11 "2024-12-20T18:58:36Z")

</div>

> Ok, will be fixed in the upcoming release.

Just curious, as AWX is not doing releases, what exactly does `upcoming release` mean?

---

<div class="post-metadata">

**Author:** ![mbutton77](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mbutton77/32/2432_2.png) [@mbutton77](https://forum.ansible.com/u/mbutton77)\
**Post date:** [January 6, 2025, 3:35pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/12 "2025-01-06T15:35:25Z")

</div>

Hi @onefourfive !

Normally, there is a release every month but I guess there has been some delay because of the problem induced by the DAB RBAC functionality.

But I may have missed something … when you say AWX is not doing releases, do you mean there was a change in the release policy ?

---

<div class="post-metadata">

**Author:** ![mbutton77](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mbutton77/32/2432_2.png) [@mbutton77](https://forum.ansible.com/u/mbutton77)\
**Post date:** [January 7, 2025, 7:22am UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/13 "2025-01-07T07:22:31Z")

</div>

Oh damn, I completely went through …  
For those in the same case :

[Streamlining AWX Releases - Project Discussions - Ansible](https://forum.ansible.com/t/streamlining-awx-releases/6894/1)

Thanks for pointing it out @onefourfive

---

<div class="post-metadata">

**Author:** ![trippinnik](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/trippinnik/32/4308_2.png) [@trippinnik](https://forum.ansible.com/u/trippinnik)\
**Post date:** [December 4, 2025, 4:21pm UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/14 "2025-12-04T16:21:07Z")

</div>

Error inception, stumbled on my one post after having a ton of jobs queue up. Anyone find a workaround besides waiting for a release that is never coming?

---

<div class="post-metadata">

**Author:** ![ildjarn](https://avatars.discourse-cdn.com/v4/letter/i/8491ac/32.png) [@ildjarn](https://forum.ansible.com/u/ildjarn)\
**Post date:** [December 5, 2025, 9:52am UTC](https://forum.ansible.com/t/awx-jobs-stuck-pending-after-upgrade-to-2-19-1-v24-6-1/10191/15 "2025-12-05T09:52:08Z")

</div>

Redeploy so that the database upgrade pod is executed again?

```auto
kubectl rollout restart deployment/<deployment-name>

```