AWX jobs pending

Ran into an issue where I can see jobs queued up for a period of time. I can see using the grafana dashboard pending jobs steadily increased
I see that jobs did continue to run.
In the logs I was able to find:

"log":"2025-12-03 15:46:04,167 ERROR    [550823dfc8e04204bfeb4e1f99e1b71f] awx.main.dispatch Worker failed to run task awx.main.scheduler.tasks.dependency_manager(*[], **{}"

I’m trying to track down what other log messages or what might cause the jobs to queue into a pending state. I can look back as far as 30 days in grafana/metrics and see that never happened previously