# Automation pod started up and not completed. **URL:** \ **Category:** AWX Project\ **Tags:** awx, kubernetes\ **Created:** [April 11, 2023, 1:02pm UTC](https://forum.ansible.com/t/automation-pod-started-up-and-not-completed/37872 "2023-04-11T13:02:49Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![prabhu\_sridhar](https://avatars.discourse-cdn.com/v4/letter/p/858c86/32.png) [@prabhu\_sridhar](https://forum.ansible.com/u/prabhu_sridhar)\ **Post date:** [April 11, 2023, 1:02pm UTC](https://forum.ansible.com/t/automation-pod-started-up-and-not-completed/37872/1 "2023-04-11T13:02:49Z")
Hi AWX: 21.14.0 Env: k8s Simple job with debug out put, started up and completed. I could see the automated pod started up , However never completed. What could be the issue? NAME READY STATUS RESTARTS AGE automation-job-2-txs4m 1/1 Running 0 21m awx-7558596859-t9fjs 4/4 Running 0 43m awx-operator-8648b876d-9fzfg 1/1 Running 0 43m postgres-5d486686d5-b28g5 1/1 Running 0 43m ![11-04-_2023_15-00-40.png](https://us1.discourse-cdn.com/ansible/original/2X/2/27a31b4561dd1343e016e5e4dc68604e26c7e1e7.png) Regards Prabhu ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [April 12, 2023, 6:57pm UTC](https://forum.ansible.com/t/automation-pod-started-up-and-not-completed/37872/2 "2023-04-12T18:57:30Z")
Hello, Can you provide us with the logs from the pod that failed to complete? Please run “kubectl get pods [pod name] -o yaml” and “kubectl logs [pod name] \> pod\_logs.txt” and provide the output. The AWX Team ---
**Author:** ![prabhu\_sridhar](https://avatars.discourse-cdn.com/v4/letter/p/858c86/32.png) [@prabhu\_sridhar](https://forum.ansible.com/u/prabhu_sridhar)\ **Post date:** [April 20, 2023, 6:57am UTC](https://forum.ansible.com/t/automation-pod-started-up-and-not-completed/37872/3 "2023-04-20T06:57:34Z")
Hi AWX: 21.14.0 Installation method: AWX operation 1.4.0 K8s version:v1.25.6 Kustomize Version: v4.5.7 Thanks for the reply. I am trying to install AWX on a k8s cluster which does not have internet access. We have an internal repository from where the images are pulled. Installation was fine, AWX UI is accessible . kubectl get pods -n awx NAME READY STATUS RESTARTS AGE awx-d485d4b8-dcrmp 4/4 Running 0 49m awx-operator-74b56bdb9c-9qr2g 1/1 Running 0 18h postgres-5d486686d5-4djpd 1/1 Running 0 18h However after AWX is started up allocated memory is consumed and sees following warning on the logs. awx-task 2023-04-13 08:17:59,680 WARNING [f310dccbf2b244a480322d5be15c69eb] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.task\_manager, load: 377 / 27 │ │ awx-task 2023-04-13 08:17:59,699 WARNING [f310dccbf2b244a480322d5be15c69eb] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.dependency\_manager, load: 378 / 27 │ │ awx-task 2023-04-13 08:17:59,700 WARNING [f310dccbf2b244a480322d5be15c69eb] awx.main.dispatch Workers maxed, queuing awx.main.analytics.analytics\_tasks.send\_subsystem\_metrics, load: 37 │ │ awx-task 2023-04-13 08:18:19,709 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.cluster\_node\_heartbeat, load: 380 / 27 │ │ awx-task 2023-04-13 08:18:19,726 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx\_k8s\_reaper, load: 381 / 27 │ │ awx-task 2023-04-13 08:18:19,726 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx\_receptor\_workunit\_reaper, load: 382 / 27 │ │ awx-task 2023-04-13 08:18:19,726 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx\_periodic\_scheduler, load: 383 / 27 │ │ awx-task 2023-04-13 08:18:19,727 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.task\_manager, load: 384 / 27 │ │ awx-task 2023-04-13 08:18:19,727 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.dependency\_manager, load: 385 / 27 │ │ awx-task 2023-04-13 08:18:19,727 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.analytics.analytics\_tasks.send\_subsystem\_metrics, load: 38 │ │ awx-task 2023-04-13 08:18:39,737 WARNING [490d104b6c9d46a1a727f152e8acb9a3] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.task\_manager, load: 387 / 27 │ │ awx-task 2023-04-13 08:18:39,754 WARNING [490d104b6c9d46a1a727f152e8acb9a3] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.dependency\_manager, load: 388 / 27 │ │ awx-task 2023-04-13 08:18:39,755 WARNING [490d104b6c9d46a1a727f152e8acb9a3] awx.main.dispatch Workers maxed, queuing awx.main.analytics.analytics\_tasks.send\_subsystem\_metrics, load: 38 │ │ awx-task 2023-04-13 08:18:49,755 WARNING [efb98d9d82fc4422bbab1bbc4d1ff717] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx\_periodic\_scheduler, load: 390 / 27 │ The jobs of project sync up are taking a long time and sometimes ending in pending status. Please also find the logs attached. Regards Prabhu > **(attachments)** > > [kubectl get pods awx-d485d4b8-dcrmp.txt](https://forum.ansible.com/uploads/short-url/hn7MEIi7p4MQizwCvJbLyoUjMcy.txt) (16.4 KB) > [redis.txt](https://forum.ansible.com/uploads/short-url/yXoVUrpFZLBuzn5fDBLW3MdZLfT.txt) (5.4 KB) > [awx-operator.txt](https://forum.ansible.com/uploads/short-url/4qdfmLftvUmDpRIOGXwdgDX03EC.txt) (365 KB) > [awx-task.txt](https://forum.ansible.com/uploads/short-url/ctI7CR6Xx5v4rr1gh1ABVH7wU0B.txt) (75.8 KB) > [awx-web.txt](https://forum.ansible.com/uploads/short-url/7VtFG1Sa8jvtqUDddhD98yE4lcC.txt) (41.2 KB) > [awx-ee.txt](https://forum.ansible.com/uploads/short-url/jC5zXItH4cTExOvDgjHQoSZ7LqV.txt) (34.1 KB) ---
**Author:** ![AWX\_Project](https://avatars.discourse-cdn.com/v4/letter/a/eada6e/32.png) [@AWX\_Project](https://forum.ansible.com/u/AWX_Project)\ **Post date:** [April 26, 2023, 6:28pm UTC](https://forum.ansible.com/t/automation-pod-started-up-and-not-completed/37872/4 "2023-04-26T18:28:55Z")
can you run “kubectl exec -it deploy/awx-task -c awx-task – bash”, then “awx-manage run\_dispatcher --status”? this will give us an idea of what is currently running on the system when the system first starts, does the demo project sync okay? do any jobs (including project updates) run successfully? ---
**Author:** ![prabhu\_sridhar](https://avatars.discourse-cdn.com/v4/letter/p/858c86/32.png) [@prabhu\_sridhar](https://forum.ansible.com/u/prabhu_sridhar)\ **Post date:** [April 27, 2023, 2:34pm UTC](https://forum.ansible.com/t/automation-pod-started-up-and-not-completed/37872/5 "2023-04-27T14:34:51Z")
Hi Thanks for the reply. AWX (awx-task/awx-web) pod memory usage was not stable , keeps growing to the max level set after the startup. No, it was not possible to sync the project or run the jobs. Jobs are going to pending status . However, we disabled the FIPS on the systems temporarily ,after which memory usage is stable and git sync and jobs are running good. Any idea why FIPS enabled systems have this issue? please advise. Regards Prabhu