Hi
AWX: 21.14.0
Env: k8s
Simple job with debug out put, started up and completed.
I could see the automated pod started up , However never completed.
What could be the issue?
NAME READY STATUS RESTARTS AGE
automation-job-2-txs4m 1/1 Running 0 21m
awx-7558596859-t9fjs 4/4 Running 0 43m
awx-operator-8648b876d-9fzfg 1/1 Running 0 43m
postgres-5d486686d5-b28g5 1/1 Running 0 43m
Regards
Prabhu
Hello,
Can you provide us with the logs from the pod that failed to complete?
Please run “kubectl get pods [pod name] -o yaml” and “kubectl logs [pod name] > pod_logs.txt” and provide the output.
The AWX Team
Hi
AWX: 21.14.0
Installation method: AWX operation 1.4.0
K8s version:v1.25.6
Kustomize Version: v4.5.7
Thanks for the reply.
I am trying to install AWX on a k8s cluster which does not have internet access.
We have an internal repository from where the images are pulled.
Installation was fine, AWX UI is accessible .
kubectl get pods -n awx
NAME READY STATUS RESTARTS AGE
awx-d485d4b8-dcrmp 4/4 Running 0 49m
awx-operator-74b56bdb9c-9qr2g 1/1 Running 0 18h
postgres-5d486686d5-4djpd 1/1 Running 0 18h
However after AWX is started up allocated memory is consumed and sees following warning on the logs.
awx-task 2023-04-13 08:17:59,680 WARNING [f310dccbf2b244a480322d5be15c69eb] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.task_manager, load: 377 / 27 │
│ awx-task 2023-04-13 08:17:59,699 WARNING [f310dccbf2b244a480322d5be15c69eb] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.dependency_manager, load: 378 / 27 │
│ awx-task 2023-04-13 08:17:59,700 WARNING [f310dccbf2b244a480322d5be15c69eb] awx.main.dispatch Workers maxed, queuing awx.main.analytics.analytics_tasks.send_subsystem_metrics, load: 37 │
│ awx-task 2023-04-13 08:18:19,709 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.cluster_node_heartbeat, load: 380 / 27 │
│ awx-task 2023-04-13 08:18:19,726 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx_k8s_reaper, load: 381 / 27 │
│ awx-task 2023-04-13 08:18:19,726 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx_receptor_workunit_reaper, load: 382 / 27 │
│ awx-task 2023-04-13 08:18:19,726 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx_periodic_scheduler, load: 383 / 27 │
│ awx-task 2023-04-13 08:18:19,727 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.task_manager, load: 384 / 27 │
│ awx-task 2023-04-13 08:18:19,727 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.dependency_manager, load: 385 / 27 │
│ awx-task 2023-04-13 08:18:19,727 WARNING [097224a155fe4465aee20ef6b6d78a12] awx.main.dispatch Workers maxed, queuing awx.main.analytics.analytics_tasks.send_subsystem_metrics, load: 38 │
│ awx-task 2023-04-13 08:18:39,737 WARNING [490d104b6c9d46a1a727f152e8acb9a3] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.task_manager, load: 387 / 27 │
│ awx-task 2023-04-13 08:18:39,754 WARNING [490d104b6c9d46a1a727f152e8acb9a3] awx.main.dispatch Workers maxed, queuing awx.main.scheduler.tasks.dependency_manager, load: 388 / 27 │
│ awx-task 2023-04-13 08:18:39,755 WARNING [490d104b6c9d46a1a727f152e8acb9a3] awx.main.dispatch Workers maxed, queuing awx.main.analytics.analytics_tasks.send_subsystem_metrics, load: 38 │
│ awx-task 2023-04-13 08:18:49,755 WARNING [efb98d9d82fc4422bbab1bbc4d1ff717] awx.main.dispatch Workers maxed, queuing awx.main.tasks.system.awx_periodic_scheduler, load: 390 / 27 │
The jobs of project sync up are taking a long time and sometimes ending in pending status.
Please also find the logs attached.
Regards
Prabhu
kubectl get pods awx-d485d4b8-dcrmp.txt (16.4 KB)
redis.txt (5.4 KB)
awx-operator.txt (365 KB)
awx-task.txt (75.8 KB)
awx-web.txt (41.2 KB)
awx-ee.txt (34.1 KB)
can you run “kubectl exec -it deploy/awx-task -c awx-task – bash”, then “awx-manage run_dispatcher --status”? this will give us an idea of what is currently running on the system
when the system first starts, does the demo project sync okay? do any jobs (including project updates) run successfully?
Hi
Thanks for the reply.
AWX (awx-task/awx-web) pod memory usage was not stable , keeps growing to the max level set after the startup.
No, it was not possible to sync the project or run the jobs. Jobs are going to pending status .
However, we disabled the FIPS on the systems temporarily ,after which memory usage is stable and git sync and jobs are running good.
Any idea why FIPS enabled systems have this issue? please advise.
Regards
Prabhu
