Running the AWX operator in AWX and we’ve noticed some times when a job targets a lot of hosts the results of the automation job will not get posted to the webUI or will show pending or running for a very long time when in reality the container for the automation job has already completed long ago.
Has anyone else run into this sort of behavior or know of an issue that would cause this?
Sounds similar to this thread Job end in Failed without error when it had large logs · Issue #1675 · ansible/awx-operator · GitHub see if tweaking those settings make a difference for you
This could be related to an old kubernetes version, update to 1.29 and I am guessing this will disappear. Pod logs stop being pulled when container log files are rotated · Issue #446 · ansible/receptor · GitHub Missing job output and log lines (Task was destroyed but it is pending!) · Issue #14158 · ansible/awx · GitHub Fixed by Fix: kubelet will not output logs after log file is rotated by xyz-li · Pull Request #115702 · kubernetes/kubernetes · GitHub
Hey, thanks for the reply. Its not that the logs are two large. This has happened with smaller jobs. Its just that they are very very delayed. We did experience the referenced issue in the past however and tweaked some memory settings and etc which largely helped.
Hey, thanks for the reply. Were actually on AWS EKS with version v1.30.0. I dont believe anything has actually been rotated because I can see the logs in the pod logs all there, they are just coming back to the UI very very slowly but are filling in.
Oh, so you eventually do get the logs but they just take forever to populate, and you’re running in the Cloud? Sorry, no experience with that kind of AWX setup but that context is certainly interesting.
if you reload the awx page do the logs appear? because then it could also be problems with the websocket