AWX-EE Exceeded retries for reading stdout

,

Hello everyone,

I am continuously getting these errors and warnings from AWX-EE container. Anyone had such experience?

Leon,

Thanks

Stream closed EOF for awx/awx-task-569c979b6c-bqcgh (init-bundle-ca-trust)
awx-task-569c979b6c-7qcdq redis 1:M 13 Oct 2024 21:29:12.098 * 100 changes in 300 seconds. Saving...
awx-task-569c979b6c-7qcdq redis 1:M 13 Oct 2024 21:29:12.098 * Background saving started by pid 288
awx-task-569c979b6c-7qcdq redis 288:C 13 Oct 2024 21:29:12.104 * DB saved on disk
awx-task-569c979b6c-7qcdq redis 288:C 13 Oct 2024 21:29:12.105 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
awx-task-569c979b6c-7qcdq redis 1:M 13 Oct 2024 21:29:12.200 * Background saving terminated with success
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:04,984 INFO     [-] awx.analytics.job_lifecycle job-145657 pre run {"type": "job", "task_id": 145657, "state": "pre_run", "work_unit_id": null, "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:06,810 INFO     [-] awx.analytics.job_lifecycle job-145657 preparing playbook {"type": "job", "task_id": 145657, "state": "preparing_playbook", "work_unit_id": null, "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:08,605 INFO     [-] awx.analytics.job_lifecycle job-145657 running playbook {"type": "job", "task_id": 145657, "state": "running_playbook", "work_unit_id": null, "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:09,607 INFO     [-] awx.analytics.job_lifecycle job-145657 work unit id received {"type": "job", "task_id": 145657, "state": "work_unit_id_received", "work_unit_id": "hNbHX8TW", "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:09,890 INFO     [-] awx.analytics.job_lifecycle job-145657 work unit id assigned {"type": "job", "task_id": 145657, "state": "work_unit_id_assigned", "work_unit_id": "hNbHX8TW", "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:30,434 INFO     [-] awx.main.commands.run_callback_receiver Starting EOF event processing for Job 145657
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:30,455 INFO     [-] awx.analytics.job_lifecycle job-145657 post run {"type": "job", "task_id": 145657, "state": "post_run", "work_unit_id": "hNbHX8TW", "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:30,885 INFO     [-] awx.analytics.job_lifecycle job-145657 finalize run {"type": "job", "task_id": 145657, "state": "finalize_run", "work_unit_id": "hNbHX8TW", "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-task 2024-10-13 21:25:31,720 INFO     [-] awx.analytics.job_lifecycle job-145657 stats wrapup finished {"type": "job", "task_id": 145657, "state": "stats_wrapup_finished", "work_unit_id": "hNbHX8TW", "task_name": "Sync alerts to AC"}
Stream closed EOF for awx/awx-task-569c979b6c-bqcgh (init-database)
Stream closed EOF for awx/awx-task-569c979b6c-7qcdq (init-bundle-ca-trust)
Stream closed EOF for awx/awx-task-569c979b6c-7qcdq (init-database)
Stream closed EOF for awx/awx-task-569c979b6c-7qcdq (init-receptor)
awx-task-569c979b6c-dzfv9 redis 1:M 13 Oct 2024 21:29:23.030 * 100 changes in 300 seconds. Saving...
awx-task-569c979b6c-dzfv9 redis 1:M 13 Oct 2024 21:29:23.030 * Background saving started by pid 288
awx-task-569c979b6c-dzfv9 redis 288:C 13 Oct 2024 21:29:23.035 * DB saved on disk
awx-task-569c979b6c-dzfv9 redis 288:C 13 Oct 2024 21:29:23.036 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
awx-task-569c979b6c-dzfv9 redis 1:M 13 Oct 2024 21:29:23.131 * Background saving terminated with success
Stream closed EOF for awx/awx-task-569c979b6c-dzfv9 (init-database)
Stream closed EOF for awx/awx-task-569c979b6c-dzfv9 (init-bundle-ca-trust)
Stream closed EOF for awx/awx-task-569c979b6c-bqcgh (init-receptor)
Stream closed EOF for awx/awx-task-569c979b6c-dzfv9 (init-receptor)
awx-task-569c979b6c-dzfv9 awx-task 2024-10-13 21:25:03,683 INFO     [-] awx.analytics.job_lifecycle job-145657 created {"type": "job", "task_id": 145657, "state": "created", "work_unit_id": null, "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-dzfv9 awx-task 2024-10-13 21:25:04,177 INFO     [-] awx.analytics.job_lifecycle job-145657 acknowledged {"type": "job", "task_id": 145657, "state": "acknowledged", "work_unit_id": null, "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-dzfv9 awx-task 2024-10-13 21:25:04,476 INFO     [-] awx.analytics.job_lifecycle job-145657 waiting {"type": "job", "task_id": 145657, "state": "waiting", "work_unit_id": null, "task_name": "Sync alerts to AC"}
awx-task-569c979b6c-7qcdq awx-ee INFO 2024/10/13 21:25:30 Context was canceled while reading logs for pod awx/automation-job-145657-42mfz. Assuming pod has finished
awx-task-569c979b6c-7qcdq awx-ee ERROR 2024/10/13 21:25:33 Exceeded retries for reading stdout /tmp/receptor/awx-task-569c979b6c-7qcdq/hNbHX8TW/stdout
awx-task-569c979b6c-7qcdq awx-ee WARNING 2024/10/13 21:25:33 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection
awx-task-569c979b6c-7qcdq awx-ee WARNING 2024/10/13 21:25:33 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection
awx-task-569c979b6c-bqcgh redis 1:M 13 Oct 2024 21:29:22.039 * 100 changes in 300 seconds. Saving...
awx-task-569c979b6c-bqcgh redis 1:M 13 Oct 2024 21:29:22.040 * Background saving started by pid 288
awx-task-569c979b6c-bqcgh redis 288:C 13 Oct 2024 21:29:22.046 * DB saved on disk
awx-task-569c979b6c-bqcgh redis 288:C 13 Oct 2024 21:29:22.048 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
awx-task-569c979b6c-bqcgh redis 1:M 13 Oct 2024 21:29:22.140 * Background saving terminated with success
awx-task-768b7bc677-h5vwq awx-ee INFO 2024/10/11 04:47:20 Context was canceled while reading logs for pod awx/automation-job-143838-k5w64. Assuming pod has finished
awx-task-768b7bc677-h5vwq awx-ee ERROR 2024/10/11 04:47:20 Error deleting pod automation-job-143838-k5w64: client rate limiter Wait returned an error: context canceled