AWX job terminated unexpectedly

epi82 · February 7, 2024, 4:48pm

@kurokobo Thank you for the reply.

For A) the “image_pull_policy” I think I can temporarily modify the awx-operator spec and set it to “Always” as I can see here:
https://docs.ansible.com/automation-controller/4.0.0/html/administration/operator_advanced_configurations.html#deploy-a-specific-version-of-awx

For B) any advice on how to remove the cached image “awx-ee:latest” from AKS?

epi82 · February 22, 2024, 8:07am

@kurokobo for point B what is the best way to do this?

Thank you very much.

kurokobo · February 23, 2024, 4:34pm

I don’t know which is the best, but there are some solutions:

Deploy dummy pod that uses awx-ee:latest with Always pull policy on each nodes (This is not a way to remove cached images but a way to renew cached images by forcibly re-pulling the images)
Connecting to AKS nodes and use crictl to remove cached images
- Connect to Azure Kubernetes Service (AKS) cluster nodes - Azure Kubernetes Service | Microsoft Learn
- Cluster configuration in Azure Kubernetes Services (AKS) - Azure Kubernetes Service | Microsoft Learn
Use Image Cleaner and ImageList
- Use Image Cleaner on Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn

epi82 · February 26, 2024, 9:03am

Thank you @kurokobo for your reply. I will try one of these.

After this there is a way to be sure that I’m using latest image? Maybe something from pod logs… I don’t know.

kurokobo · February 26, 2024, 9:21am

You can gather sha256 of the image by containerStatusess from kubectl get pod <pod-name> -o yaml:

$ kubectl -n awx get pod awx-task-97b468984-n5tmf -o yaml
...
status:
  ...
  containerStatuses:
  - ...
    image: quay.io/ansible/awx-ee:latest
    imageID: quay.io/ansible/awx-ee@sha256:a4e53d8d95a90fea0c72a62b7c42eb15e52838841f43bf2dac64dc3fdc429f56
    ...
    name: awx-ee
    ...

The sha256 of the latest latest image can be retrieved via https://quay.io/repository/ansible/awx-ee?tab=tags&tag=latest

The latest tag is updated every 12 hours, so it will quickly become not the latest and will disappear from the list, but at least it will include fixes for the issues if it is up-to-date at this time.

TheRealHaoLiu · February 27, 2024, 3:21am

im thinking pinning awx-ee image in the operator…to the versioned awx-ee image

thoughts?

kurokobo · February 27, 2024, 4:30am

Of course it should be

Related to:

github.com/ansible/awx-operator

control_plane_ee_image still uses "latest" tag in release

opened 12:36PM - 07 Jul 23 UTC

bartowl

type:enhancement good first issue help wanted needs_triage community

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx-operator/issues) for duplicates. - [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response. ### Bug Summary While `image_version` is based on `DEFAULT_AWX_VERSION` environment variable that gets injected during staging process, I have found out, that `control_plane_ee_image` via `_control_plane_ee_image` is still hardcoded to `awx-ee:latest`. This can cause unexpected behaviour, when used with default EE (which is also `awx-ee:latest`), as the default EE seems to be pulled with each job run when there is an update to that tag. After such unaware change of `awx-ee:latest` image, when the `awx-task` pod terminates for whatever reason, it will start with never code as intended. As now finally awx-ee images are tagged with tags matching awx, I would recommend to use the same tag on ee as well. For me, as a workaround, i use following: ``` spec: ... init_container_image: "{{ _image }}-ee" init_container_image_version: "{{ _image_version }}" control_plane_ee_image: "{{ _image }}-ee:{{ _image_version }}" ... ``` but this should not be needed if implemented properly. ### AWX Operator version 2.3.0 ### AWX version 22.4.0 ### Kubernetes platform other (please specify in additional information) ### Kubernetes/Platform version k3d 5.4.3, kubernetes version v1.23.6+k3s1 ### Modifications no ### Steps to reproduce * install default k3d * wait for `awx-ee:latest` tag to move * run any job, that will fetch updated `awx-ee:latest` * delete `awx-task` pod * new awx-task will have changed controlplane image ### Expected results After installation of AWX via awx-operator, no image should change automatically until deployment of new awx-operator version ### Actual results awx-ee and control_plane images might get changed when latest tag moves ### Additional information _No response_ ### Operator Logs _No response_

Denney-tech · February 28, 2024, 9:28pm

It would be nice if it was more like the other *_image specs as well:

control_plane_ee_image: quay.io/ansible/awx-ee
control_plane_ee_image_version: 23.8.1

epi82 · March 26, 2024, 8:47am

Thanks for clarifing!

I’ve checked the “sha256” for the “awx-ee:latest” and I think it’s quite updated.

By the way yesterday night it happened again the same issue with unexpected terminated job on AWX. So I think the latest mods on awx-ee:latest didn’t solve it.

This should be the error directly from pod awx-task:

INFO 2024/03/26 00:10:30 Detected Error: EOF for pod awx/automation-job-182941-9cj2d. Will retry 5 more times.

WARNING 2024/03/26 00:10:30 Error opening log stream for pod awx/automation-job-182941-9cj2d. Will retry 5 more times. Error: Get "https://10.1.42.122:10250/containerLogs/awx/automation-job-182941-9cj2d/worker?follow=true&sinceTime=2024-03-26T00%3A10%3A28Z&timestamps=true": proxy error from localhost:9443 while dialing 10.1.42.122:10250, code 503: 503 Service Unavailable

WARNING 2024/03/26 00:10:31 Error opening log stream for pod awx/automation-job-182941-9cj2d. Will retry 4 more times. Error: Get "https://10.1.42.122:10250/containerLogs/awx/automation-job-182941-9cj2d/worker?follow=true&sinceTime=2024-03-26T00%3A10%3A28Z&timestamps=true": proxy error from localhost:9443 while dialing 10.1.42.122:10250, code 503: 503 Service Unavailable

WARNING 2024/03/26 00:10:32 Error opening log stream for pod awx/automation-job-182941-9cj2d. Will retry 3 more times. Error: Get "https://10.1.42.122:10250/containerLogs/awx/automation-job-182941-9cj2d/worker?follow=true&sinceTime=2024-03-26T00%3A10%3A28Z&timestamps=true": proxy error from localhost:9443 while dialing 10.1.42.122:10250, code 503: 503 Service Unavailable

WARNING 2024/03/26 00:10:33 Error opening log stream for pod awx/automation-job-182941-9cj2d. Will retry 2 more times. Error: Get "https://10.1.42.122:10250/containerLogs/awx/automation-job-182941-9cj2d/worker?follow=true&sinceTime=2024-03-26T00%3A10%3A28Z&timestamps=true": proxy error from localhost:9443 while dialing 10.1.42.122:10250, code 503: 503 Service Unavailable

WARNING 2024/03/26 00:10:34 Error opening log stream for pod awx/automation-job-182941-9cj2d. Will retry 1 more times. Error: Get "https://10.1.42.122:10250/containerLogs/awx/automation-job-182941-9cj2d/worker?follow=true&sinceTime=2024-03-26T00%3A10%3A28Z&timestamps=true": proxy error from localhost:9443 while dialing 10.1.42.122:10250, code 503: 503 Service Unavailable

ERROR 2024/03/26 00:10:35 Error opening log stream for pod awx/automation-job-182941-9cj2d. Error: Get "https://10.1.42.122:10250/containerLogs/awx/automation-job-182941-9cj2d/worker?follow=true&sinceTime=2024-03-26T00%!A(MISSING)10%!A(MISSING)28Z&timestamps=true": proxy error from localhost:9443 while dialing 10.1.42.122:10250, code 503: 503 Service Unavailable

WARNING 2024/03/26 00:10:36 Could not read in control service: read unix /var/run/receptor/receptor.sock->@: use of closed network connection

WARNING 2024/03/26 00:10:36 Could not close connection: close unix /var/run/receptor/receptor.sock->@: use of closed network connection

ERROR 2024/03/26 00:10:36 Error deleting pod automation-job-182941-9cj2d: client rate limiter Wait returned an error: context canceled

If there is any other check I can do let me know. Thank you very much for your support.

epi82 · April 16, 2024, 7:05am

Hello,
yesterday night happened again but I think with a different message… I try to share it to you @kurokobo and @TheRealHaoLiu

Failed to JSON parse a line from worker stream. Error: Expecting value: line 1 column 1 (char 0) Line with invalid JSON data: b’’

In awx-task pod I’ve found this error:

2024-04-16T00:13:44+02:00 2024-04-15 22:13:44,409 ERROR    [6a992cbb5ae249be87f608f6d08258ba] awx.main.dispatch Worker failed to run task awx.main.tasks.system.purge_old_stdout_files(*[], **{}
2024-04-16T00:13:44+02:00 Traceback (most recent call last):
2024-04-16T00:13:44+02:00   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/worker/task.py", line 103, in perform_work
2024-04-16T00:13:44+02:00     result = self.run_callable(body)
2024-04-16T00:13:44+02:00   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/dispatch/worker/task.py", line 78, in run_callable
2024-04-16T00:13:44+02:00     return _call(*args, **kwargs)
2024-04-16T00:13:44+02:00   File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/system.py", line 378, in purge_old_stdout_files
2024-04-16T00:13:44+02:00     for f in os.listdir(settings.JOBOUTPUT_ROOT):
2024-04-16T00:13:44+02:00 FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/awx/job_status'

Any ideas? Thank you very much for your help.