AWX scheduled backup job failed after couple of days - AWX 24.6.1

Get Help
,
1

AWX scheduled backup job failed - AWX 24.6.1
I have scheduled AWX backup job by referring following steps “awx-on-k3s/backup/ansible at main · kurokobo/awx-on-k3s · GitHub”

I have used Use with AWX option and scheduled backup job.

It was working fine for 8 days then continuously getting failed with below error logs.

Kindly help me to fix the issue
@kurokobo

Logs

[root@server ~]# kubectl get awxbackup -n awx
NAME                            AGE
awxbackup-2024-11-29-12-25-22   27d
awxbackup-2024-11-29-12-30-54   27d
awxbackup-2024-11-29-12-38-30   27d
awxbackup-2024-12-02-04-30-38   25d
awxbackup-2024-12-02-23-30-38   24d
awxbackup-2024-12-03-23-30-38   23d
awxbackup-2024-12-04-23-30-46   22d
awxbackup-2024-12-05-23-30-26   21d
awxbackup-2024-12-06-23-30-28   20d
awxbackup-2024-12-07-23-30-27   19d
awxbackup-2024-12-08-23-30-26   18d
awxbackup-2024-12-09-23-30-22   17d
awxbackup-2024-12-10-23-30-22   16d
awxbackup-2024-12-11-23-30-21   15d
awxbackup-2024-12-12-11-33-29   14d
awxbackup-2024-12-12-23-30-21   14d
awxbackup-2024-12-16-23-30-22   10d
awxbackup-2024-12-17-23-30-22   9d
awxbackup-2024-12-20-23-30-20   6d5h
awxbackup-2024-12-23-23-30-21   3d5h
awxbackup-2024-12-26-10-29-32   18h
awxbackup-2024-12-26-11-05-04   18h
awxbackup-2024-12-26-11-06-20   18h
awxbackup-2024-12-26-11-10-30   18h
[root@server ~]#

[root@server ~]# ls -l /data/backup
total 15412
drwxr-xr-x 2   26   26       59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800
drwxr-xr-x 2   26   26       59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843
drwxr-xr-x 2   26   26       59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859
drwxr-xr-x 2   26   26       59 Dec  1 20:31 tower-openshift-backup-2024-12-02-043052
drwxr-xr-x 2   26   26       59 Dec  2 15:31 tower-openshift-backup-2024-12-02-233051
-rw-r--r-- 1 root root 15779840 Dec  3 03:15 tower-openshift-backup-2024-12-02-233051.tar
drwxr-xr-x 2   26   26       59 Dec  3 15:31 tower-openshift-backup-2024-12-03-233052
drwxr-xr-x 2   26   26       59 Dec  4 15:31 tower-openshift-backup-2024-12-04-233059
drwxr-xr-x 2   26   26       59 Dec  5 15:30 tower-openshift-backup-2024-12-05-233039
drwxr-xr-x 2   26   26       59 Dec  6 15:30 tower-openshift-backup-2024-12-06-233041
drwxr-xr-x 2   26   26       59 Dec  7 15:30 tower-openshift-backup-2024-12-07-233039
drwxr-xr-x 2   26   26       59 Dec  8 15:30 tower-openshift-backup-2024-12-08-233039
[root@server ~]#

Logs:-

{"uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "counter": 18, "stdout": "\r\nTASK [Create new AWXBackup resource and wait for complete] *********************", "start_line": 45, "end_line": 47, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.876625+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "name": "Create new AWXBackup resource and wait for complete", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-00000000000b"}}
{"uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741", "counter": 19, "stdout": "", "start_line": 47, "end_line": 47, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.877557+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741"}}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"uuid": "e8f324c4-941c-4d09-bcdd-87ee641e1c49", "counter": 20, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": true, \"duration\": 600, \"method\": \"create\", \"msg\": \"\\\"AWXBackup\\\" \\\"awxbackup-2024-12-26-23-30-21\\\": Timed out waiting on resource\", \"result\": {\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWXBackup\", \"metadata\": {\"creationTimestamp\": \"2024-12-26T23:30:22Z\", \"generation\": 1, \"managedFields\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:spec\": {\".\": {}, \"f:backup_pvc\": {}, \"f:clean_backup_on_delete\": {}, \"f:deployment_name\": {}, \"f:image_pull_policy\": {}, \"f:no_log\": {}, \"f:set_self_labels\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2024-12-26T23:30:22Z\"}], \"name\": \"awxbackup-2024-12-26-23-30-21\", \"namespace\": \"awx\", \"resourceVersion\": \"9000995\", \"uid\": \"03f81604-198d-44de-8e0e-0289dfa77320\"}, \"spec\": {\"backup_pvc\": \"awx-backup-claim\", \"clean_backup_on_delete\": true, \"deployment_name\": \"awx\", \"image_pull_policy\": \"IfNotPresent\", \"no_log\": true, \"set_self_labels\": true}}}\u001b[0m", "start_line": 47, "end_line": 48, "runner_ident": "1418", "event": "runner_on_failed", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:40:23.475258+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"creationTimestamp": "2024-12-26T23:30:22Z", "generation": 1, "managedFields": [{"apiVersion": "awx.ansible.com/v1beta1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:backup_pvc": {}, "f:clean_backup_on_delete": {}, "f:deployment_name": {}, "f:image_pull_policy": {}, "f:no_log": {}, "f:set_self_labels": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-12-26T23:30:22Z"}], "name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx", "resourceVersion": "9000995", "uid": "03f81604-198d-44de-8e0e-0289dfa77320"}, "spec": {"backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true, "deployment_name": "awx", "image_pull_policy": "IfNotPresent", "no_log": true, "set_self_labels": true}}, "method": "create", "duration": 600, "msg": "\"AWXBackup\" \"awxbackup-2024-12-26-23-30-21\": Timed out waiting on resource", "invocation": {"module_args": {"state": "present", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "wait_condition": {"reason": "Successful", "status": true, "type": "Running"}, "wait_timeout": 600, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "merge_type": null, "validate": null, "template": null, "delete_options": null, "label_selectors": null, "generate_name": null, "server_side_apply": null, "hidden_fields": null}}, "_ansible_no_log": false}, "start": "2024-12-26T23:30:21.877491+00:00", "end": "2024-12-26T23:40:23.475118+00:00", "duration": 601.597627, "ignore_errors": null, "event_loop": null, "uuid": "e8f324c4-941c-4d09-bcdd-87ee641e1c49"}}

backup
backup1561×576 28.3 KB

2

Logs

[root@server ~]# kubectl get pods -n awx
NAME                                               READY   STATUS      RESTARTS           AGE
automation-job-1402-r5smn                          0/1     Error       0                  18h
automation-job-1406-vw5vs                          0/1     Completed   0                  17h
automation-job-1410-qhbzm                          0/1     Error       0                  17h
automation-job-1418-rzjrj                          0/1     Error       0                  5h20m
awx-operator-controller-manager-745b55d94b-97sdw   2/2     Running     4577 (5m42s ago)   45d
awx-postgres-15-0                                  1/1     Running     12 (17d ago)       143d
awx-task-55d8c9dd9f-qzxr5                          4/4     Running     4 (17d ago)        21d
awx-web-59498d8589-98bph                           3/3     Running     24 (17d ago)       120d
[root@server ~]# kubectl logs automation-job-1418-rzjrj  -n awx
{"status": "starting", "runner_ident": "1418", "command": ["ansible-playbook", "-u", "admin", "-i", "/runner/inventory/hosts", "-e", "@/runner/env/extravars", "backup/ansible/project/backup.yml"], "env": {"KUBERNETES_SERVICE_PORT_HTTPS": "443", "KUBERNETES_SERVICE_PORT": "443", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_HOST": "10.43.106.140", "HOSTNAME": "automation-job-1418-rzjrj", "AWX_SERVICE_PORT_80_TCP": "tcp://10.43.61.35:80", "PWD": "/runner", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PORT": "8443", "PIP_BREAK_SYSTEM_PACKAGES": "1", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO": "tcp", "AWX_SERVICE_SERVICE_PORT": "80", "HOME": "/runner", "KUBERNETES_PORT_443_TCP": "tcp://10.43.0.1:443", "ANSIBLE_RUNNER_KEEPALIVE_SECONDS": "30", "AWX_SERVICE_PORT_80_TCP_ADDR": "10.43.61.35", "AWX_SERVICE_PORT": "tcp://10.43.61.35:80", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT": "tcp://10.43.106.140:8443", "AWX_SERVICE_PORT_80_TCP_PORT": "80", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT_HTTPS": "8443", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT": "8443", "SHLVL": "0", "KUBERNETES_PORT_443_TCP_PROTO": "tcp", "KUBERNETES_PORT_443_TCP_ADDR": "10.43.0.1", "AWX_SERVICE_SERVICE_HOST": "10.43.61.35", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP": "tcp://10.43.106.140:8443", "AWX_SERVICE_SERVICE_PORT_HTTP": "80", "KUBERNETES_SERVICE_HOST": "10.43.0.1", "KUBERNETES_PORT": "tcp://10.43.0.1:443", "KUBERNETES_PORT_443_TCP_PORT": "443", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR": "10.43.106.140", "AWX_SERVICE_PORT_80_TCP_PROTO": "tcp", "LC_CTYPE": "C.UTF-8", "ANSIBLE_BASE_ALLOW_CUSTOM_ROLES": "True", "ANSIBLE_BASE_ALLOW_CUSTOM_TEAM_ROLES": "False", "ANSIBLE_BASE_ALLOW_SINGLETON_ROLES_API": "False", "ANSIBLE_BASE_ALLOW_SINGLETON_TEAM_ROLES": "False", "ANSIBLE_BASE_ALLOW_SINGLETON_USER_ROLES": "True", "ANSIBLE_BASE_ALLOW_TEAM_ORG_ADMIN": "False", "ANSIBLE_BASE_ALLOW_TEAM_ORG_MEMBER": "False", "ANSIBLE_BASE_ALLOW_TEAM_ORG_PERMS": "True", "ANSIBLE_BASE_ALLOW_TEAM_PARENTS": "True", "ANSIBLE_BASE_ALL_REST_FILTERS": "('ansible_base.rest_filters.rest_framework.type_filter_backend.TypeFilterBackend', 'ansible_base.rest_filters.rest_framework.field_lookup_backend.FieldLookupBackend', 'rest_framework.filters.SearchFilter', 'ansible_base.rest_filters.rest_framework.order_backend.OrderByBackend')", "ANSIBLE_BASE_BYPASS_ACTION_FLAGS": "{}", "ANSIBLE_BASE_BYPASS_SUPERUSER_FLAGS": "['is_superuser']", "ANSIBLE_BASE_CACHE_PARENT_PERMISSIONS": "True", "ANSIBLE_BASE_CHECK_RELATED_PERMISSIONS": "['use', 'change', 'view']", "ANSIBLE_BASE_CREATOR_DEFAULTS": "['change', 'delete', 'execute', 'use', 'adhoc', 'approve', 'update', 'view']", "ANSIBLE_BASE_CUSTOM_VIEW_PARENT": "awx.api.generics.APIView", "ANSIBLE_BASE_EVALUATIONS_IGNORE_CONFLICTS": "True", "ANSIBLE_BASE_MANAGED_ROLE_REGISTRY": "{}", "ANSIBLE_BASE_ORGANIZATION_MODEL": "main.Organization", "ANSIBLE_BASE_PERMISSION_MODEL": "main.Permission", "ANSIBLE_BASE_RESOURCE_CONFIG_MODULE": "awx.resource_api", "ANSIBLE_BASE_REST_FILTERS_RESERVED_NAMES": "('page', 'page_size', 'format', 'order', 'order_by', 'search', 'type', 'host_filter', 'count_disabled', 'no_truncate', 'limit', 'validate')", "ANSIBLE_BASE_ROLE_CREATOR_NAME": "{cls.__name__} Creator", "ANSIBLE_BASE_ROLE_PRECREATE": "{}", "ANSIBLE_BASE_ROLE_SYSTEM_ACTIVATED": "True", "ANSIBLE_BASE_TEAM_MODEL": "main.Team", "ANSIBLE_FACT_CACHE_TIMEOUT": "0", "ANSIBLE_FORCE_COLOR": "True", "ANSIBLE_HOST_KEY_CHECKING": "False", "ANSIBLE_INVENTORY_UNPARSED_FAILED": "True", "ANSIBLE_PARAMIKO_RECORD_HOST_KEYS": "False", "AWX_PRIVATE_DATA_DIR": "/tmp/awx_1418_xww0hh3i", "JOB_ID": "1418", "INVENTORY_ID": "1", "PROJECT_REVISION": "7885966239d92108d9a83ce96e6cb88c4257d8e7", "ANSIBLE_RETRY_FILES_ENABLED": "False", "MAX_EVENT_RES": "700000", "AWX_HOST": "https://server.chq.ei/", "ANSIBLE_SSH_CONTROL_PATH_DIR": "/runner/cp", "ANSIBLE_COLLECTIONS_PATHS": "/runner/requirements_collections:~/.ansible/collections:/usr/share/ansible/collections", "ANSIBLE_ROLES_PATH": "/runner/requirements_roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles", "ANSIBLE_COLLECTIONS_PATH": "/runner/requirements_collections:~/.ansible/collections:/usr/share/ansible/collections", "ANSIBLE_CALLBACK_PLUGINS": "/usr/local/lib/python3.9/site-packages/ansible_runner/display_callback/callback", "ANSIBLE_STDOUT_CALLBACK": "awx_display", "AWX_ISOLATED_DATA_DIR": "/runner/artifacts/1418", "RUNNER_OMIT_EVENTS": "False", "RUNNER_ONLY_FAILED_EVENTS": "False"}, "cwd": "/runner/project"}
{"status": "running", "runner_ident": "1418"}
{"uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "counter": 1, "stdout": "", "start_line": 0, "end_line": 0, "runner_ident": "1418", "event": "playbook_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.731345+00:00", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf"}}
{"uuid": "4e329af7-df1e-0679-2e8f-000000000002", "counter": 2, "stdout": "\r\nPLAY [Backing up AWX using AWX Operator] ***************************************", "start_line": 0, "end_line": 2, "runner_ident": "1418", "event": "playbook_on_play_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.732799+00:00", "parent_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "name": "Backing up AWX using AWX Operator", "pattern": "localhost", "uuid": "4e329af7-df1e-0679-2e8f-000000000002"}}
{"uuid": "4e329af7-df1e-0679-2e8f-000000000004", "counter": 3, "stdout": "\r\nTASK [Construct default variables] *********************************************", "start_line": 2, "end_line": 4, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.744816+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct default variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:9", "name": "Construct default variables", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000004"}}
{"uuid": "6c5d59a5-4e57-4d6e-8d82-e161f2b99c86", "counter": 4, "stdout": "", "start_line": 4, "end_line": 4, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.745646+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct default variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:9", "host": "localhost", "uuid": "6c5d59a5-4e57-4d6e-8d82-e161f2b99c86"}}
{"uuid": "2eb3e739-546c-4c4c-9849-d8d398f7f9e0", "counter": 5, "stdout": "\u001b[0;32mok: [localhost]\u001b[0m", "start_line": 4, "end_line": 5, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.772016+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct default variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:9", "host": "localhost", "remote_addr": "localhost", "res": {"ansible_facts": {"_awxbackup_default": {"api_version": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "namespace": "awx", "name": "awxbackup-2024-12-26-23-30-21", "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}, "timeout": 600, "keep_days": 30}}, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.745566+00:00", "end": "2024-12-26T23:30:21.771826+00:00", "duration": 0.02626, "event_loop": null, "uuid": "2eb3e739-546c-4c4c-9849-d8d398f7f9e0"}}
{"uuid": "4e329af7-df1e-0679-2e8f-000000000005", "counter": 6, "stdout": "\r\nTASK [Construct custom variables] **********************************************", "start_line": 5, "end_line": 7, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.775500+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct custom variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:23", "name": "Construct custom variables", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000005"}}
{"uuid": "9a143b19-effa-47d9-9f5f-23ae407a10e7", "counter": 7, "stdout": "", "start_line": 7, "end_line": 7, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.776385+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct custom variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:23", "host": "localhost", "uuid": "9a143b19-effa-47d9-9f5f-23ae407a10e7"}}
{"uuid": "4267d7de-2022-45c0-9570-7375b87b7c65", "counter": 8, "stdout": "\u001b[0;32mok: [localhost]\u001b[0m", "start_line": 7, "end_line": 8, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.808275+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct custom variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:23", "host": "localhost", "remote_addr": "localhost", "res": {"ansible_facts": {"_awxbackup_config": {"api_version": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "namespace": "awx", "name": "awxbackup-2024-12-26-23-30-21", "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}, "timeout": "600", "keep_days": "30"}}, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.776317+00:00", "end": "2024-12-26T23:30:21.808076+00:00", "duration": 0.031759, "event_loop": null, "uuid": "4267d7de-2022-45c0-9570-7375b87b7c65"}}
{"uuid": "4e329af7-df1e-0679-2e8f-000000000006", "counter": 9, "stdout": "\r\nTASK [Print active variables] **************************************************", "start_line": 8, "end_line": 10, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.811961+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print active variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:34", "name": "Print active variables", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000006"}}
{"uuid": "61d8f273-d32b-4df8-b7de-162f7d94fcce", "counter": 10, "stdout": "", "start_line": 10, "end_line": 10, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.812900+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print active variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:34", "host": "localhost", "uuid": "61d8f273-d32b-4df8-b7de-162f7d94fcce"}}
{"uuid": "2504d855-b329-4cdf-a7a7-c6cf4c7ef26a", "counter": 11, "stdout": "\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m    \"_awxbackup_config\": {\u001b[0m\r\n\u001b[0;32m        \"api_version\": \"awx.ansible.com/v1beta1\",\u001b[0m\r\n\u001b[0;32m        \"keep_days\": \"30\",\u001b[0m\r\n\u001b[0;32m        \"kind\": \"AWXBackup\",\u001b[0m\r\n\u001b[0;32m        \"name\": \"awxbackup-2024-12-26-23-30-21\",\u001b[0m\r\n\u001b[0;32m        \"namespace\": \"awx\",\u001b[0m\r\n\u001b[0;32m        \"spec\": {\u001b[0m\r\n\u001b[0;32m            \"backup_pvc\": \"awx-backup-claim\",\u001b[0m\r\n\u001b[0;32m            \"clean_backup_on_delete\": true,\u001b[0m\r\n\u001b[0;32m            \"deployment_name\": \"awx\"\u001b[0m\r\n\u001b[0;32m        },\u001b[0m\r\n\u001b[0;32m        \"timeout\": \"600\"\u001b[0m\r\n\u001b[0;32m    }\u001b[0m\r\n\u001b[0;32m}\u001b[0m", "start_line": 10, "end_line": 25, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.829088+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print active variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:34", "host": "localhost", "remote_addr": "localhost", "res": {"_awxbackup_config": {"api_version": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "namespace": "awx", "name": "awxbackup-2024-12-26-23-30-21", "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}, "timeout": "600", "keep_days": "30"}, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.812828+00:00", "end": "2024-12-26T23:30:21.828883+00:00", "duration": 0.016055, "event_loop": null, "uuid": "2504d855-b329-4cdf-a7a7-c6cf4c7ef26a"}}
{"uuid": "4e329af7-df1e-0679-2e8f-000000000008", "counter": 12, "stdout": "\r\nTASK [Construct new manifest for AWXBackup resource] ***************************", "start_line": 25, "end_line": 27, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.832521+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct new manifest for AWXBackup resource", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:40", "name": "Construct new manifest for AWXBackup resource", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000008"}}
{"uuid": "d1bfdf34-a7fb-41ca-8763-3928c22c43f0", "counter": 13, "stdout": "", "start_line": 27, "end_line": 27, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.834375+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct new manifest for AWXBackup resource", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:40", "host": "localhost", "uuid": "d1bfdf34-a7fb-41ca-8763-3928c22c43f0"}}
{"uuid": "da871e5b-4dad-4ba0-95dd-a2e8eb5a50e3", "counter": 14, "stdout": "\u001b[0;32mok: [localhost]\u001b[0m", "start_line": 27, "end_line": 28, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.851735+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct new manifest for AWXBackup resource", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:40", "host": "localhost", "remote_addr": "localhost", "res": {"ansible_facts": {"_awxbackup_manifest": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}}, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.834305+00:00", "end": "2024-12-26T23:30:21.851565+00:00", "duration": 0.01726, "event_loop": null, "uuid": "da871e5b-4dad-4ba0-95dd-a2e8eb5a50e3"}}
{"uuid": "4e329af7-df1e-0679-2e8f-000000000009", "counter": 15, "stdout": "\r\nTASK [Print manifest to be created] ********************************************", "start_line": 28, "end_line": 30, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.854961+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "name": "Print manifest to be created", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000009"}}
{"uuid": "bc0f1b46-4bc3-49ab-a139-6fef09074dc6", "counter": 16, "stdout": "", "start_line": 30, "end_line": 30, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.855851+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "host": "localhost", "uuid": "bc0f1b46-4bc3-49ab-a139-6fef09074dc6"}}
{"uuid": "db16da99-fd61-4c6b-89e9-851014d27abe", "counter": 17, "stdout": "\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m    \"_awxbackup_manifest\": {\u001b[0m\r\n\u001b[0;32m        \"apiVersion\": \"awx.ansible.com/v1beta1\",\u001b[0m\r\n\u001b[0;32m        \"kind\": \"AWXBackup\",\u001b[0m\r\n\u001b[0;32m        \"metadata\": {\u001b[0m\r\n\u001b[0;32m            \"name\": \"awxbackup-2024-12-26-23-30-21\",\u001b[0m\r\n\u001b[0;32m            \"namespace\": \"awx\"\u001b[0m\r\n\u001b[0;32m        },\u001b[0m\r\n\u001b[0;32m        \"spec\": {\u001b[0m\r\n\u001b[0;32m            \"backup_pvc\": \"awx-backup-claim\",\u001b[0m\r\n\u001b[0;32m            \"clean_backup_on_delete\": true,\u001b[0m\r\n\u001b[0;32m            \"deployment_name\": \"awx\"\u001b[0m\r\n\u001b[0;32m        }\u001b[0m\r\n\u001b[0;32m    }\u001b[0m\r\n\u001b[0;32m}\u001b[0m", "start_line": 30, "end_line": 45, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.870698+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "host": "localhost", "remote_addr": "localhost", "res": {"_awxbackup_manifest": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.855783+00:00", "end": "2024-12-26T23:30:21.870498+00:00", "duration": 0.014715, "event_loop": null, "uuid": "db16da99-fd61-4c6b-89e9-851014d27abe"}}
{"uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "counter": 18, "stdout": "\r\nTASK [Create new AWXBackup resource and wait for complete] *********************", "start_line": 45, "end_line": 47, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.876625+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "name": "Create new AWXBackup resource and wait for complete", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-00000000000b"}}
{"uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741", "counter": 19, "stdout": "", "start_line": 47, "end_line": 47, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.877557+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741"}}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"uuid": "e8f324c4-941c-4d09-bcdd-87ee641e1c49", "counter": 20, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": true, \"duration\": 600, \"method\": \"create\", \"msg\": \"\\\"AWXBackup\\\" \\\"awxbackup-2024-12-26-23-30-21\\\": Timed out waiting on resource\", \"result\": {\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWXBackup\", \"metadata\": {\"creationTimestamp\": \"2024-12-26T23:30:22Z\", \"generation\": 1, \"managedFields\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:spec\": {\".\": {}, \"f:backup_pvc\": {}, \"f:clean_backup_on_delete\": {}, \"f:deployment_name\": {}, \"f:image_pull_policy\": {}, \"f:no_log\": {}, \"f:set_self_labels\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2024-12-26T23:30:22Z\"}], \"name\": \"awxbackup-2024-12-26-23-30-21\", \"namespace\": \"awx\", \"resourceVersion\": \"9000995\", \"uid\": \"03f81604-198d-44de-8e0e-0289dfa77320\"}, \"spec\": {\"backup_pvc\": \"awx-backup-claim\", \"clean_backup_on_delete\": true, \"deployment_name\": \"awx\", \"image_pull_policy\": \"IfNotPresent\", \"no_log\": true, \"set_self_labels\": true}}}\u001b[0m", "start_line": 47, "end_line": 48, "runner_ident": "1418", "event": "runner_on_failed", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:40:23.475258+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"creationTimestamp": "2024-12-26T23:30:22Z", "generation": 1, "managedFields": [{"apiVersion": "awx.ansible.com/v1beta1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:backup_pvc": {}, "f:clean_backup_on_delete": {}, "f:deployment_name": {}, "f:image_pull_policy": {}, "f:no_log": {}, "f:set_self_labels": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-12-26T23:30:22Z"}], "name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx", "resourceVersion": "9000995", "uid": "03f81604-198d-44de-8e0e-0289dfa77320"}, "spec": {"backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true, "deployment_name": "awx", "image_pull_policy": "IfNotPresent", "no_log": true, "set_self_labels": true}}, "method": "create", "duration": 600, "msg": "\"AWXBackup\" \"awxbackup-2024-12-26-23-30-21\": Timed out waiting on resource", "invocation": {"module_args": {"state": "present", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "wait_condition": {"reason": "Successful", "status": true, "type": "Running"}, "wait_timeout": 600, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null,
3

According to the logs, timeout has been exceeded, so you should increasing the timeout.

4

Thank you very much @kurokobo
I have increased timeout from 600 to 1200 but job still failing

Logs attached

project/backup.yml:50", "host": "localhost", "uuid": "bfe197a2-2450-40af-a679-b6acc0e68446"}}
{"uuid": "3625337e-99e2-4615-a17c-8bd2b05f6e94", "counter": 17, "stdout": "\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m    \"_awxbackup_manifest\": {\u001b[0m\r\n\u001b[0;32m        \"apiVersion\": \"awx.ansible.com/v1beta1\",\u001b[0m\r\n\u001b[0;32m        \"kind\": \"AWXBackup\",\u001b[0m\r\n\u001b[0;32m        \"metadata\": {\u001b[0m\r\n\u001b[0;32m            \"name\": \"awxbackup-2024-12-27-10-14-37\",\u001b[0m\r\n\u001b[0;32m            \"namespace\": \"awx\"\u001b[0m\r\n\u001b[0;32m        },\u001b[0m\r\n\u001b[0;32m        \"spec\": {\u001b[0m\r\n\u001b[0;32m            \"backup_pvc\": \"awx-backup-claim\",\u001b[0m\r\n\u001b[0;32m            \"clean_backup_on_delete\": true,\u001b[0m\r\n\u001b[0;32m            \"deployment_name\": \"awx\"\u001b[0m\r\n\u001b[0;32m        }\u001b[0m\r\n\u001b[0;32m    }\u001b[0m\r\n\u001b[0;32m}\u001b[0m", "start_line": 30, "end_line": 45, "runner_ident": "1423", "event": "runner_on_ok", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:14:37.918528+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000009", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "2297092f-0e09-2731-a1ee-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "host": "localhost", "remote_addr": "localhost", "res": {"_awxbackup_manifest": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false}, "start": "2024-12-27T10:14:37.902536+00:00", "end": "2024-12-27T10:14:37.918314+00:00", "duration": 0.015778, "event_loop": null, "uuid": "3625337e-99e2-4615-a17c-8bd2b05f6e94"}}
{"uuid": "2297092f-0e09-2731-a1ee-00000000000b", "counter": 18, "stdout": "\r\nTASK [Create new AWXBackup resource and wait for complete] *********************", "start_line": 45, "end_line": 47, "runner_ident": "1423", "event": "playbook_on_task_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:14:37.924453+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "name": "Create new AWXBackup resource and wait for complete", "is_conditional": false, "uuid": "2297092f-0e09-2731-a1ee-00000000000b"}}
{"uuid": "562aba13-93a6-43e0-a59b-da47afa1edb1", "counter": 19, "stdout": "", "start_line": 47, "end_line": 47, "runner_ident": "1423", "event": "runner_on_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:14:37.925393+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "uuid": "562aba13-93a6-43e0-a59b-da47afa1edb1"}}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"event": "keepalive", "counter": 0, "uuid": 0}
{"uuid": "f5abd608-c59b-4588-adaf-6c11ad9f08ab", "counter": 20, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": true, \"duration\": 1201, \"method\": \"create\", \"msg\": \"\\\"AWXBackup\\\" \\\"awxbackup-2024-12-27-10-14-37\\\": Timed out waiting on resource\", \"result\": {\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWXBackup\", \"metadata\": {\"creationTimestamp\": \"2024-12-27T10:14:38Z\", \"generation\": 1, \"managedFields\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:spec\": {\".\": {}, \"f:backup_pvc\": {}, \"f:clean_backup_on_delete\": {}, \"f:deployment_name\": {}, \"f:image_pull_policy\": {}, \"f:no_log\": {}, \"f:set_self_labels\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2024-12-27T10:14:38Z\"}], \"name\": \"awxbackup-2024-12-27-10-14-37\", \"namespace\": \"awx\", \"resourceVersion\": \"9015486\", \"uid\": \"7b43c40b-2029-47da-a91a-9ff567bbb2ab\"}, \"spec\": {\"backup_pvc\": \"awx-backup-claim\", \"clean_backup_on_delete\": true, \"deployment_name\": \"awx\", \"image_pull_policy\": \"IfNotPresent\", \"no_log\": true, \"set_self_labels\": true}}}\u001b[0m", "start_line": 47, "end_line": 48, "runner_ident": "1423", "event": "runner_on_failed", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:40.416567+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"creationTimestamp": "2024-12-27T10:14:38Z", "generation": 1, "managedFields": [{"apiVersion": "awx.ansible.com/v1beta1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:backup_pvc": {}, "f:clean_backup_on_delete": {}, "f:deployment_name": {}, "f:image_pull_policy": {}, "f:no_log": {}, "f:set_self_labels": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-12-27T10:14:38Z"}], "name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx", "resourceVersion": "9015486", "uid": "7b43c40b-2029-47da-a91a-9ff567bbb2ab"}, "spec": {"backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true, "deployment_name": "awx", "image_pull_policy": "IfNotPresent", "no_log": true, "set_self_labels": true}}, "method": "create", "duration": 1201, "msg": "\"AWXBackup\" \"awxbackup-2024-12-27-10-14-37\": Timed out waiting on resource", "invocation": {"module_args": {"state": "present", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "wait_condition": {"reason": "Successful", "status": true, "type": "Running"}, "wait_timeout": 1200, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "merge_type": null, "validate": null, "template": null, "delete_options": null, "label_selectors": null, "generate_name": null, "server_side_apply": null, "hidden_fields": null}}, "_ansible_no_log": false}, "start": "2024-12-27T10:14:37.925327+00:00", "end": "2024-12-27T10:34:40.416426+00:00", "duration": 1202.491099, "ignore_errors": null, "event_loop": null, "uuid": "f5abd608-c59b-4588-adaf-6c11ad9f08ab"}}
{"uuid": "2297092f-0e09-2731-a1ee-00000000000d", "counter": 21, "stdout": "\r\nTASK [Clean up failed AWXBackup resource] **************************************", "start_line": 48, "end_line": 50, "runner_ident": "1423", "event": "playbook_on_task_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:40.420119+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Clean up failed AWXBackup resource", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:80", "name": "Clean up failed AWXBackup resource", "is_conditional": false, "uuid": "2297092f-0e09-2731-a1ee-00000000000d"}}
{"uuid": "6123c395-e002-4d8a-9492-0ba6d147a4f1", "counter": 22, "stdout": "", "start_line": 50, "end_line": 50, "runner_ident": "1423", "event": "runner_on_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:40.420957+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Clean up failed AWXBackup resource", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:80", "host": "localhost", "uuid": "6123c395-e002-4d8a-9492-0ba6d147a4f1"}}
{"uuid": "098e9389-c91f-4665-972e-8165a9f39c1a", "counter": 23, "stdout": "\u001b[0;33mchanged: [localhost]\u001b[0m", "start_line": 50, "end_line": 51, "runner_ident": "1423", "event": "runner_on_ok", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.033150+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Clean up failed AWXBackup resource", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:80", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Success", "details": {"name": "awxbackup-2024-12-27-10-14-37", "group": "awx.ansible.com", "kind": "awxbackups", "uid": "7b43c40b-2029-47da-a91a-9ff567bbb2ab"}}, "method": "delete", "duration": 0, "invocation": {"module_args": {"state": "absent", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "wait_timeout": 120, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "wait_condition": null, "merge_type": null, "validate": null, "template": null, "delete_options": null, "label_selectors": null, "generate_name": null, "server_side_apply": null, "hidden_fields": null}}, "_ansible_no_log": false}, "start": "2024-12-27T10:34:40.420889+00:00", "end": "2024-12-27T10:34:41.032956+00:00", "duration": 0.612067, "event_loop": null, "uuid": "098e9389-c91f-4665-972e-8165a9f39c1a"}}
{"uuid": "2297092f-0e09-2731-a1ee-00000000000e", "counter": 24, "stdout": "\r\nTASK [Force to fail if the backup was failed] **********************************", "start_line": 51, "end_line": 53, "runner_ident": "1423", "event": "playbook_on_task_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.037726+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Force to fail if the backup was failed", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "task_action": "ansible.builtin.fail", "resolved_action": "ansible.builtin.fail", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:86", "name": "Force to fail if the backup was failed", "is_conditional": false, "uuid": "2297092f-0e09-2731-a1ee-00000000000e"}}
{"uuid": "657d800a-8400-4add-87a2-f4719ab9d0f7", "counter": 25, "stdout": "", "start_line": 53, "end_line": 53, "runner_ident": "1423", "event": "runner_on_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.038740+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Force to fail if the backup was failed", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "task_action": "ansible.builtin.fail", "resolved_action": "ansible.builtin.fail", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:86", "host": "localhost", "uuid": "657d800a-8400-4add-87a2-f4719ab9d0f7"}}
{"uuid": "40bcd7d2-e029-4d0c-a3dc-834ad32a87f5", "counter": 26, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Failed as requested from task\"}\u001b[0m", "start_line": 53, "end_line": 54, "runner_ident": "1423", "event": "runner_on_failed", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.051447+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Force to fail if the backup was failed", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "task_action": "ansible.builtin.fail", "resolved_action": "ansible.builtin.fail", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:86", "host": "localhost", "remote_addr": "localhost", "res": {"msg": "Failed as requested from task", "_ansible_no_log": false, "changed": false}, "start": "2024-12-27T10:34:41.038667+00:00", "end": "2024-12-27T10:34:41.051282+00:00", "duration": 0.012615, "ignore_errors": null, "event_loop": null, "uuid": "40bcd7d2-e029-4d0c-a3dc-834ad32a87f5"}}
{"uuid": "598d398e-8fd0-4679-9e15-5725b5dceefa", "counter": 27, "stdout": "\r\nPLAY RECAP *********************************************************************\r\n\u001b[0;31mlocalhost\u001b[0m                  : \u001b[0;32mok=6   \u001b[0m \u001b[0;33mchanged=1   \u001b[0m unreachable=0    \u001b[0;31mfailed=1   \u001b[0m skipped=0    \u001b[0;32mrescued=1   \u001b[0m ignored=0   ", "start_line": 54, "end_line": 58, "runner_ident": "1423", "event": "playbook_on_stats", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.052998+00:00", "parent_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "changed": {"localhost": 1}, "dark": {}, "failures": {"localhost": 1}, "ignored": {}, "ok": {"localhost": 6}, "processed": {"localhost": 1}, "rescued": {"localhost": 1}, "skipped": {}, "artifact_data": {}, "uuid": "598d398e-8fd0-4679-9e15-5725b5dceefa"}}
{"status": "failed", "runner_ident": "1423"}
{"zipfile": 1942}
UEsDBBQAAAAIANJRm1l9zrJ6EQYAAD0SAAAHAAAAY29tbWFuZL1YW1OjSBT+K5Yv7lZNIDGOZnxaQjqRkQAFnThZa6qrQ1rDDAEGiJqx3N++p5smQm7ruFvrQ+jLufS5fKdP+3zsx4sFjWbHl0e3xzTKgmnIGklIV9M4/n784ei4seS/dLYIIjEN+K+aLqOIpWoQPbAoj9OVOo+zPBMEjP/+UVKw6EFlT3lKH2gq9qfU/75MVKlKTdL4G/NztVhWVovw+CtQ+Y/8RGs9korzg0DYeT6+HnWRayGMPOIhd2zoiDi2i8kVxo7Hec/O2px+D12FQrv5QmwHuRq2XaLbFnZt00QuGWqWNuBfhF1Df2Uvv1e2J8S0mspZW2k1z5XWWZML5BuWNkR8ky7zeEHzII4a3+Jpo3V22m6cP+R+p9RcO3ynSbDucL7cTy5VtRB93lLaHy87QrZz06s45j3HLxSB7VzV2hcd6QzHcEjXRdo18SYeRkPiaPo1yBEubf0H+lwb29LATRdsxqcjvVl4smJyJaZCuhS+6bem0rosY2x5RtdExB1ZFpzxGiFHM40x1wkm9IR17eaBmBCt13Nfoy1Csov8UOze47lNeTLNLjuV5N113LoP35vidUR1/iVgNnPNuzLHJl9oHgjqVsbso6sHCGK/L73quK1F8t15/Sth2nLua5x2FKv6aaVZGz44kPh7vVqvgY6GrwTIllmqhrFPQzWbBtFlZb6evm6IQTGdFtfD3nTcAfv3F5F6qCuV19SJjieOqBe6MsL9RqeK/q7mIaKZpn1D9JGH7SEBtUVpw+mSHSbFSKvQ92mY7WPwDGtgImxbBTnRHOPtLO9RMwI3vckUIdx2B+DBoWH9k4I19RANIYveTA5xHb7pJI7mIgsfJCUu8jDpGyZGriD87US2DmRKM6akLMvJXRDmLM3kJKUL9hin35V8lTC5R3h/waKZgmGtL5a6xcrJh6O3S7wLWDgjIbRGy2Qtss8XTbFWkbnFWQj0GE39eXGCX1Idp7OKGTafdVelvt+3XNedOJoHqadjAxKkb2oD4b3nl32U3gjCJhJpTXx7EmQkWyYsXWZw2q9brLqmXyEZRRF1w/NA3f6AArl+DSE1NYx6mxy3J6CG+8Sf0+hejB4C9rhLLzQovHD0UF8bmVhyv7LNWMhyMWJPzF8WQymczuaxLwYJdJUPxVYyo/lBhUUJGBvoRporervHJ4UmgXLPoC8J/EwBoI9BwBY7GmvmSOOR8IgxsGwX8ZrXNw39QPIXpbAncA0uGxgedid7Ygio0yzjT6GCDO0eEtfqggYRZMo9jYKfogXd4nuNwAaXw9JFkGW7eACQ9sjVCxOMAWccmaj0B2RtvEx9RsAxu1jXWOYTKO5gIG+VJbQTWgSQf0kW/BSTuziFDpqPBAbWAzJd8XEmEMVHHO/8y98iEkYineJllJNZkFEAmsBmFJMcuklfBj0MFoGQ/0DDQGTCNp5EFMq8K5v7Zz/MFEIiwCghL0d6yig8hnbzOi4S7GhPCAWR7Lk5asccInuTQ5TPesgwo4sqXR+kSIRiY4jsEV63WmsSu4ijabu7NPHWA1rlSQFbuG123gCGNQY82O6EjCzAhgcR7WuGufvwQKANjWsboq/bbm+tYuOyg+7AcYULSE/DGukZ4oBqvoCX4+MT4c8oEsR0QVf3hHN8trvEEBr5Fl95PZZcF22Oa39G4BYXjQ2e9aLtv/vUupg2T5vtKZ22zmfT87vzj+0LOr04Y+z8bDZrdZof6Wmr9oyA9mTCMxlud2RpXWnt2oCh9gVAz6uiW6T2RZP/lbaVPd08z5MMurb702ajfHv7T4niz38oLFCrGj3vquyTCG/W1i6RL2TZVklqnXdTelFwOLVXJU3Zj2WQsgW83DPix2EIL2sAenb5l6qUz/LqctH0zWnK1B3bNbeIdmfdS+7Sl8Yhq2kqFnbokBss9+tLh+z8f8zUoS/pwrOYOOZoYBSXV6VRDoOpmqzyeRy1lU9qFuSskcA9DSUtKyUTeUYoSvx/LQTYQn61q+WgFnrcA/CutcpKSyRvmVOGZxe3ag0wUg9N8+CO+nmmlvCQD2F7aOAiU+sILLctcyLRvEX18vI3UEsDBBQAAAAIAFRUm1l3o4ueCAAAAAYAAAAGAAAAc3RhdHVzS0vMzElNAQBQSwMEFAAAAAgAVFSbWQ2+1RoDAAAAAQAAAAIAAAByYzMCAFBLAwQUAAAAAABUVJtZAAAAAAAAAAAAAAAACwAAAGpvYl9ldmVudHMvUEsBAhQDFAAAAAgA0lGbWX3OsnoRBgAAPRIAAAcAAAAAAAAAAAAAAICBAAAAAGNvbW1hbmRQSwECFAMUAAAACABUVJtZd6OLnggAAAAGAAAABgAAAAAAAAAAAAAAgIE2BgAAc3RhdHVzUEsBAhQDFAAAAAgAVFSbWQ2+1RoDAAAAAQAAAAIAAAAAAAAAAAAAAICBYgYAAHJjUEsBAhQDFAAAAAAAVFSbWQAAAAAAAAAAAAAAAAsAAAAAAAAAAAAQAMBBhQYAAGpvYl9ldmVudHMvUEsFBgAAAAAEAAQA0gAAAK4GAAAAAA=={"eof": true}
````````````
5

Adding additional details

[root@server ~]# kubectl get pods -n awx
NAME                                               READY   STATUS             RESTARTS           AGE
automation-job-1402-r5smn                          0/1     Error              0                  24h
automation-job-1406-vw5vs                          0/1     Completed          0                  23h
automation-job-1410-qhbzm                          0/1     Error              0                  23h
automation-job-1418-rzjrj                          0/1     Error              0                  11h
automation-job-1421-gvklc                          0/1     Error              0                  45m
automation-job-1423-6vtc8                          0/1     Error              0                  28m
awx-operator-controller-manager-745b55d94b-97sdw   1/2     CrashLoopBackOff   4639 (2m29s ago)   46d
awx-postgres-15-0                                  1/1     Running            12 (18d ago)       144d
awx-task-55d8c9dd9f-qzxr5                          4/4     Running            4 (18d ago)        21d
awx-web-59498d8589-98bph                           3/3     Running            24 (18d ago)       121d
[root@server ~]# kubectl get awxbackup -n awx
NAME                            AGE
awxbackup-2024-11-29-12-25-22   27d
awxbackup-2024-11-29-12-30-54   27d
awxbackup-2024-11-29-12-38-30   27d
awxbackup-2024-12-02-04-30-38   25d
awxbackup-2024-12-02-23-30-38   24d
awxbackup-2024-12-03-23-30-38   23d
awxbackup-2024-12-04-23-30-46   22d
awxbackup-2024-12-05-23-30-26   21d
awxbackup-2024-12-06-23-30-28   20d
awxbackup-2024-12-07-23-30-27   19d
awxbackup-2024-12-08-23-30-26   18d
awxbackup-2024-12-09-23-30-22   17d
awxbackup-2024-12-10-23-30-22   16d
awxbackup-2024-12-11-23-30-21   15d
awxbackup-2024-12-12-11-33-29   14d
awxbackup-2024-12-12-23-30-21   14d
awxbackup-2024-12-16-23-30-22   10d
awxbackup-2024-12-17-23-30-22   9d
awxbackup-2024-12-20-23-30-20   6d11h
awxbackup-2024-12-23-23-30-21   3d11h
awxbackup-2024-12-26-10-29-32   24h
awxbackup-2024-12-26-11-05-04   23h
awxbackup-2024-12-26-11-06-20   23h
awxbackup-2024-12-26-11-10-30   23h
awxbackup-2024-12-27-09-57-33   45m
[root@server ~]# ls -l /data/backup/
total 15412
drwxr-xr-x 2   26   26       59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800
drwxr-xr-x 2   26   26       59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843
drwxr-xr-x 2   26   26       59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859
drwxr-xr-x 2   26   26       59 Dec  1 20:31 tower-openshift-backup-2024-12-02-043052
drwxr-xr-x 2   26   26       59 Dec  2 15:31 tower-openshift-backup-2024-12-02-233051
-rw-r--r-- 1 root root 15779840 Dec  3 03:15 tower-openshift-backup-2024-12-02-233051.tar
drwxr-xr-x 2   26   26       59 Dec  3 15:31 tower-openshift-backup-2024-12-03-233052
drwxr-xr-x 2   26   26       59 Dec  4 15:31 tower-openshift-backup-2024-12-04-233059
drwxr-xr-x 2   26   26       59 Dec  5 15:30 tower-openshift-backup-2024-12-05-233039
drwxr-xr-x 2   26   26       59 Dec  6 15:30 tower-openshift-backup-2024-12-06-233041
drwxr-xr-x 2   26   26       59 Dec  7 15:30 tower-openshift-backup-2024-12-07-233039
drwxr-xr-x 2   26   26       59 Dec  8 15:30 tower-openshift-backup-2024-12-08-233039
[root@server ~]#
[root@server ~]# kubectl get events -A
NAMESPACE   LAST SEEN   TYPE      REASON                OBJECT                                                 MESSAGE
awx         48m         Normal    Scheduled             pod/automation-job-1421-gvklc                          Successfully assigned awx/automation-job-1421-gvklc to server
awx         48m         Normal    Pulled                pod/automation-job-1421-gvklc                          Container image "quay.io/ansible/awx-ee:24.6.1" already present on machine
awx         48m         Normal    Created               pod/automation-job-1421-gvklc                          Created container worker
awx         48m         Normal    Started               pod/automation-job-1421-gvklc                          Started container worker
awx         31m         Normal    Scheduled             pod/automation-job-1423-6vtc8                          Successfully assigned awx/automation-job-1423-6vtc8 to server
awx         31m         Normal    Pulled                pod/automation-job-1423-6vtc8                          Container image "quay.io/ansible/awx-ee:24.6.1" already present on machine
awx         31m         Normal    Created               pod/automation-job-1423-6vtc8                          Created container worker
awx         31m         Normal    Started               pod/automation-job-1423-6vtc8                          Started container worker
awx         3m41s       Warning   BackOff               pod/awx-operator-controller-manager-745b55d94b-97sdw   Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d)
awx         56m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_26ba2df6-f025-4717-9b4f-b1d834cf4217 became leader
awx         50m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_35152e7c-d324-4d70-b263-f22c93cba88e became leader
awx         45m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_acbf4459-5d5d-46af-90e1-75206676d2d4 became leader
awx         39m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_b3e38cf8-b1e0-426e-a165-343d125e8146 became leader
awx         33m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_c20e0124-01db-4539-b62d-8ed72b6efd31 became leader
awx         28m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_fcc73ab1-a118-484f-9509-f33f7de1903d became leader
awx         22m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_5821073f-d6fa-4f9a-beb6-38e8c7b279c8 became leader
awx         16m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_d7eea5f4-e824-49ad-a9cd-ed815959fcc6 became leader
awx         11m         Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_32d9ebcc-b497-4a5b-86f2-bddd82485090 became leader
awx         5m40s       Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_ddf5fd04-d806-4862-a3b9-a03e756a34fd became leader
awx         6s          Normal    LeaderElection        lease/awx-operator                                     awx-operator-controller-manager-745b55d94b-97sdw_bed743ee-fbaf-4734-a56f-c31bb7eef1b5 became leader
default     3m38s       Warning   ProvisioningFailed    persistentvolumeclaim/awx-projects-claim               storageclass.storage.k8s.io "awx-projects-volume" not found
default     28m         Warning   FreeDiskSpaceFailed   node/server                                   Failed to garbage collect required amount of images. Attempted to free 6804366950 bytes, but only found 0 bytes eligible to free.
default     23m         Warning   FreeDiskSpaceFailed   node/server                                   Failed to garbage collect required amount of images. Attempted to free 6805468774 bytes, but only found 0 bytes eligible to free.
default     23m         Warning   ImageGCFailed         node/server                                   Failed to garbage collect required amount of images. Attempted to free 6805468774 bytes, but only found 0 bytes eligible to free.
default     18m         Warning   FreeDiskSpaceFailed   node/server                                   Failed to garbage collect required amount of images. Attempted to free 6805063270 bytes, but only found 0 bytes eligible to free.
default     18m         Warning   ImageGCFailed         node/server                                   Failed to garbage collect required amount of images. Attempted to free 6805063270 bytes, but only found 0 bytes eligible to free.
default     13m         Warning   FreeDiskSpaceFailed   node/server                                   Failed to garbage collect required amount of images. Attempted to free 6805378662 bytes, but only found 0 bytes eligible to free.
default     13m         Warning   ImageGCFailed         node/server                                   Failed to garbage collect required amount of images. Attempted to free 6805378662 bytes, but only found 0 bytes eligible to free.
default     8m51s       Warning   FreeDiskSpaceFailed   node/server                                   Failed to garbage collect required amount of images. Attempted to free 6803998310 bytes, but only found 0 bytes eligible to free.
default     8m51s       Warning   ImageGCFailed         node/server                                   Failed to garbage collect required amount of images. Attempted to free 6803998310 bytes, but only found 0 bytes eligible to free.
default     3m51s       Warning   FreeDiskSpaceFailed   node/server                                   Failed to garbage collect required amount of images. Attempted to free 6803936870 bytes, but only found 0 bytes eligible to free.
default     3m51s       Warning   ImageGCFailed         node/server                                   Failed to garbage collect required amount of images. Attempted to free 6803936870 bytes, but only found 0 bytes eligible to free.
[root@server ~]#
[root@server ~]# df -h /
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-RootVol00   70G   52G   18G  75% /
[root@server ~]# free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       6.2Gi       1.4Gi       684Mi        24Gi        24Gi
Swap:          1.0Gi       0.0Ki       1.0Gi
[root@server ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    256 1445372   2964 25803072    0    0    41    14    0    1  1  1 98  0  0
[root@server ~]#
``````````````````````
6 
awx-operator-controller-manager-745b55d94b-97sdw   1/2     CrashLoopBackOff   4639 (2m29s ago)   46d

Thanks for updating, this indicates that your operator keeps restarting. You should investigate your operator to identify the reason of the restarting.

It is not possible to determine from the current log whether the Operator is unable to back up because it keeps restarting, or whether the backup process is causing it to restart.

7

Thank you @kurokobo,
I will check, also i have added operator log

[root@server ~]# kubectl get pods -n awx
NAME                                               READY   STATUS      RESTARTS           AGE
automation-job-1402-r5smn                          0/1     Error       0                  25h
automation-job-1406-vw5vs                          0/1     Completed   0                  25h
automation-job-1410-qhbzm                          0/1     Error       0                  25h
automation-job-1418-rzjrj                          0/1     Error       0                  12h
automation-job-1421-gvklc                          0/1     Error       0                  127m
automation-job-1423-6vtc8                          0/1     Error       0                  110m
automation-job-1425-xs87g                          0/1     Error       0                  28m
automation-job-1427-hrdf9                          0/1     Error       0                  26m
automation-job-1429-4nscb                          0/1     Error       0                  18m
automation-job-1431-wlkqd                          0/1     Error       0                  7m38s
awx-operator-controller-manager-745b55d94b-97sdw   2/2     Running     4654 (5m31s ago)   46d
awx-postgres-15-0                                  1/1     Running     12 (18d ago)       144d
awx-task-55d8c9dd9f-qzxr5                          4/4     Running     4 (18d ago)        21d
awx-web-59498d8589-98bph                           3/3     Running     24 (18d ago)       121d
[root@server ~]# kubectl get pods awx-operator-controller-manager-745b55d94b-97sdw -n awx
NAME                                               READY   STATUS    RESTARTS           AGE
awx-operator-controller-manager-745b55d94b-97sdw   2/2     Running   4654 (5m44s ago)   46d
[root@server ~]# kubectl logs awx-operator-controller-manager-745b55d94b-97sdw -n awx
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx"]}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":"2024-12-27T12:04:47Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-12-27T12:04:47Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
I1227 12:04:47.228843       7 leaderelection.go:250] attempting to acquire leader lease awx/awx-operator...
I1227 12:05:05.478317       7 leaderelection.go:260] successfully acquired lease awx/awx-operator
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awxrestore-controller"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awxmeshingress-controller"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awxbackup-controller"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awx-controller"}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awx-controller","worker count":8}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awxmeshingress-controller","worker count":8}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":8}
{"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":8}
{"level":"info","ts":"2024-12-27T12:05:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[0].message\""}
{"level":"info","ts":"2024-12-27T12:05:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[1].message\""}
{"level":"info","ts":"2024-12-27T12:05:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[2].message\""}
{"level":"info","ts":"2024-12-27T12:05:13Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"436506494050674087","EventData.Name":"Verify imagePullSecrets"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Verify imagePullSecrets] *************************************************
task path: /opt/ansible/playbooks/awx.yml:10

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"2600480675036880980","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8813032395009318914","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8861251273508755859","EventData.TaskArgs":""}
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2600480675036880980","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8630151664447508170","EventData.TaskArgs":""}
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8813032395009318914","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8861251273508755859","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"2485108430253905831","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"2061232962065953104","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"7175231377219730276","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-11-29-12-30-54","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8501857911985331229","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8630151664447508170","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-11-29-12-30-54","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8501857911985331229","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2061232962065953104","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"7175231377219730276","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2485108430253905831","EventData.Name":"backup : Run creation tasks"}
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2600480675036880980","EventData.Name":"backup : Run finalizer tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run finalizer tasks] ********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:6

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run finalizer tasks] ********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:6

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8861251273508755859","EventData.Name":"backup : Run finalizer tasks"}
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"7175231377219730276","EventData.Name":"backup : Run finalizer tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run finalizer tasks] ********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:6

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2061232962065953104","EventData.Name":"backup : Run finalizer tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run finalizer tasks] ********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:6

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8813032395009318914","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2485108430253905831","EventData.Name":"backup : Run finalizer tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run finalizer tasks] ********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:6

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-11-29-12-30-54","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8501857911985331229","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2600480675036880980","EventData.Name":"backup : Look up details for this backup object"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Look up details for this backup object] *************************
task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8861251273508755859","EventData.Name":"backup : Look up details for this backup object"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Look up details for this backup object] *************************
task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8630151664447508170","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"7175231377219730276","EventData.Name":"backup : Look up details for this backup object"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Look up details for this backup object] *************************
task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2061232962065953104","EventData.Name":"backup : Look up details for this backup object"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Look up details for this backup object] *************************
task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2485108430253905831","EventData.Name":"backup : Look up details for this backup object"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Look up details for this backup object] *************************
task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2024-12-27T12:05:18Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/awx/secrets/redhat-operators-pull-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"secrets","Subresource":"","Name":"redhat-operators-pull-secret","Parts":["secrets","redhat-operators-pull-secret"]}}
{"level":"info","ts":"2024-12-27T12:05:18Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"436506494050674087","EventData.Name":"Create imagePullSecret"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Create imagePullSecret] **************************************************
task path: /opt/ansible/playbooks/awx.yml:17

-------------------------------------------------------------------------------
[root@server ~]#
8

Hello @kurokobo
I have tried to restart K3s service but still getting same operator error.
Other job templates are successfully running, only backup job is failing.

Is there any way to troubleshoot further, Please suggest me.

awx-operator-controller-manager-745b55d94b-97sdw   1/2     OOMKilled   4879 (119s ago)   47d

[root@server ~]# kubectl get pods -n awx
NAME                                               READY   STATUS      RESTARTS          AGE
automation-job-1402-r5smn                          0/1     Error       0                 46h
automation-job-1406-vw5vs                          0/1     Completed   0                 45h
automation-job-1410-qhbzm                          0/1     Error       0                 45h
automation-job-1418-rzjrj                          0/1     Error       0                 33h
automation-job-1421-gvklc                          0/1     Error       0                 22h
automation-job-1423-6vtc8                          0/1     Error       0                 22h
automation-job-1425-xs87g                          0/1     Error       0                 21h
automation-job-1427-hrdf9                          0/1     Error       0                 21h
automation-job-1429-4nscb                          0/1     Error       0                 21h
automation-job-1431-wlkqd                          0/1     Error       0                 20h
automation-job-1433-79bnp                          0/1     Error       0                 9h
automation-job-1435-mjzfh                          0/1     Completed   0                 8m22s
automation-job-1437-kmjtw                          0/1     Completed   0                 2m54s
awx-operator-controller-manager-745b55d94b-97sdw   1/2     OOMKilled   4879 (119s ago)   47d
awx-postgres-15-0                                  1/1     Running     12 (19d ago)      144d
awx-task-55d8c9dd9f-qzxr5                          4/4     Running     4 (19d ago)       22d
awx-web-59498d8589-98bph                           3/3     Running     24 (19d ago)      121d
[root@server ~]# kubectl get events
LAST SEEN   TYPE      REASON                           OBJECT                                     MESSAGE
10m         Warning   ProvisioningFailed               persistentvolumeclaim/awx-projects-claim   storageclass.storage.k8s.io "awx-projects-volume" not found
12s         Warning   ProvisioningFailed               persistentvolumeclaim/awx-projects-claim   storageclass.storage.k8s.io "awx-projects-volume" not found
10m         Warning   FreeDiskSpaceFailed              node/server                       (combined from similar events): Failed to garbage collect required amount of images. Attempted to free 9734415974 bytes, but only found 0 bytes eligible to free.
50m         Warning   FreeDiskSpaceFailed              node/server                       Failed to garbage collect required amount of images. Attempted to free 9733240422 bytes, but only found 0 bytes eligible to free.
5m53s       Normal    Starting                         node/server                       Starting kubelet.
5m53s       Warning   InvalidDiskCapacity              node/server                       invalid capacity 0 on image filesystem
5m53s       Normal    NodeHasSufficientMemory          node/server                       Node server status is now: NodeHasSufficientMemory
5m53s       Normal    NodeHasNoDiskPressure            node/server                       Node server status is now: NodeHasNoDiskPressure
5m53s       Normal    NodeHasSufficientPID             node/server                       Node server status is now: NodeHasSufficientPID
5m53s       Normal    NodeNotReady                     node/server                       Node server status is now: NodeNotReady
5m53s       Normal    NodeAllocatableEnforced          node/server                       Updated Node Allocatable limit across pods
5m52s       Normal    Starting                         node/server
5m49s       Normal    NodePasswordValidationComplete   node/server                       Deferred node password secret validation complete
5m43s       Normal    NodeReady                        node/server                       Node server status is now: NodeReady
5m42s       Normal    RegisteredNode                   node/server                       Node server event: Registered Node server in Controller
53s         Warning   FreeDiskSpaceFailed              node/server                       Failed to garbage collect required amount of images. Attempted to free 11668506214 bytes, but only found 0 bytes eligible to free.
53s         Warning   ImageGCFailed                    node/server                       Failed to garbage collect required amount of images. Attempted to free 11668506214 bytes, but only found 0 bytes eligible to free.
[root@server ~]#
9

I have added pod logs

[root@g20-ansible-cxp ~]# kubectl describe pod awx-operator-controller-manager-745b55d94b-97sdw -n awx
Name:             awx-operator-controller-manager-745b55d94b-97sdw
Namespace:        awx
Priority:         0
Service Account:  awx-operator-controller-manager
Node:             server/IPADDRESS
Start Time:       Sun, 10 Nov 2024 20:58:47 -0800
Labels:           control-plane=controller-manager
                  pod-template-hash=745b55d94b
Annotations:      kubectl.kubernetes.io/default-container: awx-manager
Status:           Running
IP:               server
IPs:
  IP:           server
Controlled By:  ReplicaSet/awx-operator-controller-manager-745b55d94b
Containers:
  kube-rbac-proxy:
    Container ID:  containerd://1e7f895d49f51ce8f0cc6ac1baa8aae81b12c2ce6fa1242dc377a2c8f2f9a945
    Image:         gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
    Image ID:      gcr.io/kubebuilder/kube-rbac-proxy@sha256:d8cc6ffb98190e8dd403bfe67ddcb454e6127d32b87acc237b3e5240f70a20fb
    Port:          8443/TCP
    Host Port:     0/TCP
    Args:
      --secure-listen-address=0.0.0.0:8443
      --upstream=http://127.0.0.1:8080/
      --logtostderr=true
      --v=0
    State:          Running
      Started:      Mon, 30 Dec 2024 02:00:36 -0800
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Sun, 08 Dec 2024 23:08:01 -0800
      Finished:     Mon, 30 Dec 2024 02:00:33 -0800
    Ready:          True
    Restart Count:  4
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        5m
      memory:     64Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hrvhv (ro)
  awx-manager:
    Container ID:  containerd://4d79566baf0e733bcbf3276ea2d7db60f24c9650c558e37bfe1f8308bb7857dc
    Image:         quay.io/ansible/awx-operator:2.19.1
    Image ID:      quay.io/ansible/awx-operator@sha256:7302e0c8e5a79c11275848b36c63c61d8e7487840164c109b7cda9aa93456901
    Port:          <none>
    Host Port:     <none>
    Args:
      --health-probe-bind-address=:6789
      --metrics-bind-address=127.0.0.1:8080
      --leader-elect
      --leader-election-id=awx-operator
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Mon, 30 Dec 2024 02:11:33 -0800
      Finished:     Mon, 30 Dec 2024 02:13:10 -0800
    Ready:          False
    Restart Count:  5407
    Limits:
      cpu:     1500m
      memory:  960Mi
    Requests:
      cpu:      50m
      memory:   32Mi
    Liveness:   http-get http://:6789/healthz delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:6789/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ANSIBLE_GATHERING:   explicit
      ANSIBLE_DEBUG_LOGS:  false
      WATCH_NAMESPACE:     awx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hrvhv (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-hrvhv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                     From     Message
  ----     ------                  ----                    ----     -------
  Normal   Pulled                  61m (x519 over 2d1h)    kubelet  Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine
  Warning  BackOff                 41m (x12865 over 2d1h)  kubelet  Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d)
  Normal   Created                 36m (x4 over 39m)       kubelet  Created container awx-manager
  Normal   Started                 36m (x4 over 39m)       kubelet  Started container awx-manager
  Warning  BackOff                 24m (x60 over 38m)      kubelet  Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d)
  Normal   Pulled                  19m (x8 over 39m)       kubelet  Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine
  Warning  FailedCreatePodSandBox  17m                     kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fc9bee795974a4ef4351a5e2d011e50f0c13a584c844e7d0faa7e075e31a6730": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory
  Normal   Pulled                  17m                     kubelet  Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0" already present on machine
  Normal   SandboxChanged          17m (x2 over 17m)       kubelet  Pod sandbox changed, it will be killed and re-created.
  Normal   Created                 17m                     kubelet  Created container kube-rbac-proxy
  Normal   Started                 17m                     kubelet  Started container kube-rbac-proxy
  Warning  Unhealthy               16m (x3 over 16m)       kubelet  Liveness probe failed: Get "http://server:6789/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing                 16m                     kubelet  Container awx-manager failed liveness probe, will be restarted
  Normal   Pulled                  15m (x2 over 17m)       kubelet  Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine
  Normal   Created                 15m (x2 over 17m)       kubelet  Created container awx-manager
  Normal   Started                 15m (x2 over 17m)       kubelet  Started container awx-manager
  Warning  Unhealthy               11m (x21 over 16m)      kubelet  Readiness probe failed: Get "http://server:6789/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  BackOff                 2m20s (x31 over 13m)    kubelet  Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d)
[root@g20-ansible-cxp ~]#
10

Since OOMKilled is recorded, it is possible that the memory usage temporarily spiked immediately after the Operator started. Please check the Operator’s logs. The logs you provided are from when it was functioning normally and do not contain information from the moment it terminated abnormally.

It is likely that the cause is an excessive number of awxbackup resources. The Operator tries to execute Playbooks for each awxbackup resource at startup.

Please delete any unnecessary or failed awxbackup resources. However, since the deletion itself requires processing by the Operator, the delete command will likely get stuck. If it gets stuck, please remove the finalizer from the awxbackup resources that are in the process of being deleted.

11

ref:

12

Thank you @kurokobo for sharing knowledge.

As you mentioned, i have cleaned up unnecessary/failed AWX backup instances. I successfully cleaned up but still new backup job is failing and also Still AWX operator is showing same error “CrashLoopBackOff”.

[root@server ~]# kubectl get awxbackup -n awx
NAME                            AGE
awxbackup-2024-11-29-12-38-30   33d
awxbackup-2024-12-02-04-30-38   31d
awxbackup-2024-12-02-23-30-38   30d
awxbackup-2024-12-03-23-30-38   29d
awxbackup-2024-12-04-23-30-46   28d
awxbackup-2024-12-05-23-30-26   27d
awxbackup-2024-12-06-23-30-28   26d
awxbackup-2024-12-07-23-30-27   25d
awxbackup-2024-12-08-23-30-26   24d
awxbackup-2024-12-26-11-05-04   6d23h
awxbackup-2024-12-26-11-06-20   6d23h
[root@server ~]# ls -l /data/backup/
total 0
drwxr-xr-x 2 26 26 59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800
drwxr-xr-x 2 26 26 59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843
drwxr-xr-x 2 26 26 59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859
drwxr-xr-x 2 26 26 59 Dec  1 20:31 tower-openshift-backup-2024-12-02-043052
drwxr-xr-x 2 26 26 59 Dec  2 15:31 tower-openshift-backup-2024-12-02-233051
drwxr-xr-x 2 26 26 59 Dec  3 15:31 tower-openshift-backup-2024-12-03-233052
drwxr-xr-x 2 26 26 59 Dec  4 15:31 tower-openshift-backup-2024-12-04-233059
drwxr-xr-x 2 26 26 59 Dec  5 15:30 tower-openshift-backup-2024-12-05-233039
drwxr-xr-x 2 26 26 59 Dec  6 15:30 tower-openshift-backup-2024-12-06-233041
drwxr-xr-x 2 26 26 59 Dec  7 15:30 tower-openshift-backup-2024-12-07-233039
drwxr-xr-x 2 26 26 59 Dec  8 15:30 tower-openshift-backup-2024-12-08-233039
[root@server ~]# 
[root@server ~]# kubectl -n awx logs -f awx-operator-controller-manager-745b55d94b-5rpht
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx"]}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2025-01-02T11:42:50Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"}
{"level":"info","ts":"2025-01-02T11:42:50Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
I0102 11:42:50.637689       7 leaderelection.go:250] attempting to acquire leader lease awx/awx-operator...
I0102 11:43:06.436319       7 leaderelection.go:260] successfully acquired lease awx/awx-operator
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awx-controller"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awxbackup-controller"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awxrestore-controller"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: *unstructured.Unstructured"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awxmeshingress-controller"}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awxmeshingress-controller","worker count":8}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":8}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awx-controller","worker count":8}
{"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":8}
{"level":"info","ts":"2025-01-02T11:43:06Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[0].message\""}
{"level":"info","ts":"2025-01-02T11:43:06Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[1].message\""}
{"level":"info","ts":"2025-01-02T11:43:06Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[2].message\""}
{"level":"info","ts":"2025-01-02T11:43:13Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"2461380222856614183","EventData.Name":"Verify imagePullSecrets"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Verify imagePullSecrets] *************************************************
task path: /opt/ansible/playbooks/awx.yml:10

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8415435552159357307","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-04-23-30-46","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"3794599929215878019","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8415435552159357307","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-04-23-30-46","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3794599929215878019","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-02-04-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"721827200833615441","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-02-04-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"721827200833615441","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8415435552159357307","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-04-23-30-46","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3794599929215878019","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-02-04-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"721827200833615441","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-08-23-30-26","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"6180278095043985779","EventData.TaskArgs":""}
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-08-23-30-26","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"6180278095043985779","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-06-23-30-28","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"5452128470082692865","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-08-23-30-26","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"6180278095043985779","EventData.Name":"backup : Patching labels to AWXBackup kind"}
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-06-23-30-28","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"5452128470082692865","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-26-11-05-04","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"3600057349764572609","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-05-04","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3600057349764572609","EventData.Name":"backup : Run creation tasks"}
{"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-06-23-30-28","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"5452128470082692865","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-05-04","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3600057349764572609","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"4102565071948320805","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"4102565071948320805","EventData.Name":"backup : Run creation tasks"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-03-23-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"3177590642827013235","EventData.TaskArgs":""}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [debug] ********************************
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Run creation tasks] *********************************************
task path: /opt/ansible/roles/backup/tasks/main.yml:2

-------------------------------------------------------------------------------

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-03-23-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3177590642827013235","EventData.Name":"backup : Run creation tasks"}
{"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"4102565071948320805","EventData.Name":"backup : Patching labels to AWXBackup kind"}
{"level":"info","ts":"2025-01-02T11:43:18Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-03-23-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3177590642827013235","EventData.Name":"backup : Patching labels to AWXBackup kind"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [backup : Patching labels to AWXBackup kind] ******************************
task path: /opt/ansible/roles/backup/tasks/creation.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2025-01-02T11:43:18Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/awx/secrets/redhat-operators-pull-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"secrets","Subresource":"","Name":"redhat-operators-pull-secret","Parts":["secrets","redhat-operators-pull-secret"]}}
{"level":"info","ts":"2025-01-02T11:43:19Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"2461380222856614183","EventData.Name":"Create imagePullSecret"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Create imagePullSecret] **************************************************
task path: /opt/ansible/playbooks/awx.yml:17

-------------------------------------------------------------------------------
[root@server ~]#
13

Next step :- I will try to delete all backups and initiate the backup again.

14

Thank you very much @kurokobo
Once all backup resources are cleaned up, i able to run the backup job successfully.

In backup file - can I change keep_days: 30 values to “7”?

Will this help to lower the resource consumption which makes Operator became more stable.

[root@server ~]# ls -l /data/backup/
total 0
drwxr-xr-x 2 26 26 59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800
drwxr-xr-x 2 26 26 59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843
drwxr-xr-x 2 26 26 59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859
drwxr-xr-x 2 26 26 59 Dec  1 20:31 tower-openshift-backup-2024-12-02-043052
drwxr-xr-x 2 26 26 59 Dec  2 15:31 tower-openshift-backup-2024-12-02-233051
drwxr-xr-x 2 26 26 59 Dec  3 15:31 tower-openshift-backup-2024-12-03-233052
drwxr-xr-x 2 26 26 59 Dec  4 15:31 tower-openshift-backup-2024-12-04-233059
drwxr-xr-x 2 26 26 59 Dec  5 15:30 tower-openshift-backup-2024-12-05-233039
drwxr-xr-x 2 26 26 59 Dec  6 15:30 tower-openshift-backup-2024-12-06-233041
drwxr-xr-x 2 26 26 59 Dec  7 15:30 tower-openshift-backup-2024-12-07-233039
drwxr-xr-x 2 26 26 59 Dec  8 15:30 tower-openshift-backup-2024-12-08-233039
drwxr-xr-x 2 26 26 59 Jan  2 03:54 tower-openshift-backup-2025-01-02-115430
[root@g20-ansible-cxp ~]#  kubectl get awxbackup -n awx
NAME                            AGE
awxbackup-2025-01-02-11-54-17   98s
[root@server  ~]#
15

In backup file - can I change keep_days: 30 values to “7”?

Yes, you can change the variable awxbackup_keep_days to 7 to make the Operator more stable: awx-on-k3s/backup/ansible at main · kurokobo/awx-on-k3s · GitHub

16

Thanks a lot @kurokobo
With your technical support, we able to fix the issue.