# AWX scheduled backup job failed after couple of days - AWX 24.6.1 **URL:** \ **Category:** Get Help\ **Tags:** awx, kubernetes\ **Created:** [December 27, 2024, 5:22am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344 "2024-12-27T05:22:35Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 27, 2024, 5:22am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/1 "2024-12-27T05:22:35Z")
**AWX scheduled backup job failed - AWX 24.6.1** I have scheduled AWX backup job by referring following steps “[awx-on-k3s/backup/ansible at main · kurokobo/awx-on-k3s · GitHub](https://github.com/kurokobo/awx-on-k3s/tree/main/backup/ansible)” I have used **Use with AWX** option and scheduled backup job. It was working fine for 8 days then continuously getting failed with below error logs. Kindly help me to fix the issue @kurokobo Logs ```auto [root@server ~]# kubectl get awxbackup -n awx NAME AGE awxbackup-2024-11-29-12-25-22 27d awxbackup-2024-11-29-12-30-54 27d awxbackup-2024-11-29-12-38-30 27d awxbackup-2024-12-02-04-30-38 25d awxbackup-2024-12-02-23-30-38 24d awxbackup-2024-12-03-23-30-38 23d awxbackup-2024-12-04-23-30-46 22d awxbackup-2024-12-05-23-30-26 21d awxbackup-2024-12-06-23-30-28 20d awxbackup-2024-12-07-23-30-27 19d awxbackup-2024-12-08-23-30-26 18d awxbackup-2024-12-09-23-30-22 17d awxbackup-2024-12-10-23-30-22 16d awxbackup-2024-12-11-23-30-21 15d awxbackup-2024-12-12-11-33-29 14d awxbackup-2024-12-12-23-30-21 14d awxbackup-2024-12-16-23-30-22 10d awxbackup-2024-12-17-23-30-22 9d awxbackup-2024-12-20-23-30-20 6d5h awxbackup-2024-12-23-23-30-21 3d5h awxbackup-2024-12-26-10-29-32 18h awxbackup-2024-12-26-11-05-04 18h awxbackup-2024-12-26-11-06-20 18h awxbackup-2024-12-26-11-10-30 18h [root@server ~]# [root@server ~]# ls -l /data/backup total 15412 drwxr-xr-x 2 26 26 59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800 drwxr-xr-x 2 26 26 59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843 drwxr-xr-x 2 26 26 59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859 drwxr-xr-x 2 26 26 59 Dec 1 20:31 tower-openshift-backup-2024-12-02-043052 drwxr-xr-x 2 26 26 59 Dec 2 15:31 tower-openshift-backup-2024-12-02-233051 -rw-r--r-- 1 root root 15779840 Dec 3 03:15 tower-openshift-backup-2024-12-02-233051.tar drwxr-xr-x 2 26 26 59 Dec 3 15:31 tower-openshift-backup-2024-12-03-233052 drwxr-xr-x 2 26 26 59 Dec 4 15:31 tower-openshift-backup-2024-12-04-233059 drwxr-xr-x 2 26 26 59 Dec 5 15:30 tower-openshift-backup-2024-12-05-233039 drwxr-xr-x 2 26 26 59 Dec 6 15:30 tower-openshift-backup-2024-12-06-233041 drwxr-xr-x 2 26 26 59 Dec 7 15:30 tower-openshift-backup-2024-12-07-233039 drwxr-xr-x 2 26 26 59 Dec 8 15:30 tower-openshift-backup-2024-12-08-233039 [root@server ~]# ``` Logs:- ```auto {"uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "counter": 18, "stdout": "\r\nTASK [Create new AWXBackup resource and wait for complete] *********************", "start_line": 45, "end_line": 47, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.876625+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "name": "Create new AWXBackup resource and wait for complete", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-00000000000b"}} {"uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741", "counter": 19, "stdout": "", "start_line": 47, "end_line": 47, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.877557+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741"}} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"uuid": "e8f324c4-941c-4d09-bcdd-87ee641e1c49", "counter": 20, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": true, \"duration\": 600, \"method\": \"create\", \"msg\": \"\\\"AWXBackup\\\" \\\"awxbackup-2024-12-26-23-30-21\\\": Timed out waiting on resource\", \"result\": {\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWXBackup\", \"metadata\": {\"creationTimestamp\": \"2024-12-26T23:30:22Z\", \"generation\": 1, \"managedFields\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:spec\": {\".\": {}, \"f:backup_pvc\": {}, \"f:clean_backup_on_delete\": {}, \"f:deployment_name\": {}, \"f:image_pull_policy\": {}, \"f:no_log\": {}, \"f:set_self_labels\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2024-12-26T23:30:22Z\"}], \"name\": \"awxbackup-2024-12-26-23-30-21\", \"namespace\": \"awx\", \"resourceVersion\": \"9000995\", \"uid\": \"03f81604-198d-44de-8e0e-0289dfa77320\"}, \"spec\": {\"backup_pvc\": \"awx-backup-claim\", \"clean_backup_on_delete\": true, \"deployment_name\": \"awx\", \"image_pull_policy\": \"IfNotPresent\", \"no_log\": true, \"set_self_labels\": true}}}\u001b[0m", "start_line": 47, "end_line": 48, "runner_ident": "1418", "event": "runner_on_failed", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:40:23.475258+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"creationTimestamp": "2024-12-26T23:30:22Z", "generation": 1, "managedFields": [{"apiVersion": "awx.ansible.com/v1beta1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:backup_pvc": {}, "f:clean_backup_on_delete": {}, "f:deployment_name": {}, "f:image_pull_policy": {}, "f:no_log": {}, "f:set_self_labels": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-12-26T23:30:22Z"}], "name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx", "resourceVersion": "9000995", "uid": "03f81604-198d-44de-8e0e-0289dfa77320"}, "spec": {"backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true, "deployment_name": "awx", "image_pull_policy": "IfNotPresent", "no_log": true, "set_self_labels": true}}, "method": "create", "duration": 600, "msg": "\"AWXBackup\" \"awxbackup-2024-12-26-23-30-21\": Timed out waiting on resource", "invocation": {"module_args": {"state": "present", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "wait_condition": {"reason": "Successful", "status": true, "type": "Running"}, "wait_timeout": 600, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "merge_type": null, "validate": null, "template": null, "delete_options": null, "label_selectors": null, "generate_name": null, "server_side_apply": null, "hidden_fields": null}}, "_ansible_no_log": false}, "start": "2024-12-26T23:30:21.877491+00:00", "end": "2024-12-26T23:40:23.475118+00:00", "duration": 601.597627, "ignore_errors": null, "event_loop": null, "uuid": "e8f324c4-941c-4d09-bcdd-87ee641e1c49"}} ``` ![backup](https://us1.discourse-cdn.com/ansible/original/2X/7/71aa5d4fc40d1330aea52e52198911fa2fc989a8.png) ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 27, 2024, 5:38am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/2 "2024-12-27T05:38:29Z")
Logs ```auto [root@server ~]# kubectl get pods -n awx NAME READY STATUS RESTARTS AGE automation-job-1402-r5smn 0/1 Error 0 18h automation-job-1406-vw5vs 0/1 Completed 0 17h automation-job-1410-qhbzm 0/1 Error 0 17h automation-job-1418-rzjrj 0/1 Error 0 5h20m awx-operator-controller-manager-745b55d94b-97sdw 2/2 Running 4577 (5m42s ago) 45d awx-postgres-15-0 1/1 Running 12 (17d ago) 143d awx-task-55d8c9dd9f-qzxr5 4/4 Running 4 (17d ago) 21d awx-web-59498d8589-98bph 3/3 Running 24 (17d ago) 120d [root@server ~]# kubectl logs automation-job-1418-rzjrj -n awx {"status": "starting", "runner_ident": "1418", "command": ["ansible-playbook", "-u", "admin", "-i", "/runner/inventory/hosts", "-e", "@/runner/env/extravars", "backup/ansible/project/backup.yml"], "env": {"KUBERNETES_SERVICE_PORT_HTTPS": "443", "KUBERNETES_SERVICE_PORT": "443", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_HOST": "10.43.106.140", "HOSTNAME": "automation-job-1418-rzjrj", "AWX_SERVICE_PORT_80_TCP": "tcp://10.43.61.35:80", "PWD": "/runner", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PORT": "8443", "PIP_BREAK_SYSTEM_PACKAGES": "1", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO": "tcp", "AWX_SERVICE_SERVICE_PORT": "80", "HOME": "/runner", "KUBERNETES_PORT_443_TCP": "tcp://10.43.0.1:443", "ANSIBLE_RUNNER_KEEPALIVE_SECONDS": "30", "AWX_SERVICE_PORT_80_TCP_ADDR": "10.43.61.35", "AWX_SERVICE_PORT": "tcp://10.43.61.35:80", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT": "tcp://10.43.106.140:8443", "AWX_SERVICE_PORT_80_TCP_PORT": "80", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT_HTTPS": "8443", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT": "8443", "SHLVL": "0", "KUBERNETES_PORT_443_TCP_PROTO": "tcp", "KUBERNETES_PORT_443_TCP_ADDR": "10.43.0.1", "AWX_SERVICE_SERVICE_HOST": "10.43.61.35", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP": "tcp://10.43.106.140:8443", "AWX_SERVICE_SERVICE_PORT_HTTP": "80", "KUBERNETES_SERVICE_HOST": "10.43.0.1", "KUBERNETES_PORT": "tcp://10.43.0.1:443", "KUBERNETES_PORT_443_TCP_PORT": "443", "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR": "10.43.106.140", "AWX_SERVICE_PORT_80_TCP_PROTO": "tcp", "LC_CTYPE": "C.UTF-8", "ANSIBLE_BASE_ALLOW_CUSTOM_ROLES": "True", "ANSIBLE_BASE_ALLOW_CUSTOM_TEAM_ROLES": "False", "ANSIBLE_BASE_ALLOW_SINGLETON_ROLES_API": "False", "ANSIBLE_BASE_ALLOW_SINGLETON_TEAM_ROLES": "False", "ANSIBLE_BASE_ALLOW_SINGLETON_USER_ROLES": "True", "ANSIBLE_BASE_ALLOW_TEAM_ORG_ADMIN": "False", "ANSIBLE_BASE_ALLOW_TEAM_ORG_MEMBER": "False", "ANSIBLE_BASE_ALLOW_TEAM_ORG_PERMS": "True", "ANSIBLE_BASE_ALLOW_TEAM_PARENTS": "True", "ANSIBLE_BASE_ALL_REST_FILTERS": "('ansible_base.rest_filters.rest_framework.type_filter_backend.TypeFilterBackend', 'ansible_base.rest_filters.rest_framework.field_lookup_backend.FieldLookupBackend', 'rest_framework.filters.SearchFilter', 'ansible_base.rest_filters.rest_framework.order_backend.OrderByBackend')", "ANSIBLE_BASE_BYPASS_ACTION_FLAGS": "{}", "ANSIBLE_BASE_BYPASS_SUPERUSER_FLAGS": "['is_superuser']", "ANSIBLE_BASE_CACHE_PARENT_PERMISSIONS": "True", "ANSIBLE_BASE_CHECK_RELATED_PERMISSIONS": "['use', 'change', 'view']", "ANSIBLE_BASE_CREATOR_DEFAULTS": "['change', 'delete', 'execute', 'use', 'adhoc', 'approve', 'update', 'view']", "ANSIBLE_BASE_CUSTOM_VIEW_PARENT": "awx.api.generics.APIView", "ANSIBLE_BASE_EVALUATIONS_IGNORE_CONFLICTS": "True", "ANSIBLE_BASE_MANAGED_ROLE_REGISTRY": "{}", "ANSIBLE_BASE_ORGANIZATION_MODEL": "main.Organization", "ANSIBLE_BASE_PERMISSION_MODEL": "main.Permission", "ANSIBLE_BASE_RESOURCE_CONFIG_MODULE": "awx.resource_api", "ANSIBLE_BASE_REST_FILTERS_RESERVED_NAMES": "('page', 'page_size', 'format', 'order', 'order_by', 'search', 'type', 'host_filter', 'count_disabled', 'no_truncate', 'limit', 'validate')", "ANSIBLE_BASE_ROLE_CREATOR_NAME": "{cls. __name__ } Creator", "ANSIBLE_BASE_ROLE_PRECREATE": "{}", "ANSIBLE_BASE_ROLE_SYSTEM_ACTIVATED": "True", "ANSIBLE_BASE_TEAM_MODEL": "main.Team", "ANSIBLE_FACT_CACHE_TIMEOUT": "0", "ANSIBLE_FORCE_COLOR": "True", "ANSIBLE_HOST_KEY_CHECKING": "False", "ANSIBLE_INVENTORY_UNPARSED_FAILED": "True", "ANSIBLE_PARAMIKO_RECORD_HOST_KEYS": "False", "AWX_PRIVATE_DATA_DIR": "/tmp/awx_1418_xww0hh3i", "JOB_ID": "1418", "INVENTORY_ID": "1", "PROJECT_REVISION": "7885966239d92108d9a83ce96e6cb88c4257d8e7", "ANSIBLE_RETRY_FILES_ENABLED": "False", "MAX_EVENT_RES": "700000", "AWX_HOST": "https://server.chq.ei/", "ANSIBLE_SSH_CONTROL_PATH_DIR": "/runner/cp", "ANSIBLE_COLLECTIONS_PATHS": "/runner/requirements_collections:~/.ansible/collections:/usr/share/ansible/collections", "ANSIBLE_ROLES_PATH": "/runner/requirements_roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles", "ANSIBLE_COLLECTIONS_PATH": "/runner/requirements_collections:~/.ansible/collections:/usr/share/ansible/collections", "ANSIBLE_CALLBACK_PLUGINS": "/usr/local/lib/python3.9/site-packages/ansible_runner/display_callback/callback", "ANSIBLE_STDOUT_CALLBACK": "awx_display", "AWX_ISOLATED_DATA_DIR": "/runner/artifacts/1418", "RUNNER_OMIT_EVENTS": "False", "RUNNER_ONLY_FAILED_EVENTS": "False"}, "cwd": "/runner/project"} {"status": "running", "runner_ident": "1418"} {"uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "counter": 1, "stdout": "", "start_line": 0, "end_line": 0, "runner_ident": "1418", "event": "playbook_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.731345+00:00", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf"}} {"uuid": "4e329af7-df1e-0679-2e8f-000000000002", "counter": 2, "stdout": "\r\nPLAY [Backing up AWX using AWX Operator] ***************************************", "start_line": 0, "end_line": 2, "runner_ident": "1418", "event": "playbook_on_play_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.732799+00:00", "parent_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "name": "Backing up AWX using AWX Operator", "pattern": "localhost", "uuid": "4e329af7-df1e-0679-2e8f-000000000002"}} {"uuid": "4e329af7-df1e-0679-2e8f-000000000004", "counter": 3, "stdout": "\r\nTASK [Construct default variables] *********************************************", "start_line": 2, "end_line": 4, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.744816+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct default variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:9", "name": "Construct default variables", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000004"}} {"uuid": "6c5d59a5-4e57-4d6e-8d82-e161f2b99c86", "counter": 4, "stdout": "", "start_line": 4, "end_line": 4, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.745646+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct default variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:9", "host": "localhost", "uuid": "6c5d59a5-4e57-4d6e-8d82-e161f2b99c86"}} {"uuid": "2eb3e739-546c-4c4c-9849-d8d398f7f9e0", "counter": 5, "stdout": "\u001b[0;32mok: [localhost]\u001b[0m", "start_line": 4, "end_line": 5, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.772016+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct default variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000004", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:9", "host": "localhost", "remote_addr": "localhost", "res": {"ansible_facts": {"_awxbackup_default": {"api_version": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "namespace": "awx", "name": "awxbackup-2024-12-26-23-30-21", "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}, "timeout": 600, "keep_days": 30}}, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.745566+00:00", "end": "2024-12-26T23:30:21.771826+00:00", "duration": 0.02626, "event_loop": null, "uuid": "2eb3e739-546c-4c4c-9849-d8d398f7f9e0"}} {"uuid": "4e329af7-df1e-0679-2e8f-000000000005", "counter": 6, "stdout": "\r\nTASK [Construct custom variables] **********************************************", "start_line": 5, "end_line": 7, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.775500+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct custom variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:23", "name": "Construct custom variables", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000005"}} {"uuid": "9a143b19-effa-47d9-9f5f-23ae407a10e7", "counter": 7, "stdout": "", "start_line": 7, "end_line": 7, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.776385+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct custom variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:23", "host": "localhost", "uuid": "9a143b19-effa-47d9-9f5f-23ae407a10e7"}} {"uuid": "4267d7de-2022-45c0-9570-7375b87b7c65", "counter": 8, "stdout": "\u001b[0;32mok: [localhost]\u001b[0m", "start_line": 7, "end_line": 8, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.808275+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct custom variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000005", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:23", "host": "localhost", "remote_addr": "localhost", "res": {"ansible_facts": {"_awxbackup_config": {"api_version": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "namespace": "awx", "name": "awxbackup-2024-12-26-23-30-21", "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}, "timeout": "600", "keep_days": "30"}}, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.776317+00:00", "end": "2024-12-26T23:30:21.808076+00:00", "duration": 0.031759, "event_loop": null, "uuid": "4267d7de-2022-45c0-9570-7375b87b7c65"}} {"uuid": "4e329af7-df1e-0679-2e8f-000000000006", "counter": 9, "stdout": "\r\nTASK [Print active variables] **************************************************", "start_line": 8, "end_line": 10, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.811961+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print active variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:34", "name": "Print active variables", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000006"}} {"uuid": "61d8f273-d32b-4df8-b7de-162f7d94fcce", "counter": 10, "stdout": "", "start_line": 10, "end_line": 10, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.812900+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print active variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:34", "host": "localhost", "uuid": "61d8f273-d32b-4df8-b7de-162f7d94fcce"}} {"uuid": "2504d855-b329-4cdf-a7a7-c6cf4c7ef26a", "counter": 11, "stdout": "\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m \"_awxbackup_config\": {\u001b[0m\r\n\u001b[0;32m \"api_version\": \"awx.ansible.com/v1beta1\",\u001b[0m\r\n\u001b[0;32m \"keep_days\": \"30\",\u001b[0m\r\n\u001b[0;32m \"kind\": \"AWXBackup\",\u001b[0m\r\n\u001b[0;32m \"name\": \"awxbackup-2024-12-26-23-30-21\",\u001b[0m\r\n\u001b[0;32m \"namespace\": \"awx\",\u001b[0m\r\n\u001b[0;32m \"spec\": {\u001b[0m\r\n\u001b[0;32m \"backup_pvc\": \"awx-backup-claim\",\u001b[0m\r\n\u001b[0;32m \"clean_backup_on_delete\": true,\u001b[0m\r\n\u001b[0;32m \"deployment_name\": \"awx\"\u001b[0m\r\n\u001b[0;32m },\u001b[0m\r\n\u001b[0;32m \"timeout\": \"600\"\u001b[0m\r\n\u001b[0;32m }\u001b[0m\r\n\u001b[0;32m}\u001b[0m", "start_line": 10, "end_line": 25, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.829088+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print active variables", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000006", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:34", "host": "localhost", "remote_addr": "localhost", "res": {"_awxbackup_config": {"api_version": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "namespace": "awx", "name": "awxbackup-2024-12-26-23-30-21", "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}, "timeout": "600", "keep_days": "30"}, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.812828+00:00", "end": "2024-12-26T23:30:21.828883+00:00", "duration": 0.016055, "event_loop": null, "uuid": "2504d855-b329-4cdf-a7a7-c6cf4c7ef26a"}} {"uuid": "4e329af7-df1e-0679-2e8f-000000000008", "counter": 12, "stdout": "\r\nTASK [Construct new manifest for AWXBackup resource] ***************************", "start_line": 25, "end_line": 27, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.832521+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct new manifest for AWXBackup resource", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:40", "name": "Construct new manifest for AWXBackup resource", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000008"}} {"uuid": "d1bfdf34-a7fb-41ca-8763-3928c22c43f0", "counter": 13, "stdout": "", "start_line": 27, "end_line": 27, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.834375+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct new manifest for AWXBackup resource", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:40", "host": "localhost", "uuid": "d1bfdf34-a7fb-41ca-8763-3928c22c43f0"}} {"uuid": "da871e5b-4dad-4ba0-95dd-a2e8eb5a50e3", "counter": 14, "stdout": "\u001b[0;32mok: [localhost]\u001b[0m", "start_line": 27, "end_line": 28, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.851735+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Construct new manifest for AWXBackup resource", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000008", "task_action": "ansible.builtin.set_fact", "resolved_action": "ansible.builtin.set_fact", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:40", "host": "localhost", "remote_addr": "localhost", "res": {"ansible_facts": {"_awxbackup_manifest": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}}, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.834305+00:00", "end": "2024-12-26T23:30:21.851565+00:00", "duration": 0.01726, "event_loop": null, "uuid": "da871e5b-4dad-4ba0-95dd-a2e8eb5a50e3"}} {"uuid": "4e329af7-df1e-0679-2e8f-000000000009", "counter": 15, "stdout": "\r\nTASK [Print manifest to be created] ********************************************", "start_line": 28, "end_line": 30, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.854961+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "name": "Print manifest to be created", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-000000000009"}} {"uuid": "bc0f1b46-4bc3-49ab-a139-6fef09074dc6", "counter": 16, "stdout": "", "start_line": 30, "end_line": 30, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.855851+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "host": "localhost", "uuid": "bc0f1b46-4bc3-49ab-a139-6fef09074dc6"}} {"uuid": "db16da99-fd61-4c6b-89e9-851014d27abe", "counter": 17, "stdout": "\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m \"_awxbackup_manifest\": {\u001b[0m\r\n\u001b[0;32m \"apiVersion\": \"awx.ansible.com/v1beta1\",\u001b[0m\r\n\u001b[0;32m \"kind\": \"AWXBackup\",\u001b[0m\r\n\u001b[0;32m \"metadata\": {\u001b[0m\r\n\u001b[0;32m \"name\": \"awxbackup-2024-12-26-23-30-21\",\u001b[0m\r\n\u001b[0;32m \"namespace\": \"awx\"\u001b[0m\r\n\u001b[0;32m },\u001b[0m\r\n\u001b[0;32m \"spec\": {\u001b[0m\r\n\u001b[0;32m \"backup_pvc\": \"awx-backup-claim\",\u001b[0m\r\n\u001b[0;32m \"clean_backup_on_delete\": true,\u001b[0m\r\n\u001b[0;32m \"deployment_name\": \"awx\"\u001b[0m\r\n\u001b[0;32m }\u001b[0m\r\n\u001b[0;32m }\u001b[0m\r\n\u001b[0;32m}\u001b[0m", "start_line": 30, "end_line": 45, "runner_ident": "1418", "event": "runner_on_ok", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.870698+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "4e329af7-df1e-0679-2e8f-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "host": "localhost", "remote_addr": "localhost", "res": {"_awxbackup_manifest": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false}, "start": "2024-12-26T23:30:21.855783+00:00", "end": "2024-12-26T23:30:21.870498+00:00", "duration": 0.014715, "event_loop": null, "uuid": "db16da99-fd61-4c6b-89e9-851014d27abe"}} {"uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "counter": 18, "stdout": "\r\nTASK [Create new AWXBackup resource and wait for complete] *********************", "start_line": 45, "end_line": 47, "runner_ident": "1418", "event": "playbook_on_task_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.876625+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "name": "Create new AWXBackup resource and wait for complete", "is_conditional": false, "uuid": "4e329af7-df1e-0679-2e8f-00000000000b"}} {"uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741", "counter": 19, "stdout": "", "start_line": 47, "end_line": 47, "runner_ident": "1418", "event": "runner_on_start", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:30:21.877557+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "uuid": "3b1614c5-8f23-4fac-b92c-828e4e26a741"}} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"uuid": "e8f324c4-941c-4d09-bcdd-87ee641e1c49", "counter": 20, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": true, \"duration\": 600, \"method\": \"create\", \"msg\": \"\\\"AWXBackup\\\" \\\"awxbackup-2024-12-26-23-30-21\\\": Timed out waiting on resource\", \"result\": {\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWXBackup\", \"metadata\": {\"creationTimestamp\": \"2024-12-26T23:30:22Z\", \"generation\": 1, \"managedFields\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:spec\": {\".\": {}, \"f:backup_pvc\": {}, \"f:clean_backup_on_delete\": {}, \"f:deployment_name\": {}, \"f:image_pull_policy\": {}, \"f:no_log\": {}, \"f:set_self_labels\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2024-12-26T23:30:22Z\"}], \"name\": \"awxbackup-2024-12-26-23-30-21\", \"namespace\": \"awx\", \"resourceVersion\": \"9000995\", \"uid\": \"03f81604-198d-44de-8e0e-0289dfa77320\"}, \"spec\": {\"backup_pvc\": \"awx-backup-claim\", \"clean_backup_on_delete\": true, \"deployment_name\": \"awx\", \"image_pull_policy\": \"IfNotPresent\", \"no_log\": true, \"set_self_labels\": true}}}\u001b[0m", "start_line": 47, "end_line": 48, "runner_ident": "1418", "event": "runner_on_failed", "job_id": 1418, "pid": 17, "created": "2024-12-26T23:40:23.475258+00:00", "parent_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "f2d304fd-8016-4325-957e-38b1a57b17bf", "play": "Backing up AWX using AWX Operator", "play_uuid": "4e329af7-df1e-0679-2e8f-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "4e329af7-df1e-0679-2e8f-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"creationTimestamp": "2024-12-26T23:30:22Z", "generation": 1, "managedFields": [{"apiVersion": "awx.ansible.com/v1beta1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:backup_pvc": {}, "f:clean_backup_on_delete": {}, "f:deployment_name": {}, "f:image_pull_policy": {}, "f:no_log": {}, "f:set_self_labels": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-12-26T23:30:22Z"}], "name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx", "resourceVersion": "9000995", "uid": "03f81604-198d-44de-8e0e-0289dfa77320"}, "spec": {"backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true, "deployment_name": "awx", "image_pull_policy": "IfNotPresent", "no_log": true, "set_self_labels": true}}, "method": "create", "duration": 600, "msg": "\"AWXBackup\" \"awxbackup-2024-12-26-23-30-21\": Timed out waiting on resource", "invocation": {"module_args": {"state": "present", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "wait_condition": {"reason": "Successful", "status": true, "type": "Running"}, "wait_timeout": 600, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-26-23-30-21", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, ``` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [December 27, 2024, 9:51am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/3 "2024-12-27T09:51:24Z")
According to the logs, timeout has been exceeded, so you should increasing the timeout. ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 27, 2024, 10:37am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/4 "2024-12-27T10:37:37Z")
Thank you very much @kurokobo I have increased timeout from 600 to 1200 but job still failing Logs attached `````````````auto project/backup.yml:50", "host": "localhost", "uuid": "bfe197a2-2450-40af-a679-b6acc0e68446"}} {"uuid": "3625337e-99e2-4615-a17c-8bd2b05f6e94", "counter": 17, "stdout": "\u001b[0;32mok: [localhost] => {\u001b[0m\r\n\u001b[0;32m \"_awxbackup_manifest\": {\u001b[0m\r\n\u001b[0;32m \"apiVersion\": \"awx.ansible.com/v1beta1\",\u001b[0m\r\n\u001b[0;32m \"kind\": \"AWXBackup\",\u001b[0m\r\n\u001b[0;32m \"metadata\": {\u001b[0m\r\n\u001b[0;32m \"name\": \"awxbackup-2024-12-27-10-14-37\",\u001b[0m\r\n\u001b[0;32m \"namespace\": \"awx\"\u001b[0m\r\n\u001b[0;32m },\u001b[0m\r\n\u001b[0;32m \"spec\": {\u001b[0m\r\n\u001b[0;32m \"backup_pvc\": \"awx-backup-claim\",\u001b[0m\r\n\u001b[0;32m \"clean_backup_on_delete\": true,\u001b[0m\r\n\u001b[0;32m \"deployment_name\": \"awx\"\u001b[0m\r\n\u001b[0;32m }\u001b[0m\r\n\u001b[0;32m }\u001b[0m\r\n\u001b[0;32m}\u001b[0m", "start_line": 30, "end_line": 45, "runner_ident": "1423", "event": "runner_on_ok", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:14:37.918528+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000009", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Print manifest to be created", "task_uuid": "2297092f-0e09-2731-a1ee-000000000009", "task_action": "ansible.builtin.debug", "resolved_action": "ansible.builtin.debug", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:50", "host": "localhost", "remote_addr": "localhost", "res": {"_awxbackup_manifest": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "_ansible_verbose_always": true, "_ansible_no_log": false, "changed": false}, "start": "2024-12-27T10:14:37.902536+00:00", "end": "2024-12-27T10:14:37.918314+00:00", "duration": 0.015778, "event_loop": null, "uuid": "3625337e-99e2-4615-a17c-8bd2b05f6e94"}} {"uuid": "2297092f-0e09-2731-a1ee-00000000000b", "counter": 18, "stdout": "\r\nTASK [Create new AWXBackup resource and wait for complete] *********************", "start_line": 45, "end_line": 47, "runner_ident": "1423", "event": "playbook_on_task_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:14:37.924453+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "name": "Create new AWXBackup resource and wait for complete", "is_conditional": false, "uuid": "2297092f-0e09-2731-a1ee-00000000000b"}} {"uuid": "562aba13-93a6-43e0-a59b-da47afa1edb1", "counter": 19, "stdout": "", "start_line": 47, "end_line": 47, "runner_ident": "1423", "event": "runner_on_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:14:37.925393+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "uuid": "562aba13-93a6-43e0-a59b-da47afa1edb1"}} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"event": "keepalive", "counter": 0, "uuid": 0} {"uuid": "f5abd608-c59b-4588-adaf-6c11ad9f08ab", "counter": 20, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": true, \"duration\": 1201, \"method\": \"create\", \"msg\": \"\\\"AWXBackup\\\" \\\"awxbackup-2024-12-27-10-14-37\\\": Timed out waiting on resource\", \"result\": {\"apiVersion\": \"awx.ansible.com/v1beta1\", \"kind\": \"AWXBackup\", \"metadata\": {\"creationTimestamp\": \"2024-12-27T10:14:38Z\", \"generation\": 1, \"managedFields\": [{\"apiVersion\": \"awx.ansible.com/v1beta1\", \"fieldsType\": \"FieldsV1\", \"fieldsV1\": {\"f:spec\": {\".\": {}, \"f:backup_pvc\": {}, \"f:clean_backup_on_delete\": {}, \"f:deployment_name\": {}, \"f:image_pull_policy\": {}, \"f:no_log\": {}, \"f:set_self_labels\": {}}}, \"manager\": \"OpenAPI-Generator\", \"operation\": \"Update\", \"time\": \"2024-12-27T10:14:38Z\"}], \"name\": \"awxbackup-2024-12-27-10-14-37\", \"namespace\": \"awx\", \"resourceVersion\": \"9015486\", \"uid\": \"7b43c40b-2029-47da-a91a-9ff567bbb2ab\"}, \"spec\": {\"backup_pvc\": \"awx-backup-claim\", \"clean_backup_on_delete\": true, \"deployment_name\": \"awx\", \"image_pull_policy\": \"IfNotPresent\", \"no_log\": true, \"set_self_labels\": true}}}\u001b[0m", "start_line": 47, "end_line": 48, "runner_ident": "1423", "event": "runner_on_failed", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:40.416567+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Create new AWXBackup resource and wait for complete", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000b", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:56", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"creationTimestamp": "2024-12-27T10:14:38Z", "generation": 1, "managedFields": [{"apiVersion": "awx.ansible.com/v1beta1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {".": {}, "f:backup_pvc": {}, "f:clean_backup_on_delete": {}, "f:deployment_name": {}, "f:image_pull_policy": {}, "f:no_log": {}, "f:set_self_labels": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2024-12-27T10:14:38Z"}], "name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx", "resourceVersion": "9015486", "uid": "7b43c40b-2029-47da-a91a-9ff567bbb2ab"}, "spec": {"backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true, "deployment_name": "awx", "image_pull_policy": "IfNotPresent", "no_log": true, "set_self_labels": true}}, "method": "create", "duration": 1201, "msg": "\"AWXBackup\" \"awxbackup-2024-12-27-10-14-37\": Timed out waiting on resource", "invocation": {"module_args": {"state": "present", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "wait_condition": {"reason": "Successful", "status": true, "type": "Running"}, "wait_timeout": 1200, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "merge_type": null, "validate": null, "template": null, "delete_options": null, "label_selectors": null, "generate_name": null, "server_side_apply": null, "hidden_fields": null}}, "_ansible_no_log": false}, "start": "2024-12-27T10:14:37.925327+00:00", "end": "2024-12-27T10:34:40.416426+00:00", "duration": 1202.491099, "ignore_errors": null, "event_loop": null, "uuid": "f5abd608-c59b-4588-adaf-6c11ad9f08ab"}} {"uuid": "2297092f-0e09-2731-a1ee-00000000000d", "counter": 21, "stdout": "\r\nTASK [Clean up failed AWXBackup resource] **************************************", "start_line": 48, "end_line": 50, "runner_ident": "1423", "event": "playbook_on_task_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:40.420119+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Clean up failed AWXBackup resource", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:80", "name": "Clean up failed AWXBackup resource", "is_conditional": false, "uuid": "2297092f-0e09-2731-a1ee-00000000000d"}} {"uuid": "6123c395-e002-4d8a-9492-0ba6d147a4f1", "counter": 22, "stdout": "", "start_line": 50, "end_line": 50, "runner_ident": "1423", "event": "runner_on_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:40.420957+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Clean up failed AWXBackup resource", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:80", "host": "localhost", "uuid": "6123c395-e002-4d8a-9492-0ba6d147a4f1"}} {"uuid": "098e9389-c91f-4665-972e-8165a9f39c1a", "counter": 23, "stdout": "\u001b[0;33mchanged: [localhost]\u001b[0m", "start_line": 50, "end_line": 51, "runner_ident": "1423", "event": "runner_on_ok", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.033150+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Clean up failed AWXBackup resource", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000d", "task_action": "kubernetes.core.k8s", "resolved_action": "kubernetes.core.k8s", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:80", "host": "localhost", "remote_addr": "localhost", "res": {"changed": true, "result": {"kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Success", "details": {"name": "awxbackup-2024-12-27-10-14-37", "group": "awx.ansible.com", "kind": "awxbackups", "uid": "7b43c40b-2029-47da-a91a-9ff567bbb2ab"}}, "method": "delete", "duration": 0, "invocation": {"module_args": {"state": "absent", "definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "wait": true, "resource_definition": {"apiVersion": "awx.ansible.com/v1beta1", "kind": "AWXBackup", "metadata": {"name": "awxbackup-2024-12-27-10-14-37", "namespace": "awx"}, "spec": {"deployment_name": "awx", "backup_pvc": "awx-backup-claim", "clean_backup_on_delete": true}}, "api_version": "v1", "wait_sleep": 5, "wait_timeout": 120, "append_hash": false, "apply": false, "continue_on_error": false, "force": false, "delete_all": false, "kind": null, "name": null, "namespace": null, "src": null, "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "wait_condition": null, "merge_type": null, "validate": null, "template": null, "delete_options": null, "label_selectors": null, "generate_name": null, "server_side_apply": null, "hidden_fields": null}}, "_ansible_no_log": false}, "start": "2024-12-27T10:34:40.420889+00:00", "end": "2024-12-27T10:34:41.032956+00:00", "duration": 0.612067, "event_loop": null, "uuid": "098e9389-c91f-4665-972e-8165a9f39c1a"}} {"uuid": "2297092f-0e09-2731-a1ee-00000000000e", "counter": 24, "stdout": "\r\nTASK [Force to fail if the backup was failed] **********************************", "start_line": 51, "end_line": 53, "runner_ident": "1423", "event": "playbook_on_task_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.037726+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-000000000002", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Force to fail if the backup was failed", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "task_action": "ansible.builtin.fail", "resolved_action": "ansible.builtin.fail", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:86", "name": "Force to fail if the backup was failed", "is_conditional": false, "uuid": "2297092f-0e09-2731-a1ee-00000000000e"}} {"uuid": "657d800a-8400-4add-87a2-f4719ab9d0f7", "counter": 25, "stdout": "", "start_line": 53, "end_line": 53, "runner_ident": "1423", "event": "runner_on_start", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.038740+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Force to fail if the backup was failed", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "task_action": "ansible.builtin.fail", "resolved_action": "ansible.builtin.fail", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:86", "host": "localhost", "uuid": "657d800a-8400-4add-87a2-f4719ab9d0f7"}} {"uuid": "40bcd7d2-e029-4d0c-a3dc-834ad32a87f5", "counter": 26, "stdout": "\u001b[0;31mfatal: [localhost]: FAILED! => {\"changed\": false, \"msg\": \"Failed as requested from task\"}\u001b[0m", "start_line": 53, "end_line": 54, "runner_ident": "1423", "event": "runner_on_failed", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.051447+00:00", "parent_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "play": "Backing up AWX using AWX Operator", "play_uuid": "2297092f-0e09-2731-a1ee-000000000002", "play_pattern": "localhost", "task": "Force to fail if the backup was failed", "task_uuid": "2297092f-0e09-2731-a1ee-00000000000e", "task_action": "ansible.builtin.fail", "resolved_action": "ansible.builtin.fail", "task_args": "", "task_path": "/runner/project/backup/ansible/project/backup.yml:86", "host": "localhost", "remote_addr": "localhost", "res": {"msg": "Failed as requested from task", "_ansible_no_log": false, "changed": false}, "start": "2024-12-27T10:34:41.038667+00:00", "end": "2024-12-27T10:34:41.051282+00:00", "duration": 0.012615, "ignore_errors": null, "event_loop": null, "uuid": "40bcd7d2-e029-4d0c-a3dc-834ad32a87f5"}} {"uuid": "598d398e-8fd0-4679-9e15-5725b5dceefa", "counter": 27, "stdout": "\r\nPLAY RECAP ********************************************************************* \r\n\u001b[0;31mlocalhost\u001b[0m : \u001b[0;32mok=6 \u001b[0m \u001b[0;33mchanged=1 \u001b[0m unreachable=0 \u001b[0;31mfailed=1 \u001b[0m skipped=0 \u001b[0;32mrescued=1 \u001b[0m ignored=0 ", "start_line": 54, "end_line": 58, "runner_ident": "1423", "event": "playbook_on_stats", "job_id": 1423, "pid": 17, "created": "2024-12-27T10:34:41.052998+00:00", "parent_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "event_data": {"playbook": "backup/ansible/project/backup.yml", "playbook_uuid": "269a9a84-02e7-47ff-843f-950fbe730fc6", "changed": {"localhost": 1}, "dark": {}, "failures": {"localhost": 1}, "ignored": {}, "ok": {"localhost": 6}, "processed": {"localhost": 1}, "rescued": {"localhost": 1}, "skipped": {}, "artifact_data": {}, "uuid": "598d398e-8fd0-4679-9e15-5725b5dceefa"}} {"status": "failed", "runner_ident": "1423"} {"zipfile": 1942} UEsDBBQAAAAIANJRm1l9zrJ6EQYAAD0SAAAHAAAAY29tbWFuZL1YW1OjSBT+K5Yv7lZNIDGOZnxaQjqRkQAFnThZa6qrQ1rDDAEGiJqx3N++p5smQm7ruFvrQ+jLufS5fKdP+3zsx4sFjWbHl0e3xzTKgmnIGklIV9M4/n784ei4seS/dLYIIjEN+K+aLqOIpWoQPbAoj9OVOo+zPBMEjP/+UVKw6EFlT3lKH2gq9qfU/75MVKlKTdL4G/NztVhWVovw+CtQ+Y/8RGs9korzg0DYeT6+HnWRayGMPOIhd2zoiDi2i8kVxo7Hec/O2px+D12FQrv5QmwHuRq2XaLbFnZt00QuGWqWNuBfhF1Df2Uvv1e2J8S0mspZW2k1z5XWWZML5BuWNkR8ky7zeEHzII4a3+Jpo3V22m6cP+R+p9RcO3ynSbDucL7cTy5VtRB93lLaHy87QrZz06s45j3HLxSB7VzV2hcd6QzHcEjXRdo18SYeRkPiaPo1yBEubf0H+lwb29LATRdsxqcjvVl4smJyJaZCuhS+6bem0rosY2x5RtdExB1ZFpzxGiFHM40x1wkm9IR17eaBmBCt13Nfoy1Csov8UOze47lNeTLNLjuV5N113LoP35vidUR1/iVgNnPNuzLHJl9oHgjqVsbso6sHCGK/L73quK1F8t15/Sth2nLua5x2FKv6aaVZGz44kPh7vVqvgY6GrwTIllmqhrFPQzWbBtFlZb6evm6IQTGdFtfD3nTcAfv3F5F6qCuV19SJjieOqBe6MsL9RqeK/q7mIaKZpn1D9JGH7SEBtUVpw+mSHSbFSKvQ92mY7WPwDGtgImxbBTnRHOPtLO9RMwI3vckUIdx2B+DBoWH9k4I19RANIYveTA5xHb7pJI7mIgsfJCUu8jDpGyZGriD87US2DmRKM6akLMvJXRDmLM3kJKUL9hin35V8lTC5R3h/waKZgmGtL5a6xcrJh6O3S7wLWDgjIbRGy2Qtss8XTbFWkbnFWQj0GE39eXGCX1Idp7OKGTafdVelvt+3XNedOJoHqadjAxKkb2oD4b3nl32U3gjCJhJpTXx7EmQkWyYsXWZw2q9brLqmXyEZRRF1w/NA3f6AArl+DSE1NYx6mxy3J6CG+8Sf0+hejB4C9rhLLzQovHD0UF8bmVhyv7LNWMhyMWJPzF8WQymczuaxLwYJdJUPxVYyo/lBhUUJGBvoRporervHJ4UmgXLPoC8J/EwBoI9BwBY7GmvmSOOR8IgxsGwX8ZrXNw39QPIXpbAncA0uGxgedid7Ygio0yzjT6GCDO0eEtfqggYRZMo9jYKfogXd4nuNwAaXw9JFkGW7eACQ9sjVCxOMAWccmaj0B2RtvEx9RsAxu1jXWOYTKO5gIG+VJbQTWgSQf0kW/BSTuziFDpqPBAbWAzJd8XEmEMVHHO/8y98iEkYineJllJNZkFEAmsBmFJMcuklfBj0MFoGQ/0DDQGTCNp5EFMq8K5v7Zz/MFEIiwCghL0d6yig8hnbzOi4S7GhPCAWR7Lk5asccInuTQ5TPesgwo4sqXR+kSIRiY4jsEV63WmsSu4ijabu7NPHWA1rlSQFbuG123gCGNQY82O6EjCzAhgcR7WuGufvwQKANjWsboq/bbm+tYuOyg+7AcYULSE/DGukZ4oBqvoCX4+MT4c8oEsR0QVf3hHN8trvEEBr5Fl95PZZcF22Oa39G4BYXjQ2e9aLtv/vUupg2T5vtKZ22zmfT87vzj+0LOr04Y+z8bDZrdZof6Wmr9oyA9mTCMxlud2RpXWnt2oCh9gVAz6uiW6T2RZP/lbaVPd08z5MMurb702ajfHv7T4niz38oLFCrGj3vquyTCG/W1i6RL2TZVklqnXdTelFwOLVXJU3Zj2WQsgW83DPix2EIL2sAenb5l6qUz/LqctH0zWnK1B3bNbeIdmfdS+7Sl8Yhq2kqFnbokBss9+tLh+z8f8zUoS/pwrOYOOZoYBSXV6VRDoOpmqzyeRy1lU9qFuSskcA9DSUtKyUTeUYoSvx/LQTYQn61q+WgFnrcA/CutcpKSyRvmVOGZxe3ag0wUg9N8+CO+nmmlvCQD2F7aOAiU+sILLctcyLRvEX18vI3UEsDBBQAAAAIAFRUm1l3o4ueCAAAAAYAAAAGAAAAc3RhdHVzS0vMzElNAQBQSwMEFAAAAAgAVFSbWQ2+1RoDAAAAAQAAAAIAAAByYzMCAFBLAwQUAAAAAABUVJtZAAAAAAAAAAAAAAAACwAAAGpvYl9ldmVudHMvUEsBAhQDFAAAAAgA0lGbWX3OsnoRBgAAPRIAAAcAAAAAAAAAAAAAAICBAAAAAGNvbW1hbmRQSwECFAMUAAAACABUVJtZd6OLnggAAAAGAAAABgAAAAAAAAAAAAAAgIE2BgAAc3RhdHVzUEsBAhQDFAAAAAgAVFSbWQ2+1RoDAAAAAQAAAAIAAAAAAAAAAAAAAICBYgYAAHJjUEsBAhQDFAAAAAAAVFSbWQAAAAAAAAAAAAAAAAsAAAAAAAAAAAAQAMBBhQYAAGpvYl9ldmVudHMvUEsFBgAAAAAEAAQA0gAAAK4GAAAAAA=={"eof": true} ```````````` ````````````` ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 27, 2024, 10:46am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/5 "2024-12-27T10:46:43Z")
Adding additional details ```````````````````````auto [root@server ~]# kubectl get pods -n awx NAME READY STATUS RESTARTS AGE automation-job-1402-r5smn 0/1 Error 0 24h automation-job-1406-vw5vs 0/1 Completed 0 23h automation-job-1410-qhbzm 0/1 Error 0 23h automation-job-1418-rzjrj 0/1 Error 0 11h automation-job-1421-gvklc 0/1 Error 0 45m automation-job-1423-6vtc8 0/1 Error 0 28m awx-operator-controller-manager-745b55d94b-97sdw 1/2 CrashLoopBackOff 4639 (2m29s ago) 46d awx-postgres-15-0 1/1 Running 12 (18d ago) 144d awx-task-55d8c9dd9f-qzxr5 4/4 Running 4 (18d ago) 21d awx-web-59498d8589-98bph 3/3 Running 24 (18d ago) 121d [root@server ~]# kubectl get awxbackup -n awx NAME AGE awxbackup-2024-11-29-12-25-22 27d awxbackup-2024-11-29-12-30-54 27d awxbackup-2024-11-29-12-38-30 27d awxbackup-2024-12-02-04-30-38 25d awxbackup-2024-12-02-23-30-38 24d awxbackup-2024-12-03-23-30-38 23d awxbackup-2024-12-04-23-30-46 22d awxbackup-2024-12-05-23-30-26 21d awxbackup-2024-12-06-23-30-28 20d awxbackup-2024-12-07-23-30-27 19d awxbackup-2024-12-08-23-30-26 18d awxbackup-2024-12-09-23-30-22 17d awxbackup-2024-12-10-23-30-22 16d awxbackup-2024-12-11-23-30-21 15d awxbackup-2024-12-12-11-33-29 14d awxbackup-2024-12-12-23-30-21 14d awxbackup-2024-12-16-23-30-22 10d awxbackup-2024-12-17-23-30-22 9d awxbackup-2024-12-20-23-30-20 6d11h awxbackup-2024-12-23-23-30-21 3d11h awxbackup-2024-12-26-10-29-32 24h awxbackup-2024-12-26-11-05-04 23h awxbackup-2024-12-26-11-06-20 23h awxbackup-2024-12-26-11-10-30 23h awxbackup-2024-12-27-09-57-33 45m [root@server ~]# ls -l /data/backup/ total 15412 drwxr-xr-x 2 26 26 59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800 drwxr-xr-x 2 26 26 59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843 drwxr-xr-x 2 26 26 59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859 drwxr-xr-x 2 26 26 59 Dec 1 20:31 tower-openshift-backup-2024-12-02-043052 drwxr-xr-x 2 26 26 59 Dec 2 15:31 tower-openshift-backup-2024-12-02-233051 -rw-r--r-- 1 root root 15779840 Dec 3 03:15 tower-openshift-backup-2024-12-02-233051.tar drwxr-xr-x 2 26 26 59 Dec 3 15:31 tower-openshift-backup-2024-12-03-233052 drwxr-xr-x 2 26 26 59 Dec 4 15:31 tower-openshift-backup-2024-12-04-233059 drwxr-xr-x 2 26 26 59 Dec 5 15:30 tower-openshift-backup-2024-12-05-233039 drwxr-xr-x 2 26 26 59 Dec 6 15:30 tower-openshift-backup-2024-12-06-233041 drwxr-xr-x 2 26 26 59 Dec 7 15:30 tower-openshift-backup-2024-12-07-233039 drwxr-xr-x 2 26 26 59 Dec 8 15:30 tower-openshift-backup-2024-12-08-233039 [root@server ~]# [root@server ~]# kubectl get events -A NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE awx 48m Normal Scheduled pod/automation-job-1421-gvklc Successfully assigned awx/automation-job-1421-gvklc to server awx 48m Normal Pulled pod/automation-job-1421-gvklc Container image "quay.io/ansible/awx-ee:24.6.1" already present on machine awx 48m Normal Created pod/automation-job-1421-gvklc Created container worker awx 48m Normal Started pod/automation-job-1421-gvklc Started container worker awx 31m Normal Scheduled pod/automation-job-1423-6vtc8 Successfully assigned awx/automation-job-1423-6vtc8 to server awx 31m Normal Pulled pod/automation-job-1423-6vtc8 Container image "quay.io/ansible/awx-ee:24.6.1" already present on machine awx 31m Normal Created pod/automation-job-1423-6vtc8 Created container worker awx 31m Normal Started pod/automation-job-1423-6vtc8 Started container worker awx 3m41s Warning BackOff pod/awx-operator-controller-manager-745b55d94b-97sdw Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d) awx 56m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_26ba2df6-f025-4717-9b4f-b1d834cf4217 became leader awx 50m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_35152e7c-d324-4d70-b263-f22c93cba88e became leader awx 45m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_acbf4459-5d5d-46af-90e1-75206676d2d4 became leader awx 39m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_b3e38cf8-b1e0-426e-a165-343d125e8146 became leader awx 33m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_c20e0124-01db-4539-b62d-8ed72b6efd31 became leader awx 28m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_fcc73ab1-a118-484f-9509-f33f7de1903d became leader awx 22m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_5821073f-d6fa-4f9a-beb6-38e8c7b279c8 became leader awx 16m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_d7eea5f4-e824-49ad-a9cd-ed815959fcc6 became leader awx 11m Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_32d9ebcc-b497-4a5b-86f2-bddd82485090 became leader awx 5m40s Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_ddf5fd04-d806-4862-a3b9-a03e756a34fd became leader awx 6s Normal LeaderElection lease/awx-operator awx-operator-controller-manager-745b55d94b-97sdw_bed743ee-fbaf-4734-a56f-c31bb7eef1b5 became leader default 3m38s Warning ProvisioningFailed persistentvolumeclaim/awx-projects-claim storageclass.storage.k8s.io "awx-projects-volume" not found default 28m Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 6804366950 bytes, but only found 0 bytes eligible to free. default 23m Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 6805468774 bytes, but only found 0 bytes eligible to free. default 23m Warning ImageGCFailed node/server Failed to garbage collect required amount of images. Attempted to free 6805468774 bytes, but only found 0 bytes eligible to free. default 18m Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 6805063270 bytes, but only found 0 bytes eligible to free. default 18m Warning ImageGCFailed node/server Failed to garbage collect required amount of images. Attempted to free 6805063270 bytes, but only found 0 bytes eligible to free. default 13m Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 6805378662 bytes, but only found 0 bytes eligible to free. default 13m Warning ImageGCFailed node/server Failed to garbage collect required amount of images. Attempted to free 6805378662 bytes, but only found 0 bytes eligible to free. default 8m51s Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 6803998310 bytes, but only found 0 bytes eligible to free. default 8m51s Warning ImageGCFailed node/server Failed to garbage collect required amount of images. Attempted to free 6803998310 bytes, but only found 0 bytes eligible to free. default 3m51s Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 6803936870 bytes, but only found 0 bytes eligible to free. default 3m51s Warning ImageGCFailed node/server Failed to garbage collect required amount of images. Attempted to free 6803936870 bytes, but only found 0 bytes eligible to free. [root@server ~]# [root@server ~]# df -h / Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup00-RootVol00 70G 52G 18G 75% / [root@server ~]# free -h total used free shared buff/cache available Mem: 31Gi 6.2Gi 1.4Gi 684Mi 24Gi 24Gi Swap: 1.0Gi 0.0Ki 1.0Gi [root@server ~]# vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 256 1445372 2964 25803072 0 0 41 14 0 1 1 1 98 0 0 [root@server ~]# `````````````````````` ``````````````````````` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [December 27, 2024, 12:01pm UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/6 "2024-12-27T12:01:05Z")
> ```bash > awx-operator-controller-manager-745b55d94b-97sdw 1/2 CrashLoopBackOff 4639 (2m29s ago) 46d > > ``` Thanks for updating, this indicates that your operator keeps restarting. You should investigate your operator to identify the reason of the restarting. It is not possible to determine from the current log whether the Operator is unable to back up because it keeps restarting, or whether the backup process is causing it to restart. ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 27, 2024, 12:08pm UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/7 "2024-12-27T12:08:13Z")
Thank you @kurokobo, I will check, also i have added operator log ```auto [root@server ~]# kubectl get pods -n awx NAME READY STATUS RESTARTS AGE automation-job-1402-r5smn 0/1 Error 0 25h automation-job-1406-vw5vs 0/1 Completed 0 25h automation-job-1410-qhbzm 0/1 Error 0 25h automation-job-1418-rzjrj 0/1 Error 0 12h automation-job-1421-gvklc 0/1 Error 0 127m automation-job-1423-6vtc8 0/1 Error 0 110m automation-job-1425-xs87g 0/1 Error 0 28m automation-job-1427-hrdf9 0/1 Error 0 26m automation-job-1429-4nscb 0/1 Error 0 18m automation-job-1431-wlkqd 0/1 Error 0 7m38s awx-operator-controller-manager-745b55d94b-97sdw 2/2 Running 4654 (5m31s ago) 46d awx-postgres-15-0 1/1 Running 12 (18d ago) 144d awx-task-55d8c9dd9f-qzxr5 4/4 Running 4 (18d ago) 21d awx-web-59498d8589-98bph 3/3 Running 24 (18d ago) 121d [root@server ~]# kubectl get pods awx-operator-controller-manager-745b55d94b-97sdw -n awx NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-745b55d94b-97sdw 2/2 Running 4654 (5m44s ago) 46d [root@server ~]# kubectl logs awx-operator-controller-manager-745b55d94b-97sdw -n awx {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx"]} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} {"level":"info","ts":"2024-12-27T12:04:47Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2024-12-27T12:04:47Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false} I1227 12:04:47.228843 7 leaderelection.go:250] attempting to acquire leader lease awx/awx-operator... I1227 12:05:05.478317 7 leaderelection.go:260] successfully acquired lease awx/awx-operator {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awxrestore-controller"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awxmeshingress-controller"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awxbackup-controller"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting Controller","controller":"awx-controller"} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awx-controller","worker count":8} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awxmeshingress-controller","worker count":8} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":8} {"level":"info","ts":"2024-12-27T12:05:05Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":8} {"level":"info","ts":"2024-12-27T12:05:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[0].message\""} {"level":"info","ts":"2024-12-27T12:05:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[1].message\""} {"level":"info","ts":"2024-12-27T12:05:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[2].message\""} {"level":"info","ts":"2024-12-27T12:05:13Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"436506494050674087","EventData.Name":"Verify imagePullSecrets"} --------------------------- Ansible Task StdOut ------------------------------- TASK [Verify imagePullSecrets] ************************************************* task path: /opt/ansible/playbooks/awx.yml:10 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"2600480675036880980","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8813032395009318914","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8861251273508755859","EventData.TaskArgs":""} {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2600480675036880980","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8630151664447508170","EventData.TaskArgs":""} {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8813032395009318914","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8861251273508755859","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"2485108430253905831","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"2061232962065953104","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"7175231377219730276","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-11-29-12-30-54","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8501857911985331229","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8630151664447508170","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-11-29-12-30-54","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8501857911985331229","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2061232962065953104","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"7175231377219730276","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2485108430253905831","EventData.Name":"backup : Run creation tasks"} {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2600480675036880980","EventData.Name":"backup : Run finalizer tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run finalizer tasks] ******************************************** task path: /opt/ansible/roles/backup/tasks/main.yml:6 ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run finalizer tasks] ******************************************** task path: /opt/ansible/roles/backup/tasks/main.yml:6 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8861251273508755859","EventData.Name":"backup : Run finalizer tasks"} {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"7175231377219730276","EventData.Name":"backup : Run finalizer tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run finalizer tasks] ******************************************** task path: /opt/ansible/roles/backup/tasks/main.yml:6 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2061232962065953104","EventData.Name":"backup : Run finalizer tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run finalizer tasks] ******************************************** task path: /opt/ansible/roles/backup/tasks/main.yml:6 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8813032395009318914","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2485108430253905831","EventData.Name":"backup : Run finalizer tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run finalizer tasks] ******************************************** task path: /opt/ansible/roles/backup/tasks/main.yml:6 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-11-29-12-30-54","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8501857911985331229","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-10-23-30-22","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2600480675036880980","EventData.Name":"backup : Look up details for this backup object"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Look up details for this backup object] ************************* task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-11-33-29","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8861251273508755859","EventData.Name":"backup : Look up details for this backup object"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Look up details for this backup object] ************************* task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8630151664447508170","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-20-23-30-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"7175231377219730276","EventData.Name":"backup : Look up details for this backup object"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Look up details for this backup object] ************************* task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-27-09-57-33","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2061232962065953104","EventData.Name":"backup : Look up details for this backup object"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Look up details for this backup object] ************************* task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-12-23-30-21","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"2485108430253905831","EventData.Name":"backup : Look up details for this backup object"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Look up details for this backup object] ************************* task path: /opt/ansible/roles/backup/tasks/finalizer.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2024-12-27T12:05:18Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/awx/secrets/redhat-operators-pull-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"secrets","Subresource":"","Name":"redhat-operators-pull-secret","Parts":["secrets","redhat-operators-pull-secret"]}} {"level":"info","ts":"2024-12-27T12:05:18Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"436506494050674087","EventData.Name":"Create imagePullSecret"} --------------------------- Ansible Task StdOut ------------------------------- TASK [Create imagePullSecret] ************************************************** task path: /opt/ansible/playbooks/awx.yml:17 ------------------------------------------------------------------------------- [root@server ~]# ``` ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 28, 2024, 9:01am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/8 "2024-12-28T09:01:57Z")
Hello @kurokobo I have tried to restart K3s service but still getting same operator error. Other job templates are successfully running, only backup job is failing. Is there any way to troubleshoot further, Please suggest me. ```auto awx-operator-controller-manager-745b55d94b-97sdw 1/2 OOMKilled 4879 (119s ago) 47d [root@server ~]# kubectl get pods -n awx NAME READY STATUS RESTARTS AGE automation-job-1402-r5smn 0/1 Error 0 46h automation-job-1406-vw5vs 0/1 Completed 0 45h automation-job-1410-qhbzm 0/1 Error 0 45h automation-job-1418-rzjrj 0/1 Error 0 33h automation-job-1421-gvklc 0/1 Error 0 22h automation-job-1423-6vtc8 0/1 Error 0 22h automation-job-1425-xs87g 0/1 Error 0 21h automation-job-1427-hrdf9 0/1 Error 0 21h automation-job-1429-4nscb 0/1 Error 0 21h automation-job-1431-wlkqd 0/1 Error 0 20h automation-job-1433-79bnp 0/1 Error 0 9h automation-job-1435-mjzfh 0/1 Completed 0 8m22s automation-job-1437-kmjtw 0/1 Completed 0 2m54s awx-operator-controller-manager-745b55d94b-97sdw 1/2 OOMKilled 4879 (119s ago) 47d awx-postgres-15-0 1/1 Running 12 (19d ago) 144d awx-task-55d8c9dd9f-qzxr5 4/4 Running 4 (19d ago) 22d awx-web-59498d8589-98bph 3/3 Running 24 (19d ago) 121d [root@server ~]# kubectl get events LAST SEEN TYPE REASON OBJECT MESSAGE 10m Warning ProvisioningFailed persistentvolumeclaim/awx-projects-claim storageclass.storage.k8s.io "awx-projects-volume" not found 12s Warning ProvisioningFailed persistentvolumeclaim/awx-projects-claim storageclass.storage.k8s.io "awx-projects-volume" not found 10m Warning FreeDiskSpaceFailed node/server (combined from similar events): Failed to garbage collect required amount of images. Attempted to free 9734415974 bytes, but only found 0 bytes eligible to free. 50m Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 9733240422 bytes, but only found 0 bytes eligible to free. 5m53s Normal Starting node/server Starting kubelet. 5m53s Warning InvalidDiskCapacity node/server invalid capacity 0 on image filesystem 5m53s Normal NodeHasSufficientMemory node/server Node server status is now: NodeHasSufficientMemory 5m53s Normal NodeHasNoDiskPressure node/server Node server status is now: NodeHasNoDiskPressure 5m53s Normal NodeHasSufficientPID node/server Node server status is now: NodeHasSufficientPID 5m53s Normal NodeNotReady node/server Node server status is now: NodeNotReady 5m53s Normal NodeAllocatableEnforced node/server Updated Node Allocatable limit across pods 5m52s Normal Starting node/server 5m49s Normal NodePasswordValidationComplete node/server Deferred node password secret validation complete 5m43s Normal NodeReady node/server Node server status is now: NodeReady 5m42s Normal RegisteredNode node/server Node server event: Registered Node server in Controller 53s Warning FreeDiskSpaceFailed node/server Failed to garbage collect required amount of images. Attempted to free 11668506214 bytes, but only found 0 bytes eligible to free. 53s Warning ImageGCFailed node/server Failed to garbage collect required amount of images. Attempted to free 11668506214 bytes, but only found 0 bytes eligible to free. [root@server ~]# ``` ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [December 30, 2024, 10:19am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/9 "2024-12-30T10:19:45Z")
I have added pod logs ```auto [root@g20-ansible-cxp ~]# kubectl describe pod awx-operator-controller-manager-745b55d94b-97sdw -n awx Name: awx-operator-controller-manager-745b55d94b-97sdw Namespace: awx Priority: 0 Service Account: awx-operator-controller-manager Node: server/IPADDRESS Start Time: Sun, 10 Nov 2024 20:58:47 -0800 Labels: control-plane=controller-manager pod-template-hash=745b55d94b Annotations: kubectl.kubernetes.io/default-container: awx-manager Status: Running IP: server IPs: IP: server Controlled By: ReplicaSet/awx-operator-controller-manager-745b55d94b Containers: kube-rbac-proxy: Container ID: containerd://1e7f895d49f51ce8f0cc6ac1baa8aae81b12c2ce6fa1242dc377a2c8f2f9a945 Image: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0 Image ID: gcr.io/kubebuilder/kube-rbac-proxy@sha256:d8cc6ffb98190e8dd403bfe67ddcb454e6127d32b87acc237b3e5240f70a20fb Port: 8443/TCP Host Port: 0/TCP Args: --secure-listen-address=0.0.0.0:8443 --upstream=http://127.0.0.1:8080/ --logtostderr=true --v=0 State: Running Started: Mon, 30 Dec 2024 02:00:36 -0800 Last State: Terminated Reason: Unknown Exit Code: 255 Started: Sun, 08 Dec 2024 23:08:01 -0800 Finished: Mon, 30 Dec 2024 02:00:33 -0800 Ready: True Restart Count: 4 Limits: cpu: 500m memory: 128Mi Requests: cpu: 5m memory: 64Mi Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hrvhv (ro) awx-manager: Container ID: containerd://4d79566baf0e733bcbf3276ea2d7db60f24c9650c558e37bfe1f8308bb7857dc Image: quay.io/ansible/awx-operator:2.19.1 Image ID: quay.io/ansible/awx-operator@sha256:7302e0c8e5a79c11275848b36c63c61d8e7487840164c109b7cda9aa93456901 Port: Host Port: Args: --health-probe-bind-address=:6789 --metrics-bind-address=127.0.0.1:8080 --leader-elect --leader-election-id=awx-operator State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 137 Started: Mon, 30 Dec 2024 02:11:33 -0800 Finished: Mon, 30 Dec 2024 02:13:10 -0800 Ready: False Restart Count: 5407 Limits: cpu: 1500m memory: 960Mi Requests: cpu: 50m memory: 32Mi Liveness: http-get http://:6789/healthz delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:6789/readyz delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: ANSIBLE_GATHERING: explicit ANSIBLE_DEBUG_LOGS: false WATCH_NAMESPACE: awx (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hrvhv (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-hrvhv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 61m (x519 over 2d1h) kubelet Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine Warning BackOff 41m (x12865 over 2d1h) kubelet Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d) Normal Created 36m (x4 over 39m) kubelet Created container awx-manager Normal Started 36m (x4 over 39m) kubelet Started container awx-manager Warning BackOff 24m (x60 over 38m) kubelet Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d) Normal Pulled 19m (x8 over 39m) kubelet Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine Warning FailedCreatePodSandBox 17m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fc9bee795974a4ef4351a5e2d011e50f0c13a584c844e7d0faa7e075e31a6730": plugin type="flannel" failed (add): loadFlannelSubnetEnv failed: open /run/flannel/subnet.env: no such file or directory Normal Pulled 17m kubelet Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0" already present on machine Normal SandboxChanged 17m (x2 over 17m) kubelet Pod sandbox changed, it will be killed and re-created. Normal Created 17m kubelet Created container kube-rbac-proxy Normal Started 17m kubelet Started container kube-rbac-proxy Warning Unhealthy 16m (x3 over 16m) kubelet Liveness probe failed: Get "http://server:6789/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal Killing 16m kubelet Container awx-manager failed liveness probe, will be restarted Normal Pulled 15m (x2 over 17m) kubelet Container image "quay.io/ansible/awx-operator:2.19.1" already present on machine Normal Created 15m (x2 over 17m) kubelet Created container awx-manager Normal Started 15m (x2 over 17m) kubelet Started container awx-manager Warning Unhealthy 11m (x21 over 16m) kubelet Readiness probe failed: Get "http://server:6789/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning BackOff 2m20s (x31 over 13m) kubelet Back-off restarting failed container awx-manager in pod awx-operator-controller-manager-745b55d94b-97sdw_awx(8685caba-395d-4a82-8863-fd225e47d43d) [root@g20-ansible-cxp ~]# ``` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [December 31, 2024, 4:26pm UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/10 "2024-12-31T16:26:14Z")
> [@vinothravindhran](#): > > `awx-operator-controller-manager-745b55d94b-97sdw 1/2 OOMKilled 4879 (119s ago) 47d` > [@vinothravindhran](#): > > ```auto > State: Waiting > Reason: CrashLoopBackOff > Last State: Terminated > Reason: Error > Exit Code: 137 > Started: Mon, 30 Dec 2024 02:11:33 -0800 > Finished: Mon, 30 Dec 2024 02:13:10 -0800 > > ``` Since OOMKilled is recorded, it is possible that the memory usage temporarily spiked immediately after the Operator started. Please check the Operator’s logs. The logs you provided are from when it was functioning normally and do not contain information from the moment it terminated abnormally. It is likely that the cause is an excessive number of awxbackup resources. The Operator tries to execute Playbooks for each awxbackup resource at startup. Please delete any unnecessary or failed awxbackup resources. However, since the deletion itself requires processing by the Operator, the delete command will likely get stuck. If it gets stuck, please remove the finalizer from the awxbackup resources that are in the process of being deleted. ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [December 31, 2024, 4:29pm UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/11 "2024-12-31T16:29:32Z")
ref: > > > \## Environment > \- OS: Ubuntu 24.04 LTS > \- k3s version: v1.29.5+k3s1 (4e53a323) > …- AWX Operator: 2.19.0 > > \## Description > I have had a backup script running forever, that follows the \[backup guide\](https://github.com/kurokobo/awx-on-k3s/tree/main/backup). It has been working flawlessly, but it has started failing every time since \`2024-11-27\`. There have been made no modifications from the time since the backup job was functioning perfectly, until now where it has been failing every time I attempt to make a backup. > > \## Step to Reproduce > Unsure, have not attempted to reproduce on a clean setup as daily backups have worked fine. > The command that fails is \`kubectl apply -f "{{ awx\_k3s\_repo\_dir }}/awx-on-k3s/backup/awxbackup.yaml"\` > > \## Logs > When running \`kubectl apply -f "/backup/awxbackup.yaml"\` it finishes immediately instead of doing the backup, and logs displayed by \`kubectl -n awx logs -f deployments/awx-operator-controller-manager\` do not tell me anything I can make sense of. > > \`\`\` > kubectl -n awx logs -f deployments/awx-operator-controller-manager > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"cmd","msg":"Watching namespaces","namespaces":\["awx"\]} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE\_VERBOSITY\_AWX\_AWX\_ANSIBLE\_COM","default":2} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE\_VERBOSITY\_AWXBACKUP\_AWX\_ANSIBLE\_COM","default":2} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE\_VERBOSITY\_AWXRESTORE\_AWX\_ANSIBLE\_COM","default":2} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE\_VERBOSITY\_AWXMESHINGRESS\_AWX\_ANSIBLE\_COM","default":2} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"} > {"level":"info","ts":"2024-12-18T10:31:48Z","msg":"starting server","kind":"health probe","addr":"\[::\]:6789"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} > {"level":"info","ts":"2024-12-18T10:31:48Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false} > I1218 10:31:48.821221 7 leaderelection.go:250\] attempting to acquire leader lease awx/awx-operator... > I1218 10:32:05.084628 7 leaderelection.go:260\] successfully acquired lease awx/awx-operator > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: \*unstructured.Unstructured"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: \*unstructured.Unstructured"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: \*unstructured.Unstructured"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awx-controller"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awxmeshingress-controller"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awxrestore-controller"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: \*unstructured.Unstructured"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting Controller","controller":"awxbackup-controller"} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awx-controller","worker count":32} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awxmeshingress-controller","worker count":32} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":32} > {"level":"info","ts":"2024-12-18T10:32:05Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":32} > {"level":"info","ts":"2024-12-18T10:32:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \\"status.conditions\[0\].message\\""} > {"level":"info","ts":"2024-12-18T10:32:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \\"status.conditions\[1\].message\\""} > {"level":"info","ts":"2024-12-18T10:32:05Z","logger":"KubeAPIWarningLogger","msg":"unknown field \\"status.conditions\[2\].message\\""} > I1218 10:32:06.277917 7 request.go:697\] Waited for 1.028919415s due to client-side throttling, not priority and fairness, request: PUT:https://10.43.0.1:443/apis/awx.ansible.com/v1beta1/namespaces/awx/awxbackups/awx-backup-20240716080901/status > \`\`\` > > When I check the awxbackup objects I see a long list of old backup objects, and it does create a new one every time I apply the backup: > > \`\`\` > kubectl -n awx get awxbackup > NAME AGE > \<100s of other awx-backup-202x objects\> > awx-backup-20241218100946 22m > awx-backup-20241218102752 4m40s > \`\`\` > > \## Files > my awxbackup.yml is defined as > > \`\`\`yml > \--- > apiVersion: awx.ansible.com/v1beta1 > kind: AWXBackup > metadata: > name: {{ filename\_startswith }}-{{ current\_date\_utc }} > namespace: awx > spec: > deployment\_name: awx > backup\_pvc: awx-backup-claim > \`\`\` > > > Added documentation to backup readme with recommendation to clean up the awxback…up resources created when applying the backup. Include commands to recover if \`awx-operator-controller-manager\` becomes unresponsive. ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [January 2, 2025, 11:48am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/12 "2025-01-02T11:48:03Z")
Thank you @kurokobo for sharing knowledge. As you mentioned, i have cleaned up unnecessary/failed AWX backup instances. I successfully cleaned up but still new backup job is failing and also Still AWX operator is showing same error “CrashLoopBackOff”. ```auto [root@server ~]# kubectl get awxbackup -n awx NAME AGE awxbackup-2024-11-29-12-38-30 33d awxbackup-2024-12-02-04-30-38 31d awxbackup-2024-12-02-23-30-38 30d awxbackup-2024-12-03-23-30-38 29d awxbackup-2024-12-04-23-30-46 28d awxbackup-2024-12-05-23-30-26 27d awxbackup-2024-12-06-23-30-28 26d awxbackup-2024-12-07-23-30-27 25d awxbackup-2024-12-08-23-30-26 24d awxbackup-2024-12-26-11-05-04 6d23h awxbackup-2024-12-26-11-06-20 6d23h [root@server ~]# ls -l /data/backup/ total 0 drwxr-xr-x 2 26 26 59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800 drwxr-xr-x 2 26 26 59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843 drwxr-xr-x 2 26 26 59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859 drwxr-xr-x 2 26 26 59 Dec 1 20:31 tower-openshift-backup-2024-12-02-043052 drwxr-xr-x 2 26 26 59 Dec 2 15:31 tower-openshift-backup-2024-12-02-233051 drwxr-xr-x 2 26 26 59 Dec 3 15:31 tower-openshift-backup-2024-12-03-233052 drwxr-xr-x 2 26 26 59 Dec 4 15:31 tower-openshift-backup-2024-12-04-233059 drwxr-xr-x 2 26 26 59 Dec 5 15:30 tower-openshift-backup-2024-12-05-233039 drwxr-xr-x 2 26 26 59 Dec 6 15:30 tower-openshift-backup-2024-12-06-233041 drwxr-xr-x 2 26 26 59 Dec 7 15:30 tower-openshift-backup-2024-12-07-233039 drwxr-xr-x 2 26 26 59 Dec 8 15:30 tower-openshift-backup-2024-12-08-233039 [root@server ~]# [root@server ~]# kubectl -n awx logs -f awx-operator-controller-manager-745b55d94b-5rpht {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"cmd","msg":"Watching namespaces","namespaces":["awx"]} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWX_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXBACKUP_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXRESTORE_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_AWXMESHINGRESS_AWX_ANSIBLE_COM","default":2} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWX"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXBackup"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1beta1","Options.Kind":"AWXRestore"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"ansible-controller","msg":"Watching resource","Options.Group":"awx.ansible.com","Options.Version":"v1alpha1","Options.Kind":"AWXMeshIngress"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"apiserver","msg":"Starting to serve metrics listener","Address":"localhost:5050"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"} {"level":"info","ts":"2025-01-02T11:42:50Z","msg":"starting server","kind":"health probe","addr":"[::]:6789"} {"level":"info","ts":"2025-01-02T11:42:50Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false} I0102 11:42:50.637689 7 leaderelection.go:250] attempting to acquire leader lease awx/awx-operator... I0102 11:43:06.436319 7 leaderelection.go:260] successfully acquired lease awx/awx-operator {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awx-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awxrestore-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awxbackup-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awx-controller"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awxbackup-controller"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awxrestore-controller"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting EventSource","controller":"awxmeshingress-controller","source":"kind source: *unstructured.Unstructured"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting Controller","controller":"awxmeshingress-controller"} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awxmeshingress-controller","worker count":8} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awxbackup-controller","worker count":8} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awx-controller","worker count":8} {"level":"info","ts":"2025-01-02T11:43:06Z","msg":"Starting workers","controller":"awxrestore-controller","worker count":8} {"level":"info","ts":"2025-01-02T11:43:06Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[0].message\""} {"level":"info","ts":"2025-01-02T11:43:06Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[1].message\""} {"level":"info","ts":"2025-01-02T11:43:06Z","logger":"KubeAPIWarningLogger","msg":"unknown field \"status.conditions[2].message\""} {"level":"info","ts":"2025-01-02T11:43:13Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"2461380222856614183","EventData.Name":"Verify imagePullSecrets"} --------------------------- Ansible Task StdOut ------------------------------- TASK [Verify imagePullSecrets] ************************************************* task path: /opt/ansible/playbooks/awx.yml:10 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"8415435552159357307","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-04-23-30-46","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"3794599929215878019","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8415435552159357307","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-04-23-30-46","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3794599929215878019","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-02-04-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"721827200833615441","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:15Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-02-04-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"721827200833615441","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-07-23-30-27","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"8415435552159357307","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-04-23-30-46","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3794599929215878019","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-02-04-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"721827200833615441","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-08-23-30-26","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"6180278095043985779","EventData.TaskArgs":""} {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-08-23-30-26","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"6180278095043985779","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-06-23-30-28","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"5452128470082692865","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-08-23-30-26","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"6180278095043985779","EventData.Name":"backup : Patching labels to AWXBackup kind"} {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-06-23-30-28","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"5452128470082692865","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-26-11-05-04","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"3600057349764572609","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-05-04","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3600057349764572609","EventData.Name":"backup : Run creation tasks"} {"level":"info","ts":"2025-01-02T11:43:16Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-06-23-30-28","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"5452128470082692865","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-05-04","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3600057349764572609","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"4102565071948320805","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"4102565071948320805","EventData.Name":"backup : Run creation tasks"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awxbackup-2024-12-03-23-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"runner_on_ok","job":"3177590642827013235","EventData.TaskArgs":""} --------------------------- Ansible Task StdOut ------------------------------- TASK [debug] ******************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Run creation tasks] ********************************************* task path: /opt/ansible/roles/backup/tasks/main.yml:2 ------------------------------------------------------------------------------- --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-03-23-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3177590642827013235","EventData.Name":"backup : Run creation tasks"} {"level":"info","ts":"2025-01-02T11:43:17Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-26-11-06-20","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"4102565071948320805","EventData.Name":"backup : Patching labels to AWXBackup kind"} {"level":"info","ts":"2025-01-02T11:43:18Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awxbackup-2024-12-03-23-30-38","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWXBackup","event_type":"playbook_on_task_start","job":"3177590642827013235","EventData.Name":"backup : Patching labels to AWXBackup kind"} --------------------------- Ansible Task StdOut ------------------------------- TASK [backup : Patching labels to AWXBackup kind] ****************************** task path: /opt/ansible/roles/backup/tasks/creation.yml:2 ------------------------------------------------------------------------------- {"level":"info","ts":"2025-01-02T11:43:18Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/awx/secrets/redhat-operators-pull-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"secrets","Subresource":"","Name":"redhat-operators-pull-secret","Parts":["secrets","redhat-operators-pull-secret"]}} {"level":"info","ts":"2025-01-02T11:43:19Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"2461380222856614183","EventData.Name":"Create imagePullSecret"} --------------------------- Ansible Task StdOut ------------------------------- TASK [Create imagePullSecret] ************************************************** task path: /opt/ansible/playbooks/awx.yml:17 ------------------------------------------------------------------------------- [root@server ~]# ``` ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [January 2, 2025, 11:49am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/13 "2025-01-02T11:49:41Z")
Next step :- I will try to delete all backups and initiate the backup again. ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [January 2, 2025, 11:58am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/14 "2025-01-02T11:58:02Z")
Thank you very much @kurokobo Once all backup resources are cleaned up, i able to run the backup job successfully. In backup file - can I change `keep_days: 30` values to “7”? Will this help to lower the resource consumption which makes Operator became more stable. ```auto [root@server ~]# ls -l /data/backup/ total 0 drwxr-xr-x 2 26 26 59 Nov 29 04:38 tower-openshift-backup-2024-11-29-123800 drwxr-xr-x 2 26 26 59 Nov 29 04:39 tower-openshift-backup-2024-11-29-123843 drwxr-xr-x 2 26 26 59 Nov 29 04:49 tower-openshift-backup-2024-11-29-124859 drwxr-xr-x 2 26 26 59 Dec 1 20:31 tower-openshift-backup-2024-12-02-043052 drwxr-xr-x 2 26 26 59 Dec 2 15:31 tower-openshift-backup-2024-12-02-233051 drwxr-xr-x 2 26 26 59 Dec 3 15:31 tower-openshift-backup-2024-12-03-233052 drwxr-xr-x 2 26 26 59 Dec 4 15:31 tower-openshift-backup-2024-12-04-233059 drwxr-xr-x 2 26 26 59 Dec 5 15:30 tower-openshift-backup-2024-12-05-233039 drwxr-xr-x 2 26 26 59 Dec 6 15:30 tower-openshift-backup-2024-12-06-233041 drwxr-xr-x 2 26 26 59 Dec 7 15:30 tower-openshift-backup-2024-12-07-233039 drwxr-xr-x 2 26 26 59 Dec 8 15:30 tower-openshift-backup-2024-12-08-233039 drwxr-xr-x 2 26 26 59 Jan 2 03:54 tower-openshift-backup-2025-01-02-115430 [root@g20-ansible-cxp ~]# kubectl get awxbackup -n awx NAME AGE awxbackup-2025-01-02-11-54-17 98s [root@server ~]# ``` ---
**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\ **Post date:** [January 3, 2025, 2:28am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/15 "2025-01-03T02:28:45Z")
> In backup file - can I change `keep_days: 30` values to “7”? Yes, you can change the variable `awxbackup_keep_days` to `7` to make the Operator more stable: [awx-on-k3s/backup/ansible at main · kurokobo/awx-on-k3s · GitHub](https://github.com/kurokobo/awx-on-k3s/tree/main/backup/ansible) ---
**Author:** ![vinothravindhran](https://avatars.discourse-cdn.com/v4/letter/v/9fc348/32.png) [@vinothravindhran](https://forum.ansible.com/u/vinothravindhran)\ **Post date:** [January 3, 2025, 5:07am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/16 "2025-01-03T05:07:33Z")
Thanks a lot @kurokobo With your technical support, we able to fix the issue. ---
**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\ **Post date:** [February 2, 2025, 5:08am UTC](https://forum.ansible.com/t/awx-scheduled-backup-job-failed-after-couple-of-days-awx-24-6-1/39344/17 "2025-02-02T05:08:00Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.