# Install of awx-operator from helm chart generates error loop in awx-manager container **URL:** \ **Category:** Get Help\ **Tags:** awx-operator, awx\ **Created:** [October 15, 2024, 8:41am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294 "2024-10-15T08:41:50Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [October 15, 2024, 8:41am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/1 "2024-10-15T08:41:50Z")
I am attempting to deploy the awx-operator via the helm chart at [awx-operator Helm charts | awx-operator-helm](https://ansible-community.github.io/awx-operator-helm/) . I have previously successfully installed 2.9.0 and deployed AWX, but prior to the move. I am now getting the following error in awx-manager, irrespective of the version deployed: ```auto 2024-10-15T16:17:26.004798609+08:00 TASK [Verify imagePullSecrets] ************************************************* 2024-10-15T16:17:26.004803763+08:00 task path: /opt/ansible/playbooks/awx.yml:10 2024-10-15T16:17:26.004808093+08:00 2024-10-15T16:17:26.004821573+08:00 ------------------------------------------------------------------------------- 2024-10-15T16:17:26.260766825+08:00 {"level":"error","ts":"2024-10-15T08:17:26Z","logger":"logging_event_handler","msg":"","name":"awxtest15","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"7862226185777618437","EventData.Task":"Verify imagePullSecrets","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/playbooks/awx.yml:10","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"} 2024-10-15T16:17:26.260864035+08:00 2024-10-15T16:17:26.260898398+08:00 --------------------------- Ansible Task StdOut ------------------------------- 2024-10-15T16:17:26.260902750+08:00 2024-10-15T16:17:26.260907455+08:00 TASK [Verify imagePullSecrets] ******************************** 2024-10-15T16:17:26.260919623+08:00 fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2024-10-15T16:17:26.260935467+08:00 ``` This error loops, and no further pods deploy. To minimise possible confusion, I have been using the default values file, with minimal changes (enabled: true, and the name, which produced a working configuration previously): ```auto AWX: enabled: true name: awxtest15 postgres: dbName: Unset enabled: false host: Unset password: Unset port: 5678 sslmode: prefer type: unmanaged username: admin spec: admin_user: admin ``` ---
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [October 18, 2024, 1:37am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/2 "2024-10-18T01:37:44Z")
Further research suggests there is a python issue. The operator is being installed on a RKE node based on OL8.9. Running the installer manually: ```auto bash-4.4$ ansible-playbook run.yml -e @vars.yml -v Using /etc/ansible/ansible.cfg as config file [WARNING]: Found variable using reserved name: no_log PLAY [localhost] **************************************************************************************************************************************************************************** TASK [common : Get information about the cluster] ******************************************************************************************************************************************* ok: [localhost] => {"ansible_facts": {"api_groups": ["", "apiregistration.k8s.io", "apps", "events.k8s.io", "authentication.k8s.io", "authorization.k8s.io", "autoscaling", "batch", "certificates.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io", "admissionregistration.k8s.io", "apiextensions.k8s.io", "scheduling.k8s.io", "coordination.k8s.io", "node.k8s.io", "discovery.k8s.io", "flowcontrol.apiserver.k8s.io", "catalog.cattle.io", "crd.projectcalico.org", "helm.cattle.io", "k3s.cattle.io", "operator.tigera.io", "ui.cattle.io", "upgrade.cattle.io", "awx.ansible.com", "cns.vmware.com", "management.cattle.io", "metrics.k8s.io"]}, "changed": false} TASK [common : Determine the cluster type] ************************************************************************************************************************************************** ok: [localhost] => {"ansible_facts": {"is_k8s": true, "is_openshift": false}, "changed": false} TASK [common : debug] *********************************************************************************************************************************************************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } TASK [installer : Check for presence of old awx Deployment] ********************************************************************************************************************************* fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/libexec/platform-python"}, "changed": false, "msg": "Failed to import the required Python library (kubernetes) on awx-operator-controller-manager-6bd475f79-jh6nd's Python /usr/libexec/platform-python. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"} PLAY RECAP ********************************************************************************************************************************************************************************** localhost : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 bash-4.4$ uname -a Linux awx-operator-controller-manager-6bd475f79-jh6nd 5.15.0-105.125.6.2.2.el9uek.x86_64 #2 SMP Tue Sep 19 23:28:56 PDT 2023 x86_64 x86_64 x86_64 GNU/Linux bash-4.4$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 (Ootpa) ``` If I set ansible\_python\_interpreter: ‘{{ ansible\_playbook\_python }}’ in vars.yml, I get: ```auto bash-4.4$ ansible-playbook run.yml -e @vars.yml -v Using /etc/ansible/ansible.cfg as config file [WARNING]: Found variable using reserved name: no_log PLAY [localhost] **************************************************************************************************************************************************************************** TASK [common : Get information about the cluster] ******************************************************************************************************************************************* ok: [localhost] => {"ansible_facts": {"api_groups": ["", "apiregistration.k8s.io", "apps", "events.k8s.io", "authentication.k8s.io", "authorization.k8s.io", "autoscaling", "batch", "certificates.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io", "admissionregistration.k8s.io", "apiextensions.k8s.io", "scheduling.k8s.io", "coordination.k8s.io", "node.k8s.io", "discovery.k8s.io", "flowcontrol.apiserver.k8s.io", "catalog.cattle.io", "crd.projectcalico.org", "helm.cattle.io", "k3s.cattle.io", "operator.tigera.io", "ui.cattle.io", "upgrade.cattle.io", "awx.ansible.com", "cns.vmware.com", "management.cattle.io", "metrics.k8s.io"]}, "changed": false} TASK [common : Determine the cluster type] ************************************************************************************************************************************************** ok: [localhost] => {"ansible_facts": {"is_k8s": true, "is_openshift": false}, "changed": false} TASK [common : debug] *********************************************************************************************************************************************************************** ok: [localhost] => { "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True" } TASK [installer : Check for presence of old awx Deployment] ********************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "/bin/sh: line 1: 4158 Killed /usr/bin/python3 /opt/ansible/.ansible/tmp/ansible-tmp-1729214698.536406-4146-47837280803179/AnsiballZ_k8s_info.py\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 137} PLAY RECAP ********************************************************************************************************************************************************************************** localhost : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 bash-4.4$ ``` Which corresponds to the error generated by running via helm. Any suggestions as to where to go from here? ---
**Author:** ![GarbageRaccoon](https://avatars.discourse-cdn.com/v4/letter/g/48db29/32.png) [@GarbageRaccoon](https://forum.ansible.com/u/GarbageRaccoon)\ **Post date:** [October 18, 2024, 2:43pm UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/3 "2024-10-18T14:43:49Z")
I have exactly the same problem from today. AWX-Operator 2.19.1 installed on RKE2, I start working today and: ```auto [some_server]$ k get pods -n awx NAME READY STATUS RESTARTS AGE awx-operator-controller-manager-b8789c7f7-5sqwc 1/2 CrashLoopBackOff 4 (80s ago) 5m58s ``` ```auto --------------------------- Ansible Task StdOut -------------------- ----------- TASK [Verify imagePullSecrets] ************************************************************** task path: /opt/ansible/playbooks/awx.yml:10 -------------------------------------------------- ----------------------------- ``` and this certain log ends a log follower. Anybody any idea? ---
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [October 21, 2024, 8:51am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/4 "2024-10-21T08:51:04Z")
Running the playbook manually with more verbosity gives us a little more detail, but still nothing that suggests a cause or a solution: ```auto bash-4.4$ ansible-playbook run.yml -e @vars.yml -v -v -v -v -v -v ansible-playbook [core 2.15.8] config file = /etc/ansible/ansible.cfg configured module search path = ['/usr/share/ansible/openshift'] ansible python module location = /usr/local/lib/python3.9/site-packages/ansible ansible collection location = /opt/ansible/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible-playbook python version = 3.9.18 (main, Sep 22 2023, 17:58:34) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3) jinja version = 3.1.3 libyaml = True Using /etc/ansible/ansible.cfg as config file ``` and more setup information, then it starts running things. ```auto PLAY [localhost] **************************************************************************************************************************************************************************** TASK [common : Get information about the cluster] ******************************************************************************************************************************************* task path: /opt/ansible/roles/common/tasks/main.yml:3 redirecting (type: lookup) ansible.builtin.k8s to kubernetes.core.k8s ok: [localhost] => { "ansible_facts": { "api_groups": [ "", "apiregistration.k8s.io", "apps", "events.k8s.io", "authentication.k8s.io", "authorization.k8s.io", "autoscaling", "batch", "certificates.k8s.io", "networking.k8s.io", "policy", "rbac.authorization.k8s.io", "storage.k8s.io", "admissionregistration.k8s.io", "apiextensions.k8s.io", "scheduling.k8s.io", "coordination.k8s.io", "node.k8s.io", "discovery.k8s.io", "flowcontrol.apiserver.k8s.io", "catalog.cattle.io", "crd.projectcalico.org", "helm.cattle.io", "k3s.cattle.io", "operator.tigera.io", "ui.cattle.io", "upgrade.cattle.io", "awx.ansible.com", "cns.vmware.com", "management.cattle.io", "metrics.k8s.io" ] }, "changed": false } ``` And then more tasks from the common role, then it hits the installer role, and things go pear-shaped: ```auto TASK [installer : Check for presence of old awx Deployment] ********************************************************************************************************************************* task path: /opt/ansible/roles/installer/tasks/main.yml:2 ESTABLISH LOCAL CONNECTION FOR USER: ansible EXEC /bin/sh -c 'echo ~ansible && sleep 0' EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /opt/ansible/.ansible/tmp `"&& mkdir "` echo /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398 `" && echo ansible-tmp-1729497208.756366-10930-68667324237398="` echo /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398 `" ) && sleep 0' Including module_utils file ansible/ __init__.py Including module_utils file ansible/module_utils/ __init__.py Including module_utils file ansible/module_utils/basic.py Including module_utils file ansible/module_utils/_text.py Including module_utils file ansible/module_utils/common/_json_compat.py Including module_utils file ansible/module_utils/common/ __init__.py Including module_utils file ansible/module_utils/common/_utils.py Including module_utils file ansible/module_utils/common/arg_spec.py Including module_utils file ansible/module_utils/common/file.py Including module_utils file ansible/module_utils/common/locale.py Including module_utils file ansible/module_utils/common/parameters.py Including module_utils file ansible/module_utils/common/collections.py Including module_utils file ansible/module_utils/common/process.py Including module_utils file ansible/module_utils/common/sys_info.py Including module_utils file ansible/module_utils/common/text/converters.py Including module_utils file ansible/module_utils/common/text/ __init__.py Including module_utils file ansible/module_utils/common/text/formatters.py Including module_utils file ansible/module_utils/common/validation.py Including module_utils file ansible/module_utils/common/warnings.py Including module_utils file ansible/module_utils/compat/selectors.py Including module_utils file ansible/module_utils/compat/ __init__.py Including module_utils file ansible/module_utils/compat/_selectors2.py Including module_utils file ansible/module_utils/compat/selinux.py Including module_utils file ansible/module_utils/distro/ __init__.py Including module_utils file ansible/module_utils/distro/_distro.py Including module_utils file ansible/module_utils/errors.py Including module_utils file ansible/module_utils/parsing/convert_bool.py Including module_utils file ansible/module_utils/parsing/ __init__.py Including module_utils file ansible/module_utils/pycompat24.py Including module_utils file ansible/module_utils/six/ __init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/ansiblemodule.py Including module_utils file ansible_collections/ __init__.py Including module_utils file ansible_collections/kubernetes/ __init__.py Including module_utils file ansible_collections/kubernetes/core/ __init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/ __init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/ __init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/args_common.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/ __init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/client/resource.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/ __init__.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/core.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/exceptions.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py Including module_utils file ansible/module_utils/common/dict_transformations.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/apply.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/common.py Including module_utils file ansible/module_utils/urls.py Including module_utils file ansible/module_utils/compat/typing.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/exceptions.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/hashes.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8s/waiter.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/k8sdynamicclient.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/version.py Including module_utils file ansible_collections/kubernetes/core/plugins/module_utils/_version.py Using module file /opt/ansible/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py PUT /opt/ansible/.ansible/tmp/ansible-local-109200clhnr2p/tmptihkeltq TO /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/AnsiballZ_k8s_info.py EXEC /bin/sh -c 'chmod u+x /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/ /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/AnsiballZ_k8s_info.py && sleep 0' EXEC /bin/sh -c '/usr/bin/python3 /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/AnsiballZ_k8s_info.py && sleep 0' EXEC /bin/sh -c 'rm -f -r /opt/ansible/.ansible/tmp/ansible-tmp-1729497208.756366-10930-68667324237398/ > /dev/null 2>&1 && sleep 0' fatal: [localhost]: FAILED! => { "changed": false, "invocation": { "api_version": "apps/v1", "kind": "Deployment", "module_args": { "api_version": "apps/v1", "kind": "Deployment", "name": "awx", "namespace": "awx" }, "name": "awx", "namespace": "awx" }, "module_stderr": "", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": -9 } ``` What is particularly odd is that both stderr and stdout are empty. Further, manually running that first module succeeds: ```auto export PYTHONPATH=${PYTHONPATH}:/opt/ansible/.ansible/collections/ echo '{ "ANSIBLE_MODULE_ARGS": { "api_version": "apps/v1", "kind": "Deployment", "name": "awx", "namespace": "awx" } } ' |python3 /opt/ansible/.ansible/collections/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py {"changed": false, "resources": [], "api_found": true, "invocation": {"module_args": {"api_version": "apps/v1", "kind": "Deployment", "name": "awx", "namespace": "awx", "wait": false, "wait_sleep": 5, "wait_timeout": 120, "label_selectors": [], "field_selectors": [], "kubeconfig": null, "context": null, "host": null, "api_key": null, "username": null, "password": null, "validate_certs": null, "ca_cert": null, "client_cert": null, "client_key": null, "proxy": null, "no_proxy": null, "proxy_headers": null, "persist_config": null, "impersonate_user": null, "impersonate_groups": null, "wait_condition": null, "hidden_fields": null}}} ``` Any suggestions? Anyone? Bueller? ---
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [October 22, 2024, 5:24am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/5 "2024-10-22T05:24:55Z")
Also logged in github as issue #1978. ---
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [October 25, 2024, 7:46am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/6 "2024-10-25T07:46:47Z")
One minor correction: The host cluster is running OL9, not 8.9. ---
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [October 31, 2024, 7:08am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/7 "2024-10-31T07:08:27Z")
As an experiment, I set up an OL8 cluster and re-ran everything, but had the same result. So then I tried falling back to the generic Redhat kernel (instead of the UEK kernel) and still the same result. ---
**Author:** ![rmastersbun](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/rmastersbun/32/5155_2.png) [@rmastersbun](https://forum.ansible.com/u/rmastersbun)\ **Post date:** [November 14, 2024, 7:04am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/8 "2024-11-14T07:04:23Z")
Eventually identified as being caused by Crowdstrike Falcon Agent. Kill the agent, and everything runs. ---
**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\ **Post date:** [December 14, 2024, 7:04am UTC](https://forum.ansible.com/t/install-of-awx-operator-from-helm-chart-generates-error-loop-in-awx-manager-container/10294/9 "2024-12-14T07:04:44Z")
Closed on December 14, 2024, 7:04am UTC This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.