We have a setup in k8s/rancher that was working fine. It now will no longer execute jobs - or even sync git repos.
We can’t find anything in the logs on the pods. We have 2 instances, a test and a prod. We updated the operator to 15 and updated the instances with no change. In looking at the environment of the runner for a very simple playbook job (login to host, cat /etc/redhat-release) we can see both sets of our AWX environments defined. Not sure if this is expected or not, it seems rather confusing, is this correct (perhaps due to the operator)?
In the running automation-job pod we can execute the playbook command ok manually. Any ideas as to why it’s just sitting there?
process status in the container shows all the jobs in Ss, Ss+ or S+.
Thx. -Dave
Here’s the log/start of the job:
{“status”: “starting”,
“runner_ident”: “442”,
“command”: [“ssh-agent”, “sh”, “-c”, “trap ‘rm -f /runner/artifacts/442/ssh_key_data’ EXIT && ssh-add /runner/artifacts/442/ssh_key_data && rm -f /runner/artifacts/442/ssh_key_data && ansible-playbook -u ansible --ask-pass --become -i /runner/inventory/hosts -e @/runner/env/extravars rhelversion.yaml”],
“env”: {
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT”: “tcp://10.43.144.140:8443”,
“AWXTEST_SERVICE_PORT_80_TCP_ADDR”: “10.43.220.158”,
“HOSTNAME”: “automation-job-442-lngkd”,
“AWXPROD_SERVICE_PORT_80_TCP”: “tcp://10.43.29.38:80”,
“AWXPROD_SERVICE_SERVICE_PORT_HTTP”: “80”,
“AWXPROD_SERVICE_PORT_80_TCP_PORT”: “80”,
“AWXTEST_SERVICE_SERVICE_HOST”: “10.43.220.158”,
“KUBERNETES_PORT_443_TCP_PROTO”: “tcp”,
“KUBERNETES_PORT_443_TCP_ADDR”: “10.43.0.1”,
“AWXTEST_SERVICE_SERVICE_PORT_HTTP”: “80”,
“KUBERNETES_PORT”: “tcp://10.43.0.1:443”,
“PWD”: “/runner”, “HOME”: “/home/runner”,
“AWXTEST_SERVICE_PORT_80_TCP_PROTO”: “tcp”,
“AWXPROD_SERVICE_PORT_80_TCP_ADDR”: “10.43.29.38”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PORT”: “8443”,
“KUBERNETES_SERVICE_PORT_HTTPS”: “443”, “KUBERNETES_PORT_443_TCP_PORT”: “443”,
“KUBERNETES_PORT_443_TCP”: “tcp://10.43.0.1:443”,
“AWXPROD_SERVICE_SERVICE_PORT”: “80”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP”: “tcp://10.43.144.140:8443”,
“AWXPROD_SERVICE_SERVICE_HOST”: “10.43.29.38”, “AWXPROD_SERVICE_PORT_80_TCP_PROTO”: “tcp”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT_HTTPS”: “8443”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_PORT”: “8443”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_PROTO”: “tcp”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_SERVICE_HOST”: “10.43.144.140”,
“SHLVL”: “0”,
“KUBERNETES_SERVICE_PORT”: “443”,
“AWXTEST_SERVICE_PORT_80_TCP_PORT”: “80”,
“AWXTEST_SERVICE_PORT_80_TCP”: “tcp://10.43.220.158:80”,
“AWXTEST_SERVICE_SERVICE_PORT”: “80”,
“AWX_OPERATOR_CONTROLLER_MANAGER_METRICS_SERVICE_PORT_8443_TCP_ADDR”: “10.43.144.140”,
“PATH”: “/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”,
“AWXTEST_SERVICE_PORT”: “tcp://10.43.220.158:80”,
“KUBERNETES_SERVICE_HOST”: “10.43.0.1”,
“AWXPROD_SERVICE_PORT”: “tcp://10.43.29.38:80”,
“LC_CTYPE”: “C.UTF-8”,
“ANSIBLE_FORCE_COLOR”: “True”,
“ANSIBLE_HOST_KEY_CHECKING”: “False”,
“ANSIBLE_INVENTORY_UNPARSED_FAILED”: “True”,
“ANSIBLE_PARAMIKO_RECORD_HOST_KEYS”: “False”,
“AWX_PRIVATE_DATA_DIR”: “/tmp/awx_442_i6ln8dhi”,
“JOB_ID”: “442”,
“INVENTORY_ID”: “5”,
“PROJECT_REVISION”: “ef356910d99c5447f96dc15540e2bbda67cc78e2”,
“ANSIBLE_RETRY_FILES_ENABLED”: “False”,
“MAX_EVENT_RES”: “700000”,
“AWX_HOST”: “https://awxtest.k8s.eagleaccess.com/”,
“ANSIBLE_SSH_CONTROL_PATH_DIR”: “/runner/cp”,
“ANSIBLE_COLLECTIONS_PATHS”: “/runner/requirements_collections:~/.ansible/collections:/usr/share/ansible/collections”,
“ANSIBLE_ROLES_PATH”: “/runner/requirements_roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles”,
“PYTHONPATH”: “:/usr/local/lib/python3.8/site-packages/ansible_runner/config/…/callbacks”,
“ANSIBLE_CALLBACK_PLUGINS”: “/usr/local/lib/python3.8/site-packages/ansible_runner/config/…/callbacks”,
“ANSIBLE_STDOUT_CALLBACK”: “awx_display”,
“AWX_ISOLATED_DATA_DIR”: “/runner/artifacts/442”,
“RUNNER_OMIT_EVENTS”: “False”,
“RUNNER_ONLY_FAILED_EVENTS”: “False”},
“cwd”: “/runner/project”}