AWX jobs stopped to work ~10 days ago

Archives AWX Project
,
1

Hello,

I’ve a question, my AWX playbooks with modules nxos_commands and iosxr_commands started to fail ~10days ago. I’m using AWX version 20.0.1 with new custom built EE. When i’m trying to switch to and older custom EE the playbooks starts to work. The errors i’m getting is like these:

The full traceback is:
File “/runner/requirements_collections/ansible_collections/cisco/iosxr/plugins/module_utils/network/iosxr/iosxr.py”, line 122, in get_capabilities
capabilities = Connection(module._socket_path).get_capabilities()
File “/usr/local/lib/python3.9/site-packages/ansible/module_utils/connection.py”, line 200, in rpc
raise ConnectionError(to_text(msg, errors=‘surrogate_then_replace’), code=code)
fatal: [HK2ASR01]: FAILED! => {
“changed”: false,
“failed_when_result”: “The conditional check 'get_config.stdout[0] is not search(\”\\/32\“) or get_config.stdout[1] is not search(\”\\/128\“)’ failed. The error was: error while evaluating conditional (get_config.stdout[0] is not search(\”\\/32\“) or get_config.stdout[1] is not search(\”\\/128\“)): ‘dict object’ has no attribute ‘stdout’. ‘dict object’ has no attribute ‘stdout’”,
“invocation”: {
“module_args”: {
“commands”: [
“show object-group network ipv4 External_Monitoring | include /32”,
“show object-group network ipv6 External_Monitoring_IPv6 | include /128”
],
“interval”: 1,
“match”: “all”,
“retries”: 10,
“wait_for”: null
}
},
“msg”: “command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide.”

OR

ok: [TEONET01A] => {“changed”: false, “failed_when_result”: false, “msg”: “command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide.”}

seems like it’s something with Python. The older EE is using Python 3.8, and the newer one - Python 3.9. Could it be the reason ? Isn’t the EE enviroment is independent of the whole AWX ?

2

Hi,

In your newer custom EE did you change the collection version or ansible version? Are you able to provide the ansible-galaxy collection list and ansible --version from inside the EE?

Thanks,

AWX Team

3

Hello,

Thank for reply

When building an image vith ansible builder i just added the collection named:

  • name: cisco.iosxr

Now as per your requested information i see:

Newer EE when playbook fails:

“stdout”: “ansible [core 2.15.0]\n config file = None\n configured module search path = [‘/runner/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]\n ansible python module location = /usr/local/lib/python3.9/site-packages/ansible\n ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections\n executable location = /usr/local/bin/ansible\n python version = 3.9.16 (main, Dec 8 2022, 00:00:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)] (/usr/bin/python3)\n jinja version = 3.1.2\n libyaml = True”,
“stdout_lines”: [
“ansible [core 2.15.0]”,
" config file = None",
" configured module search path = [‘/runner/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]",
" ansible python module location = /usr/local/lib/python3.9/site-packages/ansible",
" ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections",
" executable location = /usr/local/bin/ansible",
" python version = 3.9.16 (main, Dec 8 2022, 00:00:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)] (/usr/bin/python3)",
" jinja version = 3.1.2",
" libyaml = True"

“stdout_lines”: [
“”,
“# /runner/requirements_collections/ansible_collections”,
“Collection Version”,
“----------------------- -------”,
"ansible.netcommon 5.1.2 ",
"ansible.utils 2.10.3 ",
"cisco.iosxr 6.0.0 ",
"community.general 7.1.0 ",
"community.mysql 3.7.2 ",
"community.proxysql 1.5.1 ",
“”,
“# /usr/share/ansible/collections/ansible_collections”,
“Collection Version”,
“----------------------- -------”,
"amazon.aws 6.0.1 ",
"ansible.netcommon 5.1.1 ",
"ansible.posix 1.5.4 ",
"ansible.utils 2.10.3 ",
"ansible.windows 1.14.0 ",
"awx.awx 22.3.0 ",
"azure.azcollection 1.15.0 ",
"cisco.iosxr 5.0.2 ",

"cisco.nxos 4.3.0 ",
"community.vmware 3.6.0 ",
"google.cloud 1.1.3 ",
"kubernetes.core 2.4.0 ",
"netbox.netbox 3.13.0 ",
"openstack.cloud 2.1.0 ",
"ovirt.ovirt 3.1.2 ",
"redhatinsights.insights 1.0.7 ",
"theforeman.foreman 3.10.0 "
]
}
}

Old EE where it works OK:

“ansible [core 2.12.5.post0]”,
" config file = None",
" configured module search path = [‘/home/runner/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]",
" ansible python module location = /usr/local/lib/python3.8/site-packages/ansible",
" ansible collection location = /runner/requirements_collections:/home/runner/.ansible/collections:/usr/share/ansible/collections",
" executable location = /usr/local/bin/ansible",
" python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]",
" jinja version = 2.11.3",
" libyaml = True"
],

“stdout_lines”: [
“”,
“# /runner/requirements_collections/ansible_collections”,
“Collection Version”,
“------------------ -------”,
"ansible.netcommon 5.1.2 ",
"ansible.utils 2.10.3 ",
"cisco.iosxr 6.0.0 ",
"community.general 7.1.0 ",
"community.mysql 3.7.2 ",
"community.proxysql 1.5.1 ",
“”,
“# /usr/share/ansible/collections/ansible_collections”,
“Collection Version”,
“----------------------- -------”,
"amazon.aws 4.1.0 ",
"ansible.posix 1.4.0 ",
"ansible.utils 2.6.1 ",
"ansible.windows 1.11.0 ",
"awx.awx 21.4.0 ",
"azure.azcollection 1.13.0 ",
"community.general 5.4.0 ",
"community.vmware 2.7.0 ",
"google.cloud 1.0.2 ",
"kubernetes.core 2.3.2 ",
"openstack.cloud 1.8.0 ",
"ovirt.ovirt 2.2.2 ",
"redhatinsights.insights 1.0.7 ",
"theforeman.foreman 3.4.0 "

Its strange that on the EE which is not working i see double entry of IOS-XR collection - maybe that could be the reason it’s not working ? I can’t imagine how it got there - and why it was working before. Also - i don’t quite sure that this could be a problem, because other playbook with cisco_nxos collection also started to fail even the entry is not doubled.

p.s - is there any easier way to get into EE for example from POD side and run these commands (ansible --version) instead of adding it to playbooks and running via AWX ?

Thank you

4

Hello,
Thank you for providing this additional information! Based on what you have provided we’d like to gather a bit more information from you. Are you building this with Ansible Builder? If so, could you please provide us with the Ansible execution environment yaml file?

Thank you for your time!

  • AWX Team
5

hello,

Today i tried to build a fresh image, i’was using git clone https://github.com/ansible/awx-ee
and my edited execution-environment.yml file looks like this: (just added some collections and Python modules)