Playbook freezes after several seconds on a task using cisco.ios to connect to Cisco Switches by using SSH in Python3.12.
I have a simple test playbook to gather facts from Cisco switches(x500, mainly Catalyst 9200) with the cisco.ios collection. In python 3.12, after some seconds on the task(several switches are answering correctly), the execution freezes without reason(without error message) and the playbook is still in running state. But it works in python 3.9. It doesn’t matter the SSH library used(libssh or paramiko)
The task is:
- name: Get Facts from Switches
cisco.ios.ios_facts:
gather_subset:
- hardware
ignore_errors: yes
register: GET_FACTS_STATUS
Collection version:
ansible.utils: 6.0.1
ansible.netcommon: 8.4.0
cisco.ios: 11.2.0
The playbook is executed from AWX with 200 forks.
detail of the 2 environments:
python 3.12
ansible 11.12.0
ansible-core 2.18.12
ansible-pylibssh 1.3.0
paramiko 4.0.0
python 3.9
ansible 8.7.0
ansible-core 2.15.8
ansible-pylibssh 1.2.2
paramiko 3.5.0
let me know if you have an idea on my issue and if you need more information
here you can find logs from debug mode but I’m pretty sure it won’t be usefull:
**[WARNING]: Persistent connection logging is enabled for A.B.C.21. This will**
**log ALL interactions to /tmp/awx/awx_196767_13r5ql5q/project/ansible.log and**
**WILL NOT redact sensitive configuration like passwords. USE WITH CAUTION!**
<A.B.C.21> ssh type is set to paramiko
<A.B.C.21> Loading collection ansible.builtin from
<A.B.C.21> local domain socket path is /var/lib/awx/.ansible/cpd/6d0d9f4419
<A.B.C.21> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
**[WARNING]: Persistent connection logging is enabled for A.B.C.110. This**
**will log ALL interactions to /tmp/awx/awx_196767_13r5ql5q/project/ansible.log**
**and WILL NOT redact sensitive configuration like passwords. USE WITH CAUTION!**
<A.B.C.110> ssh type is set to paramiko
<A.B.C.110> Loading collection ansible.builtin from
<A.B.C.110> local domain socket path is /var/lib/awx/.ansible/cpd/f191c127ba
<A.B.C.110> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
**[WARNING]: Persistent connection logging is enabled for A.B.C.11. This will**
**log ALL interactions to /tmp/awx/awx_196767_13r5ql5q/project/ansible.log and**
**WILL NOT redact sensitive configuration like passwords. USE WITH CAUTION!**
<A.B.C.11> ssh type is set to paramiko
<A.B.C.11> Loading collection ansible.builtin from
<A.B.C.11> local domain socket path is /var/lib/awx/.ansible/cpd/9d16891fad
<A.B.C.11> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
**[WARNING]: Persistent connection logging is enabled for A.B.C.91. This will**
**log ALL interactions to /tmp/awx/awx_196767_13r5ql5q/project/ansible.log and**
**WILL NOT redact sensitive configuration like passwords. USE WITH CAUTION!**
<A.B.C.91> ssh type is set to paramiko
<A.B.C.91> Loading collection ansible.builtin from
<A.B.C.91> local domain socket path is /var/lib/awx/.ansible/cpd/790594d44f
<A.B.C.91> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<A.B.C.31> local domain socket does not exist, starting it
<A.B.C.31> control socket path is /var/lib/awx/.ansible/cpd/105e4597c2
<A.B.C.31> Loading collection ansible.builtin from
<A.B.C.31> Loading collection ansible.netcommon from /tmp/awx/awx_196767_13r5ql5q/requirements_collections/ansible_collections/ansible/netcommon
<A.B.C.31> Loading collection ansible.utils from /tmp/awx/awx_196767_13r5ql5q/requirements_collections/ansible_collections/ansible/utils
<A.B.C.31> Loading collection cisco.ios from /tmp/awx/awx_196767_13r5ql5q/requirements_collections/ansible_collections/cisco/ios
<A.B.C.31> local domain socket listeners started successfully
<A.B.C.31> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /tmp/awx/awx_196767_13r5ql5q/requirements_collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os cisco.ios.ios
**[WARNING]: Persistent connection logging is enabled for A.B.C.31. This will**
**log ALL interactions to /tmp/awx/awx_196767_13r5ql5q/project/ansible.log and**
**WILL NOT redact sensitive configuration like passwords. USE WITH CAUTION!**
<A.B.C.31> ssh type is set to paramiko
<A.B.C.31> Loading collection ansible.builtin from
<A.B.C.31> local domain socket path is /var/lib/awx/.ansible/cpd/105e4597c2
Which version of AWX are you using? Does the playbook run successfully from the CLI like ansible-playbook using the environment versions which are failing when running from AWX?
If AWX runs the playbook successfully against a single target, it seems like we may have an issue related to scaling/scalability when running against a large number of targets. I see you mentioned there are ~500 targets and you are running with ~200 forks. Maybe try other options within AWX to maybe reduce the number of concurrent processes or make the exection more serialized and less parallel to see if that helps? You said running the playbook from the CLI against the same targets succeeds - is that also with 200 forks, or just using the defaults from the CLI for scalability/parallelization?
Another thing to consider is that AWX 14.1.0 was released ~August 2020 and End of Life (EOL). That release roughly corresponds to Ansible Core 2.10.x which went EOL in May of 2022.
Releases and maintenance — Ansible Community shows the General-Availability release dates and end of life dates of ansible-core which can be used to estimate support lifecycle of other things within the open-source ansible ecosystem.
If you are having the same trouble with that latest stable release, you can try submitting an issue over on the GitHub AWX Issue tracker. Or if you find a similar issue, vote that one up and add comments with specifics from your testing scenario.
Community / free support is very limited for AWX - even for the current stable release. If you need full support services, those are available with Red Hat Ansible Automation Platform (which has a commercially supported form of AWX built in).
I would like to keep a large number of fork. I do test on a basic playbook with only one task. But at the end, I have a large playbook(50 tasks) taking 6 hours to execute against those switches. It’s why I would like to reduce the execution time with a large number of forks.
Yes CLI execution was with 200 forks.
It is plannned to migrate to a new AWX version in the coming months but unfortunetely not for today :o(
Can you post your connection settings? And can you post a log using libssh instead of paramiko?
Python 3.12 should be generally be more optimized and performant over 3.9, and libssh should be 4x faster than paramiko with +200 forks. I’m curious if the logs show any kind of fall-back behavior, and you’re only seeing paramiko speeds for some reason.
<A.B.C.201> local domain socket does not exist, starting it
<A.B.C.201> control socket path is /var/lib/awx/.ansible/pc/3102d6e9d7
<A.B.C.201> Loading collection ansible.builtin from
<A.B.C.201> Loading collection ansible.netcommon from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/netcommon
<A.B.C.201> Loading collection ansible.utils from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/utils
<A.B.C.201> Loading collection cisco.ios from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios
<A.B.C.201> local domain socket listeners started successfully
<A.B.C.201> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os cisco.ios.ios
<A.B.C.161> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.161> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.161> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
<A.B.C.81> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.81> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.81> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
<A.B.C.172> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.172> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.172> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
<A.B.C.101> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.101> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.101> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
<A.B.C.13> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.13> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.29> local domain socket does not exist, starting it
<A.B.C.29> control socket path is /var/lib/awx/.ansible/pc/07c0eefd8e
<A.B.C.29> Loading collection ansible.builtin from
<A.B.C.29> Loading collection ansible.netcommon from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/netcommon
<A.B.C.29> Loading collection ansible.utils from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/utils
<A.B.C.29> Loading collection cisco.ios from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios
<A.B.C.29> local domain socket listeners started successfully
<A.B.C.29> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os cisco.ios.ios
<A.B.C.111> local domain socket does not exist, starting it
<A.B.C.111> control socket path is /var/lib/awx/.ansible/pc/b243f83b45
<A.B.C.111> Loading collection ansible.builtin from
<A.B.C.111> Loading collection ansible.netcommon from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/netcommon
<A.B.C.111> Loading collection ansible.utils from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/utils
<A.B.C.111> Loading collection cisco.ios from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios
<A.B.C.111> local domain socket listeners started successfully
<A.B.C.111> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os cisco.ios.ios
<A.B.C.21> local domain socket does not exist, starting it
<A.B.C.21> control socket path is /var/lib/awx/.ansible/pc/ab84334d46
<A.B.C.21> Loading collection ansible.builtin from
<A.B.C.21> Loading collection ansible.netcommon from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/netcommon
<A.B.C.21> Loading collection ansible.utils from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/utils
<A.B.C.21> Loading collection cisco.ios from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios
<A.B.C.21> local domain socket listeners started successfully
<A.B.C.21> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os cisco.ios.ios
<A.B.C.13> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
<A.B.C.31> local domain socket does not exist, starting it
<A.B.C.31> control socket path is /var/lib/awx/.ansible/pc/835bf59ee8
<A.B.C.31> Loading collection ansible.builtin from
<A.B.C.31> Loading collection ansible.netcommon from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/netcommon
<A.B.C.31> Loading collection ansible.utils from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/ansible/utils
<A.B.C.31> Loading collection cisco.ios from /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios
<A.B.C.31> local domain socket listeners started successfully
<A.B.C.31> loaded cliconf plugin ansible_collections.cisco.ios.plugins.cliconf.ios from path /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/cliconf/ios.py for network_os cisco.ios.ios
<A.B.C.61> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.61> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.61> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
<A.B.C.100> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.ios.ios_facts at /tmp/awx/awx_196849_28ofywyu/requirements_collections/ansible_collections/cisco/ios/plugins/modules/ios_facts.py
<A.B.C.100> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.ios.ios_facts
<A.B.C.100> ANSIBLE_NETWORK_IMPORT_MODULES: _load_params skipped for action plugin in direct execution
[WARNING]: Persistent connection logging is enabled for A.B.C.200. This
will log ALL interactions and WILL NOT redact sensitive configuration like
passwords. USE WITH CAUTION!
[WARNING]: Persistent connection logging is enabled for A.B.C.31. This will
log ALL interactions and WILL NOT redact sensitive configuration like
passwords. USE WITH CAUTION!
<A.B.C.31> ssh type is set to libssh
<A.B.C.31> Loading collection ansible.builtin from
<A.B.C.31> local domain socket path is /var/lib/awx/.ansible/pc/67ae17ac01
<A.B.C.31> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
[WARNING]: Persistent connection logging is enabled for A.B.C.201. This
will log ALL interactions and WILL NOT redact sensitive configuration like
passwords. USE WITH CAUTION!
<A.B.C.201> ssh type is set to libssh
<A.B.C.201> Loading collection ansible.builtin from
<A.B.C.201> local domain socket path is /var/lib/awx/.ansible/pc/3102d6e9d7
<A.B.C.201> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
[WARNING]: Persistent connection logging is enabled for A.B.C.29. This will
log ALL interactions and WILL NOT redact sensitive configuration like
passwords. USE WITH CAUTION!
<A.B.C.29> ssh type is set to libssh
<A.B.C.29> Loading collection ansible.builtin from
<A.B.C.29> local domain socket path is /var/lib/awx/.ansible/pc/07c0eefd8e
Regarding performance difference between paramiko and libssh, I did some tests some months ago and I got better performance with paramiko compare to libssh
Looks like you have ssh_type set to paramiko, but I’m seeing in the logs there it is trying to use “libssh”.
Are you also specifying ansible_connection=ansible.netcommon.libssh somewhere?
I would think you would want to have consistent settings that all specify the one you prefer to use.
Unfortunately, I don’t have enough experience with running so many forks or with Cisco devices to pin anything down on why newer versions of ansible/libssh/paramiko are slower than the older ones. You might compare it with what’s here:
In general, I’ve found the most significant performance improvements were increasing the socket timeouts for libssh via increasing the ControlPersists value to +600s. This is generally useful for long-running tasks or plays with high host to fork ratios, where the sockets better maintain an authenticated session with all hosts to reduce the overhead when re-authenticating later.
in fact, I’m using the variable “ansible_network_cli_ssh_type”(value: libssh or paramiko) in EXTRA VARIABLES field(AWX) to switch the connection library, I do a capture with paramiko set but for the next test, I’m setting variable to libssh(it’s why logs are showing with libssh).
I just do a test with this setting “connection: ansible.netcommon.libssh” in my playbook. initially, it was set to ansible.netcommon.network_cli
for this task:
- name: Get Facts from Switches
cisco.ios.ios_facts:
gather_subset:
- hardware
register: GET_FACTS_STATUS
I got this error:
"attempts": 1,
"changed": false,
"msg": "Connection type ansible.netcommon.libssh is not valid for this module",
"retries": 5
}
I just try also with ControlPersists value to +600s, I got the same issue(freeze after some seconds)
Unfortunately, I don’t have an environment to replicate this with, so I can’t do any testing myself, and I can’t think of anything else to ask you to try. I’m sorry I’m unable to help any further.