I am running a playbook against a network device, which is Extreme SLX switch, this playbook has been run successfully on AWX 9.0 version but I have some issues in AWX 23.0 version
With debug message from the network switch, I can see the there is no SSH request from the jump host, so the issue might be on initiating SSH session. From the logs below, I guess it might relate to SSH control master feature, but I have already disabled this feature in my ansible.cfg file.
Anyone has a suggestion or hint ?
TASK [Run show version on remote devices] **************************************
task path: /runner/project/playbooks/platform/show_version.yml:11
<test-host.test.net> local domain socket does not exist, starting it
<test-host.test.net> control socket path is /runner/.ansible/pc/09d613973f
<test-host.test.net> Loading collection ansible.builtin from
<test-host.test.net> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<test-host.test.net> Loading collection ansible.netcommon from /runner/requirements_collections/ansible_collections/ansible/netcommon
<test-host.test.net> Loading collection ansible.utils from /runner/requirements_collections/ansible_collections/ansible/utils
<test-host.test.net> redirecting (type: terminal) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> Loading collection community.network from /runner/requirements_collections/ansible_collections/community/network
<test-host.test.net> redirecting (type: cliconf) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> local domain socket listeners started successfully
<test-host.test.net> loaded cliconf plugin ansible_collections.community.network.plugins.cliconf.slxos from path /runner/requirements_collections/ansible_collections/community/network/plugins/cliconf/slxos.py for network_os slxos
<test-host.test.net> ssh type is set to auto
<test-host.test.net> autodetecting ssh_type
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<test-host.test.net> ssh type is now set to paramiko
<test-host.test.net> Loading collection ansible.builtin from
<test-host.test.net> local domain socket path is /runner/.ansible/pc/09d613973f
<test-host.test.net> Using network group action slxos for slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: found slxos_command at /runner/requirements_collections/ansible_collections/community/network/plugins/modules/slxos_command.py
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: running slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, '_ansible_parsed': False, 'module_stdout': '', 'module_stderr': 'No existing session', 'msg': 'MODULE FAILURE\nSee stdout/stderr for the exact error'}
fatal: [test-host.test.net]: FAILED! => {
"changed": false,
"module_stderr": "No existing session",
"module_stdout": "",
"msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}
...ignoring
@mapleos1123 could I ask that you donât immediately tag the AWX group in every first post? I can assure you theyâre looking at the awx tag too (which I have added to your topic now) and itâs less noisy for them. The group mentions are intended for when we need to see if something needs urgent attention, not first-round support.
Hi @mapleos1123, I am not familiar with AWX 9.x but I think thatâs before we moved to using Execution Environments for running playbooks
required modules for ansible_collections.community.network.plugins.cliconf.slxos may not be installed by default in the awx-ee image and in that case u will need to create a custom EE image
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<test-host.test.net> ssh type is now set to paramiko
and a rough google search yield
hereâs the getting started page for how to create your custom EE image and add python packages that your ansible module might need
First off, I donât use AWX so I might say something dumb here.
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
Is there any reason youâre using paramiko instead of native OpenSSH ?
Also I donât know what the package ansible-pylibssh is for, as I donât have it installed on neither of my (Debian) control nodes. Again, not using AWX.
A few more questions:
Can you run your task in debug mode (ANSIBLE_DEBUG=1), or is it already the case for this output ? It looks kind of verbose, but I canât tell if thatâs all of it
Can you list all ssh related configuration in Ansible, including envvars ? General config, and also specific to this host. You mention a jump host, so weâll need this config as well
Do you encounter the same behavior on other hosts (same device type / model and / or another one like a regular GNU/Linux box) ?
Have you already tried to login through ssh from your shell using the same parameters ? Since youâre using paramiko (which I donât know much of), you should probably try it from a python shell / script
What if you connect with OpenSSH-client instead (from your shell) ?
You mention that connection works on AWX 9 but not 23; are both of these version running on the same machine ? If not, have you check firewall rules (control node, jump box and target) ?
Also, I just found this reddit thread, which mentions a similar issue, resolved by setting an higher timeout value for connection. Might worth a try !
ansible-playbook [core 2.15.5]
config file = /runner/project/ansible.cfg
configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible-playbook
python version = 3.9.17 (main, Aug 9 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True
Using /runner/project/ansible.cfg as config file
SSH password:
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading collection community.network from /runner/requirements_collections/ansible_collections/community/network
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible_runner/display_callback/callback/awx_display.py
Attempting to use 'awx_display' callback.
Skipping callback 'awx_display', as we already have a stdout callback.
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.
PLAYBOOK: show_version.yml *****************************************************
Positional arguments: playbooks/platform/show_version.yml
verbosity: 5
remote_user: svc_opstools
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory/hosts',)
subset: test-host.test.net
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in playbooks/platform/show_version.yml
PLAY [OpsTools - Show version] *************************************************
TASK [Run show version on remote devices] **************************************
task path: /runner/project/playbooks/platform/show_version.yml:11
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
Loading collection ansible.netcommon from /runner/requirements_collections/ansible_collections/ansible/netcommon
Loading collection ansible.utils from /runner/requirements_collections/ansible_collections/ansible/utils
redirecting (type: terminal) ansible.builtin.slxos to community.network.slxos
redirecting (type: cliconf) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> attempting to start connection
<test-host.test.net> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /usr/local/bin/ansible-connection
<test-host.test.net> local domain socket does not exist, starting it
<test-host.test.net> control socket path is /runner/.ansible/pc/667920a644
<test-host.test.net> Loading collection ansible.builtin from
<test-host.test.net> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<test-host.test.net> Loading collection ansible.netcommon from /runner/requirements_collections/ansible_collections/ansible/netcommon
<test-host.test.net> Loading collection ansible.utils from /runner/requirements_collections/ansible_collections/ansible/utils
<test-host.test.net> redirecting (type: terminal) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> Loading collection community.network from /runner/requirements_collections/ansible_collections/community/network
<test-host.test.net> redirecting (type: cliconf) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> local domain socket listeners started successfully
<test-host.test.net> loaded cliconf plugin ansible_collections.community.network.plugins.cliconf.slxos from path /runner/requirements_collections/ansible_collections/community/network/plugins/cliconf/slxos.py for network_os slxos
<test-host.test.net> ssh type is set to auto
<test-host.test.net> autodetecting ssh_type
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<test-host.test.net> ssh type is now set to paramiko
<test-host.test.net> Loading collection ansible.builtin from
<test-host.test.net> local domain socket path is /runner/.ansible/pc/667920a644
<test-host.test.net> Using network group action slxos for slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: found slxos_command at /runner/requirements_collections/ansible_collections/community/network/plugins/modules/slxos_command.py
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: running slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, '_ansible_parsed': False, 'module_stdout': '', 'module_stderr': 'No existing session', 'msg': 'MODULE FAILURE\\nSee stdout/stderr for the exact error'}
fatal: [test-host.test.net]: FAILED! => {
"changed": false,
"module_stderr": "No existing session",
"module_stdout": "",
"msg": "MODULE FAILURE\\nSee stdout/stderr for the exact error"
}
...ignoring
TASK [Results [SLX]] ***********************************************************
task path: /runner/project/playbooks/platform/show_version.yml:21
fatal: [test-host.test.net]: FAILED! => {
"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout_lines'. 'dict object' has no attribute 'stdout_lines'\\n\\nThe error appears to be in '/runner/project/playbooks/platform/show_version.yml': line 21, column 7, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n - name: Results [SLX]\\n ^ here\\n"
}
PLAY RECAP *********************************************************************
test-host.test.net : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=1
General config in my Ansible
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
Host *
ProxyCommand ssh -W %h:%p noc@jump.test.net
User noc
# point to the local authorized key
IdentityFile ~/projects/.ssh/id_ed25519
Host jump.test.net
Hostname jump.test.net
User noc
# point to the local authorized key
IdentityFile ~/projects/.ssh/id_ed25519
# ControlMaster auto
# ControlPath ~/.ssh/%r@%h:%p
# ControlPersist 5m
The error appears to be in '/runner/project/playbooks/platform/show_version.yml': line 21, column 7, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n - name: Results [SLX]\\n ^ here
can you show this part of your playbook, it says you have syntax error
@fosterseth
Thanks for your reply, I donât think there is any syntax error here, also more importantly I think the issue is related to SSH login, as I canât even see from my network switch there is any SSH connection initiated.
My playbooks is
- name: OpsTools - Show version
hosts: PE, P
gather_facts: no
connection: network_cli
collections:
- community.network
tasks:
- block:
- name: Run show version on remote devices
slxos_command:
commands: show version
when:
- (inventory_hostname in groups['SLX'])
changed_when: false
ignore_errors: true
no_log: false
register: output_slx
- name: Results [SLX]
debug:
msg: "{{ output_slx.stdout_lines[0] }}"
when: output_slx.stdout_lines[0] is defined
- name: show version [MLX]
ironware_command:
commands: show version
when:
- (inventory_hostname in groups['MLX'])
changed_when: false
ignore_errors: true
no_log: true
register: output_mlx
- name: Results [MLX]
debug:
msg: "{{ output_mlx.stdout_lines[0] }}"
when: output_mlx.stdout_lines[0] is defined
hi thanks for the reply, I have level 5 (the most comprehensive level) of debugging
I am not sure how should I install ansible-pylibssh ? I have no configured anywhere in my Ansible configuration to use Paramiko at all. I just installed AWX and used the .ssh config file which I shared above
The issue should related to SSH only, as I did not see any SSH related log in my network switch when running this playbook
Your task seems to be failing due to the import_modules option, implicitly set to true. I donât know why though. Hereâs what it says:
Reduce CPU usage and network module execution time by enabling direct execution. Instead of the module being packaged and executed by the shell, it will be directly executed by the Ansible control node using the same python interpreter as the Ansible process. Note- Incompatible with asynchronous mode. Note- Python 3 and Ansible 2.9.16 or greater required. Note- With Ansible 2.9.x fully qualified modules names are required in tasks.
You could always try to explicitly setting it to false, see if it changes anything.
The below requirements are needed on the local controller node that executes this connection.
ansible-pylibssh if using ssh_type=libssh
ssh_type is by default on auto and we can see in verbose trace it fallback on paramiko because you donât have ansible-pylibssh installed on your control node. That checks out.
Depending on how you are using Ansible, there are multiple ways to install this package; now it seems you are using an Execution Environment, and Iâm not sure what would be the more appropriate way to do so in this context. See @jbericat suggestion here.
One last thing; I donât think paramiko uses traditional ssh config file. You have ansible_ssh_common_args defined in your Ansible config, which seems to work with paramiko but you donât pass your IdentityFile path in here (or anywhere else from config you showed), so Iâm not sure the connection plugin youâre using can actually login on your bastion. I might be missing something though.
This package have to be installed on your control node for it to be able to use ssh connection type instead of paramiko, not the one youâre trying to reach.
But it seems still not working, even if i tried to install âansible-pylibsshâ in the playbook as @jbericat suggested
I have in my ssh.cfg file to use SSH key in my AWX task container to log into the SSH jump host
Host *
ProxyCommand ssh -W %h:%p noc@jump.test.net
User noc
# point to the local authorized key
IdentityFile ~/projects/.ssh/id_ed25519
Host jump.test.net
Hostname jump.test.net
User noc
# point to the local authorized key
IdentityFile ~/projects/.ssh/id_ed25519
And I have configured in AWX for this playbook, the Credential of user-name and password is used
ansible_persistent_command_timeout: 300 # This key doesn't exists, you either use envvar ANSIBLE_PERSISTENT_COMMAND_TIMEOUT or command_timeout key (under [persistent_connection] section from ansible.cfg); see: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#persistent-command-timeout
import_modules: false # Ensure this key is either set up in [ansible_network] section (ansible.cfg), ANSIBLE_NETWORK_IMPORT_MODULES envvar or replace it with ansible_network_import_modules to use as a var. See: https://docs.ansible.com/ansible/latest/collections/ansible/netcommon/network_cli_connection.html#parameter-import_modules
But it seems still not working, even if i tried to install âansible-pylibsshâ in the playbook as @jbericat suggested
To be installed on your control node. See my last post (or further @jbericat precision right above).
I have in my ssh.cfg file to use SSH key in my AWX task container to log into the SSH jump host
Yeah, Iâve seen that from a previous post. What I tried to explain is that I donât think this file is read if you use paramiko, so either install missing package on your control node to use ssh connection (through OpenSSH), or define your IdentityFile path in ansible_ssh_common_args.
yes i noticed from the logs by checking more closely as well, if add into the playbook, it is trying to add this package on my remote host my playbook is running agaist
but then how can i get it installed into the AWX controller ?
i am trying to run this from AWX task container, but i donât even know the password of awx user
It is to be installed on the node running Ansible commands / playbooks. If you run these from a container (Execution Environment or else -I donât know hat you can do with AWX-), then this package have to be added to your image (or hot-installed in running container).
Edit:
i am trying to run this from AWX task container, but i donât even know the password of awx user
Assuming this is the container running Ansible commands, you could run docker exec -u root -it... to exec interactive command as root user, though it wonât work if youâre running rootless containers (I think ?), and Iâm not sure of syntax if you use podman as your container runtime. You probably should rebuild your image with added package if possible.