Can't run SSH against an network device (Extreme SLX switch)

HI @AWX

I am running a playbook against a network device, which is Extreme SLX switch, this playbook has been run successfully on AWX 9.0 version but I have some issues in AWX 23.0 version

With debug message from the network switch, I can see the there is no SSH request from the jump host, so the issue might be on initiating SSH session. From the logs below, I guess it might relate to SSH control master feature, but I have already disabled this feature in my ansible.cfg file.

Anyone has a suggestion or hint ?

TASK [Run show version on remote devices] **************************************
task path: /runner/project/playbooks/platform/show_version.yml:11
<test-host.test.net> local domain socket does not exist, starting it
<test-host.test.net> control socket path is /runner/.ansible/pc/09d613973f
<test-host.test.net> Loading collection ansible.builtin from 
<test-host.test.net> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<test-host.test.net> Loading collection ansible.netcommon from /runner/requirements_collections/ansible_collections/ansible/netcommon
<test-host.test.net> Loading collection ansible.utils from /runner/requirements_collections/ansible_collections/ansible/utils
<test-host.test.net> redirecting (type: terminal) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> Loading collection community.network from /runner/requirements_collections/ansible_collections/community/network
<test-host.test.net> redirecting (type: cliconf) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> local domain socket listeners started successfully

<test-host.test.net> loaded cliconf plugin ansible_collections.community.network.plugins.cliconf.slxos from path /runner/requirements_collections/ansible_collections/community/network/plugins/cliconf/slxos.py for network_os slxos
<test-host.test.net> ssh type is set to auto
<test-host.test.net> autodetecting ssh_type
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<test-host.test.net> ssh type is now set to paramiko
<test-host.test.net> Loading collection ansible.builtin from 
<test-host.test.net> local domain socket path is /runner/.ansible/pc/09d613973f
<test-host.test.net> Using network group action slxos for slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: found slxos_command  at /runner/requirements_collections/ansible_collections/community/network/plugins/modules/slxos_command.py
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: running slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, '_ansible_parsed': False, 'module_stdout': '', 'module_stderr': 'No existing session', 'msg': 'MODULE FAILURE\nSee stdout/stderr for the exact error'}

fatal: [test-host.test.net]: FAILED! => {
    "changed": false,
    "module_stderr": "No existing session",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}
...ignoring

1 Like

@mapleos1123 could I ask that you don’t immediately tag the AWX group in every first post? I can assure you they’re looking at the awx tag too (which I have added to your topic now) and it’s less noisy for them. The group mentions are intended for when we need to see if something needs urgent attention, not first-round support.

Please use the tags & thanks for understanding!

2 Likes

hi @gwmngilfen thanks and your suggestion is noted, excuse for the noise

Hi @mapleos1123, I am not familiar with AWX 9.x but I think that’s before we moved to using Execution Environments for running playbooks

required modules for ansible_collections.community.network.plugins.cliconf.slxos may not be installed by default in the awx-ee image and in that case u will need to create a custom EE image

[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<test-host.test.net> ssh type is now set to paramiko

and a rough google search yield

here’s the getting started page for how to create your custom EE image and add python packages that your ansible module might need

https://docs.ansible.com/ansible/devel/getting_started_ee/index.html

1 Like

hi @TheRealHaoLiu
thanks for your suggestion

but i think i don’t need to install a customised EE image as in this thread

i have followed the suggestion to create a new playbook which will help me to install the galaxy collection

from the logs i shared, i think Ansible has found the related module but it is SSH issue again who caused the failure of my playbook

and 谢谢:)

1 Like

anyone can help here ? I am getting stuck now, i think the related module is found by Ansible but the issue is SSH related

1 Like

Hi,

First off, I don’t use AWX so I might say something dumb here.

[WARNING]: ansible-pylibssh not installed, falling back to paramiko

Is there any reason you’re using paramiko instead of native OpenSSH ?
Also I don’t know what the package ansible-pylibssh is for, as I don’t have it installed on neither of my (Debian) control nodes. Again, not using AWX.

A few more questions:

  • Can you run your task in debug mode (ANSIBLE_DEBUG=1), or is it already the case for this output ? It looks kind of verbose, but I can’t tell if that’s all of it
  • Can you list all ssh related configuration in Ansible, including envvars ? General config, and also specific to this host. You mention a jump host, so we’ll need this config as well
  • Do you encounter the same behavior on other hosts (same device type / model and / or another one like a regular GNU/Linux box) ?
  • Have you already tried to login through ssh from your shell using the same parameters ? Since you’re using paramiko (which I don’t know much of), you should probably try it from a python shell / script
  • What if you connect with OpenSSH-client instead (from your shell) ?
  • You mention that connection works on AWX 9 but not 23; are both of these version running on the same machine ? If not, have you check firewall rules (control node, jump box and target) ?

Also, I just found this reddit thread, which mentions a similar issue, resolved by setting an higher timeout value for connection. Might worth a try !

2 Likes

@ptn
Thanks a lot for your suggestion,

  1. i have attached all debug logs as below
ansible-playbook [core 2.15.5]
  config file = /runner/project/ansible.cfg
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.17 (main, Aug  9 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
Using /runner/project/ansible.cfg as config file
SSH password: 
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading collection community.network from /runner/requirements_collections/ansible_collections/community/network
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible_runner/display_callback/callback/awx_display.py
Attempting to use 'awx_display' callback.
Skipping callback 'awx_display', as we already have a stdout callback.
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: show_version.yml *****************************************************
Positional arguments: playbooks/platform/show_version.yml
verbosity: 5
remote_user: svc_opstools
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory/hosts',)
subset: test-host.test.net
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in playbooks/platform/show_version.yml

PLAY [OpsTools - Show version] *************************************************

TASK [Run show version on remote devices] **************************************
task path: /runner/project/playbooks/platform/show_version.yml:11
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
Loading collection ansible.netcommon from /runner/requirements_collections/ansible_collections/ansible/netcommon
Loading collection ansible.utils from /runner/requirements_collections/ansible_collections/ansible/utils
redirecting (type: terminal) ansible.builtin.slxos to community.network.slxos
redirecting (type: cliconf) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> attempting to start connection
<test-host.test.net> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /usr/local/bin/ansible-connection
<test-host.test.net> local domain socket does not exist, starting it
<test-host.test.net> control socket path is /runner/.ansible/pc/667920a644
<test-host.test.net> Loading collection ansible.builtin from 
<test-host.test.net> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<test-host.test.net> Loading collection ansible.netcommon from /runner/requirements_collections/ansible_collections/ansible/netcommon
<test-host.test.net> Loading collection ansible.utils from /runner/requirements_collections/ansible_collections/ansible/utils
<test-host.test.net> redirecting (type: terminal) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> Loading collection community.network from /runner/requirements_collections/ansible_collections/community/network
<test-host.test.net> redirecting (type: cliconf) ansible.builtin.slxos to community.network.slxos
<test-host.test.net> local domain socket listeners started successfully
<test-host.test.net> loaded cliconf plugin ansible_collections.community.network.plugins.cliconf.slxos from path /runner/requirements_collections/ansible_collections/community/network/plugins/cliconf/slxos.py for network_os slxos
<test-host.test.net> ssh type is set to auto
<test-host.test.net> autodetecting ssh_type
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
<test-host.test.net> ssh type is now set to paramiko
<test-host.test.net> Loading collection ansible.builtin from 
<test-host.test.net> local domain socket path is /runner/.ansible/pc/667920a644
<test-host.test.net> Using network group action slxos for slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: found slxos_command  at /runner/requirements_collections/ansible_collections/community/network/plugins/modules/slxos_command.py
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: running slxos_command
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<test-host.test.net> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, '_ansible_parsed': False, 'module_stdout': '', 'module_stderr': 'No existing session', 'msg': 'MODULE FAILURE\\nSee stdout/stderr for the exact error'}
fatal: [test-host.test.net]: FAILED! => {
    "changed": false,
    "module_stderr": "No existing session",
    "module_stdout": "",
    "msg": "MODULE FAILURE\\nSee stdout/stderr for the exact error"
}
...ignoring

TASK [Results [SLX]] ***********************************************************
task path: /runner/project/playbooks/platform/show_version.yml:21
fatal: [test-host.test.net]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stdout_lines'. 'dict object' has no attribute 'stdout_lines'\\n\\nThe error appears to be in '/runner/project/playbooks/platform/show_version.yml': line 21, column 7, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n    - name: Results [SLX]\\n      ^ here\\n"
}

PLAY RECAP *********************************************************************
test-host.test.net : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=1   


  1. General config in my Ansible
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null

Host *
  ProxyCommand ssh -W %h:%p noc@jump.test.net
  User noc
  # point to the local authorized key
  IdentityFile ~/projects/.ssh/id_ed25519

Host jump.test.net
  Hostname jump.test.net
  User noc
  # point to the local authorized key
  IdentityFile ~/projects/.ssh/id_ed25519
#   ControlMaster auto
#   ControlPath ~/.ssh/%r@%h:%p
#   ControlPersist 5m

variables for this specific host

{
  "ansible_command_timeout": 300,
  "ansible_connection": "ssh",
  "ansible_host_key_checking": false,
  "ansible_persistent_command_timeout": 300,
  "ansible_python_interpreter": "/usr/bin/python",
  "ansible_ssh_common_args": "-o ProxyCommand=\"ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -W %h:%p -q noc@jump.test.net\" -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no",
  "timezone": "GMT+02"
}
  1. I only have this behaviour for Extreme network devices, for all my other Ansible playbooks, they are all Linux hosts, which are running smoothly

  2. TBH, I don’t know how it come to use paramiko, I have not configured it anywhere.

  3. I can use OpenSSH client to log in this device

  4. My AWX 9.0 version and AWX 23.0 version are running on different machines

1 Like
The error appears to be in '/runner/project/playbooks/platform/show_version.yml': line 21, column 7, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n\\n    - name: Results [SLX]\\n      ^ here

can you show this part of your playbook, it says you have syntax error

@fosterseth
Thanks for your reply, I don’t think there is any syntax error here, also more importantly I think the issue is related to SSH login, as I can’t even see from my network switch there is any SSH connection initiated.

My playbooks is

- name: OpsTools - Show version
  hosts: PE, P
  gather_facts: no
  connection: network_cli
  collections:
    - community.network
  tasks:
    - block:
        - name: Run show version on remote devices
          slxos_command:
            commands: show version
          when:
            - (inventory_hostname in groups['SLX'])
          changed_when: false
          ignore_errors: true
          no_log: false
          register: output_slx
    
        - name: Results [SLX]
          debug:
            msg: "{{ output_slx.stdout_lines[0] }}"
          when: output_slx.stdout_lines[0] is defined
    
        - name: show version [MLX]
          ironware_command:
            commands: show version
          when:
            - (inventory_hostname in groups['MLX'])
          changed_when: false
          ignore_errors: true
          no_log: true
          register: output_mlx
    
        - name: Results [MLX]
          debug:
            msg: "{{ output_mlx.stdout_lines[0] }}"
          when: output_mlx.stdout_lines[0] is defined

My first thought was you don’t have your ansible_network_os set… but I saw this from the log->

<test-host.test.net> loaded cliconf plugin ansible_collections.community.network.plugins.cliconf.slxos from path /runner/requirements_collections/ansible_collections/community/network/plugins/cliconf/slxos.py for network_os slxos

I see that the slxos is set: Platform Options — Ansible Documentation. and there “should” be slxos content

What level of debug do you have on there? The error is very strange to me… back to an old issue from 2016: Can't use Ansible ios_ core modules to manage Cisco Catalyst 3750-E with public key provided by ssh agent · Issue #16017 · ansible/ansible · GitHub

“No existing session”

Is it possible to install ansible-pylibssh? I have a feeling this is paramiko related.

1 Like

hi thanks for the reply, I have level 5 (the most comprehensive level) of debugging

I am not sure how should I install ansible-pylibssh ? I have no configured anywhere in my Ansible configuration to use Paramiko at all. I just installed AWX and used the .ssh config file which I shared above

The issue should related to SSH only, as I did not see any SSH related log in my network switch when running this playbook

Hello,

In regards of;

You can install pip packages on AWX in runtime using this module:

https://docs.ansible.com/ansible/latest/collections/ansible/builtin/pip_module.html

So, you may add this task at the beginning of your playbook (maybe a pre-task would fit better):

- name: Install python package
  ansible.builtin.pip:
    name: ansible-pylibssh

Not a very efficient way to add pip dependencies on a running AWX EE though, but good enough for debugging purposes anyway

1 Like

Your task seems to be failing due to the import_modules option, implicitly set to true. I don’t know why though. Here’s what it says:

Reduce CPU usage and network module execution time by enabling direct execution. Instead of the module being packaged and executed by the shell, it will be directly executed by the Ansible control node using the same python interpreter as the Ansible process. Note- Incompatible with asynchronous mode. Note- Python 3 and Ansible 2.9.16 or greater required. Note- With Ansible 2.9.x fully qualified modules names are required in tasks.

You could always try to explicitly setting it to false, see if it changes anything.

Now I think your issue comes from paramiko as well. Network_cli_connection module requirements states that:

The below requirements are needed on the local controller node that executes this connection.

  • ansible-pylibssh if using ssh_type=libssh

ssh_type is by default on auto and we can see in verbose trace it fallback on paramiko because you don’t have ansible-pylibssh installed on your control node. That checks out.
Depending on how you are using Ansible, there are multiple ways to install this package; now it seems you are using an Execution Environment, and I’m not sure what would be the more appropriate way to do so in this context. See @jbericat suggestion here.

One last thing; I don’t think paramiko uses traditional ssh config file. You have ansible_ssh_common_args defined in your Ansible config, which seems to work with paramiko but you don’t pass your IdentityFile path in here (or anywhere else from config you showed), so I’m not sure the connection plugin you’re using can actually login on your bastion. I might be missing something though.

1 Like

This package have to be installed on your control node for it to be able to use ssh connection type instead of paramiko, not the one you’re trying to reach.

I don’t know how you added the task, but you can either use delegate_to, local_action or add another play with connection: local, targeting localhost. See Controlling where tasks run: delegation and local actions — Ansible Documentation.

Or just <yourPossibleVenvPath>/pip install --user.... Not sure how EE work.

1 Like

@ptn

thanks a lot for your suggestion as well

a stupid question, maybe, I am not sure where I shall configure this setting, i have configured in ‘Variables’ of this inventory as below

ansible_persistent_command_timeout: 300
import_modules: false

But it seems still not working, even if i tried to install ‘ansible-pylibssh’ in the playbook as @jbericat suggested

I have in my ssh.cfg file to use SSH key in my AWX task container to log into the SSH jump host

Host *
  ProxyCommand ssh -W %h:%p noc@jump.test.net
  User noc
  # point to the local authorized key
  IdentityFile ~/projects/.ssh/id_ed25519

Host jump.test.net
  Hostname jump.test.net
  User noc
  # point to the local authorized key
  IdentityFile ~/projects/.ssh/id_ed25519

And I have configured in AWX for this playbook, the Credential of user-name and password is used

hey there,

As @ptn pointed-out, I might have overlooked that it must be installed on the controller, but on the managed nodes. Try it this way:

- name: Install python package
  ansible.builtin.pip:
    name: ansible-pylibssh
  delegate_to: localhost
2 Likes

Ok, let me correct a few things first:

ansible_persistent_command_timeout: 300 # This key doesn't exists, you either use envvar ANSIBLE_PERSISTENT_COMMAND_TIMEOUT or command_timeout key (under [persistent_connection] section from ansible.cfg); see: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#persistent-command-timeout
import_modules: false # Ensure this key is either set up in [ansible_network] section (ansible.cfg), ANSIBLE_NETWORK_IMPORT_MODULES envvar or replace it with ansible_network_import_modules to use as a var. See: https://docs.ansible.com/ansible/latest/collections/ansible/netcommon/network_cli_connection.html#parameter-import_modules

But it seems still not working, even if i tried to install ‘ansible-pylibssh’ in the playbook as @jbericat suggested

To be installed on your control node. See my last post (or further @jbericat precision right above).

I have in my ssh.cfg file to use SSH key in my AWX task container to log into the SSH jump host

Yeah, I’ve seen that from a previous post. What I tried to explain is that I don’t think this file is read if you use paramiko, so either install missing package on your control node to use ssh connection (through OpenSSH), or define your IdentityFile path in ansible_ssh_common_args.

1 Like

@jbericat

yes i noticed from the logs by checking more closely as well, if add into the playbook, it is trying to add this package on my remote host my playbook is running agaist

but then how can i get it installed into the AWX controller ?

i am trying to run this from AWX task container, but i don’t even know the password of awx user

bash-5.1$ pip3 install ansible-pylibssh
1 Like

It is to be installed on the node running Ansible commands / playbooks. If you run these from a container (Execution Environment or else -I don’t know hat you can do with AWX-), then this package have to be added to your image (or hot-installed in running container).

Edit:

i am trying to run this from AWX task container, but i don’t even know the password of awx user

Assuming this is the container running Ansible commands, you could run docker exec -u root -it... to exec interactive command as root user, though it won’t work if you’re running rootless containers (I think ?), and I’m not sure of syntax if you use podman as your container runtime. You probably should rebuild your image with added package if possible.

1 Like