NXOS "Failed to authenticate: Authentication failed."

Get Help
1

Over the last few days I’ve been troubleshooting a malfunctioning series of playbooks.

I’ve gone from a set of functional NXOS playbooks, our app team updated Rundeck//Ansible and many underlying modules, and whammo here now I sit holding the shattered parts trying to kick it back to life.

playbook:

- hosts: sansw01
  name: "Backup sansw01"
  connection: network_cli
  collections:
    - cisco.nxos
  gather_facts: no
  vars:
    host: sansw01
    iusername: ansibleuser
    ipassword: "{{ ansible_ssh_pass }}"
    transport: cli
    ansible_network_os: nxos
  tasks:
    - name: Include variables
      include_vars: '{{ item }}'
      loop:
        - /mnt/UnixAppRepo/ansible/storage/vault/mds/dc/sansw01_creds_vault.yml


    - name: Backup running config
      cisco.nxos.nxos_command:
        commands:
          - command: "copy running-config ftp://username@10.2.1.30/$(SWITCHNAME)/$(SWITCHNAME)-$(TIMESTAMP)-running-config.backup"
            prompt: 'Password'
            answer: "{{ answer }}"

    - name: Backup startup config
      cisco.nxos.nxos_command:
        commands:
          - command: "copy running-config ftp://username@10.2.1.30/$(SWITCHNAME)/$(SWITCHNAME)-$(TIMESTAMP)-startup-config.backup"
            prompt: 'Password'
            answer: "{{ answer }}"

Errors

[sansw01] => (item=/mnt/AppRepo/ansible/storage/vault/mds/dc/sansw01_creds_vault.yml) => {
    "ansible_facts": {
        "ansible_network_os": "nxos",
        "ansible_ssh_pass": "demo",
        "ansible_ssh_pass2": "demo2",
        "ansible_user": "ansibleuser",
        "answer": "ftpreplypassword"
    },
    "ansible_included_var_files": [
        "/mnt/AppRepo/ansible/storage/vault/mds/dc/sansw01_creds_vault.yml"
    ],
    "ansible_loop_var": "item",
    "changed": false,
    "item": "/mnt/AppRepo/ansible/storage/vault/mds/dc/sansw01_creds_vault.yml"
}
ASK [Backup running config] ***************************************************
task path: /mnt/AppRepo/ansible/storage/playbooks/cisco/mds/generic/mds_switch_backups.yml:21
redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
Loading collection ansible.netcommon from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/netcommon
Loading collection ansible.utils from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/utils
redirecting (type: terminal) ansible.builtin.nxos to cisco.nxos.nxos
redirecting (type: cliconf) ansible.builtin.nxos to cisco.nxos.nxos
Found a vault_id (internal-encrypt) in the vaulttext
We have a secret associated with vault id (internal-encrypt), will try to use to decrypt None
Trying to use vault secret=(ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt')) id=internal-encrypt to decrypt None
Trying secret ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt') for vault_id=internal-encrypt
Decrypt successful with secret=ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt') and vault_id=internal-encrypt
Found a vault_id (internal-encrypt) in the vaulttext
We have a secret associated with vault id (internal-encrypt), will try to use to decrypt None
Trying to use vault secret=(ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt')) id=internal-encrypt to decrypt None
Trying secret ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt') for vault_id=internal-encrypt
Decrypt successful with secret=ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt') and vault_id=internal-encrypt
Found a vault_id (internal-encrypt) in the vaulttext
We have a secret associated with vault id (internal-encrypt), will try to use to decrypt None
Trying to use vault secret=(ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt')) id=internal-encrypt to decrypt None
Trying secret ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt') for vault_id=internal-encrypt
Decrypt successful with secret=ClientScriptVaultSecret(filename='/tmp/rundeck/ansible-runner918112824797536201ansible-script-vault-client.py', vault_id='internal-encrypt') and vault_id=internal-encrypt
<sansw01> Using network group action cisco.nxos.nxos for cisco.nxos.nxos_command
<sansw01> attempting to start connection
<sansw01> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /var/lib/rundeck/.local/bin/ansible-connection
<sansw01> local domain socket does not exist, starting it
<sansw01> control socket path is /var/lib/rundeck/.ansible/pc/63b89f47fe
<sansw01> Loading collection ansible.builtin from 
<sansw01> redirecting (type: connection) ansible.builtin.network_cli to ansible.netcommon.network_cli
<sansw01> Loading collection ansible.netcommon from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/netcommon
<sansw01> Loading collection ansible.utils from /mnt/AppRepo/ansible/collections/ansible_collections/ansible/utils
<sansw01> redirecting (type: terminal) ansible.builtin.nxos to cisco.nxos.nxos
<sansw01> Loading collection cisco.nxos from /mnt/AppRepo/ansible/collections/ansible_collections/cisco/nxos
<sansw01> redirecting (type: cliconf) ansible.builtin.nxos to cisco.nxos.nxos
<sansw01> local domain socket listeners started successfully
<sansw01> loaded cliconf plugin ansible_collections.cisco.nxos.plugins.cliconf.nxos from path /mnt/AppRepo/ansible/collections/ansible_collections/cisco/nxos/plugins/cliconf/nxos.py for network_os nxos
<sansw01> ssh type is set to auto
<sansw01> autodetecting ssh_type
<sansw01> ssh type is now set to paramiko
<sansw01> Loading collection ansible.builtin from 
<sansw01> local domain socket path is /var/lib/rundeck/.ansible/pc/63b89f47fe
<sansw01> ANSIBLE_NETWORK_IMPORT_MODULES: enabled
<sansw01> ANSIBLE_NETWORK_IMPORT_MODULES: found cisco.nxos.nxos_command  at /mnt/AppRepo/ansible/collections/ansible_collections/cisco/nxos/plugins/modules/nxos_command.py
<sansw01> ANSIBLE_NETWORK_IMPORT_MODULES: running cisco.nxos.nxos_command
<sansw01> ANSIBLE_NETWORK_IMPORT_MODULES: complete
<sansw01> ANSIBLE_NETWORK_IMPORT_MODULES: Result: {'failed': True, '_ansible_parsed': False, 'module_stdout': '', 'module_stderr': 'Failed to authenticate: Authentication failed.', 'msg': 'MODULE FAILURE\nSee stdout/stderr for the exact error'}
fatal: [sansw01]: FAILED! => {
    "changed": false,
    "module_stderr": "Failed to authenticate: Authentication failed.",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}
PLAY RECAP *********************************************************************
sansw01                : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
Failed: AnsibleNonZero: ERROR: Ansible execution returned with non zero code.
[workflow] finishExecuteNodeStep(sansw01): NodeDispatch: AnsibleNonZero: ERROR: Ansible execution returned with non zero code.

Some things have been redacted, obviously, but the gist is above.

I know the username//password are good, I can SSH from the worker-nodes that execute the job itself, I’ve been jockying with this for the last 24ish hours and am at wits end.

2

Update:

Mucked most of the day with this in some way.
Playbook has stayed the same, but ansible is now running the follow

HOST~ $ ansible --version
ansible [core 2.18.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/user/.local/lib/python3.11/site-packages/ansible
  ansible collection location = /mnt/AppRepo/ansible/collections/ansible_collections
  executable location = /home/user/.local/bin/ansible
  python version = 3.11.10 (main, Sep 24 2024, 09:33:51) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python)
  jinja version = 3.1.4
  libyaml = True

Error has (d)evolved into 
TASK [Backup running config] *********************************************************************************************************************************************************************************
task path: /mnt/AppRepo/ansible/storage/playbooks/cisco/mds/generic/mds_switch_backups.yml:25
redirecting (type: terminal) ansible.builtin.nxos to cisco.nxos.nxos
redirecting (type: cliconf) ansible.builtin.nxos to cisco.nxos.nxos
[WARNING]: ansible-pylibssh not installed, falling back to paramiko
fatal: [sansw01.domain.net]: FAILED! => {
    "changed": false,
    "module_stderr": "Failed to authenticate: Authentication failed.",
    "module_stdout": "",
    "msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error"
}
3

Further progress, but I’m very much at an impasse here

fatal: [sansw01.DOMAIN.net]: FAILED! => {
    "changed": false,
    "module_stderr": "ssh connection failed: Failed to authenticate public key: Access denied for 'keyboard interactive'. Authentication that can continue: publickey,password,keyboard-interactive",
    "module_stdout": "",
    "msg": "MODULE FAILURE: No start of json char found\nSee stdout/stderr for the exact error"
}