Ansible can no longer connect via SSH

Archives Ansible Project
1

Hello,
I have a pretty simple playbook that has been working fine for a while. It appears it can no longer connect via ssh to the device. Note, I can manually ssh to the device with no problems. I ran the playbook with the -vvvv option and here is the part of the output where the problem is… Looks like it tries to connect multiple times but is failing with “mux_client_read_packet: read header failed: Broken pipe”. At that point, the playbook just hangs indefinitely. Note, I’m using the shell module to send a simple command and before the “shell task”, the playbook successfully copies a file to this device using the copy module. Note, this started happening with Ansible 2.7.1 and I upgrade to the latest 2.7.9 but still a problem. Also, the code on the target device has also been upgraded so maybe there is some “incompatibility”??? But again, I can ssh to the box manually (from the same box where the playbook is run from), with no issues.

Thx
Al

<wb-sm-1001> SSH: EXEC sshpass -d9 sftp -o BatchMode=no -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=vyatta -o ConnectTimeout=10 -o ControlPath=/home/as2863/.ansible/cp/244701d842 '[wb-sm-1001]' <wb-sm-1001> (0, 'sftp> put /home/as2863/.ansible/tmp/ansible-local-6627pINYLx/tmpqsV0W0 /home/vyatta/.ansible/tmp/ansible-tmp-1553789704.03-274547323725015/AnsiballZ_command.py\n', 'OpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 6467\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 6 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/vyatta size 0\r\ndebug3: Looking up /home/as2863/.ansible/tmp/ansible-local-6627pINYLx/tmpqsV0W0\r\ndebug3: Sent message fd 6 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/vyatta/.ansible/tmp/ansible-tmp-1553789704.03-274547323725015/AnsiballZ_command.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:8793\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 8793 bytes at 65536\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <wb-sm-1001> ESTABLISH SSH CONNECTION FOR USER: vyatta <wb-sm-1001> SSH: EXEC sshpass -d9 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=vyatta -o ConnectTimeout=10 -o ControlPath=/home/as2863/.ansible/cp/244701d842 wb-sm-1001 '/bin/sh -c '"'"'chmod u+x /home/vyatta/.ansible/tmp/ansible-tmp-1553789704.03-274547323725015/ /home/vyatta/.ansible/tmp/ansible-tmp-1553789704.03-274547323725015/AnsiballZ_command.py && sleep 0'"'"'' <wb-sm-1001> (0, '', 'OpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n 7 Dec 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 6467\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <wb-sm-1001> ESTABLISH SSH CONNECTION FOR USER: vyatta <wb-sm-1001> SSH: EXEC sshpass -d9 ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o User=vyatta -o ConnectTimeout=10 -o ControlPath=/home/as2863/.ansible/cp/244701d842 -tt wb-sm-1001 '/bin/sh -c '"'"'/usr/bin/python /home/vyatta/.ansible/tmp/ansible-tmp-1553789704.03-274547323725015/AnsiballZ_command.py && sleep 0'"'"''

2

Looks like it may be a different problem. I stripped the playbook down to bare bones and it appears only the specific Vyatta commands no longer work…

For example, this works (although the debug output still shows it disconnects a couple of times but ultimately it works each time:

  • name: Issue a test command
    shell: ls
    register: output

However, this does not work… but it used to work, so looks like may be more of a problem with the target device… I’ll investigate further on target device and post back.

  • name: Issue the add system image command
    shell: vcli -c ‘show version’

register: output