AUTOMATE SFTP USING PEXPECT IN YAML SCRIPT NOT WORKING...

Hi guys,
I am working on project to in which I am trying to backup some files from one server and than transfer that files to NAS terastation. So from my ansible box i do ssh into node. Then i make backup of some files at same node and create sftp session to transfer files to NAS and i am putting current day date and host name as name of files. The problem is that this (sftp ) command prompts me to enter password and to enter password i am trying to build a script which automatically enters password and continue the process. Even i have no error when i ran my playbook but i don’t get my files at tera station. But when i ran my playbook with level 1 verbose like -v it shows something like this but in yellow colour so not like error:

TASK [Creates Sftp session to send files to terastation.] ****************************************************************************************************************************************************************************************************************************
changed: [localhost] => {“changed”: true, “cmd”: “sftp user@hostname”, “delta”: “0:00:01.136744”, “end”: “2017-10-11 05:14:25.224441”, “failed”: false, “rc”: 0, “start”: “2017-10-11 05:14:24.087697”, “stdout”: “Connecting to terastation…\r\nPassword: \r\nsftp> exit”, “stdout_lines”: [“Connecting to terastation…”, "Password: ", “sftp> exit”]}

Here is sample of my script:

  • hosts: localhost
    sudo: true
    tasks:

  • name: Log into node
    shell: ssh user@hostname

  • name: make backup of files
    shell: tar cvf “$HOSTNAME”_date +"%Y-%m-%d".tar /etc/ansible/playbooks

  • name: Creates Sftp session to send files to terastation.
    expect:
    command: sftp user@hostname
    responses:
    (?i)Password: “1rfr2345”
    (?i)sftp>: “cd abc/def/”
    (?i)sftp>: "put “$HOSTNAME”_date +"%Y-%m-%d".tar "

(?i)sftp>: “exit”

If more info needed pls let me know. Thanks in advance.

TASK [Creates Sftp session to send files to terastation.]
****************************************************************************************************************************************************************************************************************************
changed: [localhost] => {"changed": true, "cmd": "sftp user@hostname",
"delta": "0:00:01.136744", "end": "2017-10-11 05:14:25.224441", "failed":
false, "rc": 0, "start": "2017-10-11 05:14:24.087697", "stdout":
"Connecting to terastation...\r\nPassword: \r\nsftp> exit", "stdout_lines":
["Connecting to terastation...", "Password: ", "sftp> exit"]}

It do login but just run and exit afterward, and exit is the last of you responses.

Here is sample of my script:

- hosts: localhost
  sudo: true
  tasks:

   - name: Log into node
     shell: ssh user@hostname

   - name: make backup of files
     shell: tar cvf "$HOSTNAME"_`date +"%Y-%m-%d"`.tar
/etc/ansible/playbooks

   - name: Creates Sftp session to send files to terastation.
     expect:
       command: sftp user@hostname
       responses:
           (?i)Password: "1rfr2345"
           (?i)sftp>: "cd abc/def/"
           (?i)sftp>: "put "$HOSTNAME"_`date +"%Y-%m-%d"`.tar "
           (?i)sftp>: "exit"

If more info needed pls let me know. Thanks in advance.

I think you only can have one unique question with the responses in a list like this.

   - name: Creates Sftp session to send files to terastation.
     expect:
       command: sftp user@hostname
       responses:
         (?i)Password: "1rfr2345"
         (?i)sftp>:
           - "cd abc/def/"
           - "put "$HOSTNAME"_`date +"%Y-%m-%d"`.tar "
           - "exit"

Thanks for help i have another question so as i am running script as localhost it is making backup on local host even though in ssh into another node i tried by combining backup and ssh process together but it didn’t worked error says no such file directory.

  • name: make backup
    shell: |
    ssh user@hostname
    tar cvf “$HOSTNAME”_date +"%Y-%m-%d".tar /opt/backup_test

error:

TASK [make backup] **************************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “ssh user@hostname \n tar cvf "$HOSTNAME"_date +\"%Y-%m-%d\".tar /opt/backup_test”, “delta”: “0:00:00.077642”, “end”: “2017-10-12 02:07:50.314514”, “failed”: true, “msg”: “non-zero return code”, “rc”: 2, “start”: “2017-10-12 02:07:50.236872”, “stderr”: “Pseudo-terminal will not be allocated because stdin is not a terminal.\r\ntar: Removing leading /' from member names\ntar: /opt/backup_test: Cannot stat: No such file or directory\ntar: Exiting with failure status due to previous errors", "stderr_lines": ["Pseudo-terminal will not be allocated because stdin is not a terminal.", "tar: Removing leading /’ from member names”, “tar: /opt/backup_test: Cannot stat: No such file or directory”, “tar: Exiting with failure status due to previous errors”], “stdout”: “”, “stdout_lines”: }

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=1

The error message really looks like you don’t have “/opt/backup_test” on your local system or possibly not writable by the account Ansible is using.

If you have Ansible 2.3 and you’re just building a tar file, why not use the “archive” module?http://docs.ansible.com/ansible/latest/archive_module.html

`

  • name: make backup using archive module
    archive:
    dest:“{{ansible_hostname}}_{{ansible_facts.date}}.tar.gz”
    path: /opt/backup_test
    format: gz
    `

DanL

Thanks buddy it helped a lot. Appreciated.

Hello Kai,
As soon as i used your solution now i am getting this error:

“msg”: “No remaining responses for ‘(?i)sftp>’, output was ‘: \r\nsftp>’”

TASK [Creates Sftp session to send files to terastation.]


changed: [localhost] => {“changed”: true, “cmd”: “sftp user@hostname”,
“delta”: “0:00:01.136744”, “end”: “2017-10-11 05:14:25.224441”,
“failed”:
false, “rc”: 0, “start”: “2017-10-11 05:14:24.087697”, “stdout”:
“Connecting to terastation…\r\nPassword: \r\nsftp> exit”,
“stdout_lines”:
[“Connecting to terastation…”, "Password: ", “sftp> exit”]}

It do login but just run and exit afterward, and exit is the last of you
responses.

my code

  • name: Creates Sftp session to send files to terastation.
    expect:
    command: ssh user@hostname
    responses:

(?i)root@": “sftp user@hostname”

Without the full verbose output it's difficult to say.

But some systems print out the prompt after exit and before the command terminates.
This will trigger this message in Ansible.
You could try adding a extra exit, if that doesn't help you can set ignore_errors: true or write a expect script and run that instead.

Hello Kai,
Solved the issue but i have to add 5 exit i don’t know why. Do you think it has to do something with number of commands as i am running 5 commands. new code:

  • name: crates sftp session to terastation
    expect:
    command: ssh user@host

responses:
(?i)root@: [‘sftp user@host’, ‘mkdir /tmp/backups’, ‘mv lst*tar /tmp/backups’, ‘cd /tmp/backups’, ‘rm -rf $(ls -1t /tmp/backups | tail -n +3)’, ‘exit’, ‘exit’, ‘exit’, ‘exit’,’ exit’]

(?i)Password: 213gfdshaa
(?i)sftp>: [‘cd a/o/h’, ‘put lst*tar’, ‘exit’]

Hello Kai,
                       As soon as i used your solution now i am getting

this

error:

    "msg": "No remaining responses for '(?i)sftp>', output was ':

\r\nsftp>'"

                    Solved the issue but i have to add 5 exit i don't know
why. Do you think it has to do something with number of commands as i am
running 5 commands. new code:

I you previous message it says
"No remaining responses for '(?i)sftp>'"

So adding exit on the "root@" prompt shouldn't have any effect.

- name: crates sftp session to terastation
     expect:
        command: ssh user@host

        responses:
            (?i)root@: ['sftp user@host', 'mkdir /tmp/backups', 'mv lst*tar
/tmp/backups', 'cd /tmp/backups', 'rm -rf $(ls -1t /tmp/backups | tail -n
+3)', 'exit', 'exit', 'exit', 'exit',' exit']

            (?i)Password: 213gfdshaa
            (?i)sftp>: ['cd a/o/h', 'put lst*tar', 'exit']

To understand what is happening you would need to provide a verbose output by running but

ANSIBLE_STDOUT_CALLBACK=debug ansible-playbook -vvv your-playbook.yml

Hey Kai,
First of all thanks for help. here is output:

changed: [localhost] => (item=hurry-up-1) => {
“changed”: true,
“cmd”: “ssh "hurry-up-1"”,
“delta”: “0:00:00.382495”,
“end”: “2017-10-19 15:18:03.301177”,
“failed”: false,
“invocation”: {
“module_args”: {
“chdir”: null,
“command”: “ssh "hurry-up-1"”,
“creates”: null,
“echo”: false,
“removes”: null,
“responses”: {
“(?i)root”: “tar cvf "$HOSTNAME"_date +\"%Y-%m-%d_%H.%M.%S\".tar test\nexit\n”
},
“timeout”: 30
}
},
“item”: “hurry-up-1”,
“rc”: 0,
“start”: “2017-10-19 15:18:02.918682”
}

STDOUT:

Last login: Thu Oct 19 14:33:19 2017 from localhost
hurry-up-1 ~]# tar cvf “$HOSTNAME”_date +"%Y-%m-%d_%H.%M.%S".tar
test/
[root@hurry-up-1 ~]# exit
logout
Connection to hurry-up-1 closed.

TASK [setup-sftp-session : crates sftp session to terastation] ******************************************************************************************************************************************************************************
task path: /etc/ansible/roles/setup-sftp-session/tasks/setup-sftp.yml:13
Using module file /usr/lib/python2.6/site-packages/ansible-2.4.1.0-py2.6.egg/ansible/modules/commands/expect.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c ‘echo ~ && sleep 0’
<127.0.0.1> EXEC /bin/sh -c ‘( umask 77 && mkdir -p “echo /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574” && echo ansible-tmp-1508440683.35-276654372992574=“echo /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574” ) && sleep 0’
<127.0.0.1> PUT /tmp/tmpTC63A9 TO /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/expect.py
<127.0.0.1> EXEC /bin/sh -c ‘chmod u+x /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/ /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/expect.py && sleep 0’
<127.0.0.1> EXEC /bin/sh -c ‘/usr/bin/python2.6 /root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/expect.py; rm -rf “/root/.ansible/tmp/ansible-tmp-1508440683.35-276654372992574/” > /dev/null 2>&1 && sleep 0’
changed: [localhost] => {
“changed”: true,
“cmd”: “ssh 192.168.3.40”,
“delta”: “0:00:01.737366”,
“end”: “2017-10-19 15:18:05.204508”,
“failed”: false,
“invocation”: {
“module_args”: {
“chdir”: null,
“command”: “ssh 192.168.3.40”,
“creates”: null,
“echo”: false,
“removes”: null,
“responses”: {
“(?i)Password”: “jghsjd674783”,
“(?i)root@”: [
“sftp testuser@192.168.3.43”,
“mkdir /tmp/backups”,
“mv ltt-newtar /tmp/backups",
“cd /tmp/backups”,
“rm -rf $(ls -1t /tmp/backups | tail -n +3)”,
“exit”,
“exit”,
“exit”,
“exit”,
" exit"
],
“(?i)sftp>”: [
“cd a/b/c”,
"put ltt-new
tar”,
“exit”
]
},
“timeout”: 30
}
},
“rc”: 0,
“start”: “2017-10-19 15:18:03.467142”
}

STDOUT:

Last login: Thu Oct 19 15:18:02 2017 from localhost
[root@hurry-up-1 ~]# sftp testuser@192.168.3.43
Connecting to 192.168.3.43…
mkdir /tmp/backups
Password:

cd a/b/c
put ltt-newtar
Uploading hurry-up-1_2017-10-19_15.18.03.tar to /mnt/a/b/c/hurry-up-1_2017-10-19_15.18.03.tar
hurry-up-1_2017-10-19_15.18.03.ta 100% 10KB 10.0KB/s 00:00
exit
[root@hurry-up-1 ~]# mv ltt-new
tar /tmp/backups
[root@hurry-up-1 ~]# cd /tmp/backups
)root@hurry-up-1 backups]# rm -rf $(ls -1t /tmp/backups | tail -n +3
[root@hurry-up-1 backups]# exit
logout
Connection to 192.168.3.40 closed.

            "command": "ssh \"hurry-up-1\"",
            "creates": null,
            "echo": false,
            "removes": null,
            "responses": {
                "(?i)root": "tar cvf \"$HOSTNAME\"_`date
+\"%Y-%m-%d_%H.%M.%S\"`.tar test\nexit\n"
            },
            "timeout": 30
        }
    },
    "item": "hurry-up-1",
    "rc": 0,
    "start": "2017-10-19 15:18:02.918682"
}

STDOUT:

Last login: Thu Oct 19 14:33:19 2017 from localhost
hurry-up-1 ~]# tar cvf "$HOSTNAME"_`date +"%Y-%m-%d_%H.%M.%S"`.tar
test/
[root@hurry-up-1 ~]# exit
logout
Connection to hurry-up-1 closed.

Just a tip, in this task you don't need to use the expect module.
If the host hurry-up-1 is in Ansible inventory you could just use the archive module.

            "responses": {
                "(?i)Password": "jghsjd674783",
                "(?i)root@": [
                    "sftp testuser@192.168.3.43",
                    "mkdir /tmp/backups",
                    "mv ltt-new*tar /tmp/backups",
                    "cd /tmp/backups",
                    "rm -rf $(ls -1t /tmp/backups | tail -n +3)",
                    "exit",
                    " exit"
                ],
                "(?i)sftp>": [
                    "cd a/b/c",
                    "put ltt-new*tar",
                    "exit"
                ]
            },
            "timeout": 30
        }
    },
    "rc": 0,
    "start": "2017-10-19 15:18:03.467142"
}

STDOUT:

Last login: Thu Oct 19 15:18:02 2017 from localhost
[root@hurry-up-1 ~]# sftp testuser@192.168.3.43
Connecting to 192.168.3.43...
mkdir /tmp/backups
Password:
> cd a/b/c
> put ltt-new*tar
Uploading hurry-up-1_2017-10-19_15.18.03.tar to
/mnt/a/b/c/hurry-up-1_2017-10-19_15.18.03.tar
hurry-up-1_2017-10-19_15.18.03.ta 100% 10KB 10.0KB/s 00:00
> exit
[root@hurry-up-1 ~]# mv ltt-new*tar /tmp/backups
[root@hurry-up-1 ~]# cd /tmp/backups
)root@hurry-up-1 backups]# rm -rf $(ls -1t /tmp/backups | tail -n +3
[root@hurry-up-1 backups]# exit
logout
Connection to 192.168.3.40 closed.

From this STDOUT your root@ response only need one exit.
Because after the "exit" you get "logout" and then "Connection to 192.168.3.40 closed"

And for the sftp prompt where you did get the "No remaining responses for" message it also only need one exit to disconnect.