Memory error when downloading 3.4GB file

Archives Ansible Project
1

Hello,

Has anyone come across this Memory error? It occurs when running backup.yml script which is to backup DB and SECRET KEY.
Script is completely successful, until it reaches to a point when it needs to download a common.tar.gz file which is 3.4GB --Fails with an unpredictable memory error.

Any suggestions or help please?

./setup.sh -b -i inventoryfile -e backup_dir=/tower/bkpsetup

TASK [backup : Download the common backup.] ************************************************************************************************************************

task path: /TowerSetup/tower_binaries/ansible-tower-setup-bundle-3.4.2-1.el7/roles/backup/tasks/download.yml:31

<abc.corp…com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp…com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ‘IdentityFile=“/export/home/zsjkvh0/.ssh/id_rsaDev”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp…com ‘/bin/sh -c ‘"’“‘echo ~root && sleep 0’”’"‘’

<abc.corp…com> (0, ‘/root\n’, ‘’)

<abc.corp…com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp…com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ‘IdentityFile=“/export/home/zsjkvh0/.ssh/id_rsaDev”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp…com ‘/bin/sh -c ‘"’"’( umask 77 && mkdir -p “echo /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143” && echo ansible-tmp-1553783711.94-90505061625143=“echo /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143” ) && sleep 0’“'”‘’

skipping: [abc2.corp…com] => {

“changed”: false,

“skip_reason”: “Conditional result was False”

}

skipping: [abc3.corp…com] => {

“changed”: false,

“skip_reason”: “Conditional result was False”

}

<abc.corp…com> (0, ‘ansible-tmp-1553783711.94-90505061625143=/root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143\n’, ‘’)

Using module file /usr/lib/python2.7/site-packages/ansible/modules/net_tools/basics/slurp.py

<abc.corp…com> PUT /root/.ansible/tmp/ansible-local-19291WC3_1C/tmprxDde7 TO /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py

<abc.corp…com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o ‘IdentityFile=“/export/home/zsjkvh0/.ssh/id_rsaDev”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc ‘[abc.corp…com]’

<abc.corp…com> (0, ‘sftp> put /root/.ansible/tmp/ansible-local-19291WC3_1C/tmprxDde7 /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py\n’, ‘’)

<abc.corp…com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp…com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ‘IdentityFile=“/export/home/zsjkvh0/.ssh/id_rsaDev”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp…com ‘/bin/sh -c ‘"’“‘chmod u+x /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/ /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py && sleep 0’”’"‘’

<abc.corp…com> (0, ‘’, ‘’)

<abc.corp…com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp…com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ‘IdentityFile=“/export/home/zsjkvh0/.ssh/id_rsaDev”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc -tt abc.corp…com ‘/bin/sh -c ‘"’"’/usr/bin/python /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/slurp.py && sleep 0’“'”‘’

<abc.corp…com> ESTABLISH SSH CONNECTION FOR USER: root

<abc.corp…com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o ‘IdentityFile=“/export/home/zsjkvh0/.ssh/id_rsaDev”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/e37aeee5bc abc.corp…com ‘/bin/sh -c ‘"’“‘rm -f -r /root/.ansible/tmp/ansible-tmp-1553783711.94-90505061625143/ > /dev/null 2>&1 && sleep 0’”’"‘’

<abc.corp…com> (0, ‘’, ‘’)

The full traceback is:

Traceback (most recent call last):

File “/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py”, line 138, in run

res = self._execute()

File “/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py”, line 576, in _execute

result = self._handler.run(task_vars=variables)

File “/usr/lib/python2.7/site-packages/ansible/plugins/action/fetch.py”, line 95, in run

slurpres = self._execute_module(module_name=‘slurp’, module_args=dict(src=source), task_vars=task_vars)

File “/usr/lib/python2.7/site-packages/ansible/plugins/action/init.py”, line 831, in _execute_module

res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)

File “/usr/lib/python2.7/site-packages/ansible/plugins/action/init.py”, line 938, in _low_level_execute_command

rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)

File “/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py”, line 1007, in exec_command

(returncode, stdout, stderr) = self._run(cmd, in_data, sudoable=sudoable)

File “/usr/lib/python2.7/site-packages/ansible/plugins/connection/ssh.py”, line 276, in wrapped

display.vvv(return_tuple, host=self.host)

File “/usr/lib/python2.7/site-packages/ansible/utils/display.py”, line 183, in vvv

return self.verbose(msg, host=host, caplevel=2)

File “/usr/lib/python2.7/site-packages/ansible/utils/display.py”, line 206, in verbose

self.display(“<%s> %s” % (host, msg), color=C.COLOR_VERBOSE, screen_only=True)

MemoryError

fatal: [abc.corp…com]: FAILED! => {

“msg”: “Unexpected failure during module execution.”,

“stdout”: “”

}

2

What architectures are you using? 3.4 gb is a common failure size for 32 bit processors

3

It’s Linux system with 64 bit (x86_64)
Running on 2 cpu x 8 memory, there are no other processes running except ansible. Im running the setup.sh as root user.
I can see enough disk space and available memory, no clue why it throws memory error?

Is it trying to load the entire file to some cache and fails?

4

Hi,

5

I think you might not have enough space to do this. You should contact red hat support and get them to help you.

6

Thanks for the reply Sebastian.

It(backup.yml) is a inbuilt module provided in the ansible bundle. It is also running fetch.py during the download

File “/usr/lib/python2.7/site-packages/ansible/plugins/action/fetch.py”, line 95, in run

slurpres = self._execute_module(module_name=‘slurp’, module_args=dict(src=source), task_vars=task_vars)

7

To download, ansible is using fetch module.

fetch:
dest: ’ ’
fail_on_missing: yes
flat: yes
src: ’ ’

8

I found that it’s a big in the Ansible module.
Module fetch cannot download larger files , instead we need to try synchronize module with mode set to pull and it is working.

9

fetch will automatically use slurp if become is true, fetch itself
should not have issues with large files, but slurp uses RAM > x2 the
file size due to copying the full file into memory and then base64
encoding it.

10

https://access.redhat.com/articles/3344101

https://docs.ansible.com/ansible/2.5/modules/fetch_module.html