unarchive for bulk of servers fails

Archives AWX Project
1

Hi, I have an issue where my playbook is able to push the tar.gz files to the central servers without any issues, but when it comes to large inventories it fails to upload the files to the central servers, I get an error like :

TASK [dtunix_report: report.yml - Unarchive all hosts tar file on the central server] ***
fatal: [yyyy.organe.com → xxxx.orange.com]: UNREACHABLE! => {“changed”: false, “msg”: “Failed to connect to the host via ssh: Shared connection to xxx.orange.com closed.”, “unreachable”: true}

Is it due to large number of ssh connects going to the central server, I am getting the shared connection to xxx.orange.com closed?

- name: report.yml - Unarchive all hosts tar file on the central server
  unarchive:
    src: '{{ remote_tar_local_archive }}'
    dest: '{{ remote_tar_central_results_dir.path }}'
    owner: root
    group: root
    mode: '754'
    remote_src: yes
  run_once: True
  ignore_errors: true
  delay: 10
  retries: 2
  delegate_to: '{{ custom_discovery_central_server }}'
  register: unarchive_report
  until: unarchive_report is succeeded
2

What AWX version are you running? How large are your inventories where the playbook fails? Do you have any insight into why it is unreachable.

Do the same hosts that run successfully when part of the smaller inventory fail when included in the larger inventory?

AWX Team

3

I am running ansible tower version 2.3. (its the licensed version). The inventory is around 10K servers. What I am seeing is the playbook when the multiple servers tries to establish sftp connection to dump the files in the central server the ssh connection reaches maximum and it ceases to accept new connections so it fails. So, wanted to know is there a way to chunk the large inventory into batches? Why I am asking this because if I have a batch of 1-200 I am able to receive the files. Is there a way to chunk the inventory dynamically from the main inventory file via a playbook?

4

Have you tried using the serial attribute in your playbook? From what I can see from your comments you could do something like this to chunk the inventory into blocks of 200 servers at a time and then loops until the inventory is completed

  3. Code example:
- hosts: all
  serial: 200
  become: yes
5

thank you Frank and sorry for the delay in the response. I did the same as you said and also the retry and poll was also added in the task level. All went through without any issues. Thank you Frank