ansible hang on setup stage

Hello

Sorry for the previous mail, went out accidentally before I had prepared.

ansible 2.1.1.0
centos 6.8 (control machine)
RHEL 6.6 (Managed machine)

I have a role that works fine with all the rest of the systems, but
will hang at setup stage on two systems. ssh server configuration is
identical to the working systems, selinux is on permissive mode.

What would be other potential causes of such a hang? And how have you
guys gone about debugging this kind of issues?

Here are the logs when I attempt to play ansible-play against this machine?

TASK [setup] *******************************************************************
<arsenic.eng.example.com> ESTABLISH SSH CONNECTION FOR USER: root
<arsenic.eng.example.com> SSH: ansible.cfg set ssh_args:
(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<arsenic.eng.example.com> SSH: ansible_password/ansible_ssh_pass not
set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<arsenic.eng.example.com> SSH:
ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set:
(-o)(User=root)
<arsenic.eng.example.com> SSH: ANSIBLE_TIMEOUT/timeout set:
(-o)(ConnectTimeout=10)
<arsenic.eng.example.com> SSH: PlayContext set ssh_common_args: ()
<arsenic.eng.example.com> SSH: PlayContext set ssh_extra_args: ()
<arsenic.eng.example.com> SSH: found only ControlPersist; added
ControlPath: (-o)(ControlPath=/home/william/.ansible/cp/ansible-ssh-%h-%p-%r)
<arsenic.eng.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto
-o ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o
ControlPath=/home/william/.ansible/cp/ansible-ssh-%h-%p-%r
arsenic.eng.example.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "`
echo $HOME/.ansible/tmp/ansible-tmp-1475012538.53-32071443024344 `" &&
echo ansible-tmp-1475012538.53-32071443024344="` echo
$HOME/.ansible/tmp/ansible-tmp-1475012538.53-32071443024344 `" ) &&
sleep 0'"'"''
<arsenic.eng.example.com> PUT /tmp/tmpKnd3x9 TO
/root/.ansible/tmp/ansible-tmp-1475012538.53-32071443024344/setup
<arsenic.eng.example.com> SSH: ansible.cfg set ssh_args:
(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<arsenic.eng.example.com> SSH: ansible_password/ansible_ssh_pass not
set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<arsenic.eng.example.com> SSH:
ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set:
(-o)(User=root)
<arsenic.eng.example.com> SSH: ANSIBLE_TIMEOUT/timeout set:
(-o)(ConnectTimeout=10)
<arsenic.eng.example.com> SSH: PlayContext set ssh_common_args: ()
<arsenic.eng.example.com> SSH: PlayContext set sftp_extra_args: ()
<arsenic.eng.example.com> SSH: found only ControlPersist; added
ControlPath: (-o)(ControlPath=/home/william/.ansible/cp/ansible-ssh-%h-%p-%r)
<arsenic.eng.example.com> SSH: EXEC sftp -b - -C -vvv -o
ControlMaster=auto -o ControlPersist=60s -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o
ControlPath=/home/william/.ansible/cp/ansible-ssh-%h-%p-%r
'[arsenic.eng.example.com]'
<arsenic.eng.example.com> ESTABLISH SSH CONNECTION FOR USER: root
<arsenic.eng.example.com> SSH: ansible.cfg set ssh_args:
(-o)(ControlMaster=auto)(-o)(ControlPersist=60s)
<arsenic.eng.example.com> SSH: ansible_password/ansible_ssh_pass not
set: (-o)(KbdInteractiveAuthentication=no)(-o)(PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey)(-o)(PasswordAuthentication=no)
<arsenic.eng.example.com> SSH:
ANSIBLE_REMOTE_USER/remote_user/ansible_user/user/-u set:
(-o)(User=root)
<arsenic.eng.example.com> SSH: ANSIBLE_TIMEOUT/timeout set:
(-o)(ConnectTimeout=10)
<arsenic.eng.example.com> SSH: PlayContext set ssh_common_args: ()
<arsenic.eng.example.com> SSH: PlayContext set ssh_extra_args: ()
<arsenic.eng.example.com> SSH: found only ControlPersist; added
ControlPath: (-o)(ControlPath=/home/william/.ansible/cp/ansible-ssh-%h-%p-%r)
<arsenic.eng.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto
-o ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey
-o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o
ControlPath=/home/william/.ansible/cp/ansible-ssh-%h-%p-%r -tt
arsenic.eng.example.com '/bin/sh -c '"'"'LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python
/root/.ansible/tmp/ansible-tmp-1475012538.53-32071443024344/setup; rm
-rf "/root/.ansible/tmp/ansible-tmp-1475012538.53-32071443024344/" >
/dev/null 2>&1 && sleep 0'"'"''

Regards,
William

When things hang in setup is most likely because of a file system.
Gather facts is getting a lot of different values about the file systems, if one of this command hang you get a timeout.

Stale mount, NFS, LVM lock and so on.

Log in to the node and try mount, df -h and pvs.