# Gathering Facts "hanging" for a single host **URL:** \ **Category:** Ansible Project\ **Created:** [December 6, 2013, 8:08pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093 "2013-12-06T20:08:06Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 6, 2013, 8:08pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/1 "2013-12-06T20:08:06Z")
I have a host that, when I run a playbook against it, it fails at gathering facts. Unfortunately, it just hangs: > # ansible-playbook -i test playbooks/get\_release\_versions/site.yml -f 10 -vvvv > > PLAY [Grabs all release versions for auditing purposes] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > > GATHERING FACTS \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > > ESTABLISH CONNECTION FOR USER: ddecker > > EXEC [‘ssh’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ddecker’, ‘-o’, ‘ConnectTimeout=10’, ‘testhost01’, “/bin/sh -c ‘mkdir -p $HOME/.ansible/tmp/ansible-1386355046.1-188203869680440 && chmod a+rx $HOME/.ansible/tmp/ansible-1386355046.1-188203869680440 && echo $HOME/.ansible/tmp/ansible-1386355046.1-188203869680440’”] > > REMOTE\_MODULE setup > > PUT /tmp/tmpEvKhjU TO /home/ddecker/.ansible/tmp/ansible-1386355046.1-188203869680440/setup > > EXEC [‘ssh’, ‘-tt’, ‘-vvv’, ‘-o’, ‘ControlMaster=auto’, ‘-o’, ‘ControlPersist=60s’, ‘-o’, ‘ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r’, ‘-o’, ‘Port=22’, ‘-o’, ‘KbdInteractiveAuthentication=no’, ‘-o’, ‘PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey’, ‘-o’, ‘PasswordAuthentication=no’, ‘-o’, ‘User=ddecker’, ‘-o’, ‘ConnectTimeout=10’, ‘testhost01’, ‘/bin/sh -c 'dzdo -k && dzdo -H -S -p “[sudo via ansible, key=eeszsiqmrnksgtzkolmtvmhddnepwrbh] password: " -u root /bin/sh -c '”'“'/usr/bin/python /home/ddecker/.ansible/tmp/ansible-1386355046.1-188203869680440/setup; rm -rf /home/ddecker/.ansible/tmp/ansible-1386355046.1-188203869680440/ \>/dev/null 2\>&1'”'"''’] > > ^CTraceback (most recent call last): > > File “/usr/bin/ansible-playbook”, line 268, in > > sys.exit(main(sys.argv[1:])) > > File “/usr/bin/ansible-playbook”, line 208, in main > > pb.run() > > File “/usr/lib/python2.6/site-packages/ansible/playbook/ **init**.py”, line 262, in run > > if not self.\_run\_play(play): > > File “/usr/lib/python2.6/site-packages/ansible/playbook/ **init**.py”, line 505, in \_run\_play > > self.\_do\_setup\_step(play) > > File “/usr/lib/python2.6/site-packages/ansible/playbook/ **init**.py”, line 452, in \_do\_setup\_step > > accelerate=play.accelerate, accelerate\_port=play.accelerate\_port, > > File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 968, in run > > results = [self.\_executor(h, None) for h in hosts] > > File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 382, in \_executor > > exec\_rc = self.\_executor\_internal(host, new\_stdin) > > File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 471, in \_executor\_internal > > return self.\_executor\_internal\_inner(host, self.module\_name, self.module\_args, inject, port, complex\_args=complex\_args) > > File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 659, in \_executor\_internal\_inner > > result = handler.run(conn, tmp, module\_name, module\_args, inject, complex\_args) > > File “/usr/lib/python2.6/site-packages/ansible/runner/action\_plugins/normal.py”, line 54, in run > > return self.runner.\_execute\_module(conn, tmp, module\_name, module\_args, inject=inject, complex\_args=complex\_args) > > File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 348, in \_execute\_module > > res = self.\_low\_level\_exec\_command(conn, cmd, tmp, sudoable=sudoable) > > File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 708, in \_low\_level\_exec\_command > > rc, stdin, stdout, stderr = conn.exec\_command(cmd, tmp, sudo\_user, sudoable=sudoable, executable=executable) > > File “/usr/lib/python2.6/site-packages/ansible/runner/connection\_plugins/ssh.py”, line 219, in exec\_command > > rfd, wfd, efd = select.select([p.stdout, p.stderr], , [p.stdout, p.stderr], 1) > > KeyboardInterrupt For the above, I wait more than 10 minutes and it still just hangs. If I REMOVE the host, then it proceeds (and completes all other hosts). Additionall if I add gather\_facts: False to my site.yml (for the playbook), it completes with no issue: > # ansible-playbook -i test playbooks/get\_release\_versions/site.yml -f 10 > > PLAY [Grabs all release versions for auditing purposes] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > > TASK: [Get release version] \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > > changed: [testhost01] > > PLAY RECAP \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > > testhost01 : ok=1 changed=1 unreachable=0 failed=0 Does anyone know why this happens or what I can do to better debug and find out the reason why? When I do a ansible all -u ddecker -m setup it processes all hosts except this one, so it has something to do with the host and something that is happening that the fact gathering is requesting. Thanks, Drew ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [December 6, 2013, 8:14pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/2 "2013-12-06T20:14:31Z")
Let’s narrow this down further. 1) Run ansible-playbook with ANSIBLE\_KEEP\_REMOTE\_FILES=1 and -vvvv 2) Check the debug output and file the filename that is sent to the remote host that has a “setup” filename. 3) Go to the remote machine and run the script by hand with python If the script hangs there, then we know that the setup script is having and issue, if not there may be a connection issue. ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 6, 2013, 9:31pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/3 "2013-12-06T21:31:45Z")
Yup - it hangs on the client: ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [December 6, 2013, 9:40pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/4 "2013-12-06T21:40:22Z")
OS and ansible version? If you control-C where what do you get in the traceback when running locally? ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 6, 2013, 9:44pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/5 "2013-12-06T21:44:00Z")
control-C also just hangs (as you can see from my output in my previous comment) - so no traceback at all. If you mean the traceback from where Ansible runs when I run (and there is no output or failure), then for that it shows: Traceback (most recent call last): File “/usr/bin/ansible-playbook”, line 268, in sys.exit(main(sys.argv[1:])) File “/usr/bin/ansible-playbook”, line 208, in main pb.run() File “/usr/lib/python2.6/site-packages/ansible/playbook/ **init**.py”, line 262, in run if not self.\_run\_play(play): File “/usr/lib/python2.6/site-packages/ansible/playbook/ **init**.py”, line 505, in \_run\_play self.\_do\_setup\_step(play) File “/usr/lib/python2.6/site-packages/ansible/playbook/ **init**.py”, line 452, in \_do\_setup\_step accelerate=play.accelerate, accelerate\_port=play.accelerate\_port, File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 968, in run results = [self.\_executor(h, None) for h in hosts] File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 382, in \_executor exec\_rc = self.\_executor\_internal(host, new\_stdin) File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 471, in \_executor\_internal return self.\_executor\_internal\_inner(host, self.module\_name, self.module\_args, inject, port, complex\_args=complex\_args) File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 659, in \_executor\_internal\_inner result = handler.run(conn, tmp, module\_name, module\_args, inject, complex\_args) File “/usr/lib/python2.6/site-packages/ansible/runner/action\_plugins/normal.py”, line 54, in run return self.runner.\_execute\_module(conn, tmp, module\_name, module\_args, inject=inject, complex\_args=complex\_args) File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 348, in \_execute\_module res = self.\_low\_level\_exec\_command(conn, cmd, tmp, sudoable=sudoable) File “/usr/lib/python2.6/site-packages/ansible/runner/ **init**.py”, line 708, in \_low\_level\_exec\_command rc, stdin, stdout, stderr = conn.exec\_command(cmd, tmp, sudo\_user, sudoable=sudoable, executable=executable) File “/usr/lib/python2.6/site-packages/ansible/runner/connection\_plugins/ssh.py”, line 219, in exec\_command rfd, wfd, efd = select.select([p.stdout, p.stderr], , [p.stdout, p.stderr], 1) KeyboardInterrupt OS (of Client): RHEL 6.4 x86\_64 Ansible Version: 1.3.4 Thanks! ---
**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\ **Post date:** [December 6, 2013, 10:00pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/6 "2013-12-06T22:00:32Z")
This looks like accelerate not supporting sudo with password just yet? I agree it should error at least… – Michael ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [December 6, 2013, 10:04pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/7 "2013-12-06T22:04:00Z")
No, I think one of the fact functions is stuck waiting for input. We just need to narrow it down. Drew, In the setup script on the remote system there is an init function that calls all the major groups of facts ...      def \_\_init\_\_(self):          self.facts = {}          self.get\_platform\_facts()          self.get\_distribution\_facts()          self.get\_cmdline()          self.get\_public\_ssh\_host\_keys()          self.get\_selinux\_facts()          self.get\_pkg\_mgr\_facts()          self.get\_lsb\_facts()          self.get\_date\_time\_facts()          self.get\_user\_facts()          self.get\_local\_facts()          self.get\_env\_facts() Comment out all of those self.get\_\* calls and see if the the script completes. If it does, then uncomment each get one by one till the script hangs again. ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 6, 2013, 10:18pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/8 "2013-12-06T22:18:44Z")
Guys, It still hangs, however, I tried running it with Python’s trace options: $ python -m trace --trace ./setup re.py(142): return \_compile(pattern, flags).search(string) — modulename: re, funcname: \_compile re.py(231): cachekey = (type(key[0]),) + key re.py(232): p = \_cache.get(cachekey) re.py(233): if p is not None: re.py(234): return p setup(696): if m: setup(694): for folder in os.listdir(sysdir): setup(695): m = re.search(“(” + diskname + “\d+)”, folder) — modulename: re, funcname: search re.py(142): return \_compile(pattern, flags).search(string) — modulename: re, funcname: \_compile re.py(231): cachekey = (type(key[0]),) + key re.py(232): p = \_cache.get(cachekey) re.py(233): if p is not None: re.py(234): return p setup(696): if m: setup(694): for folder in os.listdir(sysdir): setup(707): d[‘rotational’] = get\_file\_content(sysdir + “/queue/rotational”) — modulename: setup, funcname: get\_file\_content setup(2064): data = default setup(2065): if os.path.exists(path) and os.access(path, os.R\_OK): — modulename: genericpath, funcname: exists genericpath.py(17): try: genericpath.py(18): st = os.stat(path) genericpath.py(19): except os.error: genericpath.py(20): return False setup(2069): return data setup(708): d[‘scheduler\_mode’] = “” setup(709): scheduler = get\_file\_content(sysdir + “/queue/scheduler”) — modulename: setup, funcname: get\_file\_content setup(2064): data = default setup(2065): if os.path.exists(path) and os.access(path, os.R\_OK): — modulename: genericpath, funcname: exists genericpath.py(17): try: genericpath.py(18): st = os.stat(path) genericpath.py(21): return True setup(2066): data = open(path).read().strip() Let me know if I can offer any more tests that may help with debugging this issue. Thanks Drew ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 13, 2013, 12:21am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/9 "2013-12-13T00:21:31Z")
Was anyone else able to give me some pointers that would able to let me debug this issue further? ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [December 13, 2013, 1:16am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/10 "2013-12-13T01:16:12Z")
It seems that python is hanging when trying to read the scheduler file for one of your disks. Add a print statement before line 2066 … print “PATH:”,path ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 13, 2013, 3:11am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/11 "2013-12-13T03:11:12Z")
Here’s where it dies: PATH: /sys/block/…/devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.1/host14/target14:0:0/14:0:0:0/block/sr0/queue/rotational PATH: /sys/block/…/devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.1/host14/target14:0:0/14:0:0:0/block/sr0/queue/scheduler PATH: /sys/block/…/devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.1/host14/target14:0:0/14:0:0:0/block/sr0/size PATH: /sys/block/…/devices/pci0000:00/0000:00:1d.7/usb2/2-2/2-2:1.1/host14/target14:0:0/14:0:0:0/block/sr0/queue/hw\_sector\_size PATH: /sys/block/…/devices/pci0000:00/0000:00:07.0/0000:0e:00.1/host5/rport-5:0-0/target5:0:0/5:0:0:2/block/sdl/removable PATH: /sys/block/…/devices/pci0000:00/0000:00:07.0/0000:0e:00.1/host5/rport-5:0-0/target5:0:0/5:0:0:2/block/sdl/queue/scheduler ^C^C^C^C^C^C^C^C^C^C^C^C \<------- Dies right after the previous line —— ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [December 13, 2013, 3:26am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/12 "2013-12-13T03:26:17Z")
Are you able to cat that file? jtanner@u1304:~$ cat /sys/block/…/devices/pci0000:00/0000:00:06.0/virtio2/block/vda/queue/scheduler noop deadline [cfq] ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 13, 2013, 4:37am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/13 "2013-12-13T04:37:15Z")
Nope - can’t cat it at all. I can ls it, and see the following: # ls -l /sys/block/../devices/pci0000:00/0000:00:07.0/0000:0e:00.1/host5/rport-5:0-0/target5:0:0/5:0:0:2/block/sdl/queue/scheduler -rw-r–r-- 1 root root 4096 Sep 11 02:26 /sys/block/../devices/pci0000:00/0000:00:07.0/0000:0e:00.1/host5/rport-5:0-0/target5:0:0/5:0:0:2/block/sdl/queue/scheduler Is there something I can do to map this device to a physical device to know what the cause could be? ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [December 13, 2013, 4:41am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/14 "2013-12-13T04:41:36Z")
“sdl” is the device name per the path. /dev/sdl If you can’t read the scheduler file, you may have a broken device or a buggy driver. You should check your dmesg outputs and the syslog for any obvious errors. Beyond that, you need to work with the relevant OS and hardware vendors to sort out why things are hanging. ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 13, 2013, 6:02am UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/15 "2013-12-13T06:02:48Z")
James, Thanks for the input. I went ahead and tested this on another system of the same Product type (Dell R810) that runs the same apps, etc (with possible BIOS upgrades and firmware updates already installed), and the setup script ran just fine. This system that is failing might just need some firmware updates installed on them - so I’ll get on that and post results in the future. Thanks for helping me sort this out from being a Ansible issue vs a server issue, and this clearly appears to be a server issue. Thanks again! ---
**Author:** ![Drew\_Decker](https://avatars.discourse-cdn.com/v4/letter/d/6f9a4e/32.png) [@Drew\_Decker](https://forum.ansible.com/u/Drew_Decker)\ **Post date:** [December 13, 2013, 1:22pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/16 "2013-12-13T13:22:39Z")
This morning I went ahead and applied some firmware updates and rebooted the problem server. Once the server came up, I tested the “setup” script again, this time it ran all the way. Thanks for the help in troubleshooting! ---
**Author:** ![tannerjc](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tannerjc/32/985_2.png) [@tannerjc](https://forum.ansible.com/u/tannerjc)\ **Post date:** [December 13, 2013, 3:23pm UTC](https://forum.ansible.com/t/gathering-facts-hanging-for-a-single-host/14093/17 "2013-12-13T15:23:33Z")
No problem. I learned something too …