# Ansible hanging **URL:** \ **Category:** Ansible Project\ **Created:** [March 1, 2013, 5:09pm UTC](https://forum.ansible.com/t/ansible-hanging/12437 "2013-03-01T17:09:09Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Ben\_Hood](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@Ben\_Hood](https://forum.ansible.com/u/Ben_Hood)\ **Post date:** [March 1, 2013, 5:09pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/1 "2013-03-01T17:09:09Z")
Hi, I’ve been able to run my playbook using fireball and a custom inventory (that queries the ec2 API) reasonably well using 5 hosts. To completely apply a playbook from scratch takes about 160 seconds with paramiko and 65 seconds with fireball. However, when I scale this out to 100 hosts, the playbook starts off quite quickly, and outputs lots of successful tasks but then after about 1 minute, it just grounds to a halt, with no apparent diagnostics. I’m using ansible devel @ 8099e4ac. What I have noticed is that specifying the inventory on the command line and having that get dynamically evaluated slows ansible down a lot. So to mitigate this, I just ran the inventory script once and piped the output into a text file, so that I could avoid the penalty of the EC2 API. Then I ran this playbook to get fireballs running on each host: - hosts: all user: root gather\_facts: false connection: paramiko tasks: - action: fireball Running it like this: $ ansible-playbook -i hosts.txt boot-fireball.yml -f 100 -vvv took about 5 seconds to complete for 100 hosts. Then I tried to run my actual playbook via fireball using the same hosts.txt forks=100, and this kicks off very quickly, successfully executing many tasks. However, after about a minute, it completely grinds to a halt, without any diagnostics. Is there anything that I can prod to get it to tell me where it is hanging? For note, switching to paramiko appears to work, i.e. it is able to complete the entire playbook against all of the 100 hosts. Cheers, Ben ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [March 1, 2013, 5:50pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/2 "2013-03-01T17:50:02Z")
Hmm, 0mq has had some reported history of being temperamental. Do you have anything in syslog on any of the hosts that hung, perhaps? note -- we currently /do/ have a ticket open that exception logging for fireball needs to be improved. ---
**Author:** ![Ben\_Hood](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@Ben\_Hood](https://forum.ansible.com/u/Ben_Hood)\ **Post date:** [March 1, 2013, 7:19pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/3 "2013-03-01T19:19:40Z")
> Hmm, 0mq has had some reported history of being temperamental. > > Do you have anything in syslog on any of the hosts that hung, perhaps? Unfortunately not - I’ve just burned those images. But I will be spinning this up again next week, so I’ll have a look into syslog then. > note – we currently /do/ have a ticket open that exception logging > for fireball needs to be improved. OK, good to know - although if the app is blocked due to some kind of network state, it is difficult to log something out. For now, running the playbook with ssh is probably good enough, although I do see the attraction of fireball. For a single run, it more than halved the playbook time, so it did achieve some milage. Having said this, one naive question occurs to me - why do you need to make multiple network calls to execute a playbook? Would it be impractical to do the whole translation phase on the localhost first (i.e. resolve and bind all variables to tasks and templates), and then send the resulting command stream to the target host? Or would it be even insaner to ship the ansible code to the target host and execute everything remotely (i.e. ship the code and keep the data local)? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [March 1, 2013, 7:44pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/4 "2013-03-01T19:44:56Z")
Ansible can decide what to do to later based on what went before, and even relative to what went before on other hosts in rather trivial ways. It's quite useful. If you don't want that, just use ansible-pull if this appeals to you. As long as you are pushing locally and not doing it all over the internet, it should be quite reasonable to push 🙂 0mq being random is likely just a 0mq bug. Folks should be certain they want to fireball, and not everyone needs to. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [March 1, 2013, 7:47pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/5 "2013-03-01T19:47:11Z")
> > Having said this, one naive question occurs to me - why do you need to make > > multiple network calls to execute a playbook? BTW, missed this the first time. Look into "-c ssh" with ControlPersist if you want to keep connections open. ---
**Author:** ![Ben\_Hood](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@Ben\_Hood](https://forum.ansible.com/u/Ben_Hood)\ **Post date:** [March 2, 2013, 9:39am UTC](https://forum.ansible.com/t/ansible-hanging/12437/6 "2013-03-02T09:39:22Z")
OK, that looks like it could bear fruit. As I indicated beforehand, just using plain paramiko is fast enough, so this might get a little bit further without having to debug the fireball mode. On balance, this is going to be good enough for our purposes - there is no concrete need to go as fast as fireball goes. Thanks for the heads up. ---
**Author:** ![Ben\_Hood](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@Ben\_Hood](https://forum.ansible.com/u/Ben_Hood)\ **Post date:** [March 2, 2013, 9:51am UTC](https://forum.ansible.com/t/ansible-hanging/12437/7 "2013-03-02T09:51:35Z")
> Ansible can decide what to do to later based on what went before, and > even relative to what went before on other hosts in rather trivial > ways. > It’s quite useful. Good to know. Can you give you some simple examples of this in action, so that we can get an impression of how that is working under the covers? > If you don’t want that, just use ansible-pull if this appeals to you. Having ported my orchestration from Chef, I’d like to stick with the push model that attracted me to ansible in the first place. My musings about shipping the code to the remote host were inspired by a half way house between pure push and pure pull. I was wondering whether you could run the orchestration in pure Python on the remote side without installing ansible and any dependencies remotely. But this is probably not very practical, so sorry for bringing it up. > As long as you are pushing locally and not doing it all over the > internet, it should be quite reasonable to push 🙂 Unfortunately the use case is to configure 100+ nodes on AWS. Potentially I might need to spin up an orchestrator node on AWS from which I run ansible. > 0mq being random is likely just a 0mq bug. Folks should be certain > they want to fireball, and not everyone needs to. As indicated before, ssh is probably fast enough, and hence fireball is probably just nice to have. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [March 2, 2013, 2:28pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/8 "2013-03-02T14:28:40Z")
> Good to know. Can you give you some simple examples of this in action, so > that we can get an impression of how that is working under the covers? Look up "register" and "only\_if" / "when" in the advanced playbooks doc for some of this. > My musings about shipping the code to the remote host were inspired by a > half way house between pure push and pure pull. I was wondering whether you > could run the orchestration in pure Python on the remote side without > installing ansible and any dependencies remotely. But this is probably not > very practical, so sorry for bringing it up. There's room for this in the future, if we flag a playbook to run in that way, and there was a wrapping module we wrote, ansible-playbook could output pure-JSON, and ansible was also installed on the remote end. It's technically possible, just a little involved. Seth and I originally called this idea the "ansi-ball" (and I don't think we even need the tarball part of the original theory). However, for most people something like ControlPersist with -c ssh is a nicer way to go that achieves similar ends. > use case is to configure 100+ nodes on AWS. Potentially I > might need to spin up an orchestrator node on AWS from which I run ansible. Sure thing. Tons of people doing that. AWS is clearly slower from the outside, running ansible inside is a really good idea, and you also save bandwidth! ---
**Author:** ![Asad\_Khan](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@Asad\_Khan](https://forum.ansible.com/u/Asad_Khan)\ **Post date:** [March 2, 2013, 2:33pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/9 "2013-03-02T14:33:36Z")
Hi Ben, I’m in the process of prototyping/assessing a pull implementation with ansible. Your comment caught my eye - would you mind sharing what kinds of issues you ran into with implementing a pull model with Chef that made you transition to Ansible and/or consider a hybrid push/pull approach? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [March 2, 2013, 2:45pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/10 "2013-03-02T14:45:52Z")
regarding whether something worked out with Chef or not, I'd rather not discuss tool X vs tool Y, or 'should I adopt Ansible vs X' on this mailing list, folks. This is largely out of respect for people working on those other tools, but also because, eventually, I'm going to tell someone how I really feel about tool X or Y 🙂 Please reply to each other directly if you want to discuss that. ---
**Author:** ![Asad\_Khan](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@Asad\_Khan](https://forum.ansible.com/u/Asad_Khan)\ **Post date:** [March 2, 2013, 2:58pm UTC](https://forum.ansible.com/t/ansible-hanging/12437/11 "2013-03-02T14:58:09Z")
Fair enough; the question was really with regards to push vs pull vs hybrid (I should’ve been clearer) - not Chef vs Ansible vs x. I will email you directly about the latter question 🙂