# Retrying failed tasks **URL:** \ **Category:** Ansible Project\ **Created:** [August 10, 2015, 7:29pm UTC](https://forum.ansible.com/t/retrying-failed-tasks/18986 "2015-08-10T19:29:48Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Ian\_Rose](https://avatars.discourse-cdn.com/v4/letter/i/85f322/32.png) [@Ian\_Rose](https://forum.ansible.com/u/Ian_Rose)\ **Post date:** [August 10, 2015, 7:29pm UTC](https://forum.ansible.com/t/retrying-failed-tasks/18986/1 "2015-08-10T19:29:48Z")
Hi all - I’ve been pretty happy running Ansible for a few months now. The one major thorn in my side is failed tasks. Our fleet of VMs is not very large, but apparently is large enough (or our playbook is long enough) that we hit at least one spurious SSH error (e.g. “SSH Error: mux\_client\_hello\_exchange: write packet: Broken pipe”), or, more rarely, I’ll hit a spurious 500 from a third party service (e.g. adding or removing our VMs to/from load balancers via a cloud API). What’s the best practice for dealing with these kinds of transient failures? It seems like me that something like “sleep X seconds, then retry, up to Y times” would work quite well, but it isn’t obvious to me how to make that happen. I’m aware of the wait\_for module, but I don’t think that really helps in this situation since the problem isn’t that a resource is actually missing; its just spurious failures. Any suggestions? Thanks! - Ian ---
**Author:** ![Brian\_Coca](https://avatars.discourse-cdn.com/v4/letter/b/6de8d8/32.png) [@Brian\_Coca](https://forum.ansible.com/u/Brian_Coca)\ **Post date:** [August 10, 2015, 7:37pm UTC](https://forum.ansible.com/t/retrying-failed-tasks/18986/2 "2015-08-10T19:37:26Z")
You can use the .retry files as a --limit to rerun the plays. ---
**Author:** ![Ian\_Rose](https://avatars.discourse-cdn.com/v4/letter/i/85f322/32.png) [@Ian\_Rose](https://forum.ansible.com/u/Ian_Rose)\ **Post date:** [August 10, 2015, 8:11pm UTC](https://forum.ansible.com/t/retrying-failed-tasks/18986/3 "2015-08-10T20:11:02Z")
My understanding of retry files (which could certainly be wrong) is that they merely limit the hosts that are included in the run. Which I don’t think will work for me, although perhaps this indicates that my playbook is not set up well. Here is a simplified version of my site.yml: - name: copy new files to all nodes hosts: all tasks: - include: tasks/deploy\_files.yml - name: configure and deploy backend type foo hosts: tag\_foo roles: - foo - name: configure and deploy backend type bar hosts: tag\_bar roles: - bar - name: configure and deploy backend type baz hosts: tag\_baz roles: - baz (etc, for 7 total backend types) - name: clean up old deployments from all nodes hosts: all tasks: - include: tasks/remove\_old\_deployments.yml So, given this structure, pretend that the “foo” step went fine, but then some step during one of the “bar” backend deployments failed. Won’t the retry file just contain that single host? (assuming we are running “serial: 1” for that task that failed) So if I reran using that file, I might get that “bar” host to deploy correctly, but I will totally miss all of the “baz” hosts and all other backends whose deployment tasks appear after the “bar” task. I suppose one option might be to break up this single site.yml into 7 different playbooks, one for each backend type, and then execute them each in order, retrying each one as necessary if any errors occur. Would that be a better setup? That _seems_ to be a bit silly, but maybe I’m wrong on that… Thanks, Ian ---
**Author:** ![Brian\_Coca](https://avatars.discourse-cdn.com/v4/letter/b/6de8d8/32.png) [@Brian\_Coca](https://forum.ansible.com/u/Brian_Coca)\ **Post date:** [August 10, 2015, 8:39pm UTC](https://forum.ansible.com/t/retrying-failed-tasks/18986/4 "2015-08-10T20:39:42Z")
I don't know if retry works well with serial.