# Ansible deployment gets stuck/ends prematurely on unreachable host, doesn't proceed to next host **URL:** \ **Category:** Get Help\ **Tags:** awx\ **Created:** [May 22, 2025, 3:46pm UTC](https://forum.ansible.com/t/ansible-deployment-gets-stuck-ends-prematurely-on-unreachable-host-doesnt-proceed-to-next-host/42931 "2025-05-22T15:46:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![alvaroc20](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/alvaroc20/32/3485_2.png) [@alvaroc20](https://forum.ansible.com/u/alvaroc20)\ **Post date:** [May 22, 2025, 3:46pm UTC](https://forum.ansible.com/t/ansible-deployment-gets-stuck-ends-prematurely-on-unreachable-host-doesnt-proceed-to-next-host/42931/1 "2025-05-22T15:46:30Z")
Hi, I’m running deployments with `serial: 1` because I need to process hosts one by one. My goal is that if a host becomes stuck (e.g., due to an OOM error) or gets terminated (e.g., a spot instance is reclaimed) during a deployment, Ansible should: 1. Mark that specific host as failed. 2. Move on to the next host in the inventory. 3. Not block the entire deployment pipeline. 4. At the end of the play (after all hosts are attempted), a summary (like the standard PLAY RECAP) should clearly show which hosts succeeded and which failed, and `post_tasks` should run to determine the overall job status based on a custom threshold of failures. I’m using the following relevant settings in my play: - `serial: 1` - `ignore_unreachable: true` - `max_fail_percentage: 100` - A play-level `timeout: 300` (5 minutes) for the main task block. - A `block/rescue/always` structure for the main deployment tasks. - The `rescue` block is intended to catch failures/timeouts and record the failed host. - The `always` block has tasks like `unattended-upgrades` with `ignore_errors: true` and a short `ansible_connect_timeout` (e.g., 3 seconds) to ensure it doesn’t hang on a dead host. - `post_tasks` that run on `localhost` to summarize and determine the final job status. **The Problem:** When a host becomes unreachable _during_ the execution of tasks within the main `block` (e.g., a task attempts an SSH connection and it times out, resulting in an `UNREACHABLE` error for that task): - The `UNREACHABLE` error is correctly ignored at the task level (due to `ignore_unreachable: true` at the play level). - The host is marked as `failed=1` in the PLAY RECAP. - **However,** the `rescue` block associated with the main `block` does not seem to execute **(`rescued=0` in PLAY RECAP).** - Consequently, the `always` block also doesn’t execute as part of that `block/rescue/always` structure. - **Crucially, Ansible does not proceed to the next host in the inventory.** - The `post_tasks` are also not executed. - The playbook effectively ends after the first host fails with an `UNREACHABLE` state, showing a PLAY RECAP for only that single host. It seems like the `UNREACHABLE` status, even when `ignore_unreachable: true` is set, is causing the play to terminate its processing for the current `serial: 1` batch prematurely, without triggering `rescue` /`always` or moving to the next host/`post_tasks` . If, instead, a task _within_ the block fails due to a command error (while the host is still reachable) or if the play-level `timeout: 300` is hit, the `rescue` and `always` blocks _do_ seem to trigger, but the issue of not proceeding to the next host or `post_tasks` can still occur if the job ends abruptly (e.g., if AWX itself times out the job, which I’ve ruled out by setting no job template timeout). **What I’ve tried/verified:** - Ensured no AWX Job Template timeout is interfering. - Set `ansible_connect_timeout` to a low value (e.g., 3-5 seconds) for tasks in `always` and globally via `ansible_ssh_common_args` . - The playbook works correctly and processes all hosts if no hosts fail or become unreachable. **Question:** How can I ensure that when a host in a `serial: 1` batch becomes `UNREACHABLE` : 1. The `rescue` and `always` blocks for that host’s main tasks are reliably executed? 2. Ansible then robustly moves to the next host in the inventory? 3. The `post_tasks` are executed after all hosts have been attempted? Is there a different approach or a known interaction with `UNREACHABLE` in `serial: 1` plays that I might be missing? Here’s a snippet of the relevant playbook structure: ```auto - hosts: all gather_facts: no become: true max_fail_percentage: 100 ignore_unreachable: true # At play level timeout: 300 # For the main tasks block serial: 1 vars: max_allowed_failures: 1 # Example # ... pre_tasks to init a bad_hosts list ... tasks: - block: # ... multiple import_role tasks for deployment ... - name: "Import deploy role" import_role: name: deploy_awx # Example role where UNREACHABLE might occur rescue: - name: "Record failed host" set_fact: bad_hosts_accumulator: "{{ #... logic ... }}" delegate_to: localhost always: - name: "Final cleanup task (e.g., unattended-upgrades)" import_role: name: unattended-upgrades vars: ansible_connect_timeout: 3 # Short timeout for this ignore_errors: true post_tasks: # ... tasks on localhost to summarize and fail job if bad_hosts > max_allowed_failures ... - name: "Fail job if too many hosts failed" fail: msg: "Deployment failed on too many hosts: {{ final_bad_hosts_list }}" when: final_bad_hosts_list | length > max_allowed_failures run_once: true delegate_to: localhost ``` Any insights or suggestions would be greatly appreciated! Thanks! ---
**Author:** ![shertel](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/shertel/32/1241_2.png) [@shertel](https://forum.ansible.com/u/shertel)\ **Post date:** [May 22, 2025, 6:13pm UTC](https://forum.ansible.com/t/ansible-deployment-gets-stuck-ends-prematurely-on-unreachable-host-doesnt-proceed-to-next-host/42931/2 "2025-05-22T18:13:30Z")
> - Consequently, the `always` block also doesn’t execute as part of that `block/rescue/always` structure. > - **Crucially, Ansible does not proceed to the next host in the inventory.** That doesn’t seem right, but I wasn’t able to reproduce. What version of ansible are you using? Unreachable hosts don’t run tasks in the `rescue` section (as [documented](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_blocks.html#handling-errors-with-blocks)), but the `always` section should run. Could you have `any_errors_fatal` set somewhere? Maybe you could use a `fail` task at the end of the role that runs if any of the tasks were unreachable, so the host is rescued. Or, in the always if all the tasks in the role use `register` with a common naming pattern. ```auto - name: Fail unreachable hosts that haven't otherwise failed fail: when: - inventory_hostname not in ansible_failed_hosts - | tast_1_result is unreachable or task_2_result is unreachable # ... etc ``` ```auto - name: Count unreachable hosts in the always ... some logic ... when: - inventory_hostname not in ansible_failed_hosts - q('vars', *q('varnames', 'deploy_*')) | select('unreachable') | length != 0 ``` Edit: Correction - unreachable hosts are removed from the list of active hosts and do not execute `always` section tasks. Sorry, I had skimmed some ambiguous documentation without validating it first. ---
**Author:** ![alvaroc20](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/alvaroc20/32/3485_2.png) [@alvaroc20](https://forum.ansible.com/u/alvaroc20)\ **Post date:** [May 26, 2025, 10:50am UTC](https://forum.ansible.com/t/ansible-deployment-gets-stuck-ends-prematurely-on-unreachable-host-doesnt-proceed-to-next-host/42931/3 "2025-05-26T10:50:32Z")
Hi again, I’ve been running a lot of tests recently, but I’m still unable to get the behavior I’m aiming for. I’m not sure if what I want is fully achievable in Ansible as it stands. Here’s a simplified example of what I’m trying: ```auto - hosts: all gather_facts: no become: true serial: 1 vars_files: - vars/.yml collections: - tasks: - block: - name: "SSH Key" import_role: name: dynamic_ssh_key - name: "Import OS role" import_role: name: os - name: "Import aws_metadata role" import_role: name: aws_metadata - name: "Import nodejs role" import_role: name: geerlingguy_nodejs - name: "Import deploy role" import_role: name: deploy_awx always: - name: "Enable unattended-upgrades" import_role: name: unattended-upgrades vars: unattended_upgrade: "enabled" ignore_errors: true ``` At the play level, I’ve tried various combinations of: ```auto max_fail_percentage: 100 ignore_unreachable: true timeout: 300 ``` And in the AWX job template, I’m using: ```auto ansible_ssh_common_args: > -o ConnectTimeout=5 -o ServerAliveInterval=15 -o ServerAliveCountMax=2 ansible_ssh_reuse_connections: false ``` Despite all this, I still can’t get Ansible to reliably move on to the next host in the inventory when one becomes unreachable. Either the job stops entirely after the first unreachable host, or I configure it to ignore everything and the job falsely reports success for all hosts. Ideally, I want the play to: - Handle unreachable hosts gracefully. - Mark them as failed or record them as bad. - Move on to the next host (since I’m using `serial: 1`). - Finally, reflect a failed job status in AWX if one or more hosts didn’t complete successfully Is there any known pattern or workaround that can enforce this kind of “host isolation” behavior, where one failing or unreachable host doesn’t prevent the rest from being processed? Any insight would be really appreciated — thanks again! P.S: I’m not using `any_errors_fatal` ---
**Author:** ![shertel](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/shertel/32/1241_2.png) [@shertel](https://forum.ansible.com/u/shertel)\ **Post date:** [May 28, 2025, 6:09pm UTC](https://forum.ansible.com/t/ansible-deployment-gets-stuck-ends-prematurely-on-unreachable-host-doesnt-proceed-to-next-host/42931/4 "2025-05-28T18:09:53Z")
If you’re not trying to abort the play early, `serial: 1` (changing the batch size) seems like the wrong option. If a batch fails completely, the play stops instead of running subsequent batches. Have you tried [throttle: 1](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.html#restricting-execution-with-throttle) instead? Unreachable errors are fatal with `serial: 1` because unreachable hosts cannot bring themselves back into the play and there’s no non-failed host in the batch. In your original post, you’re recalcuating the play-recap, and I don’t understand what’s being added that wouldn’t be part of the default play recap. However, assuming failed hosts aren’t rescued/ignored, unreachable hosts aren’t ignored, and there’s at least 1 successful host to perform the delegation, you could give a custom recap with a `set_stats` task like this: ```yaml post_tasks: - name: Provide custom recap and extra vars for the next playbook in the workflow set_stats: data: failed_hosts: "{{ ansible_failed_hosts }}" unreachable_hosts: "{{ ansible_play_hosts_all | difference(ansible_current_hosts) | difference(ansible_failed_hosts) }}" successful_hosts: "{{ ansible_current_hosts }}" run_once: True delegate_to: localhost ``` The custom recap is displayed if [show\_custom\_stats](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/default_callback.html#parameter-show_custom_stats) is configured, but the variables will be available to the next playbook in the workflow either way.