Rescue all hosts if one host fails

Archives Ansible Project
1

There is no possibility to run the rescue part for all hosts if one host fails. I’m working on my deployment for a website. I want to deploy the site to multiple hosts. If something fails for one host than all hosts should not be deployed. Therefore, a rollback on all hosts should happen. The configuration “any_errors_fatal” is not working for it.

I am using ansible 2.4.1.0

Here the steps to reproduce:

`

- block:
    - name: Update database schema
      shell: cd '{{ ansible_web_deployment_webroot }}/releases/{{ ansible_web_deployment_release }}' && php7.1 console.php orm:schema-tool:update --force
      when: inventory_hostname in groups['schema']
    - name: Deployment
      debug:
        msg: 'Deploying'
  rescue:
    - name: Rollback
      debug:
        msg: 'Rollback'
  any_errors_fatal: true

`

Expected results:

`

PLAY [test] ********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [x1.example.com]
ok: [x2.example.com]
ok: [x3.example.com]
ok: [x4.example.com]

TASK [ansible_web_deployment : Update database schema] *************************
skipping: [x2.example.com]
skipping: [x3.example.com]
skipping: [x4.example.com]
fatal: [x1.example.com]: FAILED! => {"changed": true, "cmd": "cd '/var/www/x.example.com/releases/20171113081452' && php7.1 console.php orm:schema-tool:update --force", "delta": "0:00:00.007578", "end": "2017-11-13 08:15:05.709148", "failed": true, "msg": "non-zero return code", "rc": 2, "start": "2017-11-13 08:15:05.701570", "stderr": "/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452", "stderr_lines": ["/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452"], "stdout": "", "stdout_lines": []}

TASK [ansible_web_deployment : Rollback] *****************************
ok: [x1.example.com] => {
    "msg": "Rollback"
}
ok: [x2.example.com] => {
    "msg": "Rollback"
}
ok: [x2.example.com] => {
    "msg": "Rollback"
}
ok: [x4.example.com] => {
    "msg": "Rollback"
}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=0
x1.example.com : ok=2    changed=1    unreachable=0    failed=1
x2.example.com : ok=2    changed=1    unreachable=0    failed=1
x3.example.com : ok=2    changed=1    unreachable=0    failed=1
x4.example.com : ok=2    changed=1    unreachable=0    failed=1

`

Actual results:

`

PLAY [test] ********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [x1.example.com]
ok: [x2.example.com]
ok: [x3.example.com]
ok: [x4.example.com]

TASK [ansible_web_deployment : Update database schema] *************************
skipping: [x2.example.com]
skipping: [x3.example.com]
skipping: [x4.example.com]
fatal: [x1.example.com]: FAILED! => {"changed": true, "cmd": "cd '/var/www/x.example.com/releases/20171113081452' && php7.1 console.php orm:schema-tool:update --force", "delta": "0:00:00.007578", "end": "2017-11-13 08:15:05.709148", "failed": true, "msg": "non-zero return code", "rc": 2, "start": "2017-11-13 08:15:05.701570", "stderr": "/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452", "stderr_lines": ["/bin/sh: 1: cd: can't cd to /var/www/x.example.com/releases/20171113081452"], "stdout": "", "stdout_lines": []}

TASK [ansible_web_deployment : Restart services] *****************************
ok: [x2.example.com] => {
    "msg": "Deploying"
}
ok: [x2.example.com] => {
    "msg": "Deploying"
}
ok: [x4.example.com] => {
    "msg": "Deploying"
}

TASK [ansible_web_deployment : Rollback] *****************************
ok: [x1.example.com] => {
    "msg": "Rollback"
})

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=0
x1.example.com : ok=2    changed=1    unreachable=0    failed=1
x2.example.com : ok=2    changed=1    unreachable=0    failed=0
x3.example.com : ok=2    changed=1    unreachable=0    failed=0
x4.example.com : ok=2    changed=1    unreachable=0    failed=0

`

How can I achieve that all hosts go in rescue mode when a single host throws an exception?

If you have a look at the Github issue 14024, you’ll see that this should work.
Unfortunately I was not able to get it to work.

Thank you for your input.

2

Hi

You can use the following method (sorry I don’t have the playbook ready).

  1. Run the upgrade on each host, ignore errors, but register the results and store them in a file on the controller machine (you can also manually - fail: once the error is registered for a given host)
  2. Once all upgrades are done on all hosts - check the content of the ‘results’ files on the controller. If any of them show issues - run another role to revert the changes.

kind regards
Pshem

3

Hi Aaron,

did you managed to have ansible rollback deployment on all hosts when anyone failed in any task ?

4

I can think of a couple of ways.

You can force a fail at the end of block if any hosts failed:
- block:
   - name: Update database schema
     shell: cd '{{ ansible_web_deployment_webroot }}/releases/{{
ansible_web_deployment_release }}' && php7.1 console.php
orm:schema-tool:update --f
orce
     when: inventory_hostname in groups['schema']
   - name: Deployment
     debug:
       msg: 'Deploying'
   - fail:
     when: ansible_play_hosts|length != ansible_play_hosts_all|length
rescue:
   - name: Rollback
      debug:
       msg: 'Rollback'
any_errors_fatal: true

Or delegate rescue to all hosts:

- block:
   - name: Update database schema
     shell: cd '{{ ansible_web_deployment_webroot }}/releases/{{
ansible_web_deployment_release }}' && php7.1 console.php
orm:schema-tool:update --f
orce
     when: inventory_hostname in groups['schema']
   - name: Deployment
     debug:
       msg: 'Deploying'
rescue:
   - name: Rollback
      debug:
       msg: 'Rollback'
      delegate_to: '{{item}}'
      loop: '{{ ansible_play_hosts_all }}'
any_errors_fatal: true