# soliciting feedback on ideas about playbook error handling in regards to code deployment **URL:** \ **Category:** Ansible Project\ **Created:** [November 27, 2012, 9:27pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107 "2012-11-27T21:27:35Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Jonathan1](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@Jonathan1](https://forum.ansible.com/u/Jonathan1)\ **Post date:** [November 27, 2012, 9:27pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/1 "2012-11-27T21:27:35Z")
Hi Super People, I wanted to ping the collective about two thoughts I’ve been having about error handling. Idea One: is an idea that was briefly discussed with Michael before, which is a “notify on failure” concept, in which handler(s) would be executed on failures as opposed to successes. Thus a task could have two sets of handlers, one when things are good, one when bad. The way I am thinking of this the host where the failure occurred would be taken out of the host list after executing the notify\_on\_failure handler(s) example: > _- hosts: all_ > > _tasks:_ > > _- name: upgrade code_ > > _action: yum pkg=foo state=newest_ > > _notify: test and start foo_ > > _notify\_on\_failure: rollback foo_ My rationale for this is around code deployments. I know there are related ideas floating around about this, like: [https://github.com/ansible/ansible/issues/1312](https://github.com/ansible/ansible/issues/1312), and [https://github.com/ansible/ansible/issues/127](https://github.com/ansible/ansible/issues/127) Generally curious if this is still interest here (I think there is, but not enough for actual work yet) or if there is mental momentum to another approach on triggering handlers when things don’t go right. Idea Two: A “global” stop on a playbook task if a host, or threshold of hosts fail a task. This is different from what I believe the current default behavior is: stopping executing a play on an individual host if a task fails (and ignore\_errors is not set) but continuing on other hosts. The rationale here is again around code deployment. The desired outcome is for the task to stop executing on _all_ hosts if the task fails on one, or a threshold of hosts. The notion is that a failed code deployment (lets say the failure is detected by a test script failure post yum triggered upgrade) should not have to go through the process of failing and rollback on all the other hosts if it is obviously a bad release. This idea could also be extended to allow a percentage of hosts in the task to fail. Lets say on a farm of a 100 app servers three are broken somehow, but not removed from inventory, you wouldn’t want to stop your deployment for three failed tests, but you would want to stop it for 15 failures (out of a 100). Also I do see some extra complexity on implementing this as the play wouldn’t be able to stop executing the _exact_ time a task fails but at the “top of the loop” for the next set of hosts in line for execution. Obviously this approach is not going to work if you are deploying code to 100 servers with an argument of _-f 100_ as they would all fail in parallel. This is expecting that code deployments would be executed in smaller chunks using small _-f_ values or use of the _serial_ play argument. > _- hosts: all_ > > _stop\_on\_failure: True_ **or** > _stop\_on\_percentage\_threshold: 15_ > _tasks:_ > > _- name: upgrade code_ > > _action: yum pkg=foo state=newest_ > > _notify: test and start foo_ > _notify\_on\_failure: rollback foo_ My main goal here is to see if there is interest from the community on these ideas. From an app deployment perspective they seem useful, but it would be good to know if I am delusional or missing existing functionality. cheers Jonathan ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [November 27, 2012, 9:33pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/2 "2012-11-27T21:33:07Z")
There's been enough sporadic interest in some kind of failure handler that would run on error and allow to 'recover' the play. Unfortunately the architecture of the system is designed so that all hosts in a "batch" (or all hosts in a play, if serial is not set) are running the same task at the same time. I am generally leery of the idea that a rollback ever works, especially when data is involved. Ultimately I worry about adding too much magic and semantics to Ansible that is specific to a few niche use cases, or when others would possibly want to do something differently. Open to ideas, but I want to avoid adding additional keywords and complexity where possible. Right now you \*sort of\* can implement this using "ignore\_errors" and "only\_if" in conjunction, though it's a bit gross. Maybe some syntactic sugar around that, rather than using handlers would be better. --Michael ---
**Author:** ![Jonathan1](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@Jonathan1](https://forum.ansible.com/u/Jonathan1)\ **Post date:** [November 28, 2012, 12:07am UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/3 "2012-11-28T00:07:43Z")
comments inline > There’s been enough sporadic interest in some kind of failure handler > that would run on error and allow to ‘recover’ the play. > Unfortunately the architecture of the system is designed so that all > hosts in a “batch” (or all hosts in a play, if serial is not set) are > running the same task at the same time. > > I am generally leery of the idea that a rollback ever works, > especially when data is involved. well… I’d like to be optimistic that a rollback can work 😉 My primary goal is to make sure that post playbook execution I don’t have a service that has fallen on its face, so lets just say following that your argument that you are not technically rolling back but say running a tasks w/ delegate to tell your LB to switch to a backup pool leaving your totally horked install in place… the key thing for me is reacting, conditionally, to failures and taking different execution paths. More on that below. > Ultimately I worry about adding too much magic and semantics to > Ansible that is specific to a few niche use cases, or when others > would possibly want to do something differently. > > Open to ideas, but I want to avoid adding additional keywords and > complexity where possible. > > Right now you _sort of_ can implement this using “ignore\_errors” and > “only\_if” in conjunction, though it’s a bit gross. Maybe some > syntactic sugar around that, rather than using handlers would be > better. I see where you are going with this. I’ve been playing around with some examples and for the rather simplistic deployment process I am thinking about I think we could cobble something together with ignore\_errors, register, and then only\_if or with the new when\_[string / integer / unset, etc…]. I fear the playbook is going to be a little ugly, but could likely get the job done in terms of conditionally executing tasks if something goes wrong. So I’ll try down this road first vs. working on adding a “notify\_on\_error” and I’ll see how the results are. This still leaves a question for me about aborting playbook execution entirely, and if anyone sees value in that. Here is a more specific example that could likely apply to other users: A simple web app using a relational database and using external tool for schema updates. The goal is to deploy a schema update and then deploy our application code. What I don’t want to do is continue through the playbook if the schema update fails to execute. If I know I don’t have the required schema in place, it makes no sense to continue to the next play in the playbook… Technically it would be possible to put a _only\_if_ or _when\_string_, etc… to check for the state of the schema update result, but that will get extra ugly if I am using _only\_if, when\_string_ etc… for error handling on the deployment (which I’ve illustrated in the example below). Advice on either handling this or input on possibly playbook language extensions would be most appreciated. (example playbook for above scenario follows): ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [November 28, 2012, 12:10am UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/4 "2012-11-28T00:10:29Z")
> > Ultimately I worry about adding too much magic and semantics to > > Ansible that is specific to a few niche use cases, or when others > > would possibly want to do something differently. > > > > Open to ideas, but I want to avoid adding additional keywords and > > complexity where possible. > > > > Right now you \*sort of\* can implement this using "ignore\_errors" and > > "only\_if" in conjunction, though it's a bit gross. Maybe some > > syntactic sugar around that, rather than using handlers would be > > better. > > I see where you are going with this. I've been playing around with some > examples and for the rather simplistic deployment process I am thinking > about I think we could cobble something together with ignore\_errors, > register, and then only\_if or with the new when\_[string / integer / unset, > etc...]. I fear the playbook is going to be a little ugly, but could likely > get the job done in terms of conditionally executing tasks if something goes > wrong. So I'll try down this road first vs. working on adding a > "notify\_on\_error" and I'll see how the results are. > > This still leaves a question for me about aborting playbook execution > entirely, and if anyone sees value in that. Here is a more specific example > that could likely apply to other users: > > A simple web app using a relational database and using external tool for > schema updates. The goal is to deploy a schema update and then deploy our > application code. What I don't want to do is continue through the playbook > if the schema update fails to execute. If I know I don't have the required > schema in place, it makes no sense to continue to the next play in the > playbook... Technically it would be possible to put a only\_if or > when\_string, etc... to check for the state of the schema update result, but > that will get extra ugly if I am using only\_if, when\_string > etc... for error handling on the deployment (which I've illustrated in the > example below). I think I'd accept something like any\_errors\_fatal: True/False (default False) to extend precent behavior. Right now a whole set of hosts has to fail all the way through, or a batch size if using serial: N. ---
**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\ **Post date:** [November 28, 2012, 2:22am UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/5 "2012-11-28T02:22:47Z")
i don’t know if this makes sense, but extending only\_if outside of each task to group tasks, might go a long way to provide more branch control: ``` only_if: $result.rc -name: ladi action: whatever - name: da action: blah - include: moreactions.yml ``` ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [November 28, 2012, 2:25am UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/6 "2012-11-28T02:25:56Z")
Yeah, this is not going to happen. When ansible was created it was designed to NOT be a programming language. In most cases, if you want this kind of behavior, you are doing it to check on certain props, which is when "group\_by" starts to kick major butt. group\_by your distro type, and then make a play that does things to just your BSD hosts, or just your hosts with not a lot of RAM, etc, and completely skip needing to use only\_if. I maintain that only\_if is LARGELY a sign that something is modelled a bit wrong. There are of course, exceptions, and I know people disagree with me and make heavy usage of it. Personally, I think it looks a bit ugly, but it's a neccessary evil. In any event, the nested grouping stuff isn't going to happen, as I made a concious decision in the begining to minimize the amount of YAML or programming-ness that entered into Ansible, and I think it's done us well to keep that complexity in the language out of it. In any event, another plug for group\_by. See the module docs if this is new to you! --Michael ---
**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\ **Post date:** [November 28, 2012, 4:08am UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/7 "2012-11-28T04:08:02Z")
> I think it looks a bit ugly,but it's a necessary evil. I couldn't agree more. I remember reading about group\_by, but completely forgot about it, makes my suggestion irrelevant. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [November 28, 2012, 2:29pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/8 "2012-11-28T14:29:50Z")
I have a case where group\_by could be made more useful (but cannot be used as-is). The software is called OSSEC, and for the installation the agent requires a key that is (pre)created on the server. Since creating the key (and doing the other enterprisy stuff we do) takes a long time, we only really want to call that script if the key has not already been created. Since this is (possibly) during provisioning, the server may not be up and running (yet). So based on the output of a raw command (which is a grep of the hostname in the line-based keys file) we want to determine which systems still need a key generated. So what we really want to do is something like: //////// - name: test-key    hosts: linux    user: some\_user    gather\_facts: no    tasks:    - action: raw grep "${network\_fqdn}" ~/ossec/local.client.keys 2\>/dev/null      delegate\_to: ${ossecservers[0]}      register: ossec\_key    - action: group\_by key="need-ossec-key"      only\_if: not is\_set('${ossec\_key}') - name: create-key    hosts: need-ossec-key    user: some\_user    serial: 1    gather\_facts: no    tasks:    - action: raw /some/path/to/some/script -c '${network\_fqdn} -some -option      delegate\_to: ${ossecservers[0]}      register: script - name: test-key-again    hosts: need-ossec-key    user: some\_user    gather\_facts: no    tasks:    - action: raw grep "${network\_fqdn}" ~/ossec/local.client.keys 2\>/dev/null      delegate\_to: ${ossecservers[0]}      register: ossec\_key    - action: group\_by key="lack-ossec-key"      only\_if: not is\_set('${ossec\_key}') - name: escalate    hosts: lack-ossec-key    gather\_facts: no    tasks:    - local\_action: |        mail to="someone" subject="failed"    - action: fail msg="failed" //////// So basicly, group\_by only works if you have a clear variable that can be used to group by. In this case we do not have such a variable, we only have some evaluation, and only\_if cannot be used together with group\_by. So what I would need in this case is a way to evaluate which servers should be grouped, e.g.:    - action: group\_by key="need-ossec-key" when="not is\_set('${ossec\_key}') Would such a change be acceptable ? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [November 28, 2012, 3:10pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/9 "2012-11-28T15:10:36Z")
So I'm having some issues with that hypothetical syntax but I grasp the idea. The idea is key is what you are grouping around, it is the name of a fact or variable. The key has to change based on the name of the group. What you are passing in to the new hypothethical "when=" seems more like an "eval=" to me. However that seems like it needs more simplification too. It seems, (shudder) that what this really needs is an eval "lookup plugin" to make this work, that evaluates a python string, and then the module can stay dumb. group\_by: key="need-osc-key-$EVAL(2+2)" Basically it would assume what was inside the EVAL was a python string. Obviously this is much easier in Jinja2. Ultimately I'm drawn on this, the syntax makes my eyes bleed, and this feels unlike Ansible, especially when you throw the "is\_set" and so on in there. I think you are better of using only\_if for your specific case. --Michael ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [November 28, 2012, 3:24pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/10 "2012-11-28T15:24:37Z")
Ok, so Daniel made this patch which makes it all much easier.      [https://gist.github.com/4161687](https://gist.github.com/4161687) This makes group\_by work with conditionals as most people will expect. So I can now simply do: //////// - name: test-key    hosts: linux    user: some\_user    gather\_facts: no    tasks:    - action: raw grep "${network\_fqdn}" ossec/local.client.keys 2\>/dev/null      delegate\_to: ${ossecservers[0]}      register: ossec\_key      when\_unset: $ossec\_key    - action: group\_by key="need-ossec-key"      when\_bool: ${ossec\_key.stdout} - name: create-key    hosts: need-ossec-key    user: some\_user    gather\_facts: no    serial: 1    tasks:    - action: raw /some/path/to/some/script -c ${network\_fqdn} -more -options      delegate\_to: ${ossecservers[0]}      register: script - name: test-key-again    hosts: need-ossec-key    user: some\_user    gather\_facts: no    tasks:    - action: raw grep "${network\_fqdn}" ossec/local.client.keys 2\>/dev/null      delegate\_to: ${ossecservers[0]}      register: ossec\_key    - action: group\_by key="lack-ossec-key"      when\_bool: ${ossec\_key.stdout} - name: escalate    hosts: lack-ossec-key    gather\_facts: no    tasks:    - action: mail to="ossec-team" subject="script fails with ${script.stdout}"    - action: fail msg="Problem generating ossec key" //////// This works well. Now I need to find out why:   - the raw module stalls for some ssh connections randomly (for no     particular reason systems that work stall forever the next run, usually     between 1-3 in a run of 25) using ssh transport   - the raw module returns successful with no output when there is an     authentication error (and when delegated) using ssh transport How I love Solaris 🙂 ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [November 28, 2012, 3:25pm UTC](https://forum.ansible.com/t/soliciting-feedback-on-ideas-about-playbook-error-handling-in-regards-to-code-deployment/12107/11 "2012-11-28T15:25:42Z")
Yeah, I like this. This is clean, ship it!