Efficient deployment management of nearly identical servers

Archives Ansible Project
1

I have an application that consists of several server types, most of which are nearly the same, differing only in one top-level application package. Usually it’s the application step that’s the most time-consuming, since it could involve a database data load/update.
(Think of a java/tomcat/apache/mysql application with a differently named tomcat application rpm per server-type)

I need to update these nearly-identical server-types concurrently - fastest possible deployment speed is a critical business requirement.
I envisioned the following Ansible deployment architecture:

playbook (targeting an environment i.e. lab vs test vs staging vs prod etc., invoked with a large enough -f)

  • task java
  • task apache
    … many other common tasks and sub-tasks
  • task application <parameter?>

And then somehow get the “application” task to deploy the application that corresponds to the specific application type.
First, am I going about this correctly - my goal being to maximize concurrency, and at the same time share as much code as possible?
Second, if this is reasonable, can you suggest how best to achieve the “application” task i.e. narrow it down to a particular application sub-type?
And if this isn’t a good path, can you suggest alternatives that achieve the stated goals?

Thanks.

2

And if this isn't a good path, can you suggest alternatives that achieve the
stated goals?

Nearly all the time in your process is going to be taken up by moving
package updates over the network and waiting for services to start.

If I am understanding you correctly, you want to do multiple things on
the same host at the same time?

--forks already lets you parallelize how many hosts you talk to at once.

A very good tip is to of course use 'with_items' with yum and apt, to
minimize the number of yum transactions.

However many things you don't want to be doing at the same time, and
running seperate package manager commands is definitely one of those
things.

3

No, the question isn’t about doing actions on a given host in parallel.
It’s concurrency at the host level, updating hosts that comprise a given group all in parallel (hopefully by using -f)
The difficulty is that the hosts are not identical, they’re nearly so. Let me try to illustrate this by showing you (some) of the application stack:

server1: os-updates | java | tomcat | apache | mysql | groovy | php | identity-mgmt | users | app1
server2: os-updates | java | tomcat | apache | mysql | groovy | php | identity-mgmt | users | app2
…
server 20: …

If app1 wasn’t different from app2, app3, … this would be a piece of cake. i would simply have a main playbook that implemented tasks for each of the applications above, and simply run with -f (e.g. update all webservers)

main-playbook:yml:

  • include: ./tasks/java.yml
  • include: ./tasks/apache.yml
    …
  • include ./tasks/app.yml

Secondly our application characteristics are different; yum copy/update times and service restarts are not the dominant factor - the app1/app2/… application-tier-specific update logic is - just how it works. That’s why it’s critically important to have the actions comprising app1 on server1-cluster, app2 on server2-cluster … all happen in parallel.
I hope that’s clearer.

4

No, the question isn't about doing actions on a given host in parallel.
It's concurrency at the *host* level, updating hosts that comprise a given
group all in parallel (hopefully by using -f)

Ok, so this already exists. Your question isn't about concurrency at all then.

The difficulty is that the hosts are not identical, they're nearly so. Let
me try to illustrate this by showing you (some) of the application stack:

server1: os-updates | java | tomcat | apache | mysql | groovy | php |
identity-mgmt | users | app1
server2: os-updates | java | tomcat | apache | mysql | groovy | php |
identity-mgmt | users | app2

Sounds like you want to list what classes a server is in, rather than
list what servers you have in each class.

I'd probably recommend writing your own external inventory script to
define groups in inverted order.

Instead of

[webservers]
a.example.com
b.example.com

you would like to instead say something arbitrarily like

a.example.com groups=webservers,dbservers,etc

That's exactly why the external inventory script capability is there,
to give you freedom to organize how you want.

Secondly our application characteristics are different; yum copy/update
times and service restarts are not the dominant factor - the app1/app2/..
application-tier-specific update logic is - just how it works. That's why
it's critically important to have the actions comprising app1 on
server1-cluster, app2 on server2-cluster .. all happen in parallel.
I hope that's clearer.

Right, you don't have a concurrency issue at all, you have a
organization question.

5

Secondly our application characteristics are different; yum copy/update
times and service restarts are not the dominant factor - the app1/app2/..
application-tier-specific update logic is - just how it works. That's why
it's critically important to have the actions comprising app1 on
server1-cluster, app2 on server2-cluster .. all happen in parallel.
I hope that's clearer.

Reading this again, it seems like you want to specify different
playbooks happen at the same time. Ok, I get this. It's a bit of a
niche case.

Sorry, we don't do that now.

It would require some idea on how to use multiprocessing on the
playbooks themselves, i.e. some kind of 'concurrent_play' keyword.

I'm interested in this conceptually.

6

Should be obvious, but if your webservers and db servers don't depend
on each other, just kick off multiple playbook runs, wait for them to
return, and then run whatever steps at the end.

Essentially Ansible is a linear pipeline.

Could be done trivially with shell backgrounding and such, on top of
/usr/bin/ansible-playbook and waiting for processes to terminate.

7

Exactly, playbook concurrency would solve this.

Secondly, there’s been discussion of a capability that could help with this: group_by module (so I wonder if this is that niche a case)

Thirdly, I need to absorb your hosts organization observation better.

Fourthly, I can sort of see a possible (convoluted) path here :

  • Write a template that iterates through the magic variables and determines server sub-type (i.e. am I server-type1 or 2 or …) and write this out on the host (basically construct a host site-fact)
  • Write a module to render this fact back into ansible-server execution space (or get this back to the server via copy & $FILE)
  • Do a parametrized include of the app task based on this “server-type” grokked from the site-fact
  • include: app.yml

Just an idea - not sure if it will work, but will try this out or your hosts reorganization suggestion. This though, is a critical issue for us - and I definitely need to solve this.

8

Oh yeah, I thought about that :slight_smile:
That’s my fall-back, but if I get to that level, then I will be bringing in Rundeck to handle other aspects of orchestration in conjunction with ansible - I was rather hoping to do everything through Ansible.

9

Rundec

10

Perfect re: concurrent scheduler.

My next mini-project was to do a comparative analysis of rundeck vs ansible-commander lol - seriously!
We need something like that, since our system is more complex then I have indicated - there are indeed intra-server dependencies that require groups of servers to be deliberately and carefully sequenced, and concurrency carefully orchestrated.

I trust your technical analysis - I was only trying to say that if I sequence ansible-invocations from outside ansible, that’s really coding in a level of orchestration outside of ansible, be it in bash or some other purported orchestration engine like Rundeck. That would be a pity since Ansible is so close to handling such (& similar) use-cases (group-by is in the same vein)
Until then I rather think I can make it work with my write-host-fact-observe-host-fact-invoke-host-specific-task-with-host-fact-parameter technique - unless you see something that would impede that.

Thanks.

11

Perfect re: concurrent scheduler.

My next mini-project was to do a comparative analysis of rundeck vs
ansible-commander lol - seriously!
We need something like that, since our system is more complex then I have
indicated - there are indeed intra-server dependencies that require groups
of servers to be deliberately and carefully sequenced, and concurrency
carefully orchestrated.

I trust your technical analysis - I was only trying to say that if I
sequence ansible-invocations from outside ansible, that's really coding in a
level of orchestration outside of ansible, be it in bash or some other
purported orchestration engine like Rundeck. That would be a pity since

AFAIK, Rundeck is just a parallel SSH-to-host pusher for dumb scripts.
  Doesn't really orchestrate anything.

Not saying it doesn't serve a purpose, but it's a narrow purpose.

Ansible is so close to handling such (& similar) use-cases (group-by is in
the same vein)
Until then I rather think I can make it work with my
write-host-fact-observe-host-fact-invoke-host-specific-task-with-host-fact-parameter
technique - unless you see something that would impede that.

Well, if you put conditionals on everything, you're still linear in
the order at which things execute, no different than if you had 5
plays in a row.

basically all I'm suggesting is...

ansbile-playbook foo.yml &
ansible-playbook bar.yml &
ansible-playbook baz.yml &

If you really have a directed pipeline like:

(A and B and C at once) then (D and E) at once and then (F)

I kind of envision something for this being possible (call it
ansible-pipeline, which would wrap playbooks) and it would be super
easy to build... yet... I've got other things to do first.

12

(A and B and C at once) then (D and E) at once and then (F)
The directed pipeline is exactly what we here need to accomplish at the end of this exercise.

I should have explained what I meant by trying to use Rundeck - otherwise the Rundeck notion would seem silly.
Rundeck is indeed pretty dumb, but what it does provide is:

  • defining/tagging hosts/groups
  • defining execution pipelines on said hosts
  • start/stop/pause/step/prompt of the execution pipeline

So the thought was to couple Rundeck with ansible-pull i.e. it’s Rundeck that does the host->playbook mapping instead of Ansible. Ansible does all the rest of the heavy-lifting of running playbooks and tasks, updating servers.
What hopefully one gets with the above combo is generalized pipelining and a very intuitive interface for visualizing and describing pipelines, and debugging/executing them. The work on individual servers is all done by Ansible.
(It’s cheating of sorts; ansible-pull is really being driven in a push-mode via Rundeck - but the beauty is that clusters of servers can auto-update via ansible-pull on cron, thus yielding a hybrid)

Anyways that was the rough thought. The good thing is ansible-commander is also on it’s way - and will likely solve the above in a much better fashion - so either way it should be good.

ansible-pull is next on my list of things to look at - as soon as I finish proving parallel application deployment, which should be very soon. I dont know if people say it enough, but Ansible’s been a joy to work with, I’m having problems tearing myself away from it.

13

ansible-pull is next on my list of things to look at - as soon as I finish
proving parallel application deployment, which should be very soon. I dont
know if people say it enough, but Ansible's been a joy to work with, I'm
having problems tearing myself away from it.

glad to hear it!

14

Parallel deployment of the nearly-the-same-servers works fine with the template->ansible_facts technique, was really trivial to code up using Jan-Piet Mens’ (thank you!) example code for localsetup. This is a powerful technique, and I know I will use this pattern again and again.