Moving a docker instance between 2 hosts

Archives Ansible Project
1

Hello,

I’m trying to figure out how to best stay declarative while being able to have my system properly reconfigure itself in the following situation:

  • I have a bunch of docker hosts
  • I have a bunch of docker containers. Each docker container hosts a service for a customer and is reachable from a specific subdomain of my company.
  • I have a frontal nginx container doing virtual hosting to redirect traffic for a specific container to its current host+port.

So I have docker-hosts, and containers virtual hosts.

I’d like to write the configuration so that I have entries for the virtual hosts, with a parameter specifying on which docker host they must be set.

Now, if I move a docker container from docker host A to docker host B in the configuration, when I run the playbook I must ensure that the following is done:

  • if the docker container is currently running on host B (in the example it is running on host A), then stop the container on host A
    (to keep things simple, currently the containers data are in a centrally accessible place - from docker hosts - but I must ensure only one container at a time accesses the data - thus killing from A before starting from B).
  • start the container from B
    ( - Reconfigure nginx and restart if necessary → not a problem atm)

Do you have advices on how best to do this?

My current idea involves:

  • creating entries in the inventory for each docker container virtualhost
  • for each virtualhost, finding through docker hosts facts analysis where it is running (if at all), and then removing it if running in another host than what the config currently states

Does that make sense? Will it be easy to do? (still newbie here, so maybe not having the right grasp of what’s easy or not with ansible)

Thanks in advance,

2

It’s fine to use Ansible to build docker images, but I’m not sure it’s a good placement engine for deciding where containers run.

Ideally you should some set of software that is trying to be a “cloud” manager for Docker.

From what I’ve heard, maybe this is Flynn or Shipyard – I don’t know - I haven’t tried them.

Specifying it in Ansible just seems really rough, keeping manual lists of what is running where that you have to start and/or remove.

Anyway, somewhat off topic for this list.

I’m waiting for their tech to evolve myself, then it will fit in closer to the other cloud models, feeling more like other hypervisors.

3

It’s fine to use Ansible to build docker images, but I’m not sure it’s a good placement engine for deciding where containers run.

Then what is the ansible-docker module good for?

Ideally you should some set of software that is trying to be a “cloud” manager for Docker.

From what I’ve heard, maybe this is Flynn or Shipyard – I don’t know - I haven’t tried them.

Interesting, I’ll look at them more closely.

Specifying it in Ansible just seems really rough, keeping manual lists of what is running where that you have to start and/or remove.

It’s moving the data of the containers around which also may be challenging, indeed.

4

IMHO, the most exciting system out there for running Docker applications is
CoreOS, a Linux distribution that is essentially just Linux + systemd +
Docker + fleetd + etcd. There guys really have something interesting
going. I have every intention of writing some modules that incorporate
etcd (a replicated key/value store meant for service discovery) and fleetd
(a modern cluster manager that seems to do a lot right).

Deis also shows a lot of promise.

The Docker module's fine for deploying containers an arbitrary number of
Docker hosts. Sounds like you're looking for something PaaS-like if you're
not pleased with today's Docker support.

5

I cannot say I’m not pleased yet, in fact I hope I will ! :slight_smile:

The docker module is host-centric. I’d like to try an approach where I use it in a more instance-centric way, since all my instances are made available via a sub domain.

Only thing I don’t know yet is if I can know from the gathered facts which instance is currently running on which host, and from this knowledge, for each declared instance in turn, whether I have to do a remove before doing an update/install.

6

I wrote the Docker dynamic inventory script. It returns containers by host – all you have to do is declare each Docker URL. I think that does what you’re looking for.

https://github.com/ansible/ansible/blob/devel/plugins/inventory/docker.py

7

yeah once they are deployed it’s easier, the main thing is Ansible is not a placement engine (or someone might call this a “scheduler”)

CoreOS is a different animal for a different purpose.

8

If by “placement engine” you mean something sophisticated with logic for determining where containers should be deployed - or deployed -, then yeah, I totally get that.

My needs for the coming year are more simple : I will keep on a file a list of hosts, and a list of containers per host. I’ll keep this in my VCS, next to my CM tool. Several times a week, I will declare a new container here, kill a container there.

As I understand Ansible, what I’ve described above should be easily handled via a host per docker host in the inventory file, and a list of var container properties per host, and the existing docker module.

Remains one operation I’ll do from time to time : move a container from a host to another host, e.g. when I add a new docker host, I’ll remove some containers from the existing ones, and put them on the new one.

This is where I was thinking of doing the following :

  • a first play which would, per docker host, get the list of active containers (identified by name), compare this list with the declared containers for the host : all active containers that aren’t in the declared list are removed.

  • a second play (should only execute after the first play has processed all docker hosts, so that it is guaranteed that everything that has to be removed has been removed, on all hosts before activating containers on their - possibly new - location) that will update/add hosts from the declared list.

9

Right, I’m saying this isn’t sustainable or a good approach for most infrastructures - and while it may be fine for you I want to avoid steering people down that path as it has all the perils of “manual cloud” circa 2005-ish.

As I was discussing with Paul Durivage yesterday, fleetd in CoreOS actually does provide this functionality too – I was thinking more WRT the CoreOS base image solving a different problem.

So having a module that allows Ansible to tell CoreOS “I want X instances of this image” would be pretty neat to see. Submissions welcome!

10

Right, I’m saying this isn’t sustainable or a good approach for most infrastructures - and while it may be fine for you I want to avoid steering people down that path as it has all the perils of “manual cloud” circa 2005-ish.

For continued sanity of all involved I’d agree that avoiding manual cloud partitioning is strongly recommended. It is ok for small experiments, and not much more than that.

As I was discussing with Paul Durivage yesterday, fleetd in CoreOS actually does provide this functionality too – I was thinking more WRT the CoreOS base image solving a different problem.

So having a module that allows Ansible to tell CoreOS “I want X instances of this image” would be pretty neat to see. Submissions welcome!

I’ve been trying to work out how to leverage both Ansible and Docker containers (may they be hosted using CoreOS, Project Atomic, Shipper…I really don’t care which at this point so long as it won’t set the datacenter on fire or something) going forward. My current thinking has the existing docker module used as a system for building container images pushed to a local registry. Something else (fleet, geard, mesos, Shipper’s gadget) manages the placement of containers onto hosts, after Ansible tells it what to do. (Everyone sails off happily into the sunset…)

That’s my highly idealistic vision anyway. We’ll see what happens.

11

Right, I would agree.

Build images with Ansible - stick in registry or wherever. “Cloud” thing may provide this.

Have Ansible tell “cloud” scheduler how many images to deploy into “cloud” from registry.

12

@mpdehaan: Are you saying you see Ansible being called from Dockerfiles, as from a RUN instruction? What does this offer above using Dockerfile instructions directly? Since Docker interprets each instruction as a layer, it seems you would lose the benefits of layer reuse if you called out to Ansible for all your configuration.

I had considered Ansible to be a good tool for a Docker deployment on the scale of a handful of hosts, fixed in number, and a few dozen containers. I don’t think of this small number of container hosts as a cloud/PaaS. With respect to the containers in this context, I think of them more as processes than as VMs, where Docker is desired to provide packaging and isolation.

In this context, I think the manual placement and linking that the current Ansible Docker module provides is a good fit. Given that Ansible should already be in the toolkit for container host base configuration, it seems logical to also use it for simple container orchestration.

For the use case I described, I perceive some of the other engines such as Fleet, Flynn, Deis, Centurion to be too heavyweight - I don’t think small deployments require dynamic placement, automatic container migration, or service discovery that tools like these are trying to provide. I think growing in to such tools when the cluster reaches a certain size is wise, rather than taking on all that complexity up front just to run a relatively small number of containers.

13

@mpdehaan: Are you saying you see Ansible being called from Dockerfiles,
as from a RUN instruction? What does this offer above using Dockerfile
instructions directly? Since Docker interprets each instruction as a layer,
it seems you would lose the benefits of layer reuse if you called out to
Ansible for all your configuration.

I mean this:

http://www.ansible.com/blog/2014/02/12/installing-and-building-docker-with-ansible

What does Ansible offer over automating your entire infrastructure with
just bash scripts? Exactly that.

No one is really using much more than the base image support in docker for
"layers". So deploy Ubuntu, apply Ansible for config.

If you want to deploy ansible on top of other images, use this too.

By no means does running Ansible instead of Dockerfiles having bash in them
change the equation.

I had considered Ansible to be a good tool for a Docker deployment on the
scale of a handful of hosts, fixed in number, and a few dozen containers. I
don't think of this small number of container hosts as a cloud/PaaS. With
respect to the containers in this context, I think of them more as
processes than as VMs, where Docker is desired to provide packaging and
isolation.

Doesn't matter - they are actually really small VMs/jails and can get
arbitrarily large. Let's ignore the hype about whether it's a VM or not,
it's serving the same purpose, but optimized a bit more towards running
many containers oversubscribed rather than providing more complete
isolation. Different tradeoff.

In this context, I think the manual placement and linking that the current
Ansible Docker module provides is a good fit. Given that Ansible should
already be in the toolkit for container host base configuration, it seems
logical to also use it for simple container orchestration.

I can see why some people don't want to contend with learning other layers
of management software. I'm also not a big fan of continuing to provide
glue to make that easier, as in the end, it's going to look inferior to the
real thing.

For the use case I described, I perceive some of the other engines such as
Fleet, Flynn, Deis, Centurion to be too heavyweight - I don't think small
deployments require dynamic placement, automatic container migration, or
service discovery that tools like these are trying to provide. I think
growing in to such tools when the cluster reaches a certain size is wise,
rather than taking on all that complexity up front just to run a relatively
small number of containers.

I guess that's where our levels of interest differ - Ansible is used by a
lot of people with home setups and low-load startups, but it's really
designed for industry deployments and so forth, so we don't want to
advocate avenues that cause problems later.

As such, I'm merely cautioning that manually placing Docker-containers is a
very "pre-cloud" way to do things, akin to keeping a spreadsheet of where
your virtual machines run.

When something is too heavyweight, the question is really what does that
mean? Is it something someone doesn't want to learn, does it consume too
many resources, or is it something someone doesn't want to manage?

And then the question really arises as to who is asking and for what
purpose. In many small shops, this may be in fact totally ok. I'm just
saying it doesn't scale, and as such, is a bit less interesting to me
trying to build that out.

I guess what I'm saying is I don't want to be a baby-cloud and all the
extra bolts and underpinnings it would require to make that viable, as
pretty soon people wonder about storage and monitoring and other bits - and
better placement engines - how to migrate systems, etc.

I'd rather our provisioning modules leverage other tools that already do
these things.

And I do recognize many of these are still too new and evolving yet.

14

This is exactly where we are.
For one, tools like Fleet, Flynn, Deis, etc., etc. seem heavyweight when we
consider our current needs. Also, the tools do not seem mature enough ...
heck, even us are probably not mature enough yet!!

The road towards maturity is certainly not a "big bang", and managing our
dozen of stateless container instances, each one dedicated to a specific
need (not talking about horizontal scalability of dozens of anonymous /
interchangeable role-only based nodes here!) will be done via a static
inventory file for the time being.

It seems that the Docker module will fill most if not all our needs in the
mean time.

Thanks to all for your insights and good advices!

15

I mean this:

http://www.ansible.com/blog/2014/02/12/installing-and-building-docker-with-ansible

Awesome! I was dreading having to port our existing Ansible roles to Docker instructions. Now Docker bash instructions seem like a step backwards.

I think you are right about the non-base layers not being terribly important, but it makes sense to me to try to put the seldom-changed, slow configuration tasks into separate layers. This should speed up builds significantly. Fortunately, you can do this with separate RUN instructions to execute playbooks containing those tasks. As an example, consider installing a non-system Python environment in its own playbook. We already use roles this way: a “Python 3” role, for example.

I guess that’s where our levels of interest differ - Ansible is used by a lot of people with home setups and low-load startups, but it’s really designed for industry deployments and so forth, so we don’t want to advocate avenues that cause problems later.

As such, I’m merely cautioning that manually placing Docker-containers is a very “pre-cloud” way to do things, akin to keeping a spreadsheet of where your virtual machines run.

When something is too heavyweight, the question is really what does that mean? Is it something someone doesn’t want to learn, does it consume too many resources, or is it something someone doesn’t want to manage?

And then the question really arises as to who is asking and for what purpose. In many small shops, this may be in fact totally ok. I’m just saying it doesn’t scale, and as such, is a bit less interesting to me trying to build that out.

I guess what I’m saying is I don’t want to be a baby-cloud and all the extra bolts and underpinnings it would require to make that viable, as pretty soon people wonder about storage and monitoring and other bits - and better placement engines - how to migrate systems, etc.

I’d rather our provisioning modules leverage other tools that already do these things.

And I do recognize many of these are still too new and evolving yet.

I put a lot of value on simplicity, but I will take the more complex solution if it solves a real problem and “just works”. There’s just not enough of a problem of placing and linking several dozen containers using “unmagical” automation tools. It is obvious this more manual approach will not scale with either very large or highly dynamic clusters. But there are so many shops that will never run in to this kind of scaling problem, and I think it is unproductive for them to swallow PaaS-scale solutions. At least with the current generation of fancy container orchestrators, I really doubt there’s a good ROI.

That said, I do understand the focus on large-scale automation. And I see your point about not wanting to waste effort on building in more Docker orchestration than what the current docker module offers. But it would be nice if the current docker module received some priority with respect to bug fixes and tracked Docker releases closely. I think a lot of us using Ansible and Docker for small-medium scale setups would be very happy!

With respect to Laurent’s original problem, I guess I would accept a little downtime to run some ad-hoc commands to move the container, after putting the new configuration into Ansible. I would not expect Ansible to manage container migration for me.

16

“But there are so many shops that will never run in to this kind of scaling problem, and I think it is unproductive for them to swallow PaaS-scale solutions”

And most of these shops don’t need Docker.

“But it would be nice if the current docker module received some priority with respect to bug fixes and tracked Docker releases closely”

We do apply fixes on the docker modules pretty quickly, but they are also not so good about maintaining API compatibility, so it appears to break often - because it does. I’m hoping this stabilizes over time.

17

Given that they've just reached 1.0 recently I presume there will be
less breakage going forward.