Feedback on a mult-platform CM & deploy use-case please?

Archives Ansible Project
1

Hi!

Could anyone give me feedback in the way I’m setting up our DevOps?
I really couldn’t find much that actually dealt with multiple platforms so I had to piece together what I could and came up with the following (hopefully it will also help for anyone that comes after).

We, OpenConext.org, are looking to replace a homegrown Bash setup, OpenConext-VM, with Ansible.
OpenConext is basically a platform with about 8 different software components (Java WARs running in a Tomcat and PHP apps running on LAMP) at this time (though that tends to vary).
What it currently does is provision a machine from scratch and build a demo project with all platform components from source.

We want it to do much more:

  • Build a demo project from scratch with all released platform components (WARs for Java, tarballs for PHP) on a single vm.

  • Switch any single component from build to source from any git branch (for developers).

  • Build and manage a production platform with some of the released platform components on multiple machines.

  • Upgrade a component from one release to another (including any changes to provisioning) with as little downtime as possible.

  • Get the version for a single or all components for a given environment.

Our goals are:

  • More modularity in setup, mixing and matching components.
  • Using this setup on a setup with multiple environments (test, staging, production).
  • No more manual updates, every component is responsible for it’s own updates.

These are the terms I’m using:

  1. Platform
    A single platform is a set of environments all running set of installed components with their given configuration values.
    A Platform may be open (OpenConext) or a National Research Education Network may choose to run it closed (SURFconext).

1.1 Environment
An environment is part of a platform and is a set of hosts all running a set of installed component with their given configuration values.

1.1.1 Host
Is part of an environment running one or multiple installed components with their given configuration values.

  1. (Platform) Component
    Any piece of software that supports:

2.1. Install from build
2.2. Install from source.
2.3. Uninstall
2.4. Upgrade to newer version
2.5. Show version

Translating this to Ansible I’ve come up with the following structure:

Inventory
Inventory is per environment, listing the wiring of Hosts and Components in “inventory/platform_name/environment.ini”.
Inventory specifies the components to use and their version, which may be a semver version or a git tag / branch / commit.

Playbooks

  • install.yml
    Install every component listed in the inventory.
    Includes everything from install/install-*

  • install/install-openconext.component.yml
    Runs any setup tasks (like installing databases, configuring LDAP, etc), includes install.yml, injects configuration, includes activate.yml.

  • version.yml
    Get the version for every component listed in the inventory.

Roles
Every Component action is a role:

Examples:

  • openconext.engine
  • openconext.serviceregistry
  • openconext.api

Every component has at least the following playbooks (note that we’re not using main.yml):

  • install.yml

Checks the version and either installs from source if the version is numeric or from a build if the version is non-numeric (with an include).

  • install-build.yml
    Install a component build for a given (semver) version.

  • install-src.yml
    Install a component with development tools for a given branch / tag / commit.

  • activate.yml
    Runs any migrations, switches symlinks, restarts webservers.

  • erase.yml
    Remove a component from a machine.

  • version.yml
    Displays the currently installed version.

Branching
In order to achieve multiple different platforms the base OpenConext-vm can be forked, like so:

* OpenConext / OpenConext-VM
Base ‘demo’ project that has a single host on which it installs all components with their build.
inventory/openconext/demo.openconext.org

** Developer / OpenConext-VM
Fork of the main repository, contains a different inventory.
inventory/openconext/dev.openconext.org

** SURFconext / OpenConext-VM
Fork of the main repository contains multiple different inventories with different hosts, different configuration values, but the same roles.
inventory/surfconext/integration.surfconext.nl
inventory/surfconext/test.surfconext.nl
inventory/surfconext/staging.surfconext.nl
inventory/surfconext/prod.surfconext.nl

** SURFnet / OpenConext-VM
Fork of the main repository for a separate ‘playground’ environment.
inventory/surfnet-playground/playground.surfnet.nl

Use case: Developing a feature
A developer forks the OpenConext-vm, adds his dev.openconext.org environment with version = feature/some-feature and runs install.yml.
This installs all the build tools and the developer gets working.
He finished up, merges his work to develop. Later another developer checks out develop, tests it, merges it to master, tags and builds a new release.
This second developer then updates inventory/openconext/demo.openconext.org with version=4.2.0 (was 4.1.0).
A demo user goes a git pull, and vagrant provision, which does an ansible install, which installs OpenConext-engine-4.2.0, runs any applicable migrations, switches over the symlink, restarts Apache.
A SURFconext developer updates inventory/surfconext/test.surfconext.nl and runs install. Testers test. Then staging.surfconext.nl gets modified, then prod.surfconext.nl.

Thank you for reading! Any feedback to this setup would be appreciated!

Cheers,
Boy