Hi all!
Does anybody had experience in managing deployment of two or moredifferent projects on one system? I have such task and main issue that Im experiencing is configuration an dependencies clashes.
Best solution which I have is to make common deployment config for all projects, but this is looks ugly.
Are they the same project with different configuration (like multiple mysql servers on one host with different ports) or multiple different applications on the same host? In the latter case, my playbook deals with this kind of deployment - at the moment I can deploy GitLab, GitLab CI, phpIPAM, ownCloud, Etherpad, BoxBackup, ISC DHCPd, Mailman on the same host without clashes (of course it’s meant to be used with a cluster of hosts, but it works with just one). Check it out - https://github.com/ginas/ginas/
Maciej
In my case they are project backend and project front-end, so they are sub-projects of one project (SPA), and they have shared resources, and there they clashes sometimes.
I guess the first question would be, what shared resources? Different things will need to be treated differently in Ansible. For example, nginx is an easy case, because it has configuration files in /etc/nginx/sites-available/* which can be easily generated via templates with configuration passed via hash variables. Postfix on the other hand has only one main configuration file (/etc/postfixmain.cf) and to be able to configure it from multiple roles (different applications) I needed to write a kind of wrapper around Ansible local facts and include them in main configuration file so that previous configuration options would be preserved. So, again, what resources are we talking about?
Maciej
If anybody is interested, I’ve written some more explanation about Postfix configuration managed by multiple Ansible roles at once.
It’s probably very ugly hack and I suppose that could be done better with some external fact script with separate place for storing facts,but since I need it pretty quick (failing server needs a replacement), this is my solution to the following problem:
Postfix uses /etc/postfix/main.cf and /etc/postfix/master.cf configuration files; in these files (I’ll focus on main.cffrom now on, master.cf has a similar solution) you can write options with simple values or lists of values. Inmain.cf there can be only one option specified at a time, each subsequent “instance” of that option overwrites the previous setting. Most of the configuration options are lists of tables (files, sql queries, and so on). With that in mind, how to enable management of various Postfix configuration options from multiple roles without making conflicts all the time?
Let’s start at the beginning. In my playbook, I have ‘postfix’ role (https://github.com/ginas/ginas/tree/master/playbooks/roles/ginas.postfix/). This role is “common”, which means that it’s executed on all managed hosts.
First noticeable feature would be “capabilities” (described in defaults/main.yml). Because Postfix can be configured in multiple ways and for multiple environments, I decided to group features into “capabilities”, with a master list (variable ‘postfix’) that defines which features are enabled on a particular host. By default, Postfix is configured as a “null client”, ie. no network capability, no local mail, all mail relayed to domain MX server. With different capabilities enabled, Postfix gains more functionality - “network” enables network communication, “mx” enables filters on port 25 and ability to process mail from external servers, and so on. With this in mind you can probably see that one main “postfix” role can be used as a base on many hosts, and you can easily define MX hosts, relay hosts, mail storage with mailboxes available via POP3/IMAP and so on.
Other defining feature of this role is the decision to split ‘main.cf’ and ‘master.cf’ files into separate parts and use ‘assemble’ module to generate main configuration files from separate directories. That way, external roles, programs or even sysadmins can put their own local configuration in separate files inside main.cf.d/ and master.cf.d/ directories and it will be preserved during next Ansible run.
But unfortunately, with Postfix this is not enough to support multiple roles, since many parameters are lists and they cannot be repeated. Because of that, I decided to generate a set of lists using Ansible local facts, which can be populated using set of ‘postfix_dependent_*’ variables set when ‘postfix’ role is used as a dependency. Example role that uses this new system is ‘mailman’ (https://github.com/ginas/ginas/tree/master/playbooks/roles/ginas.mailman), specifically in ‘meta/main.yml’ you can find settings to enable Mailman support in Postfix for two cases, either with or without local mail delivery.
Whole process begins at the start of ansible-playbook run, when facts are gathered. Ansible will read facts from /etc/ansible/facts.d/postfix.fact JSON file and include them in ‘ansible_local’ hash tree. Specific template that generates /etc/ansible/facts.d/postfix.fact file (https://github.com/ginas/ginas/blob/master/playbooks/roles/ginas.postfix/templates/etc/ansible/facts.d/postfix.fact.j2) will read the hashes from ‘ansible_local.’ facts and create new hash variables with options set previously. Next, the template will check 'postfix_dependent_’ variables to see if there are any new entries to include with existing options. After that, template will add new options to existing variables, if they are configured and save the results in /etc/ansible/facts.d/postfix.fact. After the fact (pun intended), Ansible will re-read local facts using ‘setup’ module and incorporate new changes into ‘ansible_local’ fact tree. When the process is finished, ‘postfix’ role can generate new configuration files using updated hash variables from ansible_local facts + role defaults + inventory variables. This process can be repeated with each new play and roles that use ‘postfix’ role can add their own configuration options to the mix.
Because AFAIK there is no way to remove already registered facts during ansible-playbook run, the whole process is only additive - you cannot remove specific list elements, only add new ones. But because everything is stored in one file (/etc/ansible/facts.d/postfix.fact), to “reset” the configuration all you need to do is remove that file and re-run Ansible with ‘postfix’ role or tag, which will regenerate configuration (some Postfix settings can be wrong for a brief period of time). This is required for example when list of Postfix capabilites is changed (some configuration options can make sense only with some Postfix features enabled, like local delivery and alias_maps, for example).
When ‘postfix’ role generates configuration files from templates, there is a set of macros (in https://github.com/ginas/ginas/blob/master/playbooks/roles/ginas.postfix/templates/etc/postfix/main.cf.d/macros.j2) which can be used to determine if a particular option needs to be enabled (it is set from inventory or facts) and generate Postfix lists for that option. It is currently used mostly in https://github.com/ginas/ginas/blob/master/playbooks/roles/ginas.postfix/templates/etc/postfix/main.cf.d/10_basic_options.j2). Some other parts of the Postfix configuration might require modifications in the future, I’ll update them as I go.
To finish - the whole process is idempotent and allows you to configure Postfix from other roles pretty easily. I plan to use this system for virtual mail management (multiple options at that), Dovecot support, Request-Tracker and other software that requires modifications to Postfix configuration files.
If I left anything else unexplained or you need more information, let me know. 
Maciej
The assemble module can be used to simulate conf.d style configuraton for apps that don’t support conf.d and can be useful: