Sharing variables defaults across roles

Archives Ansible Project
1

Hello, first of all thank you very much for Ansible.

I’m setting up the deployment of a complex system and splitting it across roles. I haven’t found yet a satisfactory way of sharing variables across the roles. A simple example: set the database port in the db role and use it in the web role. I’m expecting to be able do do something along this line:

roles/
db/
defaults/main.yml
db_port: 5432
web/
templates/
web.conf.j2
[database]
port: {{ db_port }}

The above wants to represent the fact that the default port for the database is 5432; this may be overridden by a group_vars file for specific installation having the db on a non-standard port. Currently it doesn’t work because the db_port variable is not known by the web role. How to work around that?

  1. put all the shared variables in group_vars/all. It works but it breaks the encapsulation of the roles: without a certain unknown set of variables defined at least in “all” the db cannot be configured.
  2. add the db role as a dependency of web: this makes it execute the entire db tasks list, which is not what hoped.
  3. add “include_vars: …/…/db/defaults/main.yml” to the web’s tasks. This gives a higher preference to the default variable and overriding it in group_vars (or in the inventory) doesn’t work anymore.

As it is now I’m stuck with 1, which is also what I’ve generally found in the examples. Is there anything better? It seems a relatively straightforward use case to me:

  1. a provider role defines a variable and provides a default for it;
  2. a consumer role uses the variable provided;
  3. a group or inventory variable can override the default.

Thank you very much,

– Daniele

2

Setting defaults in group_vars/all is usually a very good place to set global defaults.

Note that role defaults are a lower priority than inventory though.

You could also consider symlinks.

3

Setting defaults in group_vars/all is usually a very good place to set global defaults.

They are not really globals: the db/web above is just an example. The variables may be shared across a few roles but other roles don’t need to see them. In a project with about a dozen of roles the “all” file is quickly becoming big and not pleasant to maintain.

Note that role defaults are a lower priority than inventory though.

Yes, the priority of the variables in defaults is as low as I need it. The priority gets increased if the default is included via include_vars though, and it’s at a level that the inventory cannot override.

You could also consider symlinks.

What can I symlink? I’ve tried linking db/defaults/main.yml as web/defaults/db.yml but ansible doesn’t read the file. It would have solved my problem indeed, but it appears only main.yml is read, not any *.yml. Did you have a different symlink in mind?

Thank you

– Daniele

4

In our environment, we have several roles that share common default “site” variables for things like the lists of domain controllers, NTP servers, the name of the kerberos realm, etc. Many of these variables are used in more than one role, but not all the roles are used by all systems. So we just created a role called “site-defaults” that only has a defaults/main.yml containing all the site-specific default variables; it doesn’t perform any tasks. Then we just add a role dependency in the other roles for the site-defaults role. Admins can always override the defaults with host or group variables if they need to, but at least we make an attempt to get everyone off on the right foot.

5

Correct, you could symlink a main.yml but not stick another file in there with a different name.

The reason for this is some folks use those for conditional includes (of vars/) directories, so auto-loading them would cause more confusion than it’s worth.

6

I see, so I think the idea of symlinking the 'main.yml’s is ruled out too (as the web role may need to declare some defaults itself).

If it is not currently possible to achieve what I need, I can imagine several ways ansible could be extended to allow that:

  • allowing some form of include command inside defaults/main.yml (which may work for vars too). I see the problem in the file being just a yaml dict, not a more generic structure. I guess conditional stuff couldn’t be executed either as you don’t know the vars yet to test them…

  • adding an include_defaults, like include_vars but with the variables precedence at lower level. This may also allow for conditionals or ignore_errors;

  • allowing multiple files in vars/defaults, disabled by default for the people who add files there for conditional inclusions as you explained. E.g. there could be a “role_vars_glob” defaulting to “main.yml” that somebody could set to “*.yml” if required. Then symlinks could be used.

I think the first solution is the one that would make more sense but I guess it’s harder to implement. So probably the second idea is more manageable (if include_vars proved to be a good idea, include_defaults should work too). The third feels more an hack to allow another hack, but it may be the easiest to implement.

Would any of these approaches be a welcome improvement? Or there is something just fundamentally wrong in my deployment approach (it could be, it is the first time I use ansible for such a complex task - but that’s what it’s for, isn’t it?). Is there a way to use just vars/main.yml and include_vars in the tasks but then be able to override them from the inventory? Sometimes I think group vars should be more powerful than role vars (that’s why I’m trying to use defaults instead).

I’ll be happy to try and hack on ansible if any of these approaches make sense; or to change my deployment configuration if there is better way to organize my rules.

Thank you for any comment.

– Daniele

7

Ultimately at some point everyone would like some different way something is handled, we work by observing the patterns and implementing those things as they come.

My advice to you is to not think too much about variables and defaults, and look at how to simplify your infrastructure. There are basic mechanisms there, but if you have to worry about what overrides what, it can get confusing.

It’s usually better to just define something in one place.

Role defaults are great for defining parameters that might not be passed for roles, but to default things in inventory, put something in group_vars/all and more specific groups will override it, etc.

We assuredly want to limit the number of ways we expand how variables work, since complexity is not something we’re after… this does mean looking at ansible differently, which can sometimes present challenges when folks are coming from, e.g., Chef and want to look at it like an outright programming language.

To me, includes inside variable files don’t make sense, because that whole structure exists to define variables, not include tasks. I’m a little less against the idea of “- include_defaults” but it would lead, IMHO, to bad role design.

I think it would be better to step back and see what you are trying to model, to understand you use case about the apps you are deploying, and then to suggest the way most people are modelling it in terms of current best practices.

8

Well, what I have is a service we deploy as a website for some customers, as an api for other customers (the website using the api internally). It is composed by parts outside the system (e.g. db), and parts inside the system (e.g. a redis server). We also host the same website in white-label version for providers who customize it and have their own customers. A host may run more than one whitelabel and there could be some services shared across whitelabels (e.g. a single redis for all the whitelabels on the same host). Bigger websites may get separate hosts for different services, tiers can be sharded etc. That’s the picture.

So, how I’m organizing the thing: I’m trying to have a single playbook, “site.yml”, for the deploy of any configuration. An inventory file would represent instead a single whitelabel with the description of what host it should run on (red.inv, blue.inv, for development I’d have vm.inv, localhost.inv etc). So, for instance, if Green and Blue are two small websites which can run on the same box while Red is so big that it needs web and api separated on two machines we could have:

green.inv
[web]
host1
[api]
host1

blue.inv
[web]
host1
[api]
host1

red.inv
[web]
host2
[api]
host3

And only one playbook to rule them all:

site.yml

  • hosts: web
    roles:

  • role: web

  • hosts: api

roles:

  • role: api

Let’s say normally the api serves on port 4000. It is usually fine but on a host running many of them it may need configuration. With the idea of a role being self-describing I think it’s normal to define api_port=4000 somewhere inside the role. If the role is used in a playbook it is conceivable that tasks in other roles may want to refer to that variable. If the default is fine then it’s ok for both the provider and the receiver; if it’s not the inventory/group may decide to override it; so I’d have:

roles/
api/
default/main.yml
api_port: 4000

group_vars/
green
info_email: info@green.com
blue
info_email: info@blue.com
api_port: 4001

As things stand now I cannot provide a self-contained api role that defines its own default port. If I put it in the vars it can be included but cannot be overridden. If I put it in group_vars/all it is no more self contained: the idea of modularized roles that can be (re)used at leisure in different playbooks breaks because they can only be used if sensible variables are provided in “all”, the inventory or somewhere else: I would have thought that the role itself is the most sensible provider of defaults for such variables.

A different organization may see a different playbook for each whitelabel, and add vars_files to each of them, but this seems much worse to maintain: if I add a new piece to the system, let’s say now it needs rabbitmq too to run, I have to add the rule to every playbook, instead of just to site.yml and then configure only the groups where its port is non-standard (which may be none of them). The idea I have in mind is the playbook describing how a system is composed and the inventory describing where the system should run; I may be wrong but in my playing around different interpretations have led to harder to maintain configurations.

Are you sure that being able to write “http_port: 80” inside roles/apache/defaults/main.yml would be bad design? I would have thought it is well encapsulated, provided it is exposed in such way that some role/firewall may read it. It serves as role interface indeed. As things stand now I’m experiencing “a migration of defaults”: first I define http_port inside apache defaults, because I think it belongs there and I just need to write the apache config file. Then I add the firewall configuration to the site and… oops: I need to know where apache wants to listen: another variable flying into groups_vars/all just because two roles need it (while 10 other roles don’t really care, but now I have to be more careful to scope everybody’s variables because they have become more global than what was needed).

If there is a better way to organize my work I’d be happy to follow suggestions. As it is now I’m following the advice to put everything inside “all”, but I don’t find it optimal and I’m wondering if it can be improved.

Thank you

– Daniele

9

Yep, roles that just cary variables in “/roles/foo/vars/main.yml” can be a perfect solution for cases like this, just include them before your other role.

Basically a role describes what something is, but they don’t entirely have to self contained.

Ansible contains some nice protections to make sure if you reference a var IN a role you’ll get that variable, no surprises, but roles can define variables for other roles.

10

Thanks John, that was just what I was looking for. This would be a great addition to the docs. I was originally trying to do:

pre_tasks:

  • include_vars: sharedvars.yml

But the variables in that case weren’t being pulled in. I feel like the docs would warn you that pre_tasks don’t affect the environment for the subsequent roles and also suggest the dependency trick for sharing variable values across roles.

Regards,
David

11

why not use the play's var loading?

vars_files:
  - sharedvars.yml

12

How about adding 3rd common role that has no tasks but just sets vars, then add that to both db and web as a dependency?