multi-role organization and re-use

The purpose of this message is to come up with a best practices for
role/playbook/library layout that encourages re-use and logically
separates projects & roles into their own repositories. Just as an
aside, I'm not a git expert by any means, so I'm always looking to
simplify my workflow and organization. If I'm overcomplicating
things, by all means let me know!

*TL;DR - what's the best way to re-use roles and playbooks across projects?

Let's say I have project voltron. Project voltron uses the following
roles -- web and db

Of course the web and db roles all live in source control with the
general layout:

ansible-web-role:

.
├── files
├── handlers
├── library
├── tasks
├── templates
└── vars

These roles are also used outside of the voltron project, so there's
also a repo that uses ansible-web-role as a subrepo and would look
something like this:

ansible-web:

.
├── bender.yml
├── roles
│ └── web
│ ├── files
│ ├── handlers
│ ├── library
│ ├── tasks
│ ├── templates
│ └── vars
└── setup_web.yml

As you can see there are some additional playbooks that I am using,
and the roles/web subdirectory is just a git subrepo, which means any
changes that happen to the ansible-web-role project can be pulled down
into the ansible-web project. Also changes made in the roles/web
directory can get pushed up to the ansible-web-role repo.

Now to tie it all together in the voltron project:

.
└── roles
    ├── db
    │ ├── files
    │ ├── handlers
    │ ├── library
    │ ├── tasks
    │ ├── templates
    │ └── vars
    └── web
        ├── files
        ├── handlers
        ├── library
        ├── tasks
        ├── templates
        └── vars

Here the web and db roles are subrepos of the voltron-repo. The
problem that comes into play is what if I want to re-use the
bender.yml and setup_web.yml playbooks from the ansible-web repo?

One (hacky) solution I can think of is running a script that would
copy those playbooks into the root of the voltron project, or
checkout the ansible-web repo

Another idea is something like this. Place all the subprojects (roles
and associative playbooks) into a "subprojects" folder that lives at
the root of the project. Subrepo all of dependent projects into the
"subprojects" folder. Symlink the roles and playbooks to their proper
locations. It would end up looking something like this:

voltron_project:

.
├── bender.yml -> subprojects/web/bender.yml
├── roles
│ ├── db -> ../subprojects/db/roles/db
│ └── web -> ../subprojects/web/roles/web
├── setup_db.yml -> subprojects/db/setup_db.yml
├── setup_web.yml -> subprojects/web/setup_web.yml
└── subprojects
    ├── db
    │ ├── roles
    │ │ └── db
    │ │ ├── files
    │ │ ├── handlers
    │ │ ├── library
    │ │ ├── tasks
    │ │ ├── templates
    │ │ └── vars
    │ └── setup_db.yml
    └── web
        ├── bender.yml
        ├── roles
        │ └── web
        │ ├── files
        │ ├── handlers
        │ ├── library
        │ ├── tasks
        │ ├── templates
        │ └── vars
        └── setup_web.yml

Symlinks in git make me feel icky, but this is just to get a
conversation started. This keeps the playbook outside of the role (as
it's not a roles purpose to house playbooks), but is messier.

Another thought would be to just include a subdir in the role that
provided playbooks that used that role exclusively. Then just those
playbooks could be symlinked into the project.

Something like this:

.
├── bender.yml -> roles/web/playbooks/bender.yml
├── roles
│ ├── db
│ │ ├── files
│ │ ├── handlers
│ │ ├── library
│ │ ├── playbooks
│ │ │ └── setup_db.yml
│ │ ├── tasks
│ │ ├── templates
│ │ └── vars
│ └── web
│ ├── files
│ ├── handlers
│ ├── library
│ ├── playbooks
│ │ ├── bender.yml
│ │ └── setup_web.yml
│ ├── tasks
│ ├── templates
│ └── vars
├── setup_db.yml -> roles/db/playbooks/setup_db.yml
├── setup_web.yml -> roles/web/playbooks/setup_web.yml
└── site.yml

If you made it this far, thanks!

James

Hi,

In my opinion you are over-complicating in trying to have re-usable playbooks. If you only have re-usable roles, than it is almost obvious how the structure should look like.

Greetings,
gw

Gw,

Yes, the structure if I'm only re-using roles is obvious. It's just
that some of the playbooks I have are complex and I'd like to only
have only one place where they need to be updated.

- James

Right, roles are the main unit of reusability.

If you want to include playbooks inside of other playbooks, you of course can.

Ultimately, I hate to be the “TLDR” guy, but it would be useful if you could boil down the question to the most salient point fo me.

Hi list,

I think I'm dealing with a similar problem where I'm managing multiple
independent sites with servers, that may resuse roles, but the primary
'site.yml' as well as a subdir with site specific vars is located in a
site specific subdir, and I use a separate common location for
reusable roles & playbooks.

e.g instead of as what's documented at
http://www.ansibleworks.com/docs/playbooks.html
  site.yml
  [..]
  roles/
    [..]

I am trying to manage it like this:
  common/webserver.yml
  common/roles/
  common/roles/[..]

  site1/site.yml
  site1/vars/{{variable}}.yml
  [..]

  site2/site.yml
  site2/vars/{{variable}}.yml
  [..]

In siteX/site.yml, I would like to specify:

Right, roles are the main unit of reusability.

If you want to include playbooks inside of other playbooks, you of course can.

Ultimately, I hate to be the “TLDR” guy, but it would be useful if you could boil down the question to the most salient point fo me.

There was a TLDR, granted it was after the first paragraph so maybe it was too late by then :slight_smile:

James

I tend to understand things better when we talk in terms of use cases rather than syntax, I guess.

If you need different variables for different sites, this is exactly what “group_vars/” is for, so this may be a question of trying to apply the wrong solution (“roles”) to a problem (“needing to be able to set variables on a site specific basis”)

Hi Michael,

Thanks for your reply.

I tend to understand things better when we talk in terms of use cases
rather than syntax, I guess.

If you need different variables for different sites, this is exactly what
"group_vars/<groupname>" is for, so this may be a question of trying to
apply the wrong solution ("roles") to a problem ("needing to be able to set
variables on a site specific basis")

When I say sites, I really meant customers. I have a project
structure per customer, that exists in a per-customer directory.
Things like the anisble config file for that customer's hosts,
customer variables etc are stored here.

I then have reusable roles that I want to store centrally without
duplicating it per customer project structure.

I'd like to avoid having separate customers variable files co-existing
under some point in one project structure tree.

Does that explain my use case more clearly?

Thanks,

Chris

I second that use case. We are in the same boat:

As a service provider, we introduce our own best practices to all different customer projects and we want to migrate all projects to Ansible. Now, as an example, there is the role “apache” which contains everything to install and configure the Apache server which of course is driven by variables to cope with customer specific setups. But the general tasks, handlers, templates, etc. are all the same.

What I was thinking is that we could have everything in one single directory tree and handle the customer specific stuff by different inventories. That would already work apart from the fact that developers and admins from the customer get involved in the maintenance of the playbooks and that would mean that they get to see everything from our other customers too - which is not a good idea.

Hope this use case describes the requirement properly enough so that other experienced Ansible users can suggest adjustments or completely new ways.

What you want here is group_vars/.

Put all machines belonging to that one customer in a single group, and a different customer can set different variables.

That is only one part of it. Yes, grouping hosts by customer makes a lot of sense.

However, the whole directory tree with many dozen files from roles, variables and files on my own developer machine contains the superset of everything across all my customers. I want to share the relevant files for each customer with them, but only what belongs to them on not what belongs to others.

Sure, this is easy to do with multiple repos.

You can specify roles dirs outside the repo if you want:

roles:

  • { role: /path/to/other/role, … }
  • { role: some_role_that_is_assumed_to_be_in_roles_subdirectory, … }

And the same with vars_files and normal task includes.

So having a repo per customer, I think, is what you’d want to do.

Each repo could include the group_vars in their repo alongside the playbook if on 1.2+ and then could path the “internal” stuff using specified paths.

This is the same way some folks like to handle secret information like passwords while also open-sourcing or sharing their main repo.

Sounds very promising, I just may have one missing piece. Let’s have a look at an example:

/somepath/ansible/
./common/
./group_vars/
./host_vars/
./roles/
/apache/
/mysql/
/security/
./master.yml
./customer1/
./group_vars/
./host_vars/
./roles/
./hosts
./customer2/
./group_vars/
./host_vars/
./roles/
./hosts

So, each customer has their own hosts file with the inventory. And they all have their group_vars and host_vars directory for their specific variables.

Question: In the structure above i have group_vars and host_vars relative to the hosts file and relative to the master.yml playbook. How do I get ansible-playbook to first read the variables from below the master.yml and then potentially overwrite them from below the inventory?

If that was possible I guess it would already be the perfect solution.

The group_vars/host_vars stuff can only go alongside the top level playbook AND the inventory file directory.

You probably want to use vars_files or roles instead, to effectively do something like the same thing.

Just path the role or vars_files you need to load in.

also sprach Michael DeHaan <michael@ansibleworks.com> [2013.06.20.1829 +0200]:

roles:
   - { role: /path/to/other/role, ... }

Nice, I didn't know this was possible. Would you consider extending
this to allow for relative paths? Right now, the behaviour is a bit
weird:

  fishbowl:…a/ansible|master|playbooks% cat roles/motd/playbook.yml
  - name: manage /etc/motd
    hosts: motd_hosts:!down
    sudo: yes
    roles:
      - { role: .. }

  fishbowl:…a/ansible|master|playbooks% grep name: roles/motd/tasks/main.yml
  - name: write the motd file
  - name: remove old motd.tail file

  fishbowl:…a/ansible|master|playbooks% ansible-playbook roles/motd/playbook.yml --list-tasks

  playbook: roles/motd/playbook.yml

    play #1 (manage /etc/motd): task count=5
      write the motd file
      remove old motd.tail file

The weirdness is in the role:.. (two dots, parent directory)
statement. What I want is a way to use a role *in the same
directory* as the playbook. However, '.' does not work ("cannot find
role in …/roles/motd/roles"). I don't know why it looks for
a directory ./roles/. Using two dots '..' works, but I cannot
explain why.

I'd like to use this functionality and would like this behaviour to
be well-defined. Is that possible?

Cheers,

Tried this and can’t confirm yet.

I have a group_vars/all.yml file anlongside the the the top level playbook AND alongside the inventory file and only the variables from alongside the inventory file are known. From the example above I’m calling ansible-playbook -i …/customer1/hosts master.yml from within the master.yml directory but only variables from …/customer1/group_vars/all.yml are known. I’m missing the variables from ./group_vars/all.yml

And even worse, it looks as if variables are resolved differently. The playbook looks like this:

  • hosts: “{{ myhost }}”
    connection: local
    gather_facts: no
    tasks:
  • name: Test
    debug: msg=“Value of test is {{ test }} or $test and xyz is {{ xyz }} or $xyz”

When I call ansible-playbook master.yml --extra-vars “myhost=all” -i hosts

I gate this output: “Value of test is 1 or 1 and xyz is 23 or 23

When I call ansible-playbook master.yml --extra-vars “myhost=all” -i …/customer1/hosts

I gate this output: “Value of test is {{test}} or 2 and xyz is {{xyz}} or $xyz

Already supports relative paths.

The group_vars all location relative to the playbook must be relative to THE playbook being executed.

This is why all top level playbooks in ansible-examples live where they do.

I think it would be confusing if you had to store them in more than one place. In your example, I’d recommend storing playbooks at top level OR keeping them with your inventory.

Hi Michael,

roles:
   - { role: /path/to/other/role, ... }
   - { role: some_role_that_is_assumed_to_be_in_roles_subdirectory, ... }

Excellent, this works well for my use case.

I'd like to avoid duplicate pathing for a location that has common
roles though.

roles:
  - { role: /path/to/other/role1, ... }
  - { role: /path/to/other/role2, ... }

If there was an environment variable or playbook variable that was observed in
role location lookup, this could be managed without the playbook having static
paths.

Does variable subst work in the { role: [..] } section?

My tests with:
  vars:
    role_base_path: /path/to/other/

  roles:
    # any of
    - { role: ${role_base_path}/role1 }
    - { role: $role_base_path/role1 }
    - { role: {{ role_base_path }}/role1 }

all failed with varying errors.

If there was some form of role search path & precedence behaviour, you
could have one or more common role locations, plus the relative roles/
path, and simplify the playbook to
  roles:
    - webserver

Overriding roles in the relative roles path might also be useful for
testing & development as you could just copy over & modify the
relative role & not have to change the playbook.

Thanks for your help so far on this - I don't really think I'm
over-complicating things, being too clever, or programmatic, but if
you think this use case doesn't fit with ansible at this point in
time, I'll make do with the duplication and watch closely & hope it
comes in the future :slight_smile:

Regards,

Chris