How to setup projects with shared resources

Get Help
1

I have two projects (lets call them P1 and P2) setup in a collection-ish style that depend on a shared set of inventory and var files. The projects are a combination of playbooks and roles; theres no custom modules or any plugins really. All three are checked into source control as 3 different repos. The inventories are pretty small and statically defined. The shared vars are numerous enough that they are broken out into different files in group_vars.

Currently, I have a bash script in each of the projects that checks out the shared resources and then moves them into place. The inventory files are no problem, but the var files needs to be orchestrated a bit carefully so I dont have conflicts or accidentally overwrite a file that exists in both repos.

It works, but its pretty clunky. I was wondering if people had other ideas or solutions for a situation like this

1 Like
2

We have a similar setup: N git repos, one per service line, and another common repo (named mw-ansible-defaults) that contains a static inventory in .ini format and a group_vars/all/_mw-ansible-defaults.yml file with some “global” variables - snat pool cidrs, notification email addresses, common directory paths, proxy configs, etc. (We could have had other group_vars or host_vars files there, but we don’t.) The common repo changes very rarely, but when it does, commits to main trigger a Jenkins job which updates all N of our other projects and injects the global vars file(s) into those worktrees, and - if the result passes linting - pushes the updated main (or master; we’ve renamed most but not all, yet) branches back into our GitLab instance.

This seems to be the equivalent of your bash script, except ours is a 1→N push to our canonical upstreams triggered by changes in the common project rather than a N←1 pull to downstream worktrees initiated by each of the downstream projects.

All of our variables — and I mean all, even one-off registered task results — follow strict prefixing conventions, whether playbook or role specific. So for example all variables in the common repo have a mw_global_ prefix. That seemed a little wordy at first, but it eliminates the risk of accidental name conflicts, we can be pretty sure at a glance where a variable comes from. Once you get used to it though, un-prefixed variables look rather suspect, like they are a not-ready-for-production work-in-progress.

I’m not necessarily recommending this method, but it’s been very reliable for us.

3

The way I workarounded this issue is that I put shared variables into inventory directory and variables that are project specific are exclusively going into vars at root level of the project. To be more clear:

<project_dir>
├── group_vars
│   └── <example-group>        # <- here goes project level variables (playbook level)
├── host_vars
│   └── <example-host>         # <- also project level variables (playbook level)
├── inventory
│   ├── group_vars
│   │   └── <example-group>    # <- here goes shared variables (inventory level)
│   └── inventory              # <- inventory file
├── playbook.yml
└── requirements.yml

Shared variables (inventory level) have lower precedence than project variables (playbook level). This allows for project variables to override shared variables if the project has some exceptions to what is specified in shared variables. We consider shared variables as immutable (MUST never be changed by hand… except by the pull of new version). Since project and shared variables are clearly separated, project variables:

  1. are clean - only contain what is relevant to the project
  2. cannot be overwritten by importing shared variables

By making inventory level variables immutable I essentially lose one variable precedence level… but hey… it’s not like Ansible doesn’t have 20 of them :slightly_smiling_face:

1 Like
4

Are the group_vars at the playbook level imported automatically, or do you need an include_vars task?

5

Everything is imported automatically. In fact, I’ve never used include_vars :slightly_smiling_face:

6

Are you populating content in and under your ./inventory level from another project? If so, how?

7

Depends on what you are considering a project? Yes, the content in ./inventory directory is versioned in a separate Git repo (project) but it is not cloned from the repo. Instead it is downloaded as tar archive and extracted (git archive command). There is a script that does that + some additional stuff like version selection. The rest of the . is versioned in Git in other repo and cloned from it. Changes are pushed.

Content of ./inventory directory in fact looks like this:

<project_dir>
└── inventory
    ├── group_vars
    │   └── <huge hierarchy of var files>
    ├── 00-global         # <- shared inventory file with predefined group hierarchy, global
    ├── 01-client         # <- shared inventory file with predefined group hierarchy, client specific
    └── inventory         # <- per project inventory file, hand populated (for now)

So group_vars and inventory files 00-global and 01-client (shown here for example) are both kept in a repo from which they are downloaded. inventory file is your plain, hand written inventory, versioned in Git with the rest of the project.

Predefined groups (they are specified empty) all have some special meaning and they work in combination with group_vars. This allows for all kind of variable definitions to be made “up front”, on any level of the group hierarchy. As a user, you just have to put your hosts in specific groups at the bottom of the hierarchy and the host automatically gains all of the required variables, all properly defined for it. This ranges from e.g. variables describing specifics of the host physical location, VM platform, OS… to vault protected secrets for everything needed.

Lastly, we use inventory directory as… well… inventory directory. To be more clear:

ansible-playbook -i inventory/ playbook.yml
2 Likes
8

@bvitnik That is really helpful. I think your structuring helps protect the variables a lot.

Im glad you both had similar setups. Good to know I was on the right track and didnt miss something obvious

9

Thanks. I’m glad too. I was using this scheme for years and saw no one else use it. I’m glad to know other people came to similar ideas. It’s good to know you are not alone… and “doing it wrong”.