idea: allow group_vars/ and host_vars/ files to be organised in sub folders

Archives Ansible Project
1

Hi list,

My use case:

At $WORK, we divise our ansible repo in multiple projects/* dirs, where all projects have their own hosts file and group_vars/; a wrapper script merges all these together in $basedir; basically having all group_vars files in one big directory is too big here (we have 800 hosts and almost 300 groups).
(This should better be handled by some backend + inventory script, but for now, that’s what we have. )

Now, one optimisation, would be for us to get rid of that part in the wrapper script, basically allowing to organise goup_vars/ and host_vars/ files into subdirectories.

Basically what I want is, instead of having

group_vars/

group_vars/project1group1

group_vars/project1group2

group_vars/project2group1

group_vars/project2group2

I want to have this:

group_vars/

group_vars/project1/project1group1

group_vars/project1/project1group2

group_vars/project1/project2group1

group_vars/project1/project2group2

The only problem I see with that, is that would allow to have multiple files for the same group, such as

group_vars/

group_vars/project1/project1group1

group_vars/project2/project1group1

Which would probably confusing, and not clear which logic is implemented on which variable gets precedence when that same variable is defined in multiple files with the same group name, hence for the same group.
(in our case, we just wouldn’t have that, by convention, but it would be possible to abuse)

group_vars/ and host_vars/ are handled by the vars_plugins/group_vars plugin, and looking at the code, that could be done with a small and simple patch.

Your ideas?

Serge

2

How would ansible know which subdirectory to use?

3

​It doesn't have to, just parse all the files it can find.
The intermediatie $project directory is just for organisational purposes.

As you said on irc, parsing all leaf nodes with matching filenames.

  Serge​

4

I’m fine with this, we should make it also ignore file extensions at the same time.

5

Is this already implemented in a current Ansible version?

6

​Yes, instead of e.g. a group_vars/groupname.yml, you can​ now have
group_vars/groupname/ directory with multiple files in it.

7

I just tried this on 1.9.1 I have /group_vars/staging/staging.yml and /group_vars/staging/secrets.yml and it seems to me that it is not merging the files properly.

I have an api variable in the staging file which doesnt exist in the secrets file. When I run the playbook it complains that the api variable doesnt exist. If I either remove the secrets file or add the api variable to the secrets file then it works.

8

​If that would be true it would be a bug, and you would need to present us
a minimal example on how to reproduce it.

That being said, I use ​this feature everywhere around and i know i works.
It might be some corner case of course, which would only become clear when
being able to reproduce it.

Can you make a minimal example that reproduces this, and show us the needed
files?

Thanks,

Serge

9

I am curious if there was follow-up to this. I am seeing something similar for a customer I am working on.

  • If I put all my variables under group_vars/all (where all is a flat file in yml syntax) it works fine.

  • If I move one device out of group_vars/all to group_vars/leaf/leaf01 all devices work great. (where leaf corresponds to a group within my Ansible hosts file).

  • As soon as I do group_vars/leaf/leaf01 and leaf02 it can’t find leaf01 anymore. leaf02 will work but not leaf01. (where leaf01 and leaf02 are hosts in my ansible hosts file both in the group leaf)
    Do they have to end with .yml? Is this not how group_vars works? Just trying to make folders to break up variables for simplification.
    (I am running on Ansible 2.1.1)

10

Are you sure that's the way things are supposed to work?

I'd have expected individual hosts to be under host_vars/leaf01 , not
nested under a group (remember, a host can be in multiple groups).

so your layout would be more:

.
├── group_vars
│ ├── all
│ └── leaf
├── host_vars
│ ├── leaf01
│ └── leaf02
└── hosts

11

In this thread though Dehaan said this should work->

I just tried this on 1.9.1 I have /group_vars/staging/staging.yml and /group_vars/staging/secrets.yml and it seems to me that it is not merging the files properly.

But others said this was a problem. I just don’t think this is documented well. If its not supposed to work I guess it shouldn’t work :frowning:

12

According to the documentation it should work
https://docs.ansible.com/ansible/intro_inventory.html#splitting-out-host-and-group-specific-data

Quote from the page:
"As an advanced use-case, you can create directories named after your groups or hosts, and Ansible will read all the files in these directories. An example with the ‘raleigh’ group:

   /etc/ansible/group_vars/raleigh/db_settings
   /etc/ansible/group_vars/raleigh/cluster_settings

All hosts that are in the ‘raleigh’ group will have the variables defined in these files available to them. This can be very useful to keep your variables organized when a single file starts to be too big, or when you want to use Ansible Vault on a part of a group’s variables. Note that this only works on Ansible 1.4 or later."

13

In this thread though Dehaan said this should work->

I just tried this on 1.9.1 I have /group_vars/staging/staging.yml and
/group_vars/staging/secrets.yml and it seems to me that it is not merging
the files properly.

But others said this was a problem. I just don't think this is documented
well. If its not supposed to work I guess it shouldn't work :frowning:

According to the documentation it should work
https://docs.ansible.com/ansible/intro_inventory.html#splitting-out-host-and-group-specific-data

And it does. I use this feature heavily since before ansible 1.9, never had a problem.
Be careful to not duplicate variables definitions. Starting from version 2 ansible issue a warning if a variable has been defined more than once, but if I remember well it was silent on older versions.

--
Kai Stian Olstad

Ciao
Andrea