Request for Comments on ideas for a new Ansible Inventory `2.0'

Dear All,

I have been brainstorming for a while, on what should or could be done in redesigning a new Ansible Inventory. I did write up my ideas, in three blog posts, scattered over the last three months, which you can read at following links. I started with writing down some generic concepts I thought should be solved, and in the thirs post, I came up with some list of features.

  1. https://serge.vanginderachter.be/2016/current-state-of-the-ansible-inventory-and-how-it-might-evolve/

  2. https://serge.vanginderachter.be/2016/first-design-ideas-ansible-inventory-2-0/

  3. https://serge.vanginderachter.be/2017/ansible-inventory-2-0-design-rules/

Now, this is far from finished, and only a first step in a thought process… I’ve been weary of overthinking this, and tried to come up with some simple rules. But even the simplest of examples, can get quite complex, quite quickly. I hope at least some basic and fundamental ideas can be parsed from this. If inventory interests you, please take some time to read those articles, and feel free to get back to me with questions. I’m also eager to hear your ideas, as this is only based on my experiences. My nick on Freenode is svg if you want to chat (I’m in ansible-devel) and my hours are CET - Central Europe.

Kind regards,

Serge van Ginderachter

Some of the things brought up in blog have already been discussed and/or are under way, I’ll try to link existing info on those, for others I’m going to comment inline.

One thing we have been planning since 2.0 (but keep delaying) is making it real plugins again: https://github.com/ansible/proposals/issues/41, this opens up to features like a common cache, cache invalidation, common config, etc.

A feature that I was contemplating, after making them all ‘real plugins’, is to state ‘enabled plugins and order’. It would default to ‘dir,script,ini,yaml’ to keep the current behaviour, each plugin requires a state from the target (is dir? is executable? is file?).

On the issue of ‘unrelated/equal depth’ groups variable precedence or ‘deterministic configurable hierarchy’, there is a new variable ‘ansible_group_priority’, introduced in 2.2 but not used yet, which was planned to allow the user to define specific ‘variable merge priority’. I’m hoping to add the merging logic in 2.3 or 2.4, keeping the current group name sorting as the ‘tie breaker’ and default when priorities are absent.


Root/normal/merge groups​: I don’t see much use of making this additional classification, I would argue that these already exist and have to do with how users define the groups. I don’t see the need for having a declaration and/or restriction on this.

Constructed/dynamic/pattern groups: this is something I see as very useful and had thought of adding a plugin that would deal with these (group3 = group1:&group2)

Role var spec/discovery ​
https://github.com/ansible/proposals/issues/39

​Role versioning ​
https://github.com/ansible/proposals/issues/23

As for the variable merging and or scoping to targeted group​, I think it is already too complex and that adding more on top is going to make it harder and not easier.
When a machine is a member of 2 groups and you duplicate variables you need to think, do I want the same config to change on the machine if I refer to it as group1 or group2?
​ This seems counter intuitive to me​, that the variable or group classification are incorrectly scoped.

I know plenty of people disagree with me on this one, in my experience most of their group structure was derived from other systems that did not look at the host as a whole but as a partial snapshot from a certain viewpoint, which can cause conflicts or ‘bouncing’ when each viewpoint is ‘executed’. I think the current view from Ansible makes more sense, it looks at the host as a whole and gives you 1 truth for it.

For example, the most common case I get is cluster/master, master machines are part of the cluster and tend to have a subset of settings that need to differ, the easy solution is making the ‘masters’ group a child of cluster, so it inherits the general settings but can override what is needed, the ansible_group_priority mentioned above should solve most other cases that do not fit parent/child. The other is application vs environment (dev/stage/prod) groups, which can be solved teh same way.

That said, I do agree that we need better information on how/where variables are set.

Correction on my previous email: i mean @svg … not @sivel, though he has contributed to these discussions my tab completion gave me the wrong nick …

Brian, all,

I always intended to get back on this thread, but it seems I forgot about
it.

As for the variable merging and or scoping to targeted group​, I think it
is already too complex and that adding more on top is going to make it
harder and not easier.
When a machine is a member of 2 groups and you duplicate variables you
need to think, do I want the same config to change on the machine if I
refer to it as group1 or group2?
​ This seems counter intuitive to me​, that the variable or group
classification are incorrectly scoped.

I know plenty of people disagree with me on this one, in my experience
most of their group structure was derived from other systems that did not
look at the host as a whole but as a partial snapshot from a certain
viewpoint, which can cause conflicts or 'bouncing' when each viewpoint is
'executed'. I think the current view from Ansible makes more sense, it
looks at the host as a whole and gives you 1 truth for it.

​First, I'm not advocating for the group scopes yielding different values.
I definitely agree with Brian on this​. Also agree on keeping things simple.

That being said, there are still a couple of problems I'd like to get
solved. On the top of my head:

* having an overview of a big inventory with hundreds of deeply nested
groups is hard, but that could be solved with alternative implementations
once we have the inventory as plugins merged in
* especially keeping track of variables, its values and where to override
them, or not, can be a problem; the problem of chaning a default value and
changing it for too many instances
* vars that hold application settings, and that should listify when
applying: e.g. generating a list of apache virtualhosts, based on separate
vars in different groups that define a single site, see also what puppet
solves with exported resources
* as for the cluster with master example: how do we handle the fact that a
different node could have become master after the inmitial deploy?

Serge

Another tool I have in PR is ansible-inventory (based on the already
existing 3rd party ansible-inventory and ansible-inventory-grapher).

My intention for this tool was to give more introspection into the
inventory , but primarily to allow for easy migration for those using
'pre plugins' inventory as it outputs valid 'script Ansible JSON
inventory' and 'static Ansible YAML inventory'.

A feature I wanted to add to this tool was 'variable origin' to show
the location from which a variable was set in inventory ... but this
might be a bit more complex than the tool can handle and would need
hooks in the inventory itself, increasing memory footprint and slowing
down execution. .

Currently I'm considering adding debugging to be able to show
when/where a variable is set.

I think this is the single most useful and flexible feature you could
implement. I've been hoping to see it for quite a while.

--g