Jeroen Hoekx implemented a very nice "group_by" module.
You can see it in use here:
https://github.com/ansible/ansible/blob/devel/examples/playbooks/group_by.yml
--Michael
Jeroen is a hero !
This is beautiful. Covers a lot of use cases for writing a ENC.
Having evaluated this, I think we should urgently get together and think about the conceptual designs.
Because both this, and ansible-provisioning are neat ways of doing stuff with the current capabilities, but conceptually will harm us in the long run because we use the tools we have, rather than create the tools we need.
In my case this is because I refrain from refactoring core code because I don't know what direction we want to go and whether a large effort will be accepted.
I am convinced we can harmonize the advanced/complex needs of larger setups with the simplicity of the current inventory mechanism. And this covers both basic inventory, facts as well as grouping.
Conceptually, group_by is only useful for limiting plays, whereas a real inventory solution to inventory would cover:
- Parallel execution (use of 'ansible')
- Adding group facts
- Would be implemented on a higher level
(works with playbooks included playbooks, as well as the parent
playbooks without the need to redefine/reinclude the same parts)
A lot has already been said, posted, chatted by different people in the past (and I only saw fragments), but it is unclear where we are going.
Get together in person asap ? 
Because both this, and ansible-provisioning are neat ways of doing stuff
with the current capabilities, but conceptually will harm us in the long run
because we use the tools we have, rather than create the tools we need.
This isn't harming us.
This is a great way of doing certain things to certain hosts that
match certain criteria, like "on machines that run this particular
software, do this". Super useful.
That being said, you have some concerns about the idea of "super
dynamic inventory", where a play can modify inventory records. My
core belief is that a play should not dynamically modify inventory
records in any sort of way that can persist.
I also believe that, as when we started, we need to hold to keeping
things reasonably simple.
That's why, for more advanced cases, we have things like the ability
to write your own inventory source, and your own modules.
We're going to accept simple things that are useful into core, but I
don't personally think the idea of using a only_if statement to
dynamically add a temporary group variable is even remotely simple,
and starts to include levels of logic in playbooks that I would find
gross.
The aesthetic of Ansible now is something that is important to
maintain -- because it's supposed to be simple, accessible, and
maximally auditable.
Because both this, and ansible-provisioning are neat ways of doing stuff
with the current capabilities, but conceptually will harm us in the long run
because we use the tools we have, rather than create the tools we need.This isn't harming us.
This is a great way of doing certain things to certain hosts that
match certain criteria, like "on machines that run this particular
software, do this". Super useful.That being said, you have some concerns about the idea of "super
dynamic inventory", where a play can modify inventory records. My
core belief is that a play should not dynamically modify inventory
records in any sort of way that can persist.
I never stated that, in fact I believe a play should NOT change the inventory. Not sure where you get this idea that I would. The fact I don't like group_by is because it does.
I send a mail a few weeks ago clearly stating what I think we should do, I don't think anyone replied to it though.
I proposed a multi-source inventory (hosts/groups) approach where hosts are merged based on facts (and rules), I proposed facts-based grouping as a separate step after the inventory.
This is all done before running tasks (ansible) or running playbooks (ansible-playbook).
I also believe that, as when we started, we need to hold to keeping
things reasonably simple.
I agree.
That's why, for more advanced cases, we have things like the ability
to write your own inventory source, and your own modules.
I agree.
We're going to accept simple things that are useful into core, but I
don't personally think the idea of using a only_if statement to
dynamically add a temporary group variable is even remotely simple,
and starts to include levels of logic in playbooks that I would find
gross.
I never proposed this. Where do you get this from ?
The aesthetic of Ansible now is something that is important to
maintain -- because it's supposed to be simple, accessible, and
maximally auditable.
Sure.
I never stated that, in fact I believe a play should NOT change the
inventory. Not sure where you get this idea that I would. The fact I don't
like group_by is because it does.
The phrase "conceptually harm", seeing to imply group_by wasn't
awesome, I think, threw me.
I proposed a multi-source inventory (hosts/groups) approach where hosts are
merged based on facts (and rules), I proposed facts-based grouping as a
separate step after the inventory.
I have some plans around allowing the inventory file to be a
directory, allowing it to be full of a collection of individual
inventory files and external
resource scripts (conf.d style), and have mentioned it. This is
already in plan.
This (group_by) seems to be a great way to do facts based grouping.
I never proposed this. Where do you get this from ?
Hard to tell what you were proposing, never mind that part then.
It's a great idea, implemented on the wrong level, lacking group facts, harming a proper solution.
That's what "conceptually harm" was meant to be.
Before sending that mail I discussed this with Jeroen and verified that group facts were not established.
The fact that it interferes with playbook includes is just something that would affect me and make it not fit for my environment, but that is not an objection to the functionality. However, if done on another level if would be acceptable for all use-cases.
I never stated that, in fact I believe a play should NOT change the
inventory. Not sure where you get this idea that I would. The fact I
don't
like group_by is because it does.The phrase "conceptually harm", seeing to imply group_by wasn't
awesome, I think, threw me.It's a great idea, implemented on the wrong level, lacking group facts,
harming a proper solution.
I completely disagree, nor have you explained what "group facts" means to me.
Discussion needs to happen on list, not in private, so we can all
understand what we are talking about, nor do I understand
your apparent need for it, because that was not communicated.
Right now, I'm pretty well in the dark, plus I love this solution, and
I'm not pulling it.
There's no harm here.
I appear to have offended you. If you don't like group_by, that's
cool, just don't use it.
Ansible is pluggable, so you can write your own module.
But you can't go in and say what harms ansible here.
The fact that it interferes with playbook includes is just something that
would affect me and make it not fit for my environment, but that is not an
objection to the functionality. However, if done on another level if would
be acceptable for all use-cases.
I think it's acceptable for nearly everyone's use cases as is.
If your use cases are different, this doesn't mean it harms anything.
I never stated that, in fact I believe a play should NOT change the
inventory. Not sure where you get this idea that I would. The fact I
don't
like group_by is because it does.The phrase "conceptually harm", seeing to imply group_by wasn't
awesome, I think, threw me.It's a great idea, implemented on the wrong level, lacking group facts,
harming a proper solution.I completely disagree, nor have you explained what "group facts" means to me.
Facts set for a group, like group_vars.
Discussion needs to happen on list, not in private, so we can all
understand what we are talking about, nor do I understand
your apparent need for it, because that was not communicated.
I mentioned it now 2 times in public already, the fact that I discussed this with Jeroen in advance was to make sure that
1. I wasn't mistaken
2. it's a polite way before making it public.
Groups are used for 2 things in Ansible as far as I know:
- limiting plays (e.g. only run this play for webservers)
- setting facts per group (e.g. we use these NTP servers in datacenter brussels for production)
So you expect that when you create a group (using group_by), the variables set for a group are taken into account.
I appear to have offended you. If you don't like group_by, that's
cool, just don't use it.
You have not offended me, if anything I get frustrated by repeating the same things over and over again and not being understood.
Ansible is pluggable, so you can write your own module.
But you can't go in and say what harms ansible here.
I never asked it to be pulled, I merely warned about it not covering all use-cases. And we get into this negative spiral about disagreements, etc...
And I didn't say it harms Ansible as such, I warned it could harm a proper solution to be devised.
Can we now please get to the technical stuff ?
- You stated to dislike plays to change inventory, why is it ok for group_by ?
- Shouldn't group_by consider group variables ?
Facts set for a group, like group_vars.
So we already support inventory sources.
We're going to allow a conf.d style directory of inventory sources for
0.9 as well, and they will auto-merge so you can define group
membership in one and vars in another and it will just work.
I mentioned it now 2 times in public already, the fact that I discussed this
with Jeroen in advance was to make sure that
on this list? We're not on IRC all the time. Only the list is official.
Jeroen is an extremely great contributor, but I need to be made aware
of these things.
Groups are used for 2 things in Ansible as far as I know:
- limiting plays (e.g. only run this play for webservers)
- setting facts per group (e.g. we use these NTP servers in datacenter
brussels for production)
not facts, but variables.
So you expect that when you create a group (using group_by), the variables
set for a group are taken into account.
group_by doesn't have a way to create any variables for that group yet, I agree.
I also don't think this is super limiting, but it would be also very
easy to allow it to take a list of key=value pairs to assign some
variables
to it, if that's what people wanted.
You have not offended me, if anything I get frustrated by repeating the same
things over and over again and not being understood.
Computers are hard
- You stated to dislike plays to change inventory, why is it ok for
group_by ?
If we were talking about yesterday, I think we were discussing
persisting inventory, such as when a call to one of Seth's EC2 modules
creates a new host and you want to persist it by adding it to a group.
That's different.
- Shouldn't group_by consider group variables ?
I can see that being added later by a pull request if you want to say
group_by key=foo a=1 b=2
Doesn't seem to be super important because you could just use a "vars"
section on the play that later referenced the group, but I'm not
against it.
I never stated that, in fact I believe a play should NOT change the
inventory. Not sure where you get this idea that I would. The fact I don't
like group_by is because it does.The phrase "conceptually harm", seeing to imply group_by wasn't
awesome, I think, threw me.
Ok, so you dislike the fact that a play can change the inventory (which I agree), yet you accept group_by which does exactly this, but doesn't cover group_facts, which is highly confusing. What you call simple, I call incomplete and unwelcome.
Even though the example is neat, this functionality belongs on a higher level. Something I did explain in a mail a few weeks back (after a brainstorm with Jeroen).
I proposed a multi-source inventory (hosts/groups) approach where hosts are
merged based on facts (and rules), I proposed facts-based grouping as a
separate step after the inventory.I have some plans around allowing the inventory file to be a
directory, allowing it to be full of a collection of individual
inventory files and external
resource scripts (conf.d style), and have mentioned it. This is
already in plan.
On IRC (which is by the way one of the worst ways to communicate ideas and discuss them) I already warned about:
- source priorities
- conflicting facts
- merge rules
This (group_by) seems to be a great way to do facts based grouping.
Except that:
- it does not take care of group facts (which is part of the inventory)
- it is only set as part of a play, so you have to repeat/include the
same facts-based groups in every playbook which breaks playbooks
including plays/playbooks
- it should really be part of the inventory phase (of course, that
requires some indepth thought about how to achieve this)
I never proposed this. Where do you get this from ?
Hard to tell what you were proposing, never mind that part then.
I send a mail once though.
- You stated to dislike plays to change inventory, why is it ok for
group_by ?If we were talking about yesterday, I think we were discussing
persisting inventory, such as when a call to one of Seth's EC2 modules
creates a new host and you want to persist it by adding it to a group.
That's different.
I don't think we discussed this yesterday, and it does not match with my needs or with the concepts. Maybe you confused me with another conversation ? (I was offline for the large part in the afternoon when you probably where online because of 3G issues)
I don't even create VMs on the fly myself (Seth and Jeroen do), it would be neat to do that and for a workshop or presentation that could be very useful. (I used Jeroen's module for this at T-DOSE). But on the fly-creation is not something I have all the answers for and personally would prefer to delay until te project has a clearer idea what this brings to the table. (Technically we would need to first understand more virtualization/cloud technologies)
What I currently already do is provision a system that is not already installed or powered on. The difference here is that the 'asset' already exists outside of Ansible. I have no need to have Ansible take care of managing assets or inventory, it only needs to source them (more dynamically).
- Shouldn't group_by consider group variables ?
I can see that being added later by a pull request if you want to say
group_by key=foo a=1 b=2
Doesn't seem to be super important because you could just use a "vars"
section on the play that later referenced the group, but I'm not
against it.
No, what I mean is, if I make a group 'brussels-production' I expect my pre-existing variables in groups_vars/brussels-production to be set for my host.
I don't necessarily want to start setting group variables in plays (maybe others do), for me it mixes inventories/variables/actions too much.
We have a clear separation of the inventory, facts/variables and actions
(except when those variables are very specific to a play in isolated cases)
For that reason I am not a big fan of vars, vars_files and even group_by.