# New "group\_by" module **URL:** \ **Category:** Ansible Project\ **Created:** [October 31, 2012, 12:06am UTC](https://forum.ansible.com/t/new-group-by-module/12054 "2012-10-31T00:06:11Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [October 31, 2012, 12:06am UTC](https://forum.ansible.com/t/new-group-by-module/12054/1 "2012-10-31T00:06:11Z")
Jeroen Hoekx implemented a very nice "group\_by" module. You can see it in use here: [https://github.com/ansible/ansible/blob/devel/examples/playbooks/group\_by.yml](https://github.com/ansible/ansible/blob/devel/examples/playbooks/group_by.yml) --Michael ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 12:09am UTC](https://forum.ansible.com/t/new-group-by-module/12054/2 "2012-10-31T00:09:10Z")
Jeroen is a hero ! ---
**Author:** ![Romeo\_Theriault](https://avatars.discourse-cdn.com/v4/letter/r/839c29/32.png) [@Romeo\_Theriault](https://forum.ansible.com/u/Romeo_Theriault)\ **Post date:** [October 31, 2012, 12:46am UTC](https://forum.ansible.com/t/new-group-by-module/12054/3 "2012-10-31T00:46:26Z")
This is beautiful. Covers a lot of use cases for writing a ENC. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 10:20am UTC](https://forum.ansible.com/t/new-group-by-module/12054/4 "2012-10-31T10:20:00Z")
Having evaluated this, I think we should urgently get together and think about the conceptual designs. Because both this, and ansible-provisioning are neat ways of doing stuff with the current capabilities, but conceptually will harm us in the long run because we use the tools we have, rather than create the tools we need. In my case this is because I refrain from refactoring core code because I don't know what direction we want to go and whether a large effort will be accepted. I am convinced we can harmonize the advanced/complex needs of larger setups with the simplicity of the current inventory mechanism. And this covers both basic inventory, facts as well as grouping. Conceptually, group\_by is only useful for limiting plays, whereas a real inventory solution to inventory would cover:   - Parallel execution (use of 'ansible')   - Adding group facts   - Would be implemented on a higher level     (works with playbooks included playbooks, as well as the parent     playbooks without the need to redefine/reinclude the same parts) A lot has already been said, posted, chatted by different people in the past (and I only saw fragments), but it is unclear where we are going. Get together in person asap ? 🙂 ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [October 31, 2012, 11:08am UTC](https://forum.ansible.com/t/new-group-by-module/12054/5 "2012-10-31T11:08:58Z")
> Because both this, and ansible-provisioning are neat ways of doing stuff > with the current capabilities, but conceptually will harm us in the long run > because we use the tools we have, rather than create the tools we need. This isn't harming us. This is a great way of doing certain things to certain hosts that match certain criteria, like "on machines that run this particular software, do this". Super useful. That being said, you have some concerns about the idea of "super dynamic inventory", where a play can modify inventory records. My core belief is that a play should not dynamically modify inventory records in any sort of way that can persist. I also believe that, as when we started, we need to hold to keeping things reasonably simple. That's why, for more advanced cases, we have things like the ability to write your own inventory source, and your own modules. We're going to accept simple things that are useful into core, but I don't personally think the idea of using a only\_if statement to dynamically add a temporary group variable is even remotely simple, and starts to include levels of logic in playbooks that I would find gross. The aesthetic of Ansible now is something that is important to maintain -- because it's supposed to be simple, accessible, and maximally auditable. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 12:32pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/6 "2012-10-31T12:32:25Z")
> > Because both this, and ansible-provisioning are neat ways of doing stuff > > with the current capabilities, but conceptually will harm us in the long run > > because we use the tools we have, rather than create the tools we need. > > This isn't harming us. > > This is a great way of doing certain things to certain hosts that > match certain criteria, like "on machines that run this particular > software, do this". Super useful. > > That being said, you have some concerns about the idea of "super > dynamic inventory", where a play can modify inventory records. My > core belief is that a play should not dynamically modify inventory > records in any sort of way that can persist. I never stated that, in fact I believe a play should NOT change the inventory. Not sure where you get this idea that I would. The fact I don't like group\_by is because it does. I send a mail a few weeks ago clearly stating what I think we should do, I don't think anyone replied to it though. I proposed a multi-source inventory (hosts/groups) approach where hosts are merged based on facts (and rules), I proposed facts-based grouping as a separate step after the inventory. This is all done before running tasks (ansible) or running playbooks (ansible-playbook). > I also believe that, as when we started, we need to hold to keeping > things reasonably simple. I agree. > That's why, for more advanced cases, we have things like the ability > to write your own inventory source, and your own modules. I agree. > We're going to accept simple things that are useful into core, but I > don't personally think the idea of using a only\_if statement to > dynamically add a temporary group variable is even remotely simple, > and starts to include levels of logic in playbooks that I would find > gross. I never proposed this. Where do you get this from ? > The aesthetic of Ansible now is something that is important to > maintain -- because it's supposed to be simple, accessible, and > maximally auditable. Sure. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [October 31, 2012, 12:38pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/7 "2012-10-31T12:38:32Z")
> I never stated that, in fact I believe a play should NOT change the > inventory. Not sure where you get this idea that I would. The fact I don't > like group\_by is because it does. The phrase "conceptually harm", seeing to imply group\_by wasn't awesome, I think, threw me. > I proposed a multi-source inventory (hosts/groups) approach where hosts are > merged based on facts (and rules), I proposed facts-based grouping as a > separate step after the inventory. I have some plans around allowing the inventory file to be a directory, allowing it to be full of a collection of individual inventory files and external resource scripts (conf.d style), and have mentioned it. This is already in plan. This (group\_by) seems to be a great way to do facts based grouping. > I never proposed this. Where do you get this from ? Hard to tell what you were proposing, never mind that part then. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 1:28pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/8 "2012-10-31T13:28:50Z")
It's a great idea, implemented on the wrong level, lacking group facts, harming a proper solution. That's what "conceptually harm" was meant to be. Before sending that mail I discussed this with Jeroen and verified that group facts were not established. The fact that it interferes with playbook includes is just something that would affect me and make it not fit for my environment, but that is not an objection to the functionality. However, if done on another level if would be acceptable for all use-cases. ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [October 31, 2012, 1:36pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/9 "2012-10-31T13:36:53Z")
> > > I never stated that, in fact I believe a play should NOT change the > > > inventory. Not sure where you get this idea that I would. The fact I > > > don't > > > like group\_by is because it does. > > > > The phrase "conceptually harm", seeing to imply group\_by wasn't > > awesome, I think, threw me. > > It's a great idea, implemented on the wrong level, lacking group facts, > harming a proper solution. I completely disagree, nor have you explained what "group facts" means to me. Discussion needs to happen on list, not in private, so we can all understand what we are talking about, nor do I understand your apparent need for it, because that was not communicated. Right now, I'm pretty well in the dark, plus I love this solution, and I'm not pulling it. There's no harm here. I appear to have offended you. If you don't like group\_by, that's cool, just don't use it. Ansible is pluggable, so you can write your own module. But you can't go in and say what harms ansible here. > The fact that it interferes with playbook includes is just something that > would affect me and make it not fit for my environment, but that is not an > objection to the functionality. However, if done on another level if would > be acceptable for all use-cases. I think it's acceptable for nearly everyone's use cases as is. If your use cases are different, this doesn't mean it harms anything. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 1:48pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/10 "2012-10-31T13:48:46Z")
> > > > I never stated that, in fact I believe a play should NOT change the > > > > inventory. Not sure where you get this idea that I would. The fact I > > > > don't > > > > like group\_by is because it does. > > > > > > The phrase "conceptually harm", seeing to imply group\_by wasn't > > > awesome, I think, threw me. > > > > It's a great idea, implemented on the wrong level, lacking group facts, > > harming a proper solution. > > I completely disagree, nor have you explained what "group facts" means to me. Facts set for a group, like group\_vars. > Discussion needs to happen on list, not in private, so we can all > understand what we are talking about, nor do I understand > your apparent need for it, because that was not communicated. I mentioned it now 2 times in public already, the fact that I discussed this with Jeroen in advance was to make sure that   1. I wasn't mistaken   2. it's a polite way before making it public. Groups are used for 2 things in Ansible as far as I know:   - limiting plays (e.g. only run this play for webservers)   - setting facts per group (e.g. we use these NTP servers in datacenter brussels for production) So you expect that when you create a group (using group\_by), the variables set for a group are taken into account. > I appear to have offended you. If you don't like group\_by, that's > cool, just don't use it. You have not offended me, if anything I get frustrated by repeating the same things over and over again and not being understood. > Ansible is pluggable, so you can write your own module. > > But you can't go in and say what harms ansible here. I never asked it to be pulled, I merely warned about it not covering all use-cases. And we get into this negative spiral about disagreements, etc... And I didn't say it harms Ansible as such, I warned it could harm a proper solution to be devised. Can we now please get to the technical stuff ?   - You stated to dislike plays to change inventory, why is it ok for group\_by ?   - Shouldn't group\_by consider group variables ? ---
**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\ **Post date:** [October 31, 2012, 1:56pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/11 "2012-10-31T13:56:01Z")
> Facts set for a group, like group\_vars. So we already support inventory sources. We're going to allow a conf.d style directory of inventory sources for 0.9 as well, and they will auto-merge so you can define group membership in one and vars in another and it will just work. > I mentioned it now 2 times in public already, the fact that I discussed this > with Jeroen in advance was to make sure that on this list? We're not on IRC all the time. Only the list is official. Jeroen is an extremely great contributor, but I need to be made aware of these things. > Groups are used for 2 things in Ansible as far as I know: > > - limiting plays (e.g. only run this play for webservers) > - setting facts per group (e.g. we use these NTP servers in datacenter > brussels for production) not facts, but variables. > So you expect that when you create a group (using group\_by), the variables > set for a group are taken into account. group\_by doesn't have a way to create any variables for that group yet, I agree. I also don't think this is super limiting, but it would be also very easy to allow it to take a list of key=value pairs to assign some variables to it, if that's what people wanted. > You have not offended me, if anything I get frustrated by repeating the same > things over and over again and not being understood. Computers are hard 🙂 > - You stated to dislike plays to change inventory, why is it ok for > group\_by ? If we were talking about yesterday, I think we were discussing persisting inventory, such as when a call to one of Seth's EC2 modules creates a new host and you want to persist it by adding it to a group.   That's different. > - Shouldn't group\_by consider group variables ? I can see that being added later by a pull request if you want to say group\_by key=foo a=1 b=2 Doesn't seem to be super important because you could just use a "vars" section on the play that later referenced the group, but I'm not against it. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 1:57pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/12 "2012-10-31T13:57:18Z")
> > I never stated that, in fact I believe a play should NOT change the > > inventory. Not sure where you get this idea that I would. The fact I don't > > like group\_by is because it does. > > The phrase "conceptually harm", seeing to imply group\_by wasn't > awesome, I think, threw me. Ok, so you dislike the fact that a play can change the inventory (which I agree), yet you accept group\_by which does exactly this, but doesn't cover group\_facts, which is highly confusing. What you call simple, I call incomplete and unwelcome. Even though the example is neat, this functionality belongs on a higher level. Something I did explain in a mail a few weeks back (after a brainstorm with Jeroen). > > I proposed a multi-source inventory (hosts/groups) approach where hosts are > > merged based on facts (and rules), I proposed facts-based grouping as a > > separate step after the inventory. > > I have some plans around allowing the inventory file to be a > directory, allowing it to be full of a collection of individual > inventory files and external > resource scripts (conf.d style), and have mentioned it. This is > already in plan. On IRC (which is by the way one of the worst ways to communicate ideas and discuss them) I already warned about:   - source priorities   - conflicting facts   - merge rules > This (group\_by) seems to be a great way to do facts based grouping. Except that:   - it does not take care of group facts (which is part of the inventory)   - it is only set as part of a play, so you have to repeat/include the     same facts-based groups in every playbook which breaks playbooks     including plays/playbooks   - it should really be part of the inventory phase (of course, that     requires some indepth thought about how to achieve this) > > I never proposed this. Where do you get this from ? > > Hard to tell what you were proposing, never mind that part then. I send a mail once though. ---
**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\ **Post date:** [October 31, 2012, 2:10pm UTC](https://forum.ansible.com/t/new-group-by-module/12054/13 "2012-10-31T14:10:35Z")
> > - You stated to dislike plays to change inventory, why is it ok for > > group\_by ? > > If we were talking about yesterday, I think we were discussing > persisting inventory, such as when a call to one of Seth's EC2 modules > creates a new host and you want to persist it by adding it to a group. > That's different. I don't think we discussed this yesterday, and it does not match with my needs or with the concepts. Maybe you confused me with another conversation ? (I was offline for the large part in the afternoon when you probably where online because of 3G issues) I don't even create VMs on the fly myself (Seth and Jeroen do), it would be neat to do that and for a workshop or presentation that could be very useful. (I used Jeroen's module for this at T-DOSE). But on the fly-creation is not something I have all the answers for and personally would prefer to delay until te project has a clearer idea what this brings to the table. (Technically we would need to first understand more virtualization/cloud technologies) What I currently already do is provision a system that is not already installed or powered on. The difference here is that the 'asset' already exists outside of Ansible. I have no need to have Ansible take care of managing assets or inventory, it only needs to source them (more dynamically). > > - Shouldn't group\_by consider group variables ? > > I can see that being added later by a pull request if you want to say > > group\_by key=foo a=1 b=2 > > Doesn't seem to be super important because you could just use a "vars" > section on the play that later referenced the group, but I'm not > against it. No, what I mean is, if I make a group 'brussels-production' I expect my pre-existing variables in groups\_vars/brussels-production to be set for my host. I don't necessarily want to start setting group variables in plays (maybe others do), for me it mixes inventories/variables/actions too much. We have a clear separation of the inventory, facts/variables and actions (except when those variables are very specific to a play in isolated cases) For that reason I am not a big fan of vars, vars\_files and even group\_by.