# RFC: with\_nested\_dependents plugin (and the future of nested loops) **URL:** \ **Category:** Ansible Developer\ **Tags:** aws\ **Created:** [October 29, 2015, 3:48am UTC](https://forum.ansible.com/t/rfc-with-nested-dependents-plugin-and-the-future-of-nested-loops/19646 "2015-10-29T03:48:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Abhijit\_Menon-Sen1](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Abhijit\_Menon-Sen1](https://forum.ansible.com/u/Abhijit_Menon-Sen1)\ **Post date:** [October 29, 2015, 3:48am UTC](https://forum.ansible.com/t/rfc-with-nested-dependents-plugin-and-the-future-of-nested-loops/19646/1 "2015-10-29T03:48:30Z")
I wrote a loop (== lookup) plugin to solve a problem I had, and felt it may be worth explaining the motivation in some detail. My examples here are AWS-based, but please note that neither the problem nor the solution I describe are in any way AWS-specific. Suppose I start with a list of regions:     regions: ['us-east-1', 'us-west-2', 'eu-west-1'] With the ec2 module, I can do this (don't worry about how the instance ids for a single region are retrieved for now):     - ec2:         state: absent         instance\_ids: …list of instances in this region…         region: item       with\_items: regions But with ec2\_vpc or ec2\_key (or many other modules), I can't pass in a list of ids to remove, only one at a time. So I want to do this:     foreach (regions) {         vpcs = vpcs\_in\_region(«this region»)         foreach (vpcs) {             remove(«this vpc» in «this region»)         }     } But there's no way to write a nested loop of this kind in Ansible, where evaluating the sub-list requires the value of the current item of the outer list(s). Ideally, I would like to be able to do this:     - block:         - …get list of vpcs in current region…         - ec2\_vpc:             state: absent             vpc\_id: "{{ vpc }}"             region: "{{ region }}"           with\_items: vpcs       with\_items: regions This has two problems: (1) the loop variable is always named "item", so there's a conflict; (2) blocks can't currently take with\_items, and the PlayIterator would need significant changes to allow nesting using that construct. But a solution to those problems would be the ideal: easier to explain and use than with\_nested/with\_subelements etc., and more general. I wish that's how it had worked from the start. My first attempt to solve this involved a custom "with\_dict\_of\_arrays" lookup plugin. If I could build a dict like this:     { 'us-east-1': [x,y,z],       'us-west-2': [p,q,r],       'eu-west-1': [a,b,c] } where a…z are vpc\_ids, then I could do this:     - ec2\_vpc:         state: absent         region: "{{ item.key }}"         vpc\_id: "{{ item.value }}"       with\_dict\_of\_arrays: vpcs\_by\_region We would iterate over (us-east-1,x), (us-east-1,y), …, (eu-west-1,c) in this case, and it would work fine. But building the required dict is a bit horrid. I have to find the intersection of all hosts in the cluster (via a unique tag) and the hosts in each region I'm interested in, and then extract the vpc\_id from each one in turn. That looks like this:     - set\_fact:         vpcs\_by\_region: "{{ vpcs\_by\_region|default({})|combine({item: groups[cluster\_tag]|intersect(groups[item])|map('lookup', hostvars, 'ec2\_vpc\_id')|unique|list}) }}"       with\_items: regions groups[cluster\_tag] is all my hosts. groups[item] is all the hosts in the "item" region. The intersection is the hosts in the cluster and the region. Then I look up hostvars[h].ec2\_vpc\_id for each host, via another custom plugin (the details of which aren't important here). Note that all this is coming from the inventory (i.e. ec2.py), so I can't statically define a dict-of-lists or list-of-lists beforehand. I have to build it up somehow (and set\_fact/combine in a loop is the only way I could find to do that). With the new nested\_dependents lookup plugin, the above becomes a little easier to follow:     - ec2\_vpc:         state: absent         region: "{{ item.0 }}"         vpc\_id: "{{ item.1 }}"       with\_nested\_dependents:         - regions         - groups[cluster\_tag]|intersect(groups[item.0])|map('lookup', hostvars, 'ec2\_vpc\_id')|unique|list The basic idea is that:     - foo: …       with\_nested\_dependents:         - expr\_a         - expr\_b         - expr\_c translates to this:     item =     list = evaluate(expr\_a, item)     for l in list:         item.append(l)         nextlist = evaluate(expr\_b, item)         for l2 in nextlist:             item.append(l2)             nextnextlist = evaluate(expr\_c, item)             for l3 in nextnextlist:                 …run task foo… In other words, when evaluating expr\_b, you can use item.0 to refer to the current element in the expr\_a; when evaluating expr\_c, item.1 refers to the current element in expr\_b, etc. The expression is re-evaluated each time the outer loop variable changes. Here's another (real) example from one of my playbooks, which would be quite painful to express in some other way:     - name: Remove VPC subnets in each region       ec2\_vpc\_subnet:         state: absent         region: "{{ item.0 }}"         vpc\_id: "{{ item.1 }}"         cidr: "{{ item.2 }}"       with\_nested\_dependents:         - regions         - groups[cluster\_tag]|intersect(groups[item.0])|map('lookup', hostvars, 'ec2\_vpc\_id')|unique|list         - instances|selectattr('region', 'equalto', item.0)|map(attribute='subnet')|unique|list       tags: ec2\_vpcs The plugin itself is pretty simple, just straightforward recursion. I've attached the source here for anyone who is interested. Comments and suggestions welcome. -- Abhijit > **(attachments)** > > [nested\_dependents.py](https://forum.ansible.com/uploads/short-url/e0JsDvShEqS4K4sPBOtLlSAZGjO.py) (3 KB) ---
**Author:** ![Abhijit\_Menon-Sen1](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Abhijit\_Menon-Sen1](https://forum.ansible.com/u/Abhijit_Menon-Sen1)\ **Post date:** [October 29, 2015, 4:10am UTC](https://forum.ansible.com/t/rfc-with-nested-dependents-plugin-and-the-future-of-nested-loops/19646/2 "2015-10-29T04:10:11Z")
> Ideally, I would like to be able to do this: > >     - block: >         - …get list of vpcs in current region… >         - ec2\_vpc: >             state: absent >             vpc\_id: "{{ vpc }}" >             region: "{{ region }}" >           with\_items: vpcs >       with\_items: regions > > This has two problems: (1) the loop variable is always named "item", so > there's a conflict; (2) blocks can't currently take with\_items, and the > PlayIterator would need significant changes to allow nesting using that > construct. But a solution to those problems would be the ideal: easier > to explain and use than with\_nested/with\_subelements etc., and more > general. I wish that's how it had worked from the start. Note that (1) is a problem that many people have complained about over the years, and I believe the maintainers have said that they would like to fix it someday. I think it would be best to leave the existing with\_ loop plugins alone, and instead suggest the use of a new language construct like this:     - block:         - ec2\_vpc: state=absent vpc\_id="{{ v }}" region="{{ r }}"           foreach: v in vpcs       foreach: r in regions Maybe it should be called "loop" instead of "foreach"; maybe we want to use «['v', vpcs]» instead of «v in vpcs». Whatever. Those are relatively minor details. What's important is that: (a) the looping construct looks entirely different from with\_x, (b) that it has a way to specify the name of the loop variable, and (c) that it integrates with the nesting already provided by blocks. Since with\_x loops can only apply to tasks, there are no additional backwards-compatibility problems to doing it this way, and existing loops can work unmodified. In terms of implementation, (a) and (b) are relatively simple. The real changes would be in PlayIterator/executor to support the looping. Thoughts? >     item = >     list = evaluate(expr\_a, item) >     for l in list: >         item.append(l) (Aside: this pseudocode is a bit broken, in that it keeps appending stuff to item forever. The real code doesn't do that.) -- Abhijit ---
**Author:** ![Hagai\_Kariti](https://avatars.discourse-cdn.com/v4/letter/h/4491bb/32.png) [@Hagai\_Kariti](https://forum.ansible.com/u/Hagai_Kariti)\ **Post date:** [October 29, 2015, 5:20am UTC](https://forum.ansible.com/t/rfc-with-nested-dependents-plugin-and-the-future-of-nested-loops/19646/3 "2015-10-29T05:20:31Z")
Heh, I wrote something similar a while ago: [http://grokbase.com/t/gg/ansible-devel/156s9cqqth/with-recursive-a-lookup-plugin-that-chains-other-lookups](http://grokbase.com/t/gg/ansible-devel/156s9cqqth/with-recursive-a-lookup-plugin-that-chains-other-lookups). Why should loops over blocks be diffrerent than loops over include? They’re the same thing basically… ---
**Author:** ![Abhijit\_Menon-Sen1](https://avatars.discourse-cdn.com/v4/letter/a/8e7dd6/32.png) [@Abhijit\_Menon-Sen1](https://forum.ansible.com/u/Abhijit_Menon-Sen1)\ **Post date:** [October 29, 2015, 5:53am UTC](https://forum.ansible.com/t/rfc-with-nested-dependents-plugin-and-the-future-of-nested-loops/19646/4 "2015-10-29T05:53:05Z")
I don't think they should be different. But the naïve approach to allowing with\_items on a block would simply push the loop clause down into each task in the block, i.e.     - block:         - a         - b         - c       with\_items: [1,2] …would result in a being executed twice, followed by b being executed twice, and so on, rather than "a,b,c" being executed twice. (I didn't try very hard to implement this yet; I'm taking @jimi-c's word for it that it would require extensive PlayIterator changes following an IRC discussion a couple of weeks ago.) Instead, I \*want\* this to work in the same way that a loop over an include would work: execute all tasks for each element in the list. But I definitely want to be able to do this without creating another include file for each level of nesting. In both cases, the 'item' naming problem would need to be solved. If this is easier to accomplish than I had thought, I will be thrilled. -- Abhijit