Nested loops with subelements

Hi,

As others have before me, I find the syntax for nested loops somewhat confusing :frowning:

Consider my group_vars:
ec2_specs:

  • { region: “us-east-1”, ami: “ami-b66ed3de”, count: 1, type: “t2.micro” }
  • { region: “us-west-1”, ami: “ami-b56e64f0”, count: 2, type: “t2.micro” }
  • { region: “sa-east-1”, ami: “ami-9337828e”, count: 1, type: “t2.micro” }

nginx_containers:

  • { name: “NGINX_A”, hostport: “443” }
  • { name: “NGINX_B”, hostport: “444” }
  • { name: “NGINX_B”, hostport: “445” }

I’m setting up a security group for these instances like so:

  • name: Setup AWS EC2 Security group
    ec2_group:
    name: microserver_security_group
    description: “Security group for microservices”
    region: “{{ item.region }}”
    rules:
  • proto: tcp
    from_port: 22
    to_port: 22
    cidr_ip: 0.0.0.0/0
  • proto: tcp
    from_port: 443
    to_port: 445
    cidr_ip: 0.0.0.0/0
    rules_egress:
  • proto: all
    cidr_ip: 0.0.0.0/0
    with_items: ec2_specs

This works because the host ports are continuous 443-445. I need to retain the flexibility of changing this to something like 443, 1443, 2443 … etc. One way to express this might be:

  • name: Setup AWS EC2 Security group
    ec2_group:
    name: microserver_security_group
    description: “Security group for microservices”
    region: “{{ item.region }}”
    rules:
  • proto: tcp
    from_port: 22
    to_port: 22
    cidr_ip: 0.0.0.0/0
  • proto: tcp
    from_port: “{{ subitem.hostport }}”
    to_port: “{{ subitem.hostport }}”
    cidr_ip: 0.0.0.0/0
    with_subitems: nginx_containers
    rules_egress:
  • proto: all
    cidr_ip: 0.0.0.0/0
    with_items: ec2_specs

Something like: with_XYZ at arbitrary levels with XYZ representing each item would be intuitive to me here. No doubt there are other ways of expressing this as well - where the syntax makes visual sense. As of now though, I don’t know how to accomplish this short of hard-coding it in the role tasks.

A second problem arising from the same problem (or my lack of syntactic know-how) is the way I call ec2 and use its somewhat clunky output.

  • name: Setup AWS EC2 Instances
    ec2:
    instance_type: “{{ item.type }}”
    image: “{{ item.ami }}”
    monitoring: no
    wait: yes
    group: microserver_security_group
    key_name: microserver_keypair
    instance_tags:
    nvtype: microserver
    Name: “MicroServer”
    count: “{{ item.count }}”
    region: “{{ item.region }}”
    with_items: ec2_specs
    register: ec2

The only way I’ve found of using the output of the above is the following sanity test example:

  • name: Test microsites 443
    shell: chdir={{ tempdir.stdout }} /usr/bin/curl -v -s -k --key test.key --cert test.crt https://{{ item.1.public_ip }}:443
    with_subelements:
    - ec2.results
    - instances
    register: curl_out

Since the curl_test itself runs from localhost, I can’t use the dynamic inventory within the site.yml file against the test role. I now have to hard-code N tests here - one for each host-port in the nginx_containers variable. I’d much rather express this as a single task with a nested loop - but I haven’t been able to figure out how to.

Is this possible?

Many thanks,
Ananda

I guess I could express it as a nested dict in the first place?

ec2_specs:

  • { region: “us-east-1”, ami: “ami-b66ed3de”, count: 1, type: “t2.micro”, containers: [{ name: “NGINX_A”, hostport: “443” }, { name: “NGINX_B”, hostport: “444” }] }
  • etc

I suspect though that if I need to express a large number of these, it can get error-prone - so a set of M properties and a set of N sub-properties would still be a more compact and less error-prone representation of this matrix.

Any recommendations?

Sorry, I’m not sure what your specific question is.

I see a lot of text without a question mark and am having trouble parsing.

Thanks for looking Michael,

I was looking for an intuitive way for expressing nested variables in this use case.

In my group vars, I specify 3 ec2 region/ami specs:
ec2_specs:

  • { region: “us-east-1”, ami: “ami-b66ed3de”, count: 1, type: “t2.micro” }
  • { region: “us-west-1”, ami: “ami-b56e64f0”, count: 2, type: “t2.micro” }
  • { region: “sa-east-1”, ami: “ami-9337828e”, count: 1, type: “t2.micro” }

I want to spin up count instances of each ami. In each of them I want to setup N separate processes that run on different ports:
process_ports:

  • { name: “Process_A”, hostport: “1443” }
  • { name: “Process_B”, hostport: “2974” }
  • { name: “Process_C”, hostport: “3555” }

To make sure these ports are accessible from the outside world I need to set up ec2 security groups for these instances. How can I express this by using my group vars rather than hard coding them as I have done below?

  • name: Setup AWS EC2 Security group
    ec2_group:
    name: microserver_security_group
    description: “Security group for microservices”
    region: “{{ item.region }}”
    rules:
  • proto: tcp
    from_port: 22
    to_port: 22
    cidr_ip: 0.0.0.0/0
  • proto: tcp
    from_port: 1443
    to_port: 1443
    cidr_ip: 0.0.0.0/0
  • proto: tcp
    from_port: 2754
    to_port: 2754
    cidr_ip: 0.0.0.0/0
  • proto: tcp
    from_port: 3555
    to_port: 3555
    cidr_ip: 0.0.0.0/0
    rules_egress:
  • proto: all
    cidr_ip: 0.0.0.0/0
    with_items: ec2_specs

Using nesting variables trivially doesn’t work:

  • name: Setup AWS EC2 Security group
    ec2_group:
    name: microserver_security_group
    description: “Security group for microservices”
    region: “{{ item[0].region }}”
    rules:
  • proto: tcp
    from_port: 22
    to_port: 22
    cidr_ip: 0.0.0.0/0
  • proto: tcp
    from_port: “{{ item[1].hostport }}”
    to_port: “{{ item[1].hostport }}”
    cidr_ip: 0.0.0.0/0
    rules_egress:
  • proto: all
    cidr_ip: 0.0.0.0/0
    with_nested:
    - ec2_specs
    - process_ports

This simply causes 3 separate runs of the same ec2_group command and I end up with a single security group in each region where only the last hostport is exposed (each run overwriting the previous run).

Hello Ananda,

Is your number of processes (containers in your case) static? What about the ports? do they have a particular generation rule that matters to you?