Aggregating data from complex hierarchal sources

I’m developing an Ansible playbook to help me automate the management of several dozen zones in Bind using the Community nsupdate module https://docs.ansible.com/ansible/latest/collections/community/general/nsupdate_module.html. Generally speaking, things have gone really well but I’ve found myself in a pickle with the data structure I’ve designed to store the configuration of the zone. I am pretty new to Ansible, and have a software engineering background, but I feel like either my data structure needs a tweak or my understanding of how to handle looping and combining data just isn’t there (probably both).

First, here’s the relevant portion of my inventory; nothing too surprising here:

nameservers:
  children:
    ns_masters:
    ns_slaves:

ns_masters:
  hosts:
    ns1.mydomain.com:
      hostname: ns1
      ip_address: x.x.x.x
      interface_name: eno0
      type: master

ns_slaves:
  hosts:
    ns2.mydomain.com:
      hostname: ns2
      ip_address: x.x.x.x
      interface_name: eno0
      type: slave

Next, I have created a template for zone files. My goal was to have a format that’s easy to read and maintain, and that intuitively follows the standard for zone files. Each file is named for the zone it manages.

As an example:

---
mydomain_com:
  - zone: mydomain.com
    default_ttl: 5m
    soa:
      primary_nameserver: ns1
      admin: me.mydomain.com
      serial: 2025033100
      refresh: 30m
      retry: 15m
      expire: 4w
      minimum: 2m
    ns_records:
      - ns1
      - ns2
    a_records:
      - {label: "@", rdata: "x.x.x.x"}
      - {label: "www", rdata: "x.x.x.x"}
      - {label: "ns1", rdata: "x.x.x.x"}
      - {label: "ns2", rdata: "x.x.x.x"}
    cname_records:
      - {label: "my-cname", rdata: "www.mydomain.com"}
      - {label: "another-cname", rdata: "www.mydomain.com"}

Next, I have a configuration file that holds information about the overall Bind configuration, including a property that aggregates the separate zones into a massive list (at least I’m pretty sure it’s a list and not a dictionary). Or is it a list of dictionaries in this case?

# Bind9 user and group that named runs under
bind9_user: bind
bind9_group: bind

# Bind9 configuration paths
bind_directory: "/etc/bind"
keys_directory: "/etc/bind/keys"
zones_directory: "/etc/bind/zones"
cache_directory: "/var/cache/bind"

# Bind9 acls
acl:
  - name: ACL1
    networks:
      - localhost
      - localnets
  - name: ACL2
    networks:
      - localhost
      - localnets

zones: >
  {{ mydomain_com
  | union (mydomain2_com) }}
  | union (mydomain3_com) }}
  | union (mydomain4_com) }}
  | union (mydomain5_com) }}
  | union (mydomain6_com) }}
  | union (mydomain7_com) }}
  | union (mydomain8_com) }}
  | union (mydomain9_com) }}
  | union (mydomain10_com) }}

This configuration file is loaded into the playbook, and the “zones” property is populated by all of the zone specific configurations defined in the individual zone files. Up to this point, everything seems to be working perfectly fine, and as you’d expect.

However, I am having an issue iterating over the zones and their associated records (a, cname, etc.). The playbook is really pretty simple and I don’t have any issues with it except for the task that needs to iterate the records and issue the “nsupdate” module executions.

The task as I’ve landed on it looks like this:

tasks: 
  - name: Updating authoritative forward zone files
    ansible.builtin.debug:
      msg:
        - "{{ item }}"
    loop: "{{ zones | map(attribute='a_records') | flatten(levels=1) }} "
    loop_control:
      label: "{{ item.label }}"
    when:
      - inventory_hostname in groups['ns_masters']
    tags:
      - zone_update

This task only executes against my primary name servers, and only when I’m performing an actual zone update (via tag and when); it works great, and outputs the following debug information:

ok: [name-of-my-dns-server-redacted] => (item=www) => {
    "msg": [
        {
            "label": "www",
            "rdata": "x.x.x.x"
        }
    ]
}

Unfortunately, I need more than just the “label” and “rdata” to run the nsupdate module. Ultimately I need the output of this task to look like this:

ok: [name-of-my-dns-server-redacted] => (item=www) => {
    "msg": [
        {
            "label": "www",
            "rdata": "x.x.x.x"
            "zone": "mydomain.com"
            "server": "my-nameserver-ip-address-goes-here"
        }
    ]
}

The zone value should come from the zone property of the zone configuration file, and the server should come from the inventory. In theory, each of these is trivial to get to, but I’m really really stuck on how to pull this all together in the task.

With all that said, how do I go about inserting these values into this list? The value for “zone” needs to come from the same file that the records themselves come from, and the value for “server” comes from the “ip_address” property that goes along with the inventory item.

I investigated set_fact, but couldn’t figure out how to structure that, and I’m not sure it would work to begin with since the “server” property is global to all zones and the “zone” property is specific to the zone being updated.

I’m not sure what to do to solve this, and I’m hopeful I’m just missing some fancy syntax somewhere.

Any thoughts or wisdom are greatly appreciated :slight_smile:

This can be achieved by stacking loops via include_tasks.

For example, such playbook:

- hosts: ns_masters
  gather_facts: false
  vars:
    zones:
      - zone: mydomain.com
        a_records:
          - {label: "@", rdata: "x.x.x.x"}
          - {label: "www", rdata: "x.x.x.x"}
          - {label: "ns1", rdata: "x.x.x.x"}
          - {label: "ns2", rdata: "x.x.x.x"}
      - zone: myotherdomain.com
        a_records:
          - {label: "@", rdata: "x.x.x.x"}
          - {label: "www", rdata: "x.x.x.x"}
          - {label: "ns1", rdata: "x.x.x.x"}
          - {label: "ns2", rdata: "x.x.x.x"}
  tasks:
    - ansible.builtin.include_tasks:
        file: create-zone-records.yml
      loop: "{{ zones }}"
      loop_control:
        loop_var: zone

And then tasks file create-zone-records.yml which will operate with each zone:

- ansible.builtin.debug:
    msg: "Creating A record {{ item.label }} for zone {{ zone.zone }}"
  loop: "{{ zone.a_records }}"
  loop_control:
    label: "{{ item.label }}"

Note the loop_var that aloows inner loop to access outer item.

And to execute only against ns_masters group, you could just limit this by hosts option of the playbook.

Also not really sure why you need the unions, you could just make a single list as shown in the playbook. (And yes, you are correct, they are not dictionaries, they are lists.)

1 Like

Huh, I had no idea you could nest tasks like that, absolutely going to give that a shot today!

Regarding the Unions…

I used to have pretty much the exact structure to organize the zones you’re showing in this example, but found that even with a few domains, the forward and reverse zone configurations bloat out the file and it gets hard to keep track of things. The forward zones are likely manageable, but my old eyes blur when navigating all the reverse zones and their naming (e.g., 32.85.214.in-addr.arpa vs 28.81.98.in-addr.arpa).

So, what I ended up doing is taking each of those zones and putting them in their own files storing them in a directory next to the configuration yaml file. It’s not lost on me that I still have the same amount of data to manage, and at a certain point its a “six of one, half a dozen of the other” sort of situation. I guess I just thought that storing them individually was a little easier to keep track of both on disk, and when reviewing history in Git.

vars/
├─ forward-zones/
│  ├─ mydomain.com.yaml
│  ├─ sub-domain.mydomain.com.yaml
├─ reverse-zones/
│  ├─ 32.85.214.in-addr.arpa.yaml
│  ├─ 28.81.98.in-addr.arpa.yaml
├─ config.yaml

Honestly, I don’t really like the union situation. It feels kludgy and prone to user error when managing how the various files aggregate together into a list. I spent entirely way too much time fighting what I think is a yaml parsing problem; if I didn’t have all the spaces and line breaks perfect, it would union things together “wrongly”… I’m not sure how else to describe the issue; the system ended up seeing everything as a gigantic string instead of the lists of items.

It would be nice to have a mechanism to take each of the files in those forward-zone and reverse-zone directories and aggregate them into that ‘zones’ variable in a way that doesn’t rely on me manually union’ing things together.

If your goal is to to have something like this:

- hosts: ns_masters
  gather_facts: false
  vars:
    zones:
      - zone: mydomain.com
        a_records:
          - {label: "@", rdata: "x.x.x.x"}
          - {label: "www", rdata: "x.x.x.x"}
          - {label: "ns1", rdata: "x.x.x.x"}
          - {label: "ns2", rdata: "x.x.x.x"}
      - zone: myotherdomain.com
        a_records:
          - {label: "@", rdata: "x.x.x.x"}
          - {label: "www", rdata: "x.x.x.x"}
          - {label: "ns1", rdata: "x.x.x.x"}
          - {label: "ns2", rdata: "x.x.x.x"}
     - zone:  32.85.214.in-addr.arpa
     - zone: 28.81.98.in-addr.arpa

You could use set_fact in a loop and append to your list of dicts to where all of your zone files are in the zones variable. Then based on the need of the task, only use the a_records var as needed. I’m not sure if this is what you’re trying to achieve or rot, but this would take “union” ouf of the equation and leave you to just loop over a list of dicts.