Writing/Augmenting Host Variables from a Playbook.

Archives Ansible Project
1

Let's assume I have a service that can listen on different combinations
or host/port:

2

Hello Marc,

nice to see you here :-). This is certainly doable:

3

Hi Stefan,

thanks for your answer. I busied myself with working around the issue
temporarily because I had to get a project forward.

> Let's assume I have a service that can listen on different combinations
> or host/port:
>

Hello Marc,

nice to see you here :-). This is certainly doable:

---
- hosts: all

  vars:
    host:
      service:
        hostname: "google.com"
        ports:
          - "80"
          - "443"
  tasks:
    - name: Get DNS records
      set_fact:
        ip_addresses: "{{ ip_addresses | default() + lookup('dig', host.service.hostname + '/' + item, wantlist=True) }}"
      loop:
        - A
        - AAAA
       when: "'listen' not in host.service"

This sets a local fact to the list of IP addreses pulled from DNS.
Adding the IPv6 addresses is an easy enogh exercise.

    - name: Determine all combinations of IP address and ports
      set_fact:
        ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
      when: "'listen' not in host.service"

That's magic. Cute, indeed. This returns a list, alternating between IP
adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]

    - name: Turn list members into dictionaries
      set_fact:
        host:
          service:
            listen: "{{ host.service.listen | default() + [{ 'ip': item[0], 'port': item[1] }] }}"
      loop: "{{ ips_ports }}"
      when: ips_ports is defined

This is where you're losing me. You're iterating over an array and get
array members, so item will be "ip1" in the first iteration, "port1" in
the second, "ip2" in the third. How can you index into an array
element???

Anyway, things have become a bit more complicated since my original
request. This is what I have now:

service:
  sites:
    default:
      protocols:
      - "http"
      - "https"
  listen:
  - ip: "2a01:4f8:161:3541::32:100"
    port: "80"
    protocol: "http"
  - ip: "2a01:4f8:161:3541::32:100"
    port: "443"
    protocol: "https"

The templates pull site configuration from the "sites" part of the
definition; the listen configuration from the "listen" part of the
definition. This is a strongly simplified example with reality being
much more complex, I just reduced this to the relevant parts. This is,
however, from a working setup (and yes, it's IPv6 only).

For better readability, I'd like to write this like:

service:
  sites:
    default:
      protocols:
      - "http"
      - "https"
      listen:
      - ip: "2a01:4f8:161:3541::32:100"

With the possibility of defining explicit ports and protocols in the
listen clause. I think to massage one format into the other one in a
task is awfully hard, and to use local facts makes it impossible to
override the automatism by explicitly writing the desired result to
inventory proper.

Can a custom inventory plugin access what the previously running
inventory plugins have parsed, and can it augment structure that was
build by the predecessors? This way, I could have all processing power
and flexibility of imperative programming. I could think of a gazillion
of other places where this could be useful to simplify my templates
_AND_ my inventory.

Having this done inside ansible would allow me to take advantage of the
inventory reading logic that is already present in ansible. I could
write a preprocessor writing out the "augmented inventory" before
ansible is started, but I'd have to manually process the inventory file
-and- the contents of the host_vars and group_vars directories. I'd like
to avoid this.

Greetings
Marc

4

Hi Stefan,

thanks for your answer. I busied myself with working around the issue
temporarily because I had to get a project forward.

Let's assume I have a service that can listen on different combinations
or host/port:

Hello Marc,

nice to see you here :-). This is certainly doable:

---
- hosts: all

  vars:
    host:
      service:
        hostname: "google.com"
        ports:
          - "80"
          - "443"
  tasks:
    - name: Get DNS records
      set_fact:
        ip_addresses: "{{ ip_addresses | default() + lookup('dig', host.service.hostname + '/' + item, wantlist=True) }}"
      loop:
        - A
        - AAAA
       when: "'listen' not in host.service"

This sets a local fact to the list of IP addreses pulled from DNS.
Adding the IPv6 addresses is an easy enogh exercise.

It already contains the IPv6 addresses (AAAA record).

    - name: Determine all combinations of IP address and ports
      set_fact:
        ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
      when: "'listen' not in host.service"

That's magic. Cute, indeed. This returns a list, alternating between IP
adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]

It contains all IP and port combinations as nested list:

[[ip1,port1],[ip1,port2],[ip2,port1]]

    - name: Turn list members into dictionaries
      set_fact:
        host:
          service:
            listen: "{{ host.service.listen | default() + [{ 'ip': item[0], 'port': item[1] }] }}"
      loop: "{{ ips_ports }}"
      when: ips_ports is defined

This is where you're losing me. You're iterating over an array and get
array members, so item will be "ip1" in the first iteration, "port1" in
the second, "ip2" in the third. How can you index into an array
element???

See above, each entry is a list of IP (item[0]) and port (item[1]).

Regards
         Racke

5

>>> Let's assume I have a service that can listen on different combinations
>>> or host/port:
>>>
>>
>> Hello Marc,
>>
>> nice to see you here :-). This is certainly doable:
>>
>> ---
>> - hosts: all
>>
>> vars:
>> host:
>> service:
>> hostname: "google.com"
>> ports:
>> - "80"
>> - "443"
>> tasks:
>> - name: Get DNS records
>> set_fact:
>> ip_addresses: "{{ ip_addresses | default() + lookup('dig', host.service.hostname + '/' + item, wantlist=True) }}"
>> loop:
>> - A
>> - AAAA
>> when: "'listen' not in host.service"
>
> This sets a local fact to the list of IP addreses pulled from DNS.
> Adding the IPv6 addresses is an easy enogh exercise.

It already contains the IPv6 addresses (AAAA record).

Idiot me, of course, I missed the loop.

>> - name: Determine all combinations of IP address and ports
>> set_fact:
>> ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
>> when: "'listen' not in host.service"
>
> That's magic. Cute, indeed. This returns a list, alternating between IP
> adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]

It contains all IP and port combinations as nested list:

[[ip1,port1],[ip1,port2],[ip2,port1]]

Now it all makes sense to me. Thanks for explaining. I'm still wondering
whether it would be possible for a more complex of variable data
definition.

> Can a custom inventory plugin access what the previously running
> inventory plugins have parsed, and can it augment structure that was
> build by the predecessors? This way, I could have all processing power
> and flexibility of imperative programming. I could think of a gazillion
> of other places where this could be useful to simplify my templates
> _AND_ my inventory.
>
> Having this done inside ansible would allow me to take advantage of the
> inventory reading logic that is already present in ansible. I could
> write a preprocessor writing out the "augmented inventory" before
> ansible is started, but I'd have to manually process the inventory file
> -and- the contents of the host_vars and group_vars directories. I'd like
> to avoid this.

Any idea whether this would work?

Greetings
Marc

6

Let's assume I have a service that can listen on different combinations
or host/port:

Hello Marc,

nice to see you here :-). This is certainly doable:

---
- hosts: all

  vars:
    host:
      service:
        hostname: "google.com"
        ports:
          - "80"
          - "443"
  tasks:
    - name: Get DNS records
      set_fact:
        ip_addresses: "{{ ip_addresses | default() + lookup('dig', host.service.hostname + '/' + item, wantlist=True) }}"
      loop:
        - A
        - AAAA
       when: "'listen' not in host.service"

This sets a local fact to the list of IP addreses pulled from DNS.
Adding the IPv6 addresses is an easy enogh exercise.

It already contains the IPv6 addresses (AAAA record).

Idiot me, of course, I missed the loop.

    - name: Determine all combinations of IP address and ports
      set_fact:
        ips_ports: "{{ ip_addresses | product(host.service.ports) | list }}"
      when: "'listen' not in host.service"

That's magic. Cute, indeed. This returns a list, alternating between IP
adress and port, like [ ip1, port1, ip2, port2, ip3, port3 ]

It contains all IP and port combinations as nested list:

[[ip1,port1],[ip1,port2],[ip2,port1]]

Now it all makes sense to me. Thanks for explaining. I'm still wondering
whether it would be possible for a more complex of variable data
definition.

It is possible, but the Jinja expressions may become really complex
which you might want to prevent.

Can a custom inventory plugin access what the previously running
inventory plugins have parsed, and can it augment structure that was
build by the predecessors? This way, I could have all processing power
and flexibility of imperative programming. I could think of a gazillion
of other places where this could be useful to simplify my templates
_AND_ my inventory.

Having this done inside ansible would allow me to take advantage of the
inventory reading logic that is already present in ansible. I could
write a preprocessor writing out the "augmented inventory" before
ansible is started, but I'd have to manually process the inventory file
-and- the contents of the host_vars and group_vars directories. I'd like
to avoid this.

Any idea whether this would work?

You could also write a custom module in Python which can transform the structure
as you wish. You can call your custom module in the first task to achieve the
augmenting.

Regards
         Racke

7

_That_ sounds totally interesting, can you point me to some example code
please?

Greetings
Marc

8

I don't have a custom module around which I could share here. Searching the web
for "ansible custom module" should give you plenty of insights though.

Regards
           Racke

9

Are you trying to say that a plain custom module called from a task (a)
has access to the full inventory including all variables and (b) can
write to it with following tasks seeing the changes it did?

Greetings
Marc

10

That's a good question :-). But you can also write filter plugins which is probably a better
idea.

This would allow you do augment your data structure with

"{{ data | augment }}"

You can find an example here: https://blog.oddbit.com/post/2019-04-25-writing-ansible-filter-plugins/

Regards
         Racke

11

I am not sure whether this is a better idea, making the augmentation
dependent to the actual task being in quesiton. An independent approach
would, for example, allow people to use generic apache, letsencrypt,
proxy and DNS modules from The Galaxy while just writing the URL and
other web site data in the host definition, with independent code taking
the job of converting the simple host definition into input data
structures the generic modules can grok.

Greetings
Marc