Using Jinja2 loops inside inventory variables files

[N-Mi (Nicolas)]

Hi,

I’m trying to organize my inventory variables, and I’m struggling with the syntax.

I have an inventory structure similar to one suggested in documentation :

admin@ansible:~/ansible/environments$ find .
.
./000_cross_env_vars.yml
./prod
./prod/group_vars
./prod/group_vars/all
./prod/group_vars/all/000_cross_env_vars
./prod/group_vars/all/001_env_specific.yml
./prod/hosts
./prod/host_vars
./prod/host_vars/host-a.yml
./prod/host_vars/host-b.yml
./staging
./staging/group_vars
./staging/group_vars/all
./staging/group_vars/all/000_cross_env_vars
./staging/group_vars/all/001_env_specific.yml
./staging/hosts

In my 001_env_specific.yml file, I define a list of hosts/networks/ports that I will use to setup my firewalling rules using UFW (servers are running Ubuntu 22.04). Then I have some rules defined in host_vars, that use variables defined in group_vars.

Example group_vars :

# Hosts
ufw_host_bastion: 192.168.1.2

# Networks
ufw_net_postes_adm:
- 192.168.10.0/24
- 192.168.20.0/24

Example host_vars :

ufw_rules:
  - rule: allow
    from_ip: "{{ ufw_host_bastion }}"
    to_port: "{{ ufw_port_ssh }}"
    protocol: tcp
    interface: "{{ if_adm }}"
    comment: 'allow incoming ssh on admin interface from bastion'
{%- for item in ufw_net_postes_adm -%}
  - rule: allow
    from_ip: {{ item }}
    to_port: {{ ufw_port_https }}
    protocol: tcp
    interface: {{ if_adm }}
    comment: 'allow incoming HTTPS on admin interface from admin network {{ item }}'
{%- endfor -%}"

This fails with the following message :

ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
JSON: Expecting value: line 1 column 1 (char 0)

Syntax Error while loading YAML.
  found character that cannot start any token

The error appears to be in '/home/admin/ansible/environments/prod/host_vars/host-a.yml': line 14, column 2, but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


{%- for item in ufw_net_postes_adm -%}
^ here

I searched Ansible’s documentation if there are restrictions on how I can use Jinja2 inside variable files, but found nothing. It seems I can only use {{ variable }} inside inventory files but no {% %} instruction.

How can I achieve this in a correct syntax way ?

[mikemorency]

The file needs to be valid YAML or JSON, and having the jinja templating conditionals as you do makes it not valid YAML.

You could template the variable as a string and then parse the string using from_json/from_yaml in your playbook. Or you can just do the templating in your playbook.
For the first situation (keep templating in vars file)

# host_vars
ufw_rules: >-
    [
      {
        rule: allow,
        from_ip: "{{ ufw_host_bastion | default('') }}",
        to_port: "{{ ufw_port_ssh | d('') }}",
        protocol: tcp,
        interface: "{{ if_adm | d('') }}",
        comment: 'allow incoming ssh on admin interface from bastion',
    },
    {% for item in ufw_net_postes_adm %}
    {
        rule: allow,
        from_ip: "{{ item }}",
        to_port: "{{ ufw_port_https | d('') }}",
        protocol: tcp,
        interface: "{{ if_adm | d('') }}",
        comment: 'allow incoming HTTPS on admin interface from admin network {{ item }}',
    },
    {% endfor %}
    ]

# playbook
----
- hosts: localhost
  gather_facts: false
  tasks:
    - debug:
        var: ufw_rules | from_yaml

[mikemorency]

Also you might consider re-organizing your inventory file structure to something like

inventories/
  production.yml # or production.ini, its your prod hosts
  staging.yml    # same as above, but staging hosts
  group_vars/
    all.yml      # or all/ .... these vars are included for all hosts
    production/  # all files in here are automatically included for production hosts
      foo.yml
      bar.yml
    staging/
      something.yml
      foo.yml
  host_vars/
    my_host
    other_host

[utoddl (Todd Lewis)]

YAML knows nothing about Jinja, so you can’t insert YAML-generating Jinja expressions into YAML files and expect the YAML parser to look into the future to divine what the following text would look like if it were run through Jinja.

YAML parsing results in a nested tree of lists, dicts, strings, and numbers (with optional !TAGS) and hands that off to Ansible.

Ansible looks at the strings, and under certain circumstances decides whether they look like they contain Jinja expressions. If so, it may invoke Jinja on them, or it may defer that step. “Lazy evaluation” means it defers as long as possible.

The trick, then, is to create your variable value – at the YAML level – so that it looks like a string containing Jinja at the Ansible level, and when evaluated as Jinja will result in the data you intended.

The following playbook does just that. I’m not recommending this practice, nor am I recommending against it. If you’re willing to maintain something like this and it works for you, hey, you do you! I’m showing this simply as an example of what I described above.

Note that the expression in ufw_rules below produces JSON, not YAML. While it’s true that JSON is a subset of YAML, that’s not why this works! It works because after Ansible runs a string through Jinja, if the resulting text looks like JSON, Ansible re-interprets it as JSON. Sort of. Sometimes. Apparently. In other words, this is an empirical observation; the actual rules governing when and what Ansible does to and with strings are buried in the code and not easily understood through a casual pass over the sources. So I’m providing this working playbook for you to compare against whatever you might try until you get something working to your satisfaction. I hope this helps.

---
# test.yml
- name: Jinja in variable definitions
  hosts: localhost
  gather_facts: false
  vars:
    ufw_host_bastion: UFW_HOST_BASTION
    ufw_port_ssh: UFW_PORT_SSH
    if_adm: IF_ADM
    ufw_port_https: UFW_PORT_HTTPS
    ufw_net_postes_adm:
      - ALPHA
      - BETA
    ufw_rules: >-
       [{"rule":     "allow",
        "from_ip":   "{{ ufw_host_bastion }}",
        "to_port":   "{{ ufw_port_ssh }}",
        "protocol":  "tcp",
        "interface": "{{ if_adm }}",
        "comment":   "allow incoming ssh on admin interface from bastion"
       },
       {%- for item in ufw_net_postes_adm -%}
       {"rule":      "allow",
        "from_ip":   "{{ item }}",
        "to_port":   "{{ ufw_port_https }}",
        "protocol":  "tcp",
        "interface": "{{ if_adm }}",
        "comment":   "allow incoming HTTPS on admin interface from admin network {{ item }}"
       }{% if not loop.last %},{% endif %}
       {%- endfor -%}]
  tasks:
    - name: Dump ufw_rules
      ansible.builtin.debug:
        var: ufw_rules