How to get `AnsibleUnsafeText` to template?

First of all, I really appreciate any help you all can offer me! This is my first post here.

We have an ansible role that is executed in an AWX environment. This role authenticates with a secure internal endpoint that returns additional configuration data to be configured on the target hosts in the inventory. This data contains variables that need to be templated with variables in the inventory file. This part is not happening - note the {{ host_seq }} below:

TASK [bmc : Debug profiledata_validated_api_action] ****************************
task path: /tmp/awx_478711_5zqo0k99/requirements_roles/bmc/tasks/main.yaml:150
Thursday 26 September 2024  17:53:13 +0000 (0:00:00.051)       0:00:21.291 **** 
ok: [ctv-mobile-strtus-m-91-01.vi.comcast.net] => (item=.links.Systems.href) => {
    "msg": "AnsibleUnsafeText"
}
ok: [ctv-mobile-strtus-m-91-01.vi.comcast.net] => (item=.links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href) => {
    "msg": "AnsibleUnsafeText"
}
ok: [ctv-mobile-strtus-m-91-01.vi.comcast.net] => (item={ "MACAddress": .HostCorrelation.HostMACAddress[0]}) => {
    "msg": "AnsibleUnsafeText"
}
TASK [bmc : Debug profiledata_validated_api_action] ****************************
task path: /tmp/awx_478711_5zqo0k99/requirements_roles/bmc/tasks/main.yaml:155
Thursday 26 September 2024  17:53:13 +0000 (0:00:00.072)       0:00:21.364 **** 
ok: [ctv-mobile-strtus-m-91-01.vi.comcast.net] => (item=.links.Systems.href) => {
    "ansible_loop_var": "item",
    "item": ".links.Systems.href"
}
ok: [ctv-mobile-strtus-m-91-01.vi.comcast.net] => (item=.links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href) => {
    "ansible_loop_var": "item",
    "item": ".links.Member[] | select(.href | endswith(\"C{{ host_seq }}N1\")) | .href"
}
ok: [ctv-mobile-strtus-m-91-01.vi.comcast.net] => (item={ "MACAddress": .HostCorrelation.HostMACAddress[0]}) => {
    "ansible_loop_var": "item",
    "item": "{ \"MACAddress\": .HostCorrelation.HostMACAddress[0]}"
}

From my reading this is by design since the data in the variable profiledata_validated_api_action was acquired via the uri module and since that is an external data source, ansible prevents Jinja2 templating by marking the data type as AnsibleUnsafeText.

I have read, among other things:

ChatGPT seemed to believe that I could use a | map('template') filter to force Jinja2 templating, but that doesn’t appear to exist in ansible-core 2.16. I cannot find any documentation regarding a template filter. I did find this, but it doesn’t fit my use case: ansible.builtin.template lookup – retrieve contents of file after templating with Jinja2 — Ansible Community Documentation

ChatGPT also believed that if I used | to_json | from_json it would cause the variable to be treated as standard JSON instead of a protected string, but that didn’t work either.

So I tried adding allow_jinja_in_extra_vars = always as ChatGPT seems to think this should be supported in ansible-core release 2.14+, but it doesn’t seem to be effective. Also it doesn’t seem to be a related configuration option based on the limited documentation I could find on the setting:

[root@CHQS-EniCEJ55G6 ansible]# export ANSIBLE_CONFIG=/root/ansible/ansible.cfg
[root@CHQS-EniCEJ55G6 ansible]# cat /root/ansible/ansible.cfg
[defaults]
allow_jinja_in_extra_vars = always
ALLOW_JINJA_IN_EXTRA_VARS = always

[root@CHQS-EniCEJ55G6 ansible]#
[root@CHQS-EniCEJ55G6 ansible]# ansible-config dump -t all --only-changed
ERROR: Error reading config file (/root/ansible/ansible.cfg): While reading from '<string>' [line  3]: option 'allow_jinja_in_extra_vars' in section 'defaults' already exists
[root@CHQS-EniCEJ55G6 ansible]# cat >ansible.cfg<<EOF
[defaults]
allow_jinja_in_extra_vars = always
EOF

[root@CHQS-EniCEJ55G6 ansible]# ansible-config dump -t all --only-changed
CONFIG_FILE() = /root/ansible/ansible.cfg
[root@CHQS-EniCEJ55G6 ansible]# ansible-config dump -t all | grep -i allow_jinja_in_extra_vars
[root@CHQS-EniCEJ55G6 ansible]#

I setup an example playbook (below) to demo the AnsibleUnsafeText problem with my attempt at using | to_json | from_json to trigger templating:

ansible-playbook -i localhost, -c local /dev/stdin <<EOF
---
- name: variable within variable expansion
  hosts: all
  become: false
  gather_facts: false

  vars:
    host_seq: 1

    jq_filters:
      - ".links.Systems.href"
      - !unsafe .links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href
      - '{ "MACAddress": .HostCorrelation.HostMACAddress[0]}'


  tasks:
    - debug:
        var: jq_filters

    - debug:
        msg: "{{ item | string | type_debug }}"
      loop: "{{ jq_filters }}"

    - set_fact:
        jq_filters_new: "{{ jq_filters_new | default( [], true ) + [ item | to_json | from_json ] }}"
      loop: "{{ jq_filters }}"

    - name: debug jq_filters_new
      debug:
        var: jq_filters_new

    - name: each item in jq_filters_new
      debug:
        msg: "{{ item | type_debug }}"
      loop: "{{ jq_filters_new }}"
EOF

Details about my AWX worker node / ansible version:

bash-4.4# ansible-playbook --version
ansible-playbook [core 2.13.11]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/etc/ansible/library']
  ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
  ansible collection location = /usr/local/lib/python3.10/site-packages/ansible_collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.10.12 (main, Jul 24 2023, 21:10:19) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]
  jinja version = 3.1.2
  libyaml = True
bash-4.4# cat /etc/redhat-release
CentOS Stream release 8
bash-4.4# uname -r
4.18.0-513.18.1.el8_9.x86_64
bash-4.4# cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
bash-4.4#

Details about my test machine (WSL2 instance of Rocky 8.10):

[root@CHQS-EniCEJ55G6 ansible]# ansible-playbook --version
ansible-playbook [core 2.16.3]
  config file = /root/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.12/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.12.3 (main, Jul  2 2024, 20:57:30) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python3.12)
  jinja version = 3.1.2
  libyaml = True
[root@CHQS-EniCEJ55G6 ansible]# cat /etc/redhat-release
Rocky Linux release 8.10 (Green Obsidian)
[root@CHQS-EniCEJ55G6 ansible]# uname -r
5.15.146.1-microsoft-standard-WSL2
[root@CHQS-EniCEJ55G6 ansible]# cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.10 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.10 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.10"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"
[root@CHQS-EniCEJ55G6 ansible]#

I just tried to implement a solution based on this documentation: AnsibleUnsafe notes · mitogen-hq/mitogen Wiki · GitHub

It doesn’t seem to be working correctly, but might be on the right path?

Playbook:

---
- name: Ensure compatibility for AnsibleUnsafeText
  hosts: localhost
  gather_facts: false

  vars:
    unsafe_text: !unsafe .links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href
    ansible_version: "{{ ansible_version.full }}"

  tasks:

    - debug:
        var: ansible_version

    - debug:
        msg: "ansible_version: {{ ansible_version | type_debug }}"

    - name: Check if ansible-core is <= 2.13
      set_fact:
        is_pre_2_14: "{{ ansible_version.full is version('2.14.0', '<') }}"

    - name: Strip unsafe in Ansible <= 2.13
      set_fact:
        stripped_value: "{{ unsafe_text | string }}"
      when: is_pre_2_14

    - name: Strip unsafe in Ansible 2.14 - 2.16
      set_fact:
        stripped_value: "{{ unsafe_text._strip_unsafe() }}"
      when: not is_pre_2_14

    - name: Debug stripped value
      debug:
        var: stripped_value

Output:

[root@CHQS-EniCEJ55G6 ansible]# ansible-playbook -i ctv-al-mobile-99.ini -l localhost ./AnsibleUnsafeText-expansion.yaml -vvv
ansible-playbook [core 2.16.3]
  config file = /root/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.12/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.12.3 (main, Jul  2 2024, 20:57:30) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] (/usr/bin/python3.12)
  jinja version = 3.1.2
  libyaml = True
Using /root/ansible/ansible.cfg as config file
host_list declined parsing /root/ansible/ctv-al-mobile-99.ini as it did not pass its verify_file() method
script declined parsing /root/ansible/ctv-al-mobile-99.ini as it did not pass its verify_file() method
auto declined parsing /root/ansible/ctv-al-mobile-99.ini as it did not pass its verify_file() method
yaml declined parsing /root/ansible/ctv-al-mobile-99.ini as it did not pass its verify_file() method
Parsed /root/ansible/ctv-al-mobile-99.ini inventory source with ini plugin
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: AnsibleUnsafeText-expansion.yaml ************************************************************************************************************************************************************************************************************************************
1 plays in ./AnsibleUnsafeText-expansion.yaml

PLAY [Ensure compatibility for AnsibleUnsafeText] *****************************************************************************************************************************************************************************************************************************

TASK [debug] ******************************************************************************************************************************************************************************************************************************************************************
task path: /root/ansible/AnsibleUnsafeText-expansion.yaml:12
ok: [localhost] => {
    "ansible_version": {
        "full": "2.16.3",
        "major": 2,
        "minor": 16,
        "revision": 3,
        "string": "2.16.3"
    }
}

TASK [debug] ******************************************************************************************************************************************************************************************************************************************************************
task path: /root/ansible/AnsibleUnsafeText-expansion.yaml:15
ok: [localhost] => {
    "msg": "ansible_version: dict"
}

TASK [Check if ansible-core is <= 2.13] ***************************************************************************************************************************************************************************************************************************************
task path: /root/ansible/AnsibleUnsafeText-expansion.yaml:18
ok: [localhost] => {
    "ansible_facts": {
        "is_pre_2_14": false
    },
    "changed": false
}

TASK [Strip unsafe in Ansible <= 2.13] ****************************************************************************************************************************************************************************************************************************************
task path: /root/ansible/AnsibleUnsafeText-expansion.yaml:22
skipping: [localhost] => {
    "changed": false,
    "false_condition": "is_pre_2_14",
    "skip_reason": "Conditional result was False"
}

TASK [Strip unsafe in Ansible 2.14 - 2.16] ************************************************************************************************************************************************************************************************************************************
task path: /root/ansible/AnsibleUnsafeText-expansion.yaml:27
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/ansible/executor/task_executor.py", line 526, in _execute
    self._task.post_validate(templar=templar)
  File "/usr/lib/python3.12/site-packages/ansible/playbook/task.py", line 290, in post_validate
    super(Task, self).post_validate(templar)
  File "/usr/lib/python3.12/site-packages/ansible/playbook/base.py", line 543, in post_validate
    value = method(attribute, getattr(self, name), templar)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ansible/playbook/task.py", line 298, in _post_validate_args
    args = templar.template(value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ansible/template/__init__.py", line 791, in template
    d[k] = self.template(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ansible/template/__init__.py", line 764, in template
    result = self.do_template(
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ansible/template/__init__.py", line 1010, in do_template
    res = myenv.concat(rf)
          ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/ansible/template/native_helpers.py", line 43, in ansible_eval_concat
    head = list(islice(nodes, 2))
           ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 12, in root
  File "/usr/lib/python3.12/site-packages/ansible/template/__init__.py", line 380, in call
    raise SecurityError(f"{obj!r} is not safely callable")
jinja2.exceptions.SecurityError: <bound method AnsibleUnsafeText._strip_unsafe of '.links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href'> is not safely callable
fatal: [localhost]: FAILED! => {
    "changed": false
}

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0

[root@CHQS-EniCEJ55G6 ansible]#

Disregard the below, as it does not work with ansible v2.13.11 - I forgot to include the !unsafe in the variable definition of `raw_unsafe_text…

The above solution does seem to work in ansible v2.13.11:

bash-4.4# ansible-playbook -i localhost, -l localhost -vvv /dev/stdin <<EOF
> ---
> - name: Ensure compatibility for AnsibleUnsafeText
>   hosts: localhost
>   gather_facts: false
>   vars:
>     host_seq: 1
>     raw_unsafe_text: '.links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href'
>     ansible_version: "{{ ansible_version.full }}"
>
>   tasks:
>     - debug:
>         var: ansible_version
>
>     - name: Check if ansible-core is <= 2.13
>       set_fact:
>         is_pre_2_14: "{{ ansible_version.full is version('2.14.0', '<') }}"
>
>     - name: Render unsafe_text with host_seq
>       set_fact:
>         unsafe_text: "{{ raw_unsafe_text | string | replace('{{ host_seq }}', host_seq | string) }}"
>
>     - name: Debug unsafe_text data type
>       debug:
>         msg: "unsafe_text: {{ unsafe_text | type_debug }}"
>
>     - name: Strip unsafe in Ansible <= 2.13
>       set_fact:
>         stripped_value: "{{ unsafe_text | string }}"
>       when: is_pre_2_14
>
>     - name: Strip unsafe in Ansible 2.14 - 2.16
>       set_fact:
>         stripped_value: "{{ unsafe_text._strip_unsafe() }}"
>       when: not is_pre_2_14
>
>     - name: Debug stripped value
>       debug:
>         var: stripped_value
>
>     - name: Debug stripped_value data type
>       debug:
>         msg: "stripped_value: {{ stripped_value | type_debug }}"
>
> EOF

ansible-playbook [core 2.13.11]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/etc/ansible/library']
  ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
  ansible collection location = /usr/local/lib/python3.10/site-packages/ansible_collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.10.12 (main, Jul 24 2023, 21:10:19) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]
  jinja version = 3.1.2
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
Parsed localhost, inventory source with host_list plugin
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: stdin *****************************************************************************************************************************************************************************************************************************
1 plays in /dev/stdin

PLAY [Ensure compatibility for AnsibleUnsafeText] *******************************************************************************************************************************************************************************************
[top  1200134] 19:36:54.181045 D ansible_mitogen.affinity: CPU mask for Ansible top-level process: 0x000002
[top  1200134] 19:36:54.181558 D ansible_mitogen.process: inherited open file limits: soft=262144 hard=262144
[top  1200134] 19:36:54.181700 D ansible_mitogen.process: max open files already set to hard limit: 262144
[mux  1200139] 19:36:54.185996 D ansible_mitogen.affinity: CPU mask for MuxProcess 0: 0x000001
[mux  1200139] 19:36:54.202834 D mitogen.service: Pool(3dc0, size=32, th='MainThread'): initialized
[mux  1200139] 19:36:54.203567 D ansible_mitogen.process: Service pool configured: size=32
META: ran handlers

TASK [debug] ********************************************************************************************************************************************************************************************************************************
task path: /dev/stdin:11
Friday 27 September 2024  19:36:54 +0000 (0:00:00.257)       0:00:00.257 ******
[task 1200173] 19:36:54.235282 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000004
[task 1200173] 19:36:54.248852 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] =>
  ansible_version:
    full: 2.13.11
    major: 2
    minor: 13
    revision: 11
    string: 2.13.11

TASK [Check if ansible-core is <= 2.13] *****************************************************************************************************************************************************************************************************
task path: /dev/stdin:14
Friday 27 September 2024  19:36:54 +0000 (0:00:00.033)       0:00:00.290 ******
[task 1200175] 19:36:54.262441 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000008
[task 1200175] 19:36:54.278833 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] => changed=false
  ansible_facts:
    is_pre_2_14: true

TASK [Render unsafe_text with host_seq] *****************************************************************************************************************************************************************************************************
task path: /dev/stdin:18
Friday 27 September 2024  19:36:54 +0000 (0:00:00.029)       0:00:00.319 ******
[task 1200177] 19:36:54.291621 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000010
[task 1200177] 19:36:54.321580 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] => changed=false
  ansible_facts:
    unsafe_text: .links.Member[] | select(.href | endswith("C1N1")) | .href

TASK [Debug unsafe_text data type] **********************************************************************************************************************************************************************************************************
task path: /dev/stdin:22
Friday 27 September 2024  19:36:54 +0000 (0:00:00.041)       0:00:00.361 ******
[task 1200179] 19:36:54.333199 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000020
[task 1200179] 19:36:54.361661 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] =>
  msg: 'unsafe_text: str'

TASK [Strip unsafe in Ansible <= 2.13] ******************************************************************************************************************************************************************************************************
task path: /dev/stdin:26
Friday 27 September 2024  19:36:54 +0000 (0:00:00.040)       0:00:00.401 ******
[task 1200181] 19:36:54.373349 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000040
[task 1200181] 19:36:54.401580 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] => changed=false
  ansible_facts:
    stripped_value: .links.Member[] | select(.href | endswith("C1N1")) | .href

TASK [Strip unsafe in Ansible 2.14 - 2.16] **************************************************************************************************************************************************************************************************
task path: /dev/stdin:31
Friday 27 September 2024  19:36:54 +0000 (0:00:00.039)       0:00:00.441 ******
[task 1200183] 19:36:54.412944 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000080
skipping: [localhost] => changed=false
  skip_reason: Conditional result was False

TASK [Debug stripped value] *****************************************************************************************************************************************************************************************************************
task path: /dev/stdin:36
Friday 27 September 2024  19:36:54 +0000 (0:00:00.020)       0:00:00.462 ******
[task 1200185] 19:36:54.433728 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000100
[task 1200185] 19:36:54.448090 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] =>
  stripped_value: .links.Member[] | select(.href | endswith("C1N1")) | .href

TASK [Debug stripped_value data type] *******************************************************************************************************************************************************************************************************
task path: /dev/stdin:40
Friday 27 September 2024  19:36:54 +0000 (0:00:00.026)       0:00:00.489 ******
[task 1200187] 19:36:54.462134 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000200
[task 1200187] 19:36:54.494570 D ansible_mitogen.mixins: _remove_tmp_path(None)
ok: [localhost] =>
  msg: 'stripped_value: NativeJinjaText'
META: ran handlers
META: ran handlers

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=7    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Friday 27 September 2024  19:36:54 +0000 (0:00:00.050)       0:00:00.539 ******
===============================================================================
Debug stripped_value data type ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.05s
/dev/stdin:40 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Render unsafe_text with host_seq ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.04s
/dev/stdin:18 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug unsafe_text data type ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.04s
/dev/stdin:22 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Strip unsafe in Ansible <= 2.13 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 0.04s
/dev/stdin:26 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
debug -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.03s
/dev/stdin:11 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Check if ansible-core is <= 2.13 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.03s
/dev/stdin:14 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Debug stripped value ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.03s
/dev/stdin:36 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Strip unsafe in Ansible 2.14 - 2.16 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 0.02s
/dev/stdin:31 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
[mux  1200139] 19:36:54.507037 D mitogen: Waker(fd=6/8): disconnecting
[mux  1200139] 19:36:54.507816 D mitogen: Router(Broker(3610)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
[mux  1200139] 19:36:54.508042 D mitogen.service: thread mitogen.Pool.3dc0.0 exiting gracefully
[mux  1200139] 19:36:54.508468 D mitogen.service: thread mitogen.Pool.3dc0.1 exiting gracefully
[mux  1200139] 19:36:54.508672 D mitogen.service: thread mitogen.Pool.3dc0.2 exiting gracefully
[mux  1200139] 19:36:54.508900 D mitogen.service: thread mitogen.Pool.3dc0.3 exiting gracefully
[mux  1200139] 19:36:54.509091 D mitogen.service: thread mitogen.Pool.3dc0.4 exiting gracefully
[mux  1200139] 19:36:54.509290 D mitogen.service: thread mitogen.Pool.3dc0.5 exiting gracefully
[mux  1200139] 19:36:54.509490 D mitogen.service: thread mitogen.Pool.3dc0.6 exiting gracefully
[mux  1200139] 19:36:54.509691 D mitogen.service: thread mitogen.Pool.3dc0.7 exiting gracefully
[mux  1200139] 19:36:54.510020 D mitogen.service: thread mitogen.Pool.3dc0.8 exiting gracefully
[mux  1200139] 19:36:54.510224 D mitogen.service: thread mitogen.Pool.3dc0.9 exiting gracefully
[mux  1200139] 19:36:54.510422 D mitogen.service: thread mitogen.Pool.3dc0.10 exiting gracefully
[mux  1200139] 19:36:54.510627 D mitogen.service: thread mitogen.Pool.3dc0.11 exiting gracefully
[mux  1200139] 19:36:54.510847 D mitogen.service: thread mitogen.Pool.3dc0.12 exiting gracefully
[mux  1200139] 19:36:54.511056 D mitogen.service: thread mitogen.Pool.3dc0.13 exiting gracefully
[mux  1200139] 19:36:54.511255 D mitogen.service: thread mitogen.Pool.3dc0.14 exiting gracefully
[mux  1200139] 19:36:54.511564 D mitogen.service: thread mitogen.Pool.3dc0.15 exiting gracefully
[mux  1200139] 19:36:54.511781 D mitogen.service: thread mitogen.Pool.3dc0.16 exiting gracefully
[mux  1200139] 19:36:54.511986 D mitogen.service: thread mitogen.Pool.3dc0.17 exiting gracefully
[mux  1200139] 19:36:54.512194 D mitogen.service: thread mitogen.Pool.3dc0.18 exiting gracefully
[mux  1200139] 19:36:54.512399 D mitogen.service: thread mitogen.Pool.3dc0.19 exiting gracefully
[mux  1200139] 19:36:54.512653 D mitogen.service: thread mitogen.Pool.3dc0.20 exiting gracefully
[mux  1200139] 19:36:54.512899 D mitogen.service: thread mitogen.Pool.3dc0.21 exiting gracefully
[mux  1200139] 19:36:54.513216 D mitogen.service: thread mitogen.Pool.3dc0.22 exiting gracefully
[mux  1200139] 19:36:54.513424 D mitogen.service: thread mitogen.Pool.3dc0.23 exiting gracefully
[mux  1200139] 19:36:54.513651 D mitogen.service: thread mitogen.Pool.3dc0.24 exiting gracefully
[mux  1200139] 19:36:54.513892 D mitogen.service: thread mitogen.Pool.3dc0.25 exiting gracefully
[mux  1200139] 19:36:54.514091 D mitogen.service: thread mitogen.Pool.3dc0.28 exiting gracefully
[mux  1200139] 19:36:54.514397 D mitogen.service: thread mitogen.Pool.3dc0.29 exiting gracefully
[mux  1200139] 19:36:54.514592 D mitogen.service: thread mitogen.Pool.3dc0.27 exiting gracefully
[mux  1200139] 19:36:54.514811 D mitogen.service: thread mitogen.Pool.3dc0.26 exiting gracefully
[mux  1200139] 19:36:54.515298 D mitogen.service: thread mitogen.Pool.3dc0.30 exiting gracefully
[mux  1200139] 19:36:54.515530 D mitogen.service: thread mitogen.Pool.3dc0.31 exiting gracefully
[mux  1200139] 19:36:54.515702 D mitogen.service: FileService().on_shutdown()
[top  1200134] 19:36:54.520627 D ansible_mitogen.process: multiplexer 0 PID 1200139 exited with return code 0
bash-4.4#

Now I just have to figure out how to support newer versions of ansible as it does not work for 2.16.

A lot of smart people studied a subtle but serious problem and as a result went to great lengths to prevent the kind of thing you’re trying to do. Not to say you don’t have a legitimate use case, but maybe there’s a way to accomplish what you need without having to circumvent security measures.

Because your problem isn’t to defeat !unsafe. Your problem is to interpolate values of some variables into strings. So focus on that.

For example, rather than relying on the full power of the templating engine (while hoping the supplied template-containing strings are “safe enough”), why not explicitly handle a restricted set of variable substitutions that you can vet in advance, like so:

    - name: Interpolate vetted var into a string in all Ansible versions
      set_fact:
        interpolated_value: '{{ unsafe_text | regex_replace("^(.*)\{\{.*\}\}(.*)", "\g<1>" ~ inner_val ~ "\g<2>") }}'
      when:
        - inner_var in allowed_vars
      vars:
        inner_var: '{{ unsafe_text | regex_replace("^.*\{\{ *([a-z0-9_]+) *\}\}.*", "\1") }}'
        inner_val: '{{ lookup("ansible.builtin.vars", inner_var, default="BROKEN") }}'
        allowed_vars:
          - host_seq
          - guest_lease
          - secret_sauce
1 Like

@utoddl that seems to work just fine! Thank you!

[root@CHQS-EniCEJ55G6 ansible]# cat inventory.ini
### Ansible Inventory file for Orch-Spray Legacy
#
localhost host_seq=99
anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast host_seq=1
xcdr-as-a-22.sys.comcast host_seq=2
test host_seq=3
host host_seq=4
[root@CHQS-EniCEJ55G6 ansible]#
[root@CHQS-EniCEJ55G6 ansible]#
[root@CHQS-EniCEJ55G6 ansible]#
[root@CHQS-EniCEJ55G6 ansible]# ansible-playbook -i inventory.ini -c local /dev/stdin <<EOF
---
- name: Variable templating example
  hosts: all
  become: false
  gather_facts: false

  vars:
    unsafe_text: !unsafe .links.Member[] | select(.href | endswith("C{{ host_seq }}N1")) | .href

  tasks:
    - name: Debug unsafe_text data type
      debug:
        msg: "unsafe_text: {{ unsafe_text | type_debug }}"

    - name: Interpolate vetted var into a string in all Ansible versions
      set_fact:
        interpolated_value: '{{ unsafe_text | regex_replace("^(.*)\{\{.*\}\}(.*)", "\g<1>" ~ inner_val ~ "\g<2>") }}'
      when:
        - inner_var in allowed_vars
      vars:
        inner_var: '{{ unsafe_text | regex_replace("^.*\{\{ *([a-z0-9_]+) *\}\}.*", "\1") }}'
        inner_val: '{{ lookup("ansible.builtin.vars", inner_var, default="BROKEN") }}'
        allowed_vars:
          - host_seq
          - guest_lease
          - secret_sauce

    - name: Debug interpolated_value data type
      debug:
        msg: "interpolated_value: {{ interpolated_value | type_debug }}"

    - name: Debug interpolated_value value
      debug:
        var: interpolated_value
EOF


PLAY [Variable templating example] ********************************************************************************************************************************************************************************************************************************************

TASK [Debug unsafe_text data type] ********************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "unsafe_text: AnsibleUnsafeText"
}
ok: [test] => {
    "msg": "unsafe_text: AnsibleUnsafeText"
}
ok: [anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast] => {
    "msg": "unsafe_text: AnsibleUnsafeText"
}
ok: [xcdr-as-a-22.sys.comcast] => {
    "msg": "unsafe_text: AnsibleUnsafeText"
}
ok: [host] => {
    "msg": "unsafe_text: AnsibleUnsafeText"
}

TASK [Interpolate vetted var into a string in all Ansible versions] ***********************************************************************************************************************************************************************************************************
ok: [localhost]
ok: [anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast]
ok: [xcdr-as-a-22.sys.comcast]
ok: [test]
ok: [host]

TASK [Debug interpolated_value data type] *************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "interpolated_value: AnsibleUnsafeText"
}
ok: [anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast] => {
    "msg": "interpolated_value: AnsibleUnsafeText"
}
ok: [xcdr-as-a-22.sys.comcast] => {
    "msg": "interpolated_value: AnsibleUnsafeText"
}
ok: [test] => {
    "msg": "interpolated_value: AnsibleUnsafeText"
}
ok: [host] => {
    "msg": "interpolated_value: AnsibleUnsafeText"
}

TASK [Debug interpolated_value value] *****************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "interpolated_value": ".links.Member[] | select(.href | endswith(\"C99N1\")) | .href"
}
ok: [anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast] => {
    "interpolated_value": ".links.Member[] | select(.href | endswith(\"C1N1\")) | .href"
}
ok: [xcdr-as-a-22.sys.comcast] => {
    "interpolated_value": ".links.Member[] | select(.href | endswith(\"C2N1\")) | .href"
}
ok: [test] => {
    "interpolated_value": ".links.Member[] | select(.href | endswith(\"C3N1\")) | .href"
}
ok: [host] => {
    "interpolated_value": ".links.Member[] | select(.href | endswith(\"C4N1\")) | .href"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************
anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host                       : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
test                       : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
xcdr-as-a-22.sys.comcast   : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[root@CHQS-EniCEJ55G6 ansible]#

I tried something somewhat similar (below) which it worked without !unsafe and replaced host_seq with a number; however, when I added !unsafe it just removed the curly braces and left me with Chost_seqN1:

[root@CHQS-EniCEJ55G6 ansible]# ansible-playbook -i inventory.ini -c local /dev/stdin <<EOF
---
- name: Variable templating example
  hosts: all
  become: false
  gather_facts: false

  vars:
    api_action: "get-pxe-mac-address"
    host_seq: 1
    jq_filters:
      - ".links.Systems.href"
      - !unsafe ".links.Member[] | select(.href | endswith('C{{ host_seq }}N1')) | .href"
      - '{ "MACAddress": .HostCorrelation.HostMACAddress[0]}'

  tasks:
    - name: Apply manual templating
      set_fact:
        jq_filters_templated: >
          {{ jq_filters | map('regex_replace', '{{\s*(.+?)\s*}}', '\\1') }}

    - debug:
        var: jq_filters_templated
EOF


PLAY [Variable templating example] ********************************************************************************************************************************************************************************************************************************************

TASK [Apply manual templating] ************************************************************************************************************************************************************************************************************************************************
ok: [localhost]
ok: [anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast]
ok: [test]
ok: [host]
ok: [xcdr-as-a-22.sys.comcast]

TASK [debug] ******************************************************************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "jq_filters_templated": [
        ".links.Systems.href",
        ".links.Member[] | select(.href | endswith('Chost_seqN1')) | .href",
        "{ \"MACAddress\": .HostCorrelation.HostMACAddress[0]}"
    ]
}
ok: [anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast] => {
    "jq_filters_templated": [
        ".links.Systems.href",
        ".links.Member[] | select(.href | endswith('Chost_seqN1')) | .href",
        "{ \"MACAddress\": .HostCorrelation.HostMACAddress[0]}"
    ]
}
ok: [test] => {
    "jq_filters_templated": [
        ".links.Systems.href",
        ".links.Member[] | select(.href | endswith('Chost_seqN1')) | .href",
        "{ \"MACAddress\": .HostCorrelation.HostMACAddress[0]}"
    ]
}
ok: [xcdr-as-a-22.sys.comcast] => {
    "jq_filters_templated": [
        ".links.Systems.href",
        ".links.Member[] | select(.href | endswith('Chost_seqN1')) | .href",
        "{ \"MACAddress\": .HostCorrelation.HostMACAddress[0]}"
    ]
}
ok: [host] => {
    "jq_filters_templated": [
        ".links.Systems.href",
        ".links.Member[] | select(.href | endswith('Chost_seqN1')) | .href",
        "{ \"MACAddress\": .HostCorrelation.HostMACAddress[0]}"
    ]
}

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************
anv2-1007-1-stor-qaperf3-01.anvil2.vpi.comcast : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
test                       : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
xcdr-as-a-22.sys.comcast   : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

[root@CHQS-EniCEJ55G6 ansible]#

What is it that is different about your approach that ansible allows for it to template when specifying only allowed vars?

It isn’t templating. What’s different is this (what you were doing):

    - name: Substitute only the name of the variable
      set_fact:
        jq_filters_templated: >
          {{ jq_filters | map('regex_replace', '{{\s*(.+?)\s*}}', '\1') }}

versus this:

    - name: Substitute the looked-up value of the named variable
      set_fact:
        jq_filters_templated: >
          {% set final_filters = [] -%}
          {% for filter in jq_filters -%}
          {%   set inner_var = filter | regex_replace('^.*{{\s*(.+?)\s*}}.*', '\\1') -%}
          {%   set inner_val = lookup("ansible.builtin.vars", inner_var, default="") | string -%}
          {%   set _ = final_filters.append(filter | regex_replace('{{\s*(.+?)\s*}}', inner_val)) -%}
          {% endfor -%}{{ final_filters }}

Note the example here doesn’t restrict the variables, so a bad actor could feed it a string containing '{{vault_master_password}}`. And then they’ve won.

I know where you are because I’ve been there myself: You just want to get it to work, and it feels like the whole “don’t allow templating !unsafe strings” thing was invented specifically to make your life miserable. But it wasn’t. You need (we all need) to move from that place to a mindset of: “These restrictions are in place to protect me, my company, and our customers. How can I accomplish the end goal without compromising or circumventing these protections?”

That’s why the bit in my prior example – the part that vets which variables you’re willing to interpolate into unsafe strings – is in my opinion the most important part of this whole topic. You can accomplish what you need to while working within, and arguably enhancing, the security restrictions.

I’d go so far as to suggest switching from “{{…}}” delimiters in these jq_filters to something else, just to walk away from the idea of using Jinja on them at all. My 2¢.