Using community.general.json_query in vars/main.yml with an Ansible role and ansible-core 2.19.x

I have a Ansible PHP role which uses json_query in the vars/main.yml file.

With ansible-core prior to 2.19.x this has been fine but with 2.19.x it doesn’t work, I have created a minimal test case to demonstrate this.

In the test case there is a roles/test/defaults/main.yml which defines a list of PHP packages like this:

test_php_versions:
  - name: PHP 8.4 packages
    version: "8.4"
    state: present
    pkg_absent:
      - libapache2-mod-php8.4
      - php8.4-phpdbg
    pkg_present:
      - php8.4-apcu
      - php8.4-bcmath
      - php8.4-bz2
      - php8.4-cli
      - php8.4-common
      - php8.4-curl
      - php8.4-gd
      - php8.4-gmp
      - php8.4-fpm
      - php8.4-imagick
      - php8.4-imap
      - php8.4-intl
      - php8.4-ldap
      - php8.4-mbstring
      - php8.4-mysql
      - php8.4-opcache
      - php8.4-readline
      - php8.4-redis
      - php8.4-sqlite3
      - php8.4-soap
      - php8.4-uploadprogress
      - php8.4-xml
      - php8.4-xmlrpc
      - php8.4-xsl
      - php8.4-zip
      - php-pear
  - name: PHP 8.3 packages
    version: "8.3"
    state: present
    pkg_absent:
      - libapache2-mod-php8.3
      - php8.3-phpdbg
    pkg_present:
      - php8.3-apcu
      - php8.3-bcmath
      - php8.3-bz2
      - php8.3-cli
      - php8.3-common
      - php8.3-curl
      - php8.3-gd
      - php8.3-gmp
      - php8.3-fpm
      - php8.3-imagick
      - php8.3-imap
      - php8.3-intl
      - php8.3-ldap
      - php8.3-mbstring
      - php8.3-mysql
      - php8.3-opcache
      - php8.3-readline
      - php8.3-redis
      - php8.3-sqlite3
      - php8.3-soap
      - php8.3-uploadprogress
      - php8.3-xml
      - php8.3-xmlrpc
      - php8.3-xsl
      - php8.3-zip
      - php-pear

And a roles/test/vars/main.yml which contains:

test_php_ver_present: "{{ test_php_versions | community.general.json_query('sort([?state==`present`].version)') }}"

Running the role fails with the task in roles/test/tasks/main.yml to print the test_php_ver_present variable:

- name: PHP versions that are set to be present
  ansible.builtin.debug:
    var: test_php_ver_present

The error is:

TASK [test : PHP versions that are set to be present] ********************************************************************************
[ERROR]: Task failed: Error while resolving `var` expression: The filter plugin 'community.general.json_query' failed: JMESPathError in json_query filter plugin:
In function sort(), invalid type for value: 8.4, expected one of: ['array-string', 'array-number'], received: "_AnsibleTaggedStr"

Task failed.
Origin: /test/roles/test/tasks/main.yml:5:7

3   block:
4
5     - name: PHP versions that are set to be present
        ^ column 7

<<< caused by >>>

Error while resolving `var` expression: The filter plugin 'community.general.json_query' failed.
Origin: /test/roles/test/tasks/main.yml:7:14

5     - name: PHP versions that are set to be present
6       ansible.builtin.debug:
7         var: test_php_ver_present
               ^ column 14

<<< caused by >>>

JMESPathError in json_query filter plugin:
In function sort(), invalid type for value: 8.4, expected one of: ['array-string', 'array-number'], received: "_AnsibleTaggedStr"

fatal: [localhost]: FAILED! => 
    msg: |-
        Task failed: Error while resolving `var` expression: The filter plugin 'community.general.json_query' failed: JMESPathError in json_query filter plugin:
        In function sort(), invalid type for value: 8.4, expected one of: ['array-string', 'array-number'], received: "_AnsibleTaggedStr"

This can also be tested using yq and jp:

cat roles/test/defaults/main.yml | yq -o=json -P | jp "sort(test_php_versions[?state=='present'].version)"
[
  "7.4",
  "8.0",
  "8.1",
  "8.2",
  "8.3",
  "8.4"
]

Is this error behaviour correct or is this a bug?

If it is correct how could I update the role to work with ansible-core 2.19.x?

  1. Replace the use of JMESPath via community.general.json_query with Jinja2 in /vars/main.yml?
  2. Move all the setting of variables in /vars/main.yml to tasks?
  3. Something else?

FWIW.

  • Do not use the json_query sort
test_php_ver_presen3: "{{ test_php_versions |
                          community.general.json_query('[?state==`present`].version') |
                          sort }}"
  • You don’t need json_query
test_php_ver_presen2: "{{ test_php_versions |
                          selectattr('state', '==', 'present') |
                          map(attribute='version') |
                          sort }}"
1 Like

Thanks @vbotka you are right, the JMESPath sort does appear to be the cause, is this a bug or a feature?

You are also right that it can be done using Jinja2, I find JMESPath easier so tend to use it a lot…

I don’t know. I also use JMESPath a lot. It works for me in 2.18 too. (I don’t have 2.19). You might want to look at the result of [?state==`present`].version in 2.19. How could this be _AnsibleTaggedStr ?

1 Like

Thanks @vbotka I don’t understand why 2.19 thinks a list is a _AnsibleTaggedStr.

I have also just come across another issue with a JMESPath query that doesn’t use sort that also generates an error, I’ll distil it to a minimal test case and post it, perhaps there is a general issue with the json_query filter and ansible-core 2.19.x?

I’ve another case where json_query fails with Ansible 2.19.0 but is fine with earlier versions, given a list like this:

test_php_fpm_pool_file_contents:
  - www74:
      group: www-data
      listen: /run/php/php7.4-fpm.sock
      listen.group: www-data
      listen.mode: '0666'
      listen.owner: www-data
      pm: dynamic
      pm.max_children: '4'
      pm.max_requests: '5000'
      pm.max_spare_servers: '2'
      pm.min_spare_servers: '2'
      pm.start_servers: '2'
      pm.status_path: /fpm-www74-status
      user: www-data
  - www81:
      group: www-data
      listen: /run/php/php8.1-fpm.sock
      listen.group: www-data
      listen.mode: '0666'
      listen.owner: www-data
      pm: dynamic
      pm.max_children: '4'
      pm.max_requests: '5000'
      pm.max_spare_servers: '2'
      pm.min_spare_servers: '2'
      pm.start_servers: '2'
      pm.status_path: /fpm-www81-status
      user: www-data

The following task to set a fact:

    - name: Set a fact based on a list of PHP-FPM socket paths and pool names
      ansible.builtin.set_fact:
        test_php_fpm_pool_names_and_socket_paths: "{{ test_php_fpm_pool_file_contents | community.general.json_query('[].{name: keys(@)|[0],socket: *.listen|[0]}') }}"

Fails with:

TASK [test : Set a fact based on a list of PHP-FPM socket paths and pool names] ******************************************************
[ERROR]: Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'test_php_fpm_pool_names_and_socket_paths': The filter plugin 'community.general.json_query' failed: JMESPathError in json_query filter plugin:
In function keys(), invalid type for value: {'www74': {'group': 'www-data', 'listen': '/run/php/php7.4-fpm.sock', 'listen.group': 'www-data', 'listen.mode': '0666', 'listen.owner': 'www-data', 'pm': 'dynamic', 'pm.max_children': '4', 'pm.max_requests': '5000', 'pm.max_spare_servers': '2', 'pm.min_spare_servers': '2', 'pm.start_servers': '2', 'pm.status_path': '/fpm-www74-status', 'user': 'www-data'}}, expected one of: ['object'], received: "unknown"

Task failed.
Origin: /test/roles/test/tasks/main.yml:29:7

27         var: test_php_fpm_pool_file_contents
28
29     - name: Set a fact based on a list of PHP-FPM socket paths and pool names
        ¦^ column 7

<<< caused by >>>

Finalization of task args for 'ansible.builtin.set_fact' failed.
Origin: /test/roles/test/tasks/main.yml:30:7

28
29     - name: Set a fact based on a list of PHP-FPM socket paths and pool names
30       ansible.builtin.set_fact:
        ¦^ column 7

<<< caused by >>>

Error while resolving value for 'test_php_fpm_pool_names_and_socket_paths': The filter plugin 'community.general.json_query' failed.
Origin: /test/roles/test/tasks/main.yml:31:51

29     - name: Set a fact based on a list of PHP-FPM socket paths and pool names
30       ansible.builtin.set_fact:
31         test_php_fpm_pool_names_and_socket_paths: "{{ test_php_fpm_pool_file_contents | community.general.json_que...
        ¦       ¦       ¦       ¦       ¦       ¦    ^ column 51

<<< caused by >>>

JMESPathError in json_query filter plugin:
In function keys(), invalid type for value: {'www74': {'group': 'www-data', 'listen': '/run/php/php7.4-fpm.sock', 'listen.group': 'www-data', 'listen.mode': '0666', 'listen.owner': 'www-data', 'pm': 'dynamic', 'pm.max_children': '4', 'pm.max_requests': '5000', 'pm.max_spare_servers': '2', 'pm.min_spare_servers': '2', 'pm.start_servers': '2', 'pm.status_path': '/fpm-www74-status', 'user': 'www-data'}}, expected one of: ['object'], received: "unknown"

fatal: [localhost]: FAILED! =>
    changed: false
    msg: |-
        Task failed: Finalization of task args for 'ansible.builtin.set_fact' failed: Error while resolving value for 'test_php_fpm_pool_names_and_socket_paths': The filter plugin 'community.general.json_query' failed: JMESPathError in json_query filter plugin:
        In function keys(), invalid type for value: {'www74': {'group': 'www-data', 'listen': '/run/php/php7.4-fpm.sock', 'listen.group': 'www-data', 'listen.mode': '0666', 'listen.owner': 'www-data', 'pm': 'dynamic', 'pm.max_children': '4', 'pm.max_requests': '5000', 'pm.max_spare_servers': '2', 'pm.min_spare_servers': '2', 'pm.start_servers': '2', 'pm.status_path': '/fpm-www74-status', 'user': 'www-data'}}, expected one of: ['object'], received: "unknown"

With an older version of ansible-core the task works:

TASK [test : Set a fact based on a list of PHP-FPM socket paths and pool names] ******************************************************
ok: [localhost] => 
    ansible_facts:
        test_php_fpm_pool_names_and_socket_paths:
        -   name: www74
            socket: /run/php/php7.4-fpm.sock
        -   name: www81
            socket: /run/php/php8.1-fpm.sock
    changed: false

I have added a task for the above to the demo repo and a couple of others, it does looks to me that there is an issue with community.general.json_query and ansible-core 2.19.0?

EDIT: I have opened an issue at GitHub for this problem.

2 Likes

Thanks @bcoca I’m now looking forward to the next release of community.general so I can easily test this fix — after manually editing json_query.py locally to apply this fix I still get the same error :frowning: , I also tried clobbering the local file like this:

wget https://raw.githubusercontent.com/felixfontein/community.general/a52c88a6bb72e3d06701e69bd842ced8f66ccba4/plugins/filter/json_query.py -O ~/.local./pipx/venvs/ansible/lib/python3.11/site-packages/ansible_collections/community/general/plugins/filter/json_query.py

And I still have the same error, @felixfontein any suggestions?

I wasn’t able to reproduce this with the fix (only without it). community.general 11.1.2 just got released with the fix, can you try it from there? Maybe that works better.

3 Likes

Thanks @felixfontein community.general 11.1.2 does fix it for me.