Support for Jinja2 test plugins

Archives Ansible Developer
1

Ansible does not provide a way for users to register Jinja2 test plugins. Furthermore, many of Ansible’s filter plugins are logically tests, but they are not registered as such. I have submitted a Pull Request (there is some discussion there already, but I’ll repeat myself here):

https://github.com/ansible/ansible/pull/8217

Ansible has a concept of “filter_plugins” that are registered with Jinja2 in order to provide new functionality in variable templates. This enables custom filters to be applied with a | character. Users can even create their own custom ones.

Jinja2, though, actually has two separate notions: “tests” transform variables into booleans (used by is, is not, and some filters); all other transformations are “filters” (which use the | syntax). See http://jinja.pocoo.org/docs/templates/#builtin-filters … the Tests documentation immediately follows the Filters documentation.

Ansible conflates these two notions, registering all of its Jinja2 plugins only as filters, even the ones (like match and version_compare) that are logically tests. This means that Ansible’s boolean “filters” cannot be used as arguments to filter functions like reject. It also is the reason for the inconsistency between “var | success” and “var is defined”; the former is an Ansible filter, and the latter is a built-in Jinja2 one.

My proposal is for Ansible to have two separate Jinja2 plugin groups: tests and filters. For backwards compatibility, all tests must also be registered as filters. Additionally, users must be able to register new tests (as they currently can with filters).

My current implementation (the aforementioned Pull Request) does this. It places the plugins in two entirely separate groups (test_plugins and filter_plugins) because that seemed to me to offer the most clarity and flexibility for Ansible users, but there is no reason that the definitions could not be combined into a single file.

This change allows entirely new patterns of usage. For example, it can allow the user to pull certain lines from stdout_lines (e.g., those matching a pattern), and apply with_items to each of those. The user could even add new tests like “startswith(prefix)” for string matching. At present, none of this is possible (AFAICT) without using external scripts.

Again, I already have a working implementation at https://github.com/ansible/ansible/pull/8217 that’s ready to go (documentation and all), but I’m happy to discuss other approaches.

Cheers,

Brandon

2

Yes, absolutely. This is due to these lines in lib/ansible/utils/template.py, which prepare the Jinja2 environment:

environment.filters.update(_get_filters())
environment.filters.update(_get_tests())
environment.tests.update(_get_tests())

Also, at present, the “test_plugins” are simply the “filter_plugins” that returned boolean values, nothing more. Of course, more could be added easily, either to core.py or by the user.

3

cool

4

(A)

First question - explain what Jinja2 test plugins are.

(B)

"Ansible conflates these two notions, registering all of its Jinja2 plugins only as filters, even the ones (like match and version_compare) that are logically tests. "

Follow up questions:

  • conflates has a connotation. Why is this a problem?
  • Are you proposing a backward compatible break?

(C)

Seems like no to the break, in which case, I’m failing to understand why to not just register all tests into the filter namespace.

(D)

"For example, it can allow the user to pull certain lines from stdout_lines (e.g., those matching a pattern), and apply with_items to each of those. "

An example would help me understand more here.

5

Hi Michael,

6

Thanks for replies.

First off, I should point out that in Ansible our usage of Jinja2 is MOSTLY just to have a way to do variables and simple loops.

I don’t really embrace all of it’s messed up syntax in playbooks, though if some people want to do that it’s fine.

I just want that consideration to be apparent as we go through this - parts of Jinja2 are not easy to use, most parts of it are completely fine, etc.

We want to keep the barrier to learning ansible very low, and day to day usage easy and not full of a lot of idioms that don’t feel right together.

This is why, for instance, we have our own loops, rather than just doing something like rendering YAML in a template, so it stays a parseable data format, etc.

Replies below.

7

Excerpts from Michael DeHaan's message of 2014-07-22 08:06:30 -0400:

> # ec2 is now a list of instance tags
>
> - name: filter web tags
> set_fact:
> web_tag: "{{ ec2.data | selectattr('value', 'match', 'WebServer.*') |
> first }}"
>

I'm unclear how the above is using a test. I'm not seeing the "is" syntax
anywhere. This looks like filters to me.

Just chiming in, because I ran into an almost identical issue recently,
and wouldn't mind having a way to add Jinja "tests" (which are indeed
a bit confusing.

The above Jinja expression is using a test called 'match', and is
equivalent to, say, the following list comprehension:

    [ i for i in ec2['data'] when i.value|match('WebServer.*') ]

Pardon my mixing of a Jinja filter with raw Python, but perhaps that
helps.

8

Ok…

I don’t find the selecattr syntax easy to understand at all, but in a sense Jinja2 is already limited because it doesn’t allow list comprehensions, which would be cleaner to an extent.

I’m ok with just exposing what’s there in the Environment with the equivalent of whatever “enable_tests” is, which should enable the above, but I’m not sold on test plugins at all yet. It’s something else for people to understand, and even if 3% of the users want it, that may not be enough for the additional cognitive overhead.

(Random aside on why all template engines devolve into horrible monstrosities and how I need to write my own (but know better because that’s a trap (no I don’t count $foo support earlier in that, but it’s instructional)) in 3… 2… )

–Michael

9

Excerpts from Michael DeHaan's message of 2014-07-22 09:00:28 -0400:

Ok...

I don't find the selecattr syntax easy to understand at all, but in a sense
Jinja2 is already limited because it doesn't allow list comprehensions,
which would be cleaner to an extent.

I agree it's basically baffling.

I'm ok with just exposing what's there in the Environment with the
equivalent of whatever "enable_tests" is, which should enable the above,
but I'm not sold on test plugins at all yet. It's something else for
people to understand, and even if 3% of the users want it, that may not be
enough for the additional cognitive overhead.

I also don't know that there's anything to be gained by allowing folks
to create test plugins directly, but being able to use existing filters
that return booleans as the predicate in the select/selectattr/reject/rejectattr
filters would be very useful occasionally.

(Random aside on why all template engines devolve into horrible
monstrosities and how I need to write my own (but know better because
that's a trap (no I don't count $foo support earlier in that, but it's
instructional)) in 3... 2... )

admiral_ackbar.jpg

10

Thanks for everyone's replies. I've opened a new Pull Request (
https://github.com/ansible/ansible/pull/8235) that I hope addresses the
remaining concerns.

Thanks for replies.

First off, I should point out that in Ansible our usage of Jinja2 is
*MOSTLY* just to have a way to do variables and simple loops.

I don't really embrace all of it's messed up syntax in playbooks, though
if some people want to do that it's fine.

I just want that consideration to be apparent as we go through this -
parts of Jinja2 are not easy to use, most parts of it are completely fine,
etc.

We want to keep the barrier to learning ansible very low, and day to day
usage easy and not full of a lot of idioms that don't feel right together.

This is why, for instance, we have our own loops, rather than just doing
something like rendering YAML in a template, so it stays a parseable data
format, etc.

I totally agree. I just want a simple way to extract values from a variable
list using Jinja's select, selectattr, reject, and rejectattr, and that
requires tests.

(A)

First question - explain what Jinja2 test plugins are.

http://jinja.pocoo.org/docs/templates/#tests

Thanks. So it seems most of this is the "is" operator.

To me, this doesn't feel a lot different that " x | foo" returning a
True/False value, is there value to this other than syntax?

I also don't think I care for the non-python chaos which is:

{% if foo.expression is equalto 42 %}

Why not? Because it could already be written as:

{% if foo == 42 %}

Maybe I'm still missing something.

Totally agree again. I don't have a problem with | instead of 'is'. The
only thing I really care about is being able to use tests in other filter
expressions and, as a user, to be able to define my own tests.

I'm going to skip some questions that appeared to be cleared up in later
discussions.

(B)

"Ansible conflates these two notions, registering all of its Jinja2
plugins only as filters, even the ones (like match and version_compare)
that are logically tests. "

Follow up questions:

- conflates has a connotation. Why is this a problem?

It's a problem because the Jinja paradigm (but I'm by no means an expert
here) seems to be to use `is` for tests and `|` for other filters. Ansible
departs from that. This is confusing from a user perspective, when, e.g.,
`var is defined` (a Jinja2 test) and `res | success` (an Ansible boolean
filter) are inconsistent, even though both are effectively using tests. It
could easily be `res is successful` if successful was an Ansible test.

Why does the Jinja2 paradigm even remotely matter?

"when x is defined" doesn't feel too inconsistent to me, as python
normally has "when x is None", so it's just introducing the new
"variableness" of defined.

I was pointing out that Ansible is already using the Jinja2 paradigm in
some cases. The paradigm matters only when users are trying to use Jinja's
builtin tests, of which "defined" is one. Consistency is generally a nice
goal, but I recognize that arguments could be made in either way (i.e.,
"It's nice to always use |" vs. "We're using Jinja2 here, so we may as well
stick to their conventions").

Furthermore, some filters (select is another example) accept a test as an

argument. As an Ansible user, I was surprised to find that I could not pass
`match` to a `selectattr` filter.

- Are you proposing a backward compatible break?

(C)

Seems like no to the break, in which case, I'm failing to understand why
to not just register all tests into the filter namespace.

Because I would also like to get the benefits of Jinja2 tests, and allow
Ansible users to define new tests. Maybe I'm misunderstanding the question
here.

It seems the core tests could just be added to the Environment without
introducing a new degree of test plugins, regardless, which few people
need/use (based on extremely large user base, and this having not come up
before). An extra plugin is something else to keep in one's mental
capacity, something else to know, more syntax, something at least to decide
to know when you don't need.

I agree. My new Pull Request addresses this by simply adding a
FilterModule.tests method alongside FilterModule.filters. Users can add new
filters and tests by implementing filters, tests, or both, using the
existing filter_plugins infrastructure.

Cheers,
Brandon

11

Thanks!

My general thought is “Ansible users should be able to use Jinja2 without knowing much about Jinja2”.

I feel {{ x }} should just work, they shouldn’t know with_items is Jinja2 powered at all, and maybe they learn loops if they are using the template module.

This is looks like a good compromise, though will have to review/test things obviously.

Thanks again!

12

Thanks!

My general thought is "Ansible users should be able to use Jinja2 without
knowing much about Jinja2".

I feel {{ x }} should just work, they shouldn't know with_items is Jinja2
powered at all, and maybe they learn loops if they are using the template
module.

As a new user, I appreciated this approach.

This is looks like a good compromise, though will have to review/test

things obviously.

Thanks again!

Great! Thank you for your quick feedback!

Users can add new filters and tests by implementing filters, tests, or
both, using the existing filter_plugins infrastructure.

Here's a silly-but-working example of this (perhaps useful for a new test?):

# plugins/filters.py
def length(value, n):
    return len(value) == n

class FilterModule(object):
    def tests(self):
        return {
            "length": length
        }

# filter-test.yml
- hosts: localhost
  gather_facts: no
  vars:
    lines:
      - hello world
      - goodbye world
      - foo bar
  tasks:
    - set_fact:
        worlds: "{{ lines | select('match', '.* world') | list }}"
    - debug: var=worlds
    - debug: msg=Success
      when: worlds is length(2)

$ ANSIBLE_FILTER_PLUGINS=plugins ansible-playbook -i localhost, -e
connection=local filter-test.yml

PLAY [localhost]