# Support for Jinja2 test plugins **URL:** \ **Category:** Ansible Developer\ **Created:** [July 21, 2014, 6:44pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727 "2014-07-21T18:44:55Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Brandon\_Mintern1](https://avatars.discourse-cdn.com/v4/letter/b/57b2e6/32.png) [@Brandon\_Mintern1](https://forum.ansible.com/u/Brandon_Mintern1)\ **Post date:** [July 21, 2014, 6:44pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/1 "2014-07-21T18:44:55Z")
Ansible does not provide a way for users to register Jinja2 test plugins. Furthermore, many of Ansible’s filter plugins are logically tests, but they are not registered as such. I have submitted a Pull Request (there is some discussion there already, but I’ll repeat myself here): [https://github.com/ansible/ansible/pull/8217](https://github.com/ansible/ansible/pull/8217) Ansible has a concept of “filter\_plugins” that are registered with Jinja2 in order to provide new functionality in variable templates. This enables custom filters to be applied with a | character. Users can even create their own custom ones. Jinja2, though, actually has two separate notions: “tests” transform variables into booleans (used by is, is not, and some filters); all other transformations are “filters” (which use the | syntax). See [http://jinja.pocoo.org/docs/templates/#builtin-filters](http://jinja.pocoo.org/docs/templates/#builtin-filters) … the Tests documentation immediately follows the Filters documentation. Ansible conflates these two notions, registering all of its Jinja2 plugins only as filters, even the ones (like match and version\_compare) that are logically tests. This means that Ansible’s boolean “filters” cannot be used as arguments to filter functions like reject. It also is the reason for the inconsistency between “var | success” and “var is defined”; the former is an Ansible filter, and the latter is a built-in Jinja2 one. My proposal is for Ansible to have two separate Jinja2 plugin groups: tests and filters. For backwards compatibility, all tests must also be registered as filters. Additionally, users must be able to register new tests (as they currently can with filters). My current implementation (the aforementioned Pull Request) does this. It places the plugins in two entirely separate groups (test\_plugins and filter\_plugins) because that seemed to me to offer the most clarity and flexibility for Ansible users, but there is no reason that the definitions could not be combined into a single file. This change allows entirely new patterns of usage. For example, it can allow the user to pull certain lines from stdout\_lines (e.g., those matching a pattern), and apply with\_items to each of those. The user could even add new tests like “startswith(prefix)” for string matching. At present, none of this is possible (AFAICT) without using external scripts. Again, I already have a working implementation at [https://github.com/ansible/ansible/pull/8217](https://github.com/ansible/ansible/pull/8217) that’s ready to go (documentation and all), but I’m happy to discuss other approaches. Cheers, Brandon ---
**Author:** ![Brandon\_Mintern1](https://avatars.discourse-cdn.com/v4/letter/b/57b2e6/32.png) [@Brandon\_Mintern1](https://forum.ansible.com/u/Brandon_Mintern1)\ **Post date:** [July 21, 2014, 6:58pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/2 "2014-07-21T18:58:18Z")
Yes, absolutely. This is due to these lines in lib/ansible/utils/template.py, which prepare the Jinja2 environment: environment.filters.update(\_get\_filters()) environment.filters.update(\_get\_tests()) environment.tests.update(\_get\_tests()) Also, at present, the “test\_plugins” are simply the “filter\_plugins” that returned boolean values, nothing more. Of course, more could be added easily, either to core.py or by the user. ---
**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\ **Post date:** [July 21, 2014, 7:06pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/3 "2014-07-21T19:06:07Z")
cool ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 21, 2014, 11:17pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/4 "2014-07-21T23:17:04Z")
(A) First question - explain what Jinja2 test plugins are. (B) "Ansible conflates these two notions, registering all of its Jinja2 plugins only as filters, even the ones (like match and version\_compare) that are logically tests. " Follow up questions: - conflates has a connotation. Why is this a problem? - Are you proposing a backward compatible break? (C) Seems like no to the break, in which case, I’m failing to understand why to not just register all tests into the filter namespace. (D) "For example, it can allow the user to pull certain lines from stdout\_lines (e.g., those matching a pattern), and apply with\_items to each of those. " An example would help me understand more here. ---
**Author:** ![Brandon\_Mintern1](https://avatars.discourse-cdn.com/v4/letter/b/57b2e6/32.png) [@Brandon\_Mintern1](https://forum.ansible.com/u/Brandon_Mintern1)\ **Post date:** [July 22, 2014, 1:15am UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/5 "2014-07-22T01:15:02Z")
Hi Michael, ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 22, 2014, 12:06pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/6 "2014-07-22T12:06:30Z")
Thanks for replies. First off, I should point out that in Ansible our usage of Jinja2 is _MOSTLY_ just to have a way to do variables and simple loops. I don’t really embrace all of it’s messed up syntax in playbooks, though if some people want to do that it’s fine. I just want that consideration to be apparent as we go through this - parts of Jinja2 are not easy to use, most parts of it are completely fine, etc. We want to keep the barrier to learning ansible very low, and day to day usage easy and not full of a lot of idioms that don’t feel right together. This is why, for instance, we have our own loops, rather than just doing something like rendering YAML in a template, so it stays a parseable data format, etc. Replies below. ---
**Author:** ![C\_Morgan\_Hamill](https://avatars.discourse-cdn.com/v4/letter/c/ebca7d/32.png) [@C\_Morgan\_Hamill](https://forum.ansible.com/u/C_Morgan_Hamill)\ **Post date:** [July 22, 2014, 12:49pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/7 "2014-07-22T12:49:12Z")
Excerpts from Michael DeHaan's message of 2014-07-22 08:06:30 -0400: > \> # ec2 is now a list of instance tags > \> > \> - name: filter web tags > \> set\_fact: > \> web\_tag: "{{ ec2.data | selectattr('value', 'match', 'WebServer.\*') | > \> first }}" > \> > > I'm unclear how the above is using a test. I'm not seeing the "is" syntax > anywhere. This looks like filters to me. Just chiming in, because I ran into an almost identical issue recently, and wouldn't mind having a way to add Jinja "tests" (which are indeed a bit confusing. The above Jinja expression is using a test called 'match', and is equivalent to, say, the following list comprehension:     [i for i in ec2['data'] when i.value|match('WebServer.\*') ] Pardon my mixing of a Jinja filter with raw Python, but perhaps that helps. ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 22, 2014, 1:00pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/8 "2014-07-22T13:00:28Z")
Ok… I don’t find the selecattr syntax easy to understand at all, but in a sense Jinja2 is already limited because it doesn’t allow list comprehensions, which would be cleaner to an extent. I’m ok with just exposing what’s there in the Environment with the equivalent of whatever “enable\_tests” is, which should enable the above, but I’m not sold on test plugins at all yet. It’s something else for people to understand, and even if 3% of the users want it, that may not be enough for the additional cognitive overhead. (Random aside on why all template engines devolve into horrible monstrosities and how I need to write my own (but know better because that’s a trap (no I don’t count $foo support earlier in that, but it’s instructional)) in 3… 2… ) –Michael ---
**Author:** ![C\_Morgan\_Hamill](https://avatars.discourse-cdn.com/v4/letter/c/ebca7d/32.png) [@C\_Morgan\_Hamill](https://forum.ansible.com/u/C_Morgan_Hamill)\ **Post date:** [July 22, 2014, 1:15pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/9 "2014-07-22T13:15:26Z")
Excerpts from Michael DeHaan's message of 2014-07-22 09:00:28 -0400: > Ok... > > I don't find the selecattr syntax easy to understand at all, but in a sense > Jinja2 is already limited because it doesn't allow list comprehensions, > which would be cleaner to an extent. I agree it's basically baffling. > I'm ok with just exposing what's there in the Environment with the > equivalent of whatever "enable\_tests" is, which should enable the above, > but I'm not sold on test plugins at all yet. It's something else for > people to understand, and even if 3% of the users want it, that may not be > enough for the additional cognitive overhead. I also don't know that there's anything to be gained by allowing folks to create test plugins directly, but being able to use existing filters that return booleans as the predicate in the select/selectattr/reject/rejectattr filters would be very useful occasionally. > (Random aside on why all template engines devolve into horrible > monstrosities and how I need to write my own (but know better because > that's a trap (no I don't count $foo support earlier in that, but it's > instructional)) in 3... 2... ) admiral\_ackbar.jpg ---
**Author:** ![Brandon\_Mintern1](https://avatars.discourse-cdn.com/v4/letter/b/57b2e6/32.png) [@Brandon\_Mintern1](https://forum.ansible.com/u/Brandon_Mintern1)\ **Post date:** [July 22, 2014, 1:34pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/10 "2014-07-22T13:34:37Z")
Thanks for everyone's replies. I've opened a new Pull Request ( [https://github.com/ansible/ansible/pull/8235](https://github.com/ansible/ansible/pull/8235)) that I hope addresses the remaining concerns. > Thanks for replies. > > First off, I should point out that in Ansible our usage of Jinja2 is > \*MOSTLY\* just to have a way to do variables and simple loops. > > I don't really embrace all of it's messed up syntax in playbooks, though > if some people want to do that it's fine. > > I just want that consideration to be apparent as we go through this - > parts of Jinja2 are not easy to use, most parts of it are completely fine, > etc. > > We want to keep the barrier to learning ansible very low, and day to day > usage easy and not full of a lot of idioms that don't feel right together. > > This is why, for instance, we have our own loops, rather than just doing > something like rendering YAML in a template, so it stays a parseable data > format, etc. I totally agree. I just want a simple way to extract values from a variable list using Jinja's select, selectattr, reject, and rejectattr, and that requires tests. > > (A) > > > > > First question - explain what Jinja2 test plugins are. > > > > [http://jinja.pocoo.org/docs/templates/#tests](http://jinja.pocoo.org/docs/templates/#tests) > > Thanks. So it seems most of this is the "is" operator. > > To me, this doesn't feel a lot different that " x | foo" returning a > True/False value, is there value to this other than syntax? > > I also don't think I care for the non-python chaos which is: > > {% if foo.expression is equalto 42 %} > > Why not? Because it could already be written as: > > {% if foo == 42 %} > > Maybe I'm still missing something. Totally agree again. I don't have a problem with | instead of 'is'. The only thing I really care about is being able to use tests in other filter expressions and, as a user, to be able to define my own tests. I'm going to skip some questions that appeared to be cleared up in later discussions. (B) > > > "Ansible conflates these two notions, registering all of its Jinja2 > > > plugins only as filters, even the ones (like match and version\_compare) > > > that are logically tests. " > > > > > > Follow up questions: > > > > > > - conflates has a connotation. Why is this a problem? > > > > It's a problem because the Jinja paradigm (but I'm by no means an expert > > here) seems to be to use `is` for tests and `|` for other filters. Ansible > > departs from that. This is confusing from a user perspective, when, e.g., > > `var is defined` (a Jinja2 test) and `res | success` (an Ansible boolean > > filter) are inconsistent, even though both are effectively using tests. It > > could easily be `res is successful` if successful was an Ansible test. > > Why does the Jinja2 paradigm even remotely matter? > > "when x is defined" doesn't feel too inconsistent to me, as python > normally has "when x is None", so it's just introducing the new > "variableness" of defined. I was pointing out that Ansible is already using the Jinja2 paradigm in some cases. The paradigm matters only when users are trying to use Jinja's builtin tests, of which "defined" is one. Consistency is generally a nice goal, but I recognize that arguments could be made in either way (i.e., "It's nice to always use |" vs. "We're using Jinja2 here, so we may as well stick to their conventions"). Furthermore, some filters (select is another example) accept a test as an > > argument. As an Ansible user, I was surprised to find that I could not pass > > `match` to a `selectattr` filter. > > > > - Are you proposing a backward compatible break? > > > > > (C) > > > > > > Seems like no to the break, in which case, I'm failing to understand why > > > to not just register all tests into the filter namespace. > > > > Because I would also like to get the benefits of Jinja2 tests, and allow > > Ansible users to define new tests. Maybe I'm misunderstanding the question > > here. > > It seems the core tests could just be added to the Environment without > introducing a new degree of test plugins, regardless, which few people > need/use (based on extremely large user base, and this having not come up > before). An extra plugin is something else to keep in one's mental > capacity, something else to know, more syntax, something at least to decide > to know when you don't need. I agree. My new Pull Request addresses this by simply adding a FilterModule.tests method alongside FilterModule.filters. Users can add new filters and tests by implementing filters, tests, or both, using the existing filter\_plugins infrastructure. Cheers, Brandon ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [July 22, 2014, 1:51pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/11 "2014-07-22T13:51:47Z")
Thanks! My general thought is “Ansible users should be able to use Jinja2 without knowing much about Jinja2”. I feel {{ x }} should just work, they shouldn’t know with\_items is Jinja2 powered at all, and maybe they learn loops if they are using the template module. This is looks like a good compromise, though will have to review/test things obviously. Thanks again! ---
**Author:** ![Brandon\_Mintern1](https://avatars.discourse-cdn.com/v4/letter/b/57b2e6/32.png) [@Brandon\_Mintern1](https://forum.ansible.com/u/Brandon_Mintern1)\ **Post date:** [July 22, 2014, 2:05pm UTC](https://forum.ansible.com/t/support-for-jinja2-test-plugins/15727/12 "2014-07-22T14:05:05Z")
> Thanks! > > My general thought is "Ansible users should be able to use Jinja2 without > knowing much about Jinja2". > > I feel {{ x }} should just work, they shouldn't know with\_items is Jinja2 > powered at all, and maybe they learn loops if they are using the template > module. As a new user, I appreciated this approach. This is looks like a good compromise, though will have to review/test > things obviously. > > Thanks again! Great! Thank you for your quick feedback! > > Users can add new filters and tests by implementing filters, tests, or > > both, using the existing filter\_plugins infrastructure. Here's a silly-but-working example of this (perhaps useful for a new test?): # plugins/filters.py def length(value, n):     return len(value) == n class FilterModule(object):     def tests(self):         return {             "length": length         } # filter-test.yml - hosts: localhost   gather\_facts: no   vars:     lines:       - hello world       - goodbye world       - foo bar   tasks:     - set\_fact:         worlds: "{{ lines | select('match', '.\* world') | list }}"     - debug: var=worlds     - debug: msg=Success       when: worlds is length(2) $ ANSIBLE\_FILTER\_PLUGINS=plugins ansible-playbook -i localhost, -e connection=local filter-test.yml PLAY [localhost]