Expressions vs. templates

Project Discussions
, ,
1

One thing that frequently causes quite some pain in Ansible are automatic conversion of templated strings to types (here’s yet another occurence which triggered me finally writing this post, after seeing many similar “bug reports” or user questions in many different collection repos, on IRC, matrix, mailing list, …, and having thought about this multiple times already).

The main problem here is that if Ansible encounters a string that uses Jinja2 templating, it does not know whether a) it’s a expression or b) a template. Before I continue, let me make clear what I mean with that:

  1. An expression is something like {{ expression }} where the user wants expression to be evaluated and hopes to obtain the value returned by this expression without further modifications.
  2. A template is something that can also look like {{ expression }}, for example {{ lookup("ansible.builtin.file", "/path/to/file.json") }}, but can also contain more text around Jinja2 constructs (also conditionals etc.). Users generally expect this to be a string.

Now since Ansible cannot really distinguish between the two, it does a lot of magic to try to get this right. For example, if {{ expression }} evaluates to something that looks like a Python dict (JSON dicts often do that), it will parse it. This makes it really hard to handle strings containing JSON dicts in Ansible, like in {{ lookup("ansible.builtin.file", "/path/to/file.json") }}, since the unexpected conversions (especially the conversion back to strings if passed to a string parameter) usually results in something that isn’t JSON.

Sometimes these conversions are also used, for example by combining JSON with a template that’s full of Jinja2 constructs like {% for %} etc., which Ansible in some situations happily parses without further instructions. This causes more confusion since that parsing only happens in some cases, not in others. (I remember a forum thread recently with such a problem.)

So it would be great if there would be an easy way to tell Ansible whether something is an expression or a template. How about using YAML tags for that? Similar to !unsafe, which tells Ansible a value is “unsafe” and shouldn’t be templated; for example, !unsafe "{{ 'foo' }}" will always be the string {{ 'foo' }} and will never get evaluated. So I’m proposing something like:

  - community.general.foo:
      dict_parameter: !expression lookup("ansible.builtin.file", "/path/to/file.json")
      string_parameter: !template "{{ some_dictionary | to_json }}"

Then Ansible safely knows that the result of lookup("ansible.builtin.file", "/path/to/file.json") is whatever the file lookup returned and won’t automatically try to parse it before passing it on, and it knows that the result of "{{ some_dictionary | to_json }}" is a string that it shouldn’t try to parse.

If these tags are used consequently, I think all problems would go away, assuming I didn’t overlooked something :slight_smile:

So, my questions:

  1. Am I the first with this idea? (I would be a bit surprised if yes.) (Or does it even already exist in Ansible and I managed to not notice?)
  2. Are there some special cases I forgot, where this breaks down even if you use these tags every time you’re templating something?
  3. Bonus question: can you think of nicer tag names? For expressions I think !expr is shorter to type (maybe even !e?), and for templates maybe !templ or even !t.
2 Likes
2

Not really a direct answer to your questions, but just a heads up about data tagging. When data tagging lands, which we hope to have a PR up for maybe within the next 4 weeks (but cannot guarantee that), it will remove the ast.literal_eval that happens at the almost last step of templating. This is the biggest cause to “weirdness” we’ve always had, that was necessitated long before jinja2 native came about. Although before we had a much more complicated process of doing the same thing (safe_eval).

Additionally, jinja2 native will be the only option, but it will be improved so that the entire process of going through jinja2 is native. Right now it is not, because there are some sneaky str() uses within jinja2. We’ve located those, and are going to be overriding the functions that do it, so that we can keep an actual native type through the entire pipeline.

Because of these 2 changes, there will (hopefully) be no need to concern oneself with these differences outlined in this topic, and no unexpected conversion of a string that looks like a dict/list/set/bool being turned into a dict.

If you run the following command now, you’ll notice it is converted to a list:

ansible localhost -m debug -a 'msg={{ "[1,2,3]" }}'

localhost | SUCCESS => {
    "msg": [
        1,
        2,
        3
    ]
}

Running that on data tagging produces:

localhost | SUCCESS => {
    "msg": "[1,2,3]"
}

This also just yields the string "True" with DT but returns a boolean now:

ansible localhost -m debug -a 'msg={{ "True" }}'

So as to some slightly more direct answers to your YAML tags question, is that I don’t think they are going to be needed.

And for those who may not seen the docs PR, data tagging did get delayed until 2.18. We ultimately decided to do this, because it became a much larger change than we initially expected, and how Templar works is largely being rewritten. We’ll be asking for a lot of testing within the community, and likely will release it first as part of a fallible release that we will lovingly attend to, before we merge it into devel.

3 Likes
3

Hmm, that indeed sounds very promising. The only downside I can see right now is that it will introduce a larger number of breaking changes for many existing roles and playbooks, which relied on some of the conversions that now will no longer take place. The interesting question how large that number is. I’m looking forward to try this out!

2 Likes