I’m curious if the 2.19 templating changes might affect (in a good way; fingers crossed) our chances of someday being able to use macro/endmacro in useful ways in Ansible templates. They work as expected in template files, but in templated string values, not so much. Behold:
If you squint you might notice that both foodef_alpha and bardef contain the definitions of otherwise identical macros named foo and bar. foodef_alpha also contains a templated expression that uses its macro definition successfully. That is, stuff.my_alpha ends up with the value of A.
What doesn’t work is the bardef macro definition without an invocation. It doesn’t “fail” in the Ansible task sense. It just doesn’t accomplish anything. As far as I’ve found there’s no way to define a macro in one string and invoke it in others. (Exception: included files, as per normal Jinja usage. But this is about strings defined in Ansible vars etc.)
This has been touched on a couple of times before. The TL;DR boils down to, “It’s complicated, because of the way templates are done.” Fair enough. But seeing how templating is getting a makeover, it seems reasonable to ask about it again.
The primary problem is one of scoping- while playbooks can contain Jinja templates and expressions, the playbook itself is not a Jinja template and doesn’t execute under a Jinja scope (as anyone who’s tried to affect playbook shapes with Jinja has discovered the hard way). Most of our Jinja integration lives in the variable subsystem- the numerous ephemeral Jinja scopes we use are created dynamically beneath each host/task/loop-item and only live for the duration of rendering a single template. So there’s not currently a supported mechanism to preserve Jinja locals (like macros), since that stuff all disappears with the Jinja context.
I’m kinda burying the lede here, but several of the templating engine improvements in 2.19 actually allow what you’re doing to work now. You just need a tiny tweak to your assignment to an Ansible variable and Bob’s Your Uncle:
vars:
# ...
# note that the template result is not the macro block itself or an invocation of the macro, but the generated macro callable from the Jinja local `bar`
bardef: '{% macro bar(key_) %}{{ ddd[key_] | default(key_) }}{% endmacro %}{{ bar }}'
tasks:
- debug:
var: bardef("beta") # woot, works in 2.19.0b5
- debug:
var: bardef("boguskey") # doh, broken in 2.19.0b5
The default fallback case in your sample actually exposed a small bug in the datatag access mechanism that prematurely halts expression eval when an error occurs outside a filter/test/operator invocation- I’ve fixed that locally, so with that fix applied, both cases work correctly:
This works, but it’s somewhat inefficient, since we don’t currently allow storage of callables (eg via set_fact), so it actually has to recompile the macro on every invocation. I’ve got a compiled template cache sitting on a shelf that bypasses all that, but it’s not ready for prime-time.
Thanks for the testing and feedback! I’m going to polish up the access fix and add some explicit test coverage for this use case, but seems like between the bugfix (which should be available in beta6) and the template tweak to return the macro callable, this gives you what you need, yeah?
Wow! That’s fantastic. I felt kind of bad about bringing it up again, but if it helped you work out a corner case bug, that makes it worth it. That it will solve my problem is, frankly, an unexpected bonus.
When last discussed (links up-thread), there was talk of “collection scoped macro plugins”. Like maybe that would be the clean way to expose Jinja macros in template strings. Still a neat idea - scoping just feels somehow less fragile.
Now that I see how you got this to work - without scoping - um, hm. Care to speculate on collection scoped macro plugins ever becoming a thing?
(This is all fascinating to me, lifting the corners and peeking at these implementation details. Ansible has grown up so much in the last decade!)
I wanted to address this point separately. It makes sense that this would impose some costs. However, the alternative is to duplicate the macro implementation logic - and keep those copies in sync - everywhere we would otherwise call mymacro('param'). Wouldn’t that incur the same efficiency penalties? Let the machines work for us, I say.
It makes sense that this would impose some costs. However, the alternative is to duplicate the macro implementation logic
Oh yeah, that wasn’t an indictment of its use- it still works WAY better as a macro like this than inlined all over the place, so I’m thrilled that we “accidentally” fixed it in 2.19 . I was mainly noting that there are several things we could do to further increase its efficiency under the covers. The lazy templating and intra-operation indirect template caching we’ve already added with data tagging in 2.19 should make its use in loops much cheaper than it would’ve been in older releases (had the other things blocking it from working ever been fixed).
Now that I see how you got this to work - without scoping - um, hm. Care to speculate on collection scoped macro plugins ever becoming a thing?
I’m still not opposed to the idea, but TBH (and especially with this working as a stop-gap), it’s unlikely to be a priority for at least the next couple releases. I recently figured out a couple other low-hanging-fruit templating tricks that make it much easier to do arbitrary-expression list/dict comprehensions entirely within Jinja + Ansible, but it’s pretty late to try and slam those in for 2.19, so it’ll probably be 2.20…
)
. I was mainly noting that there are several things we could do to further increase its efficiency under the covers. The lazy templating and intra-operation indirect template caching we’ve already added with data tagging in 2.19 should make its use in loops much cheaper than it would’ve been in older releases (had the other things blocking it from working ever been fixed).