Is it possible to define Jinja2 macros such that they are available across all Jinja expressions in a playbook? I’m thinking of how you can create various types of plugins that magically extend what’s available in terms of lookups, filters, tests, etc., but a lot of repetitive grunt work of some complex expressions could be nicely encapsulated in Jinja macros. It would be nice not to have to say {% import 'macros.j2' as macros with context %} all over the place, just like you don’t have to import custom filters and tests. I suppose scoping them to a collection, role, or playbook would make sense.
I’m thinking that just like my ansible_managed (and its associated variables) in ansible/template/__init__.py it ought to be possible, but it’s been a while since I’ve been close to that code. Skimming over the API — Jinja Documentation (3.0.x) nothing jumps out at me, but I’m sure there are people here who’d know how to do it.
An ansible.cfg variable pointing to a file with macros comes to mind…
Turns out, this was discussed - sort of - back in February, 2018, in “Add ability to define filter macros” The suggestion was to add the {% import … %} to your jinja expressions, which is what I’m trying to avoid!
Given that the internals of Ansible, and possibly Jinja itself, have evolved quite a bit in the intervening 5 years, it sure would be neat if plugins/macros became A Thing. I have a gut feeling that macros are underutilized, taking a back seat to custom python plugins, mostly because they don’t have the same level of supporting implementation. Which is ironic since they are Jinja2-native right out of the box.
I proposed adding jinja2 macros as a plugin type a long time ago, but was not accepted, but that was a long time ago. Open a Feature Request on github if you want to get core to look at it again.
The following are just some off-the-cuff thoughts, after spending much of the last several months replumbing most of Ansible’s templating guts and interface with Jinja (the reasons for which we’ll be talking about soon )…
Making this kind of feature work is not hard- it’s a few lines of code in the simplest cases. That said, making it work in a way that doesn’t actively degrade Ansible’s content portability, performance, and security is likely a lot harder.
TL;DR: Jinja macros could probably work as a new collection-only plugin type, but I don’t know if the overhead of hosting them in the collection dir structure would be off-putting to the users that want them.
Longer discussion follows:
Many would be amazed appalled at the amount of wall-clock time Ansible burns on startup scanning directories and pre-emptively loading plugins “just in case”; an unrestricted macro plugin that just blindly loads everything in directories would definitely be another set of logs on that fire. It would also create new distribution problems- what happens when you want to share a playbook/role/collection that uses custom macros? You have to distribute the macros to the target env, and potentially the config to locate/load them. What happens when macro names accidentally (or worse, maliciously) collide?
Supporting macros only as collection plugins solves most of these problems:
no config settings or pre-emptive arbitrary directory scans required (loaded on demand)
name collision issues are solved
distribution and documentation are solved
There are also readability and maintainability issues to consider; speaking from much experience fixing up our own integration tests over the past several months, debugging “clever” extended templating like this in playbooks when something isn’t working as you expect can be maddeningly difficult even with an interactive Python debugger attached to Ansible- doing it without one might be basically impossible in gnarlier cases. I wouldn’t necessarily let that block the feature, but it should definitely give folks pause before they start meta-templating the crap out of their playbooks .
Anyway, probably worth some discussion, but just want to make sure everyone’s considering the blast radius to Ansible’s entire ecosystem…
Thanks, @nitzmahone . I could seriously get behind making this a collection-only feature. In fact I’ve been putting in quite some time lately moving our miscellaneous plugins into a collection. It just feels more sane, for reasons I’ve not been able to articulate. Unless I’m seriously misreading you, there’s supporting technical merit for such moves. I’ve a mind to move pretty much any project components that can be moved into a project-specific local collection. But I digress.
WRT macros specifically: Since my original post 20 days ago, I’ve hit an issue where this would be a help for us. We have an application specific config role/foo/file/blah.conf that we’ve deployed with ansible.builtin.copy for years. Lots of similar tedious fiddly bits which would be easier to maintain as YML than as the .conf file the app requires, particularly as a mildly clever template can generate lots of the required obfuscating repetitive bits. This was implemented a week ago, and in the process revealed several inconsistencies and oversights that we would have never spotted in the .conf form.
Now we have a wildly different need for the same upstream .yml to feed another use case. Doable, but requiring another template which must implement the same cleverness as the first one but with a totally different output. While that’s given us a much easier to maintain .yml source-of-truth, we now have two dangerously clever templates, the cleverness of which must be maintained in parallel.
If macro plugins were available – and restricted to collections would be fine – then we could embed Teh Clever right in the upstream .yml definition where macros could consistently handle the parts humans have proven bad at. Given this fully fleshed out and consistent upstream data, the downstream templates could focus on straightforward output. Regardless of the internal overhead (which you rightly point to), this would make the users’ experience much cleaner than what we’re about to do: choose between re-implementing and maintaining data expansion logic in multiple templates vs. throwing the Don’t Repeat Yourself principle out the window and filling our source-of-truth .yml with the same blindingly almost-but-not-quite boilerplate text that made the original .conf file so hard to maintain.
This is only a use case to highlight a problem that could be addressed by a macro plugin feature. It’s not a request for assistance on this particular problem. Sorry if it got kinda long.
I’d be delighted to see that discussed over in Project Discussions, for the record. Discoverable reasoning, linkable explanations (thank you@nitzmahone!) and ecosystem-wide involvement is exactly why I want this forum to take off
Meanwhile I’m marking Nitz’s answer as the solution, because the question was “is it possible” and the answer is “yes it’s possible”
)…
