Clarifying when to develop a module vs an action plugin

[samccann (Sandra McCann)]

Hi folks -

This came up in a recent docs issue - How does a developer decide when to create a module vs an action plugin. What are the pros and cons of each type, and so on.

We talk about the different plugins here, but don’t mention modules are a type of plugin, nor why a developer would want to create one vs the other. So we’re looking for ideas on how to improve this documentation.

[bcoca (Brian Coca)]

The short answer:

• Use modules to execute the action on the target machine. Example ‘yum’ to install packages.
• Choose an action plugin when all or part of the work should take place on the controller, the action plugin can then also call a module to finish off the work on the remote target. Example ‘template’, the templating happens on controller and the file copy is to the target, this last step is pretty complex as it is calling the copy action plugin which can call the stat, copy and file modules.

[mutech (Michael Utech)]

Hey Guys!

Thanks again for taking the time to react to my ramblings.

I’m currently working on other things, so I’m looking at this topic from a slightly different perspective. I meanwhile gave up on the concrete plugins I wanted to write because the impact of using dependencies was imho not worth the price and the same goes for the effort to write all functionality from scratch.

One thing that I dearly missed on the way from a complete rookie a somewhat fluent playwright was - as mentioned on github - the lack of guidance for all things outside of playbooks. This particular topic, module vs. action plugin was my most recent issue, but I think it’s just one incarnation of a somewhat larger problem I had:

Ansible is extremely awesome while you are writing playbooks and find suitable modules. I rarely had more fun getting things done than in this context. However, when you have to leave this scope, things get really bad. One thing that pops up is that automation very often is doing the same thing slightly differently over and over again. So reusing solutions is necessary. Ansible has no good tool to allow for reuse, that is as easy to use as playbooks. Roles are the closest you get to that, but that’s not what they are designed for. The lack of a parameter interface for roles makes them unusable for that purpose. That leaves other Ansible constructs to choose from. There are two candidates (for different tasks): Jinja macros as a quick and dirty means to create complex queries and modules. There are three reasons that kill Jinja as viable option: Diagnostics (no way to locate the source of errors), paths (no way to import “…/macros.j2” or “/macros.j2” at least I didn’t find a way), quirky syntax for complex queries. Not that I want to program in Jinja anyway.

Modules: The price for making a playbook functionality reusable by using a role is initially 80 result units per time compared to 100 for playbook hacking. The 20 lost units are spend solving issues with parameters. Over time, the 80-20 becomes 20-80. The price for making such a functionality a plugin is initially 1-99 and over time maybe 40-60. These are emotional estimates, not real numbers.

The problems in my view are:

• The plugin interface grew over time and has not been consistently designed. Hacks became quasi-standards and they can no longer be changed. Things can only get worse from here on out and they are already bad.
• The amount of scaffolding a clean implementation of a module needs is utterly disproportional compared f.e. to what you would need to do when creating a Jinja macro or a role abused as module. So much so that it’s very attractive to do it anyway, no matter how bad that is and no matter that you know it will bite you back.
• The dependency issue combined with the practice (here of 3rd party module authors) to use modules when you should use actions and many other circumstances that limit choices only leaves a very narrow path for some things that can be implemented easily, while most other things are so complex that I could implement them faster in plain ANSI C from scratch.
• The fact that you can’t debug modules live is a severe limitation. It’s not even possible (to my knowledge) to interactively step through actions at the interface between controller and target.
• I have many many other similar issues, but don’t have the time right now to complete the list, so I’ll leave it at this

What I’m doing right now is utterly ridiculous. To avoid all these issues, I’m using Emacs-Org-Mode with its literate programming features to generate Ansible Playbooks, Shell- and Python scripts. I’m using Ansible Playbooks as automation assembly language. I know that this is idiotic, but it is about 2-5 times more time efficient than anything else I did in the context of Ansible. The only reason why I even try to do that is because I get excellent documentation from this (this was how I got the idea) and because a lot of functionality is already implemented in Ansible and I have no better way to get things done in time.

With this new experience, I start to believe that my original issue is more than just a documentation issue. I think the real problem is that Ansible is pure awesomeness as long as you write playbooks and at the same time a nightmare when you develop modules. And you have to eventually develop modules.

Please don’t misunderstand this as an attempt to complain or criticize your work. It’s frustration because I would love to work with Ansible, but it’s really getting too painful.

It’s fun to fix most issues with a 35 year old Elisp-Editor. But I don’t think that this is a viable approach. Nevertheless it works surprisingly well.

[bcoca (Brian Coca)]

@mutech

• The plugin interface grew over time and has not been consistently designed. Hacks became quasi-standards and they can no longer be changed. Things can only get worse from here on out and they are already bad.

Very true, we have been looking at ways to improve this, redesign plugins to be more ‘isolated’ and have a clear and defined API and data access. Over time we have limited the data passed to the different plugins, but they can still ‘poke’ into the internals to gain more data than they should have.

@mutech

• The amount of scaffolding a clean implementation of a module needs is utterly disproportional compared f.e. to what you would need to do when creating a Jinja macro or a role abused as module. So much so that it’s very attractive to do it anyway, no matter how bad that is and no matter that you know it will bite you back.

I’m not sure what you mean here, I’m going to assume you mean the AnsibleModule class and other code in moduel_utils. There is a lot of code available for python/powershell modules, but this is not required and there are some modules (not many) that avoid this, specially those in other languages. The basic requirement is that the module reads JSON input and outputs JSON back, with only a few fields ‘required’.

@mutech

• The dependency issue combined with the practice (here of 3rd party module authors) to use modules when you should use actions and many other circumstances that limit choices only leaves a very narrow path for some things that can be implemented easily, while most other things are so complex that I could implement them faster in plain ANSI C from scratch.

This is mostly a ‘marketing’ issue, modules were pushed as the easy to customize part of Ansible and action plugins here
“hidden” from view. While many modules out there, specially cloud/API related ones, would work better as action plugins most of the time, they also are more flexible with access rules, like when the controller does not have access to the API but the remote target/delegated_to does.

@mutech

• The fact that you can’t debug modules live is a severe limitation. It’s not even possible (to my knowledge) to interactively step through actions at the interface between controller and target.

Modules were designed as ‘simple scripts’ so they would not be hard to debug. Sadly the way we package and execute them has grown a lot in complexity and makes this step a bit harder. But you can find instructions to deal with our packaging and execute the modules directly with debugging, there are even instructions on how to execute remotely under a debugger and even from an IDE. One problem is that many debuggers do not handle forks or threads well, but there are those that do, you just need to find the combination that works for you.

As for reuse, yes, roles are a flawed vehicle for this, but if you craft them well, not just roles, but plays are reusable. As I understand it I think most encounter this problem because Ansible doesn’t force you into reusability and most guides I’ve seen are either just partial or very tied to specific contexts. I have not encountered a good and comprehensive guide that shows authors how to succeed at this, it does not mean it is not possible, but that we are missing a good resource to point people at. Also given the nature of some problems the increase of reusability scope will end up complicating the play/role, for example a ‘website installer’ sounds simple, until you try to handle firewalls, different webservers and OS/distributions, each one adds a scaling difficulty and complexity. So I would also limit reusability to specific targets and narrow my aim to achieve some success (also something that should be in the guides).