Add a with_batching feature so that any module can batch variable input

Archives Ansible Developer
1

Ansible currently handles batching items for installing packages with the apt, yum, and pkgng modules. This works well with the aforementioned packages because people do not have much reason to use with_items for any other input other than for package names. However I recently ran into a problem with where I was trying to create user permissions for multiple different mysql databases. Naively I tried to do something like

mysql_db:
name=catalog{{ item }}
with_sequence:
start=0
end=255

However this ended with the mysql_db module being invoked 255 times, which is a drag on performance.

I resorted to the alternative of using jinja syntax which allowed my module to be called once, however I think that this can be done better with ansible without having to resort to jinja. I want to propose using a with_batching feature be added to ansible. This would have the effect of modifying with_items input so that all items are appended into some kind of comma/space/whatever separated string where the module will then know how to handle that input appropriately.

With the above example it would probably look like this:

mysql_db: name=catalog{{ item }}
with_sequence: start=0 end=255

with_batching: separator=comma

I believe initially this could be accomplished with minimal code modification to ansible/runner/init.py and would also serve to genericize the following code chunk in ansible/runner/init.py

if len(items) and utils.is_list_of_strings(items) and self.module_name in [ ‘apt’, ‘yum’, ‘pkgng’ ]:

hack for apt, yum, and pkgng so that with_items maps back into a single module call

use_these_items =
for x in items:
inject[‘item’] = x
if not self.conditional or utils.check_conditional(self.conditional, self.basedir, inject, fail_on_undefined=self.error_on_undefined_vars):
use_these_items.append(x)
inject[‘item’] = “,”.join(use_these_items)
items = None

So instead of doing a basic check for which module we were using we would verify we were using the batching plugin. In order to maintain backwards compatibility, apt, yum, and pkgng would automatically have with_items batching even if we did not specify it.

2

I’m not liking the idea of adding new syntax for this, and really it’s not going to work too well generically - the modules themselves need to support comma joined parameters in certain cases, which means it would only work for certain modules.

While some people might think it might make sense for a MySQL DB, others might wish it to work with another parameter - which quickly runs down a rabbit hole.

First off, I’d be curious if you had looked into ControlPersist/pipelining as that will cover most of the connection overhead parts.

We could consider something like “batch_params: name”, though the default for yum/apt should still hold.

The problem is then we’d need a list of which parameters are batchable, and that would result in a lot of software plumbing for something I think probably wouldn’t be too widely used.

3

The other thing of course is continuing language creep.

This may be better served by just calling a script that creates your 255 databases in this case - though I do wonder why it is creating 255 databases.