# Add a with\_batching feature so that any module can batch variable input

**URL:** <https://forum.ansible.com/t/add-a-with-batching-feature-so-that-any-module-can-batch-variable-input/15522>\
**Category:** Ansible Developer\
**Created:** [June 23, 2014, 2:14am UTC](https://forum.ansible.com/t/add-a-with-batching-feature-so-that-any-module-can-batch-variable-input/15522 "2014-06-23T02:14:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gregory\_Rehm](https://avatars.discourse-cdn.com/v4/letter/g/dc4da7/32.png) [@Gregory\_Rehm](https://forum.ansible.com/u/Gregory_Rehm)\
**Post date:** [June 23, 2014, 2:14am UTC](https://forum.ansible.com/t/add-a-with-batching-feature-so-that-any-module-can-batch-variable-input/15522/1 "2014-06-23T02:14:40Z")

</div>

Ansible currently handles batching items for installing packages with the apt, yum, and pkgng modules. This works well with the aforementioned packages because people do not have much reason to use _with\_items_ for any other input other than for package names. However I recently ran into a problem with where I was trying to create user permissions for multiple different mysql databases. Naively I tried to do something like

mysql\_db:  
name=catalog{{ item }}  
with\_sequence:  
start=0  
end=255

However this ended with the mysql\_db module being invoked 255 times, which is a drag on performance.

I resorted to the alternative of using jinja syntax which allowed my module to be called once, however I think that this can be done better with ansible without having to resort to jinja. I want to propose using a _with\_batching_ feature be added to ansible. This would have the effect of modifying _with\_items_ input so that all items are appended into some kind of comma/space/whatever separated string where the module will then know how to handle that input appropriately.

With the above example it would probably look like this:

mysql\_db: name=catalog{{ item }}  
with\_sequence: start=0 end=255

with\_batching: separator=comma

I believe initially this could be accomplished with minimal code modification to ansible/runner/ **init**.py and would also serve to genericize the following code chunk in ansible/runner/ **init**.py

if len(items) and utils.is\_list\_of\_strings(items) and self.module\_name in [‘apt’, ‘yum’, ‘pkgng’]:

# hack for apt, yum, and pkgng so that with\_items maps back into a single module call

use\_these\_items =   
for x in items:  
inject[‘item’] = x  
if not self.conditional or utils.check\_conditional(self.conditional, self.basedir, inject, fail\_on\_undefined=self.error\_on\_undefined\_vars):  
use\_these\_items.append(x)  
inject[‘item’] = “,”.join(use\_these\_items)  
items = None

So instead of doing a basic check for which module we were using we would verify we were using the batching plugin. In order to maintain backwards compatibility, apt, yum, and pkgng would automatically have with\_items batching even if we did not specify it.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [June 23, 2014, 2:29pm UTC](https://forum.ansible.com/t/add-a-with-batching-feature-so-that-any-module-can-batch-variable-input/15522/2 "2014-06-23T14:29:27Z")

</div>

I’m not liking the idea of adding new syntax for this, and really it’s not going to work too well generically - the modules themselves need to support comma joined parameters in certain cases, which means it would only work for certain modules.

While some people might think it might make sense for a MySQL DB, others might wish it to work with another parameter - which quickly runs down a rabbit hole.

First off, I’d be curious if you had looked into ControlPersist/pipelining as that will cover most of the connection overhead parts.

We could consider something like “batch\_params: name”, though the default for yum/apt should still hold.

The problem is then we’d need a list of which parameters are batchable, and that would result in a lot of software plumbing for something I think probably wouldn’t be too widely used.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\
**Post date:** [June 23, 2014, 2:30pm UTC](https://forum.ansible.com/t/add-a-with-batching-feature-so-that-any-module-can-batch-variable-input/15522/3 "2014-06-23T14:30:07Z")

</div>

The other thing of course is continuing language creep.

This may be better served by just calling a script that creates your 255 databases in this case - though I do wonder _why_ it is creating 255 databases.