Can we run roles in parallel for multiple hosts?

Archives Ansible Project
1

New ansible user here.

After deploying VMs, I want to run a folder called configs on the Jump server to components like Syslog, NTP etc. for post configurations.

This is the structure of how files are in the config folder

configs/├─ hosts
├─ group_vars
├─ playbook.yml
├─ roles/
├─ Ntp/
├─ tasks/
├─ vars/
├─ Syslog/
├─ vars/
├─ tasks/

This is how it is my main playbook - playbook.yml

  • name: Setup config for ntp
    hosts: comp1
    roles: - comp1

  • name: Setup config for syslog
    hosts: comp2
    roles: - comp2

Right now I run this command

ansible-playbook -i hosts playbook.yml --limit “ntp,syslog”

this runs the tasks sequentially. How can I execute this in parallel considering there is no dependency between the components?

2

Morning Shweta,

This is documented here https://docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html

Regards
David

3

Hello David!

Thank you for the link. Would this strategy and fork option work for roles as well?
In the user guide, there was no mention of roles, that’s why I asked.

4

I put the setting in my ansible.cfg. I tend to use small roles and import each role as needed in a playbook so I’m not sure how that would work with just roles. If you are trying to use RH Satellite then that has a different ansible.cfg than the standard (in /etc/foreman-proxy/ansible.cfg)

You can also put it in the playbook

hosts: all
strategy: free

I put the forks in the command-line as --forks 50 or whatever. There is a small calculation that you can find on line (basically ncpus * 4 = number of forks but also if you have a chunk of memory then you can increase it)

Regards
David

5

I reread your original post, I think you will find each task/role within the playbook will be run sequentially. The strategy links that I gave you will help with getting them running on multiple hosts in roughly parallel. I’m unsure if you can parallelize the tasks.

6

wondering about the ‘why’ of your question I I guess it is about speeding up execution?

There is an extension mitogen4ansible which claims to speed up playbook execution significantly. If I understand that correctly it is related to parallelizing things in some way on a python level. Here is some youtube food for thought on this

7

Using mitogen is a no-brainer to sped up ansible runs. I was able to
reduce my ansible running times to like a fifth of the time running with
stock ansible with ZERO changes to my roles, tasks, playbooks and no
additional software or configuration needed on the targets. It all
happens on the controller and is enabled in a minute.

Highly recommended!

Greetings
Marc

8

I can confirm the speed-up achieved by mitogen but there are two issues:

1. Some tasks won't work properly, e.g. if you have to set ansible_python_interpreter for a task.

2. IIRC it's not maintained anymore and it does not work with ansible 2.10+.

Ciao, Michael.

9

2. IIRC it's not maintained anymore and it does not work with ansible 2.10+.

This might be weirdness with Debian version numbers, but my ansible
installed from debian is Version 5.5.0, and it works great with mitogen:

[4/5063]mh@drop:~ $ dpkg --list '*ansible*' | grep -v '^un'
Desired=Unknown/Install/Remove/Purge/Hold
> Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
>/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
>>/ Name Version Architecture Description
+++-===============-============-============-===============================================================
ii ansible 5.5.0-1 all Configuration management, deployment, and task execution system
ii ansible-core 2.12.4-1 all Configuration management, deployment, and task execution system
ii ansible-mitogen 0.3.2-1 all Fast connection strategy for Ansible
[5/5064]mh@drop:~ $

Greetings
Marc

10

Glad to see that there are mitogen updates. Thanks for the heads up.

Currently it seems to not work with ansible-core 2.13.x though. But hopefully there will be an update soon.

Ciao, Michael.

11

Hello, I tried mitogen for the execution of some other ansible script and saw a significant difference in the time taken to complete the play. So thank you for this tool!

But is there no way to execute these roles in parallel?

As I described before as well -
My main playbook.yml file calls role-wise to initiate setup and then inside roles I have tasks for different components. All hosts are different and execution is independent of each other.

12

Yes and No, you can run playbooks in parallel, you cannot run roles in
parallel in the same play nor playbook.

While all tasks run in parallel and you can do things like async or
free strategy this wont parallelize the roles themselves, they get
executed in the order in which they are address in the same play. So
if you want to run roles in parallel i suggest executing parallel
instances of ansible-playbook, parallel, xargs and other utilities
make this simple, or via a scheduler cron/incron/awx|tower|aap,
jenkins, etc). Theoretically you can write a strategy plugin that does
this, but this is far from trivial.

A note about mitogen, since it was mentioned, it does not change how
parallelization works, but it does accelerate execution in several
ways, one is by turning pipelining always on, others have to do with
optimizing execution for a threaded model and a 'semi persistent'
agent on the target among other things. Before you ask, this cannot be
integrated into the main ansible w/o loosing functionality and the
threading model is not always an improvement over the forks, depends a
lot on the size of the load and how it is structured, it gets very
complicated at on point.