Large task Issue

1 playbook, 20 tasks, 10 targets/hosts. On 9 host tasks runs similar time - a couple of minutes. On 1 server 15’th tasks takes 1,5h. After that time last 5 task runs quickly and end result report is produced for whole 10 hosts.

I have two problems/questions.

Is there any simple way to debug/check which exactly task and on which host is a blocker? Now I have to analyze log output and check what task is “missing” from log - that task is running currently obviously.

I use “strategy: free” so my 9 hosts are done quite quickly(10 minutes) but I have report about it after my largest host/task - 2h after. I can’t exclude that host and run it in separate job because I am not able to tell which one will be the blocker that day (I know it sounds weird - but it is). So is it possible to get some final report about a single host? It would be ofcourse between regular tasks but it is enough. Easy to grep and “tag”(in my excel) host as done (even without final successfull report).

Maybe I'm stating the obvious but if one task is taking 1.5h then that
would be very visible while running the playbook.
So that task would be the culprit.
As to why it happens on different hosts, that's impossible to tell
without more information, such as what that task looks like.

Dick

1 playbook, 20 tasks, 10 targets/hosts. On 9 host tasks runs similar time -
a couple of minutes. On 1 server 15'th tasks takes 1,5h. After that time
last 5 task runs quickly and end result report is produced for whole 10
hosts.

Is there any simple way to debug/check which exactly task and on which host
is a blocker?

ansible-runner provides profiling
https://ansible-runner.readthedocs.io/en/latest/intro.html#runner-profiling-data-directory

So is it possible to get some final report about a single host?

ansible-runner creates "artifacts" directory with the results
https://ansible-runner.readthedocs.io/en/latest/intro.html#runner-artifacts-directory-hierarchy

FWIW. See more details in "Ansible Logging Extreme Detail How-To?"
https://stackoverflow.com/questions/59184097/ansible-logging-extreme-detail-how-to/

Cheers,

  -vlado

Nope, It is not very visible. 10 hosts 20 tasks. 10x20 = 200 lines/tasks in output not in line. Please take a note that I am using “strategy free” → So tasks are not realized one by one for all hosts in one step. Even without strategy free I woludn’t know which exactly host is a blocker. I could see only previous - already done task. So the next one is running right now (I have to take a look in a code/playbook to check which one is next after already done task - info from output) - but on which host it takes the longest time - which one is still running when the rest is already done? I compare it (ansible/awx) to jenkins buid flow where in a split of second I can find really active task on a specyfic node. I would like to find out if it was possible on awx?

Nope, It is not very visible. 10 hosts 20 tasks. 10x20 = 200 lines/tasks in output not in line. Please take a note that I am using “strategy free” → So tasks are not realized one by one for all hosts in one step. Even without strategy free I woludn’t know which exactly host is a blocker. I could see only previous - already done task. So the next one is running right now (I have to take a look in a code/playbook to check which one is next after already done task - info from output) - but on which host it takes the longest time - which one is still running when the rest is already done? I compare it (ansible/awx) to jenkins buid flow where in a split of second I can find really active task on a specyfic node. I would like to find out if it was possible on awx?

Thank You. Do You have any other advice in case of AWX usage (ansible-runner is command line tool) ?