In a loop everything executes at the same time?

Archives Ansible Project
1

I want to limit the number of parllel executions for a certain task inside a playbook (I have some constrains with downloading artifacts)

In that way the rest of tasks wil execute in all hosts at the same time but when the download task executes it will be limited to batches of N.

For this I’m creating a sequence with the # of iterations needed. If I do this it seems to work quite well

  • debug:
    msg: “Sequence {{ item }} with modulus {{ (( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) }}”
    with_sequence: start=0 end={{ (( play_hosts | length ) / 2 ) | round (0, ‘floor’) | int }}
    loop_control:
    pause: 5
    when: “(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == (item | int)”

Output

TASK [debug] ********************************************************************************************************************************************************************************************************skipping: [PCT2EMFR0HP.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HQ.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HR.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
ok: [PCT2EMFR0HO.pct2m.local] => (item=0) => {
“changed”: false,
“item”: “0”,
“msg”: “Sequence 0 with modulus 0.0”
}
skipping: [PCT2W00RDSH001.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
ok: [PCT2EMFR0HP.pct2m.local] => (item=1) => {
“changed”: false,
“item”: “1”,
“msg”: “Sequence 1 with modulus 1.0”
}
skipping: [PCT2EMFR0HQ.pct2m.local] => (item=1) => {“changed”: false, “item”: “1”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HO.pct2m.local] => (item=1) => {“changed”: false, “item”: “1”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2W00RDSH001.pct2m.local] => (item=1) => {“changed”: false, “item”: “1”, “skip_reason”: “Conditional result was False”}
ok: [PCT2EMFR0HR.pct2m.local] => (item=1) => {
“changed”: false,
“item”: “1”,
“msg”: “Sequence 1 with modulus 1.0”
}
skipping: [PCT2EMFR0HP.pct2m.local] => (item=2) => {“changed”: false, “item”: “2”, “skip_reason”: “Conditional result was False”}
ok: [PCT2EMFR0HQ.pct2m.local] => (item=2) => {
“changed”: false,
“item”: “2”,
“msg”: “Sequence 2 with modulus 2.0”
}
skipping: [PCT2EMFR0HO.pct2m.local] => (item=2) => {“changed”: false, “item”: “2”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HR.pct2m.local] => (item=2) => {“changed”: false, “item”: “2”, “skip_reason”: “Conditional result was False”}
ok: [PCT2W00RDSH001.pct2m.local] => (item=2) => {
“changed”: false,
“item”: “2”,
“msg”: “Sequence 2 with modulus 2.0”
}

But if I execute something in the remote server (in this case just a sleep or a download) it seems that at the end everything is executed in parallel. See output generated times for each task, there should be some timespan seconds between loop iterations

win_shell: (0…5) | foreach { (date).ToString(“hh:mm:ss:ffff”);Start-Sleep -Seconds 1 }

with_sequence: start=0 end={{ (( play_hosts | length ) / 2 ) | round (0, ‘floor’) | int }}
when: “(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == (item | int)”

Output

TASK [Download at ratio three at most] ******************************************************************************************************************************************************************************skipping: [PCT2EMFR0HP.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HQ.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HQ.pct2m.local] => (item=1) => {“changed”: false, “item”: “1”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HR.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2W00RDSH001.pct2m.local] => (item=0) => {“changed”: false, “item”: “0”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2W00RDSH001.pct2m.local] => (item=1) => {“changed”: false, “item”: “1”, “skip_reason”: “Conditional result was False”}
changed: [PCT2EMFR0HP.pct2m.local] => (item=1) => {“changed”: true, “cmd”: “(0…5) | foreach { (date).ToString("hh:mm:ss:ffff");Start-Sleep -Seconds 1 }”, “delta”: “0:00:06.531287”, “end”: “2018-02-08 10:26:58.403828”, “item”: “1”, “rc”: 0, “start”: “2018-02-08 10:26:51.872540”, “stderr”: “”, “stderr_lines”: , “stdout”: “11:26:52:3098\r\n11:26:53:3099\r\n11:26:54:3255\r\n11:26:55:3255\r\n11:26:56:3411\r\n11:26:57:3411\r\n”, “stdout_lines”: [“11:26:52:3098”, “11:26:53:3099”, “11:26:54:3255”, “11:26:55:3255”, “11:26:56:3411”, “11:26:57:3411”]}
changed: [PCT2W00RDSH001.pct2m.local] => (item=2) => {“changed”: true, “cmd”: “(0…5) | foreach { (date).ToString("hh:mm:ss:ffff");Start-Sleep -Seconds 1 }”, “delta”: “0:00:06.484390”, “end”: “2018-02-08 10:26:58.421047”, “item”: “2”, “rc”: 0, “start”: “2018-02-08 10:26:51.936657”, “stderr”: “”, “stderr_lines”: , “stdout”: “11:26:52:3116\r\n11:26:53:3429\r\n11:26:54:3585\r\n11:26:55:3585\r\n11:26:56:3741\r\n11:26:57:3898\r\n”, “stdout_lines”: [“11:26:52:3116”, “11:26:53:3429”, “11:26:54:3585”, “11:26:55:3585”, “11:26:56:3741”, “11:26:57:3898”]}
skipping: [PCT2EMFR0HP.pct2m.local] => (item=2) => {“changed”: false, “item”: “2”, “skip_reason”: “Conditional result was False”}
changed: [PCT2EMFR0HQ.pct2m.local] => (item=2) => {“changed”: true, “cmd”: “(0…5) | foreach { (date).ToString("hh:mm:ss:ffff");Start-Sleep -Seconds 1 }”, “delta”: “0:00:06.505370”, “end”: “2018-02-08 10:26:58.493954”, “item”: “2”, “rc”: 0, “start”: “2018-02-08 10:26:51.988583”, “stderr”: “”, “stderr_lines”: , “stdout”: “11:26:52:4271\r\n11:26:53:4332\r\n11:26:54:4349\r\n11:26:55:4355\r\n11:26:56:4366\r\n11:26:57:4378\r\n”, “stdout_lines”: [“11:26:52:4271”, “11:26:53:4332”, “11:26:54:4349”, “11:26:55:4355”, “11:26:56:4366”, “11:26:57:4378”]}
changed: [PCT2EMFR0HO.pct2m.local] => (item=0) => {“changed”: true, “cmd”: “(0…5) | foreach { (date).ToString("hh:mm:ss:ffff");Start-Sleep -Seconds 1 }”, “delta”: “0:00:06.567788”, “end”: “2018-02-08 10:26:58.581462”, “item”: “0”, “rc”: 0, “start”: “2018-02-08 10:26:52.013673”, “stderr”: “”, “stderr_lines”: , “stdout”: “02:26:52:4824\r\n02:26:53:4933\r\n02:26:54:4943\r\n02:26:55:4971\r\n02:26:56:5069\r\n02:26:57:5225\r\n”, “stdout_lines”: [“02:26:52:4824”, “02:26:53:4933”, “02:26:54:4943”, “02:26:55:4971”, “02:26:56:5069”, “02:26:57:5225”]}
skipping: [PCT2EMFR0HO.pct2m.local] => (item=1) => {“changed”: false, “item”: “1”, “skip_reason”: “Conditional result was False”}
skipping: [PCT2EMFR0HO.pct2m.local] => (item=2) => {“changed”: false, “item”: “2”, “skip_reason”: “Conditional result was False”}
changed: [PCT2EMFR0HR.pct2m.local] => (item=1) => {“changed”: true, “cmd”: “(0…5) | foreach { (date).ToString("hh:mm:ss:ffff");Start-Sleep -Seconds 1 }”, “delta”: “0:00:06.546889”, “end”: “2018-02-08 10:26:58.606632”, “item”: “1”, “rc”: 0, “start”: “2018-02-08 10:26:52.059742”, “stderr”: “”, “stderr_lines”: , “stdout”: “11:26:52:5128\r\n11:26:53:5129\r\n11:26:54:5285\r\n11:26:55:5441\r\n11:26:56:5441\r\n11:26:57:5442\r\n”, “stdout_lines”: [“11:26:52:5128”, “11:26:53:5129”, “11:26:54:5285”, “11:26:55:5441”, “11:26:56:5441”, “11:26:57:5442”]}
skipping: [PCT2EMFR0HR.pct2m.local] => (item=2) => {“changed”: false, “item”: “2”, “skip_reason”: “Conditional result was False”}

Any idea why this is happening?

2

No, in a loop everything executes sequentially, but EACH host executes
their own version of the task AND the loop in parallel.

debug will give you misleading results as it is a 'local' process and
probably finishes before other forks can start, once you intorduce the
delay of remote tranfer+execution you start seeing the real and
correct behaviour.

You might want to put the task in a play with 'serial' to handle the
'global limit' correctly.

3

Thanks Brian for helping me understand how with item task work under the hoods.

I was expecting “with_items” to work as different tasks, as per documentation “To save some typing, repeated tasks can be written in short-hand like so:”

thus I was expecting it to be “transformed” to something like this

  • name: Long task iteration 1
    win_shell: (0…5) | foreach { (date).ToString(“hh:mm:ss:ffff”);Start-Sleep -Seconds 1 }
    when: “(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == 0”
  • name: Long task iteration 2
    win_shell: (0…5) | foreach { (date).ToString(“hh:mm:ss:ffff”);Start-Sleep -Seconds 1 }
    when: “(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == 1”
  • name: Long task iteration 2
    win_shell: (0…5) | foreach { (date).ToString(“hh:mm:ss:ffff”);Start-Sleep -Seconds 1 }
    when: “(( ansible_play_batch.index(inventory_hostname) % (( play_hosts | length ) / 2 )) | round) == 2”

which has my desired behaviour,

From what you told me I understand that loops are executed as a single task for each host

4

kind of, the result is similar to what you expected but not the same,
a loop gets 'unrolled' into N tasks (1 per item) per host, but they
are grouped as a single task per host for the results.

The loop itself is sequential (within the same host) but in parallel
with the same task/loop with other hosts.

5

Very interesting, so then each host individually needs 0 sec for the iterations it doesn’t execute which ends up with all the hosts executing the tasks at the same time.

I was inspired by this comment https://github.com/ansible/ansible/issues/12170#issuecomment-283950548 but now I understand it is not possible then.

Thanks

https://github.com/ansible/ansible/issues/12170#issuecomment-283950548