I have a simple playbook that I’m testing with in an attempt to update a file that has some test strings mapped to mac addresses. When I run my playbook I can only seem to get one of my two hosts to write the test string into the playbook and not both. If I run it a second time, I can get the second host in there.
The plybook looks similar to the following:
`
Remove "connection: local". See "local – execute on controller"
https://docs.ansible.com/ansible/latest/plugins/connection/local.html#local-execute-on-controller
"... allows ansible to execute tasks on the Ansible ‘controller’ instead of
on a remote host."
HTH,
-vlado
The file '/var/tmp/test.yaml' is on the remote host. Each host updates its
own instance. I can only assume the file should be on the controller, e.g.
- name: Update test file
lineinfile:
path: /var/tmp/test.yaml
regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
line: " test-{{ test_string|lower }}: {{ mac }}"
delegate_to: localhost
HTH,
-vlado
My apologies for not being a little more clear, Vlado.
I do want this to execute on the local host and not on the remote hosts. Ultimately this will be setting up a file that will be used later for building a remote host’s config file using the template module. It’s all a part of an attempt at a somewhat zero touch provisioning model for switches we deploy.
Essentially the work flow would look similar to:
a new device needs to be added => user builds initial config file specific to device location and requirements => user runs playbook
The playbook would then locally:
load the device variables from the config file
take user’s config file and make a backup to the host_vars directory for later use, if necessary
take a short hostname (test_string) and MAC address and append them to the file
Later in the playbook other plays will have tasks that load the variables in these files generate a config from a template that gets copied to the remote host. This one file and its contents is, sort of, a living file that gets updated every time we add or replace a device.
Hope that makes sense.
If the file '/var/tmp/test.yaml' shall be created on both 'host1' and 'host2'
try this
- name: Update test file
lineinfile:
path: /var/tmp/test.yaml
regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
line: " test-{{ test_string|lower }}: {{ mac }}"
loop: "{{ ansible_play_hosts }}"
vars:
mac: "{{ hostvars[item]['mac'] }}"
test_string: "{{ hostvars[item]['test_string'] }}"
HTH,
-vlado
So I added the delegate_to: localhost but I still only get one added on the first pass:
`
PLAY [Test Playbook] ************************************************************************************************************************************
TASK [Load device vars from config] *********************************************************************************************************************
ok: [host1]
ok: [host2]
TASK [Update tmp file] **********************************************************************************************************************************
changed: [host1 → localhost]
changed: [host2 → localhost]
PLAY RECAP **********************************************************************************************************************************************
host1 : ok=2 changed=1 unreachable=0 failed=0
host2 : ok=2 changed=1 unreachable=0 failed=0
`
The file contents after that pass are:
1 macs: 2 test-host222: ab:09:87:65:43:21
And when I run it a second time:
`
PLAY [Test Playbook] ************************************************************************************************************************************
TASK [Load device vars from config] *********************************************************************************************************************
ok: [host1]
ok: [host2]
TASK [Update tmp file] **********************************************************************************************************************************
changed: [host1 → localhost]
ok: [host2 → localhost]
PLAY RECAP **********************************************************************************************************************************************
host1 : ok=2 changed=1 unreachable=0 failed=0
host2 : ok=2 changed=0 unreachable=0 failed=0
`
With the file contents now the following:
1 macs: 2 test-host222: ab:09:87:65:43:21 3 test-host111: 12:34:56:78:90:ab
Since you have two host you will have two tasks that tries to write to the same
file at the same time, and only one of them will win.
Add "throttle: 1" to you lineinfile task and it will work.
> [test_group]
> host1
> host2Since you have two host you will have two tasks that tries to write to the same
file at the same time, and only one of them will win.Add "throttle: 1" to you lineinfile task and it will work.
I don't need "throttle" with 2.9.6. The playbook
cat pb.yml
- hosts:
- test_01
- test_02
gather_facts: false
tasks:
- lineinfile:
path: test-file
line: "{{ inventory_hostname }}"
delegate_to: localhost
gives
cat test-file
ansible-playbook pb.yml
PLAY [test_01,test_02]
The two hosts I have to test with are running 2.7.4 and 2.8.2. I don’t have a 2.9.x host available to test the throttle command but it makes sense.
However, when I updated the playbook to the following:
`
The file contents after that pass are:
1 macs:
2 test-host222: ab:09:87:65:43:21And when I run it a second time:
1 macs:
2 test-host222: ab:09:87:65:43:21
3 test-host111: 12:34:56:78:90:ab
It's working for me as expected
cat test_01.conf
test_string: abc
mac: 01:23:45:67:89:ab
cat test_02.conf
test_string: xyz
mac: ab:09:87:65:43:21
cat pb.yml
- hosts:
- test_01
- test_02
tasks:
- include_vars: "{{ inventory_hostname }}.conf"
- name: Update test file
lineinfile:
path: test-file
regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
line: " test-{{ test_string|lower }}: {{ mac }}"
delegate_to: localhost
cat test-file
ansible-playbook pb.yml
PLAY [test_01,test_02]
That is just pure luck, try running it 100 times and count how many times it
fails.
Here also you have two task writing to the same file at the same time.
So here you need to add "run_once: yes" to the lineinfile task.
The file contents after that pass are:
1 macs:
2 test-host222: ab:09:87:65:43:21With the file contents now the following:
1 macs:
2 test-host222: ab:09:87:65:43:21
3 test-host111: 12:34:56:78:90:ab
Bingo! I'm able to reproduce the problem with Ansible running on controller
with Python 2.7
ansible --version
ansible 2.9.6
config file = /export/scratch/tmp/ansible.cfg
configured module search path = [u'/home/vlado/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules'] ansible python module location =
/usr/lib/python2.7/dist-packages/ansible executable location =
/usr/bin/ansible python version = 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0]
There is no such problem with Ansible running on controller with Python 3
ansible --version
ansible 2.9.6
config file = /home/vlado/.ansible.cfg
configured module search path = ['/home/vlado/.ansible/my_modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 3.8.2 (default, Apr 27 2020, 15:53:34) [GCC 9.3.0]
HTH,
-vlado
The main one I’m testing with is running python 3.6
$ ansible --version
ansible 2.7.4
config file = /etc/ansible/ansible.cfg
configured module search path = [‘/home/ofoo/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
Well, I tried and it's working properly with Python3. There is no problem with
Ansible 2.9.6 and Python 3.8 on the controller ( the redirection in the script
'2>/dev/null' is there because of annoying 3.8 RuntimeWarning).
#!/bin/bash
export ANSIBLE_STDOUT_CALLBACK=null
for i in {1..99}; do
ansible-playbook pb.yml 2>/dev/null
no_lines=`cat test-file | wc -l`
[ $no_lines -ne 2 ] && echo "ERROR File 2 lines missing"
cat /dev/null > test-file
no_lines=`cat test-file | wc -l`
[ $no_lines -ne 0 ] && echo "ERROR File not empty"
done
But, you were right with "throttle". With Ansible 2.9.6 and Python 2.7 on the
controller "throttle: 1" fixed the problem
- name: Update test file
lineinfile:
path: test-file
regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
line: " test-{{ test_string|lower }}: {{ mac }}"
delegate_to: localhost
throttle: 1
Thank you,
-vlado
> The file contents after that pass are:
>
> 1 macs:
> 2 test-host222: ab:09:87:65:43:21
>
> With the file contents now the following:
>
> 1 macs:
> 2 test-host222: ab:09:87:65:43:21
> 3 test-host111: 12:34:56:78:90:abBingo! I'm able to reproduce the problem with Ansible running on controller
with Python 2.7
<snip />
There is no such problem with Ansible running on controller with Python 3
That's just not true, your test method is flawed, you need to test it more than one time.
$ ansible-playbook --version | awk 'NR==1; END{print}'
ansible-playbook 2.9.7
python version = 3.7.5 (default, Apr 19 2020, 20:18:17) [GCC 9.2.1 20191008]
$ cat test.yml
- hosts: a1,a2
tasks:
- lineinfile:
path: /tmp/txt
line: "{{ inventory_hostname }}"
delegate_to: localhost
$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
4 763950971c8c6d8df8a87a1e752799a9 /tmp/txt
11 1597a5a9948014489de663c8fb4438db /tmp/txt
36 ac34af7b876f793d09a2225e23f43088 /tmp/txt
49 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt
So as you can see I get 4 different result as expected.
$ cat a1; md5sum a1
a1
763950971c8c6d8df8a87a1e752799a9 a1
$ cat a2; md5sum a2
a2
1597a5a9948014489de663c8fb4438db a2
$ cat a1a2; md5sum a1a2
a1
a2
317f53fd9236220d0ab65e4aac4b3c5a a1a2
$ cat a2a1; md5sum a2a1
a2
a1
ac34af7b876f793d09a2225e23f43088 a2a1
> There is no such problem with Ansible running on controller with Python 3
That's just not true, your test method is flawed, you need to test it more than one time.
$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
4 763950971c8c6d8df8a87a1e752799a9 /tmp/txt
11 1597a5a9948014489de663c8fb4438db /tmp/txt
36 ac34af7b876f793d09a2225e23f43088 /tmp/txt
49 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txtSo as you can see I get 4 different result as expected.
$ cat a1; md5sum a1
a1
763950971c8c6d8df8a87a1e752799a9 a1
$ cat a2; md5sum a2
a2
1597a5a9948014489de663c8fb4438db a2
$ cat a1a2; md5sum a1a2
a1
a2
317f53fd9236220d0ab65e4aac4b3c5a a1a2
$ cat a2a1; md5sum a2a1
a2
a1
ac34af7b876f793d09a2225e23f43088 a2a1
Well, it is true. There are no problems reported with your method either. I
get correct results only.
for i in {1..100}; do >test-file; ansible-playbook pb.yml 2>/dev/null; md5sum test-file; done | sort | uniq -c | sort -n
18 82de37d1deac8c90573c35e5637839d7 test-file
82 6f4e9c06436257db5ba34d527bd3b211 test-file
cat test_01test_02; md5sum test_01test_02
test_01
test_02
6f4e9c06436257db5ba34d527bd3b211 test_01test_02
cat test_02test_01; md5sum test_02test_01
test_02
test_01
82de37d1deac8c90573c35e5637839d7 test_02test_01
ansible --version
ansible 2.9.6
config file = /home/vlado/.ansible.cfg
configured module search path = ['/home/vlado/.ansible/my_modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
executable location = /usr/bin/ansible
python version = 3.8.2 (default, Apr 27 2020, 15:53:34) [GCC 9.3.0]
> > There is no such problem with Ansible running on controller with Python 3
>
> That's just not true, your test method is flawed, you need to test it more than one time.
>
> $ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
> 4 763950971c8c6d8df8a87a1e752799a9 /tmp/txt
> 11 1597a5a9948014489de663c8fb4438db /tmp/txt
> 36 ac34af7b876f793d09a2225e23f43088 /tmp/txt
> 49 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txtWell, it is true. There are no problems reported with your method either. I
get correct results only.> for i in {1..100}; do >test-file; ansible-playbook pb.yml 2>/dev/null; md5sum test-file; done | sort | uniq -c | sort -n
18 82de37d1deac8c90573c35e5637839d7 test-file
82 6f4e9c06436257db5ba34d527bd3b211 test-file
To be sure, I let it run overnight
for i in {1..10000}; do >test-file; ansible-playbook -i hosts77 test77.yml 2>/dev/null; md5sum test-file; done | sort | uniq -c | sort -n
2689 82de37d1deac8c90573c35e5637839d7 test-file
7311 6f4e9c06436257db5ba34d527bd3b211 test-file
That is not true and has nothing to do with Python version, and everything with
a little luck and a fast computer.
The test case is flawed for your Ansible controller, if it had more load and/or
more hosts you would get a completely different result.
The reason why this i not reliable is that Ansible is default running forks of
5, that mean that if you had 5 or more hosts it will run/fork out 5 commands.
This 5 commands will try to edit the same file at the same time.
Reason it works for you is that the first fork is finished before the second
has time to start.
This is how Ansible work, and now Python version is going to alter that.
And I'm going to prove it.
Since my machine doesn't have Python 3.8 I created a VM, with 2 cores.
$ ansible-playbook --version | awk 'NR==1; END{print}'
ansible-playbook 2.9.7
python version = 3.8.2 (default, Mar 24 2020, 03:08:36) [GCC 9.3.0]
So when I run my playbook with two host I get this
$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
1 1597a5a9948014489de663c8fb4438db /tmp/txt
39 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt
60 ac34af7b876f793d09a2225e23f43088 /tmp/txt
and the next time
$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
1 1597a5a9948014489de663c8fb4438db /tmp/txt
43 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt
56 ac34af7b876f793d09a2225e23f43088 /tmp/txt
As you can see it fails i 1 % of the cases.
Lets up the stake and go for 6 hosts, the playbook is the same, but since 6 hosts
make potensial a lot of combination, I sort the file before I run md5sum.
So if everything is working OK I should only get one line with one hash like this
100 aae4eb078da77ff549495a60be1a52c2
$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; sort /tmp/txt | md5sum; done | sort | uniq -c | sort -n
1 2465fc0f03254f2bf915a04c762c9d49 -
2 254316531e34aa3ce563c22cf636ea01 -
2 31f07322fb0283ee5475e5acc74de059 -
2 ab8ecb5209d15e9852be208b79f309eb -
3 7abbd5c371c5e4ccb8bc965105148605 -
90 aae4eb078da77ff549495a60be1a52c2 -
Not even close, it fails in 10% of the runs.
So lets make the machine even more busy.
To do that I shutdown the VM and reduced the cores count from 2 to 1.
Now we have 5 forks at the same time "fighting" for this 1 core and trying to
write to the same file at the same time.
This is the result.
1 254316531e34aa3ce563c22cf636ea01 -
1 6c5ffba74557139441576f9f6536c536 -
1 8da4fee8643e840ae60169eac2d60afd -
1 9a7861beaa38064dd7179cf66b90173f -
1 aae4eb078da77ff549495a60be1a52c2 -
1 d5fad8b2b8f31c4d201309e2261ee362 -
2 2465fc0f03254f2bf915a04c762c9d49 -
2 2ea77912754b2a786046ece007bfd0a7 -
2 6503321b7879267a6a5eb06411e09b5e -
2 9ce72c8f1a3f059db908868df22c2618 -
2 ab8ecb5209d15e9852be208b79f309eb -
2 adc72709b7b7cdc8aaad85c01b66e38d -
2 c4b05365f19a397fc9b2dda817219d49 -
2 cf2b918385c6d1f75d7a0630d8881422 -
2 d678deb816d9433ff1c61d60ec1073f1 -
3 83758fc71d9927b3141c325692534add -
3 989fe3db59b18149e58657daab4721dc -
3 a421781f0d1512c9c66e9f7714bed570 -
4 7bc2be35864ae017608466e62dd680aa -
5 83ba567764a618ec0a0b405d49de8fa9 -
5 c98ae97d4888d870d9a05fe7f9b339a3 -
5 d336136909e4067c35858990dd4f9211 -
6 31f07322fb0283ee5475e5acc74de059 -
6 76aabdf63a1f041f6572fb51fb496283 -
6 cce45b22f8bc7806e3dc1128980f33f6 -
6 cce83e09262e3efced931d81c85ac809 -
6 decd8e81f11339ba387cb7cf61a28f34 -
7 aed3d7850c9c5478d84bf78fdc922328 -
11 d8ee7367ba9763647a58ab23a6f01c5f -
Yeah, it fails miserably, in 99% of the cases.
The correct md5sum is this one
$ cat sorted.txt
a1
a2
a3
a4
a5
a6
$ md5sum sorted.txt
aae4eb078da77ff549495a60be1a52c2 sorted.txt
So no combination of Ptyhon and Ansible released will alter this,
it's just how it work.
That is why we have serial, fork and throttle to tune the behavior to avoid race
conditions.
I have explained in my previous mail why this is by change/luck, and it will
fail if you controller had some more load that just 2 hosts.