Ansible lininfile behavior

I have a simple playbook that I’m testing with in an attempt to update a file that has some test strings mapped to mac addresses. When I run my playbook I can only seem to get one of my two hosts to write the test string into the playbook and not both. If I run it a second time, I can get the second host in there.

The plybook looks similar to the following:

`

Remove "connection: local". See "local – execute on controller"
https://docs.ansible.com/ansible/latest/plugins/connection/local.html#local-execute-on-controller

  "... allows ansible to execute tasks on the Ansible ‘controller’ instead of
  on a remote host."

HTH,

  -vlado

The file '/var/tmp/test.yaml' is on the remote host. Each host updates its
own instance. I can only assume the file should be on the controller, e.g.

     - name: Update test file
       lineinfile:
         path: /var/tmp/test.yaml
         regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
         line: " test-{{ test_string|lower }}: {{ mac }}"
       delegate_to: localhost

HTH,

  -vlado

My apologies for not being a little more clear, Vlado.

I do want this to execute on the local host and not on the remote hosts. Ultimately this will be setting up a file that will be used later for building a remote host’s config file using the template module. It’s all a part of an attempt at a somewhat zero touch provisioning model for switches we deploy.

Essentially the work flow would look similar to:

a new device needs to be added => user builds initial config file specific to device location and requirements => user runs playbook

The playbook would then locally:
load the device variables from the config file

take user’s config file and make a backup to the host_vars directory for later use, if necessary
take a short hostname (test_string) and MAC address and append them to the file

Later in the playbook other plays will have tasks that load the variables in these files generate a config from a template that gets copied to the remote host. This one file and its contents is, sort of, a living file that gets updated every time we add or replace a device.

Hope that makes sense.

If the file '/var/tmp/test.yaml' shall be created on both 'host1' and 'host2'
try this

    - name: Update test file
      lineinfile:
        path: /var/tmp/test.yaml
        regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
        line: " test-{{ test_string|lower }}: {{ mac }}"
      loop: "{{ ansible_play_hosts }}"
      vars:
        mac: "{{ hostvars[item]['mac'] }}"
        test_string: "{{ hostvars[item]['test_string'] }}"

HTH,

  -vlado

So I added the delegate_to: localhost but I still only get one added on the first pass:

`
PLAY [Test Playbook] ************************************************************************************************************************************

TASK [Load device vars from config] *********************************************************************************************************************
ok: [host1]
ok: [host2]

TASK [Update tmp file] **********************************************************************************************************************************
changed: [host1 → localhost]
changed: [host2 → localhost]

PLAY RECAP **********************************************************************************************************************************************
host1 : ok=2 changed=1 unreachable=0 failed=0
host2 : ok=2 changed=1 unreachable=0 failed=0

`

The file contents after that pass are:

1 macs: 2 test-host222: ab:09:87:65:43:21

And when I run it a second time:

`
PLAY [Test Playbook] ************************************************************************************************************************************

TASK [Load device vars from config] *********************************************************************************************************************
ok: [host1]
ok: [host2]

TASK [Update tmp file] **********************************************************************************************************************************
changed: [host1 → localhost]
ok: [host2 → localhost]

PLAY RECAP **********************************************************************************************************************************************
host1 : ok=2 changed=1 unreachable=0 failed=0
host2 : ok=2 changed=0 unreachable=0 failed=0
`

With the file contents now the following:

1 macs: 2 test-host222: ab:09:87:65:43:21 3 test-host111: 12:34:56:78:90:ab

Since you have two host you will have two tasks that tries to write to the same
file at the same time, and only one of them will win.

Add "throttle: 1" to you lineinfile task and it will work.

> [test_group]
> host1
> host2

Since you have two host you will have two tasks that tries to write to the same
file at the same time, and only one of them will win.

Add "throttle: 1" to you lineinfile task and it will work.

I don't need "throttle" with 2.9.6. The playbook

cat pb.yml

- hosts:
    - test_01
    - test_02
  gather_facts: false
  tasks:
    - lineinfile:
        path: test-file
        line: "{{ inventory_hostname }}"
      delegate_to: localhost

gives

cat test-file
ansible-playbook pb.yml

PLAY [test_01,test_02]

The two hosts I have to test with are running 2.7.4 and 2.8.2. I don’t have a 2.9.x host available to test the throttle command but it makes sense.

However, when I updated the playbook to the following:

`

The file contents after that pass are:

  1 macs:
  2 test-host222: ab:09:87:65:43:21

And when I run it a second time:

  1 macs:
  2 test-host222: ab:09:87:65:43:21
  3 test-host111: 12:34:56:78:90:ab

It's working for me as expected

cat test_01.conf

test_string: abc
mac: 01:23:45:67:89:ab

cat test_02.conf

test_string: xyz
mac: ab:09:87:65:43:21

cat pb.yml

- hosts:
    - test_01
    - test_02
  tasks:
    - include_vars: "{{ inventory_hostname }}.conf"
    - name: Update test file
      lineinfile:
        path: test-file
        regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
        line: " test-{{ test_string|lower }}: {{ mac }}"
      delegate_to: localhost

cat test-file
ansible-playbook pb.yml

PLAY [test_01,test_02]

That is just pure luck, try running it 100 times and count how many times it
fails.

Here also you have two task writing to the same file at the same time.
So here you need to add "run_once: yes" to the lineinfile task.

The file contents after that pass are:

  1 macs:
  2 test-host222: ab:09:87:65:43:21

With the file contents now the following:

  1 macs:
  2 test-host222: ab:09:87:65:43:21
  3 test-host111: 12:34:56:78:90:ab

Bingo! I'm able to reproduce the problem with Ansible running on controller
with Python 2.7

ansible --version

ansible 2.9.6
  config file = /export/scratch/tmp/ansible.cfg
  configured module search path = [u'/home/vlado/.ansible/plugins/modules',
u'/usr/share/ansible/plugins/modules'] ansible python module location =
/usr/lib/python2.7/dist-packages/ansible executable location =
/usr/bin/ansible python version = 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0]

There is no such problem with Ansible running on controller with Python 3

ansible --version

ansible 2.9.6
  config file = /home/vlado/.ansible.cfg
  configured module search path = ['/home/vlado/.ansible/my_modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Apr 27 2020, 15:53:34) [GCC 9.3.0]

HTH,

  -vlado

The main one I’m testing with is running python 3.6

$ ansible --version
ansible 2.7.4
config file = /etc/ansible/ansible.cfg
configured module search path = [‘/home/ofoo/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]

Well, I tried and it's working properly with Python3. There is no problem with
Ansible 2.9.6 and Python 3.8 on the controller ( the redirection in the script
'2>/dev/null' is there because of annoying 3.8 RuntimeWarning).

  #!/bin/bash
  export ANSIBLE_STDOUT_CALLBACK=null
  for i in {1..99}; do
      ansible-playbook pb.yml 2>/dev/null
      no_lines=`cat test-file | wc -l`
      [ $no_lines -ne 2 ] && echo "ERROR File 2 lines missing"
      cat /dev/null > test-file
      no_lines=`cat test-file | wc -l`
      [ $no_lines -ne 0 ] && echo "ERROR File not empty"
  done

But, you were right with "throttle". With Ansible 2.9.6 and Python 2.7 on the
controller "throttle: 1" fixed the problem

    - name: Update test file
      lineinfile:
        path: test-file
        regexp: "^\ \ test[.-]{{ test_string|lower }}:.*"
        line: " test-{{ test_string|lower }}: {{ mac }}"
      delegate_to: localhost
      throttle: 1

Thank you,

  -vlado

> The file contents after that pass are:
>
> 1 macs:
> 2 test-host222: ab:09:87:65:43:21
>
> With the file contents now the following:
>
> 1 macs:
> 2 test-host222: ab:09:87:65:43:21
> 3 test-host111: 12:34:56:78:90:ab

Bingo! I'm able to reproduce the problem with Ansible running on controller
with Python 2.7

<snip />

There is no such problem with Ansible running on controller with Python 3

That's just not true, your test method is flawed, you need to test it more than one time.

$ ansible-playbook --version | awk 'NR==1; END{print}'
ansible-playbook 2.9.7
  python version = 3.7.5 (default, Apr 19 2020, 20:18:17) [GCC 9.2.1 20191008]

$ cat test.yml
- hosts: a1,a2
  tasks:
    - lineinfile:
        path: /tmp/txt
        line: "{{ inventory_hostname }}"
      delegate_to: localhost

$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
      4 763950971c8c6d8df8a87a1e752799a9 /tmp/txt
     11 1597a5a9948014489de663c8fb4438db /tmp/txt
     36 ac34af7b876f793d09a2225e23f43088 /tmp/txt
     49 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt

So as you can see I get 4 different result as expected.

$ cat a1; md5sum a1
a1
763950971c8c6d8df8a87a1e752799a9 a1

$ cat a2; md5sum a2
a2
1597a5a9948014489de663c8fb4438db a2

$ cat a1a2; md5sum a1a2
a1
a2
317f53fd9236220d0ab65e4aac4b3c5a a1a2

$ cat a2a1; md5sum a2a1
a2
a1
ac34af7b876f793d09a2225e23f43088 a2a1

> There is no such problem with Ansible running on controller with Python 3

That's just not true, your test method is flawed, you need to test it more than one time.

$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
      4 763950971c8c6d8df8a87a1e752799a9 /tmp/txt
     11 1597a5a9948014489de663c8fb4438db /tmp/txt
     36 ac34af7b876f793d09a2225e23f43088 /tmp/txt
     49 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt

So as you can see I get 4 different result as expected.
$ cat a1; md5sum a1
a1
763950971c8c6d8df8a87a1e752799a9 a1
$ cat a2; md5sum a2
a2
1597a5a9948014489de663c8fb4438db a2
$ cat a1a2; md5sum a1a2
a1
a2
317f53fd9236220d0ab65e4aac4b3c5a a1a2
$ cat a2a1; md5sum a2a1
a2
a1
ac34af7b876f793d09a2225e23f43088 a2a1

Well, it is true. There are no problems reported with your method either. I
get correct results only.

for i in {1..100}; do >test-file; ansible-playbook pb.yml 2>/dev/null; md5sum test-file; done | sort | uniq -c | sort -n

     18 82de37d1deac8c90573c35e5637839d7 test-file
     82 6f4e9c06436257db5ba34d527bd3b211 test-file

cat test_01test_02; md5sum test_01test_02

test_01
test_02
6f4e9c06436257db5ba34d527bd3b211 test_01test_02

cat test_02test_01; md5sum test_02test_01

test_02
test_01
82de37d1deac8c90573c35e5637839d7 test_02test_01

ansible --version

ansible 2.9.6
  config file = /home/vlado/.ansible.cfg
  configured module search path = ['/home/vlado/.ansible/my_modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.2 (default, Apr 27 2020, 15:53:34) [GCC 9.3.0]

> > There is no such problem with Ansible running on controller with Python 3
>
> That's just not true, your test method is flawed, you need to test it more than one time.
>
> $ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
> 4 763950971c8c6d8df8a87a1e752799a9 /tmp/txt
> 11 1597a5a9948014489de663c8fb4438db /tmp/txt
> 36 ac34af7b876f793d09a2225e23f43088 /tmp/txt
> 49 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt

Well, it is true. There are no problems reported with your method either. I
get correct results only.

> for i in {1..100}; do >test-file; ansible-playbook pb.yml 2>/dev/null; md5sum test-file; done | sort | uniq -c | sort -n
     18 82de37d1deac8c90573c35e5637839d7 test-file
     82 6f4e9c06436257db5ba34d527bd3b211 test-file

To be sure, I let it run overnight

for i in {1..10000}; do >test-file; ansible-playbook -i hosts77 test77.yml 2>/dev/null; md5sum test-file; done | sort | uniq -c | sort -n

   2689 82de37d1deac8c90573c35e5637839d7 test-file
   7311 6f4e9c06436257db5ba34d527bd3b211 test-file

That is not true and has nothing to do with Python version, and everything with
a little luck and a fast computer.

The test case is flawed for your Ansible controller, if it had more load and/or
more hosts you would get a completely different result.

The reason why this i not reliable is that Ansible is default running forks of
5, that mean that if you had 5 or more hosts it will run/fork out 5 commands.
This 5 commands will try to edit the same file at the same time.

Reason it works for you is that the first fork is finished before the second
has time to start.

This is how Ansible work, and now Python version is going to alter that.

And I'm going to prove it.
Since my machine doesn't have Python 3.8 I created a VM, with 2 cores.

$ ansible-playbook --version | awk 'NR==1; END{print}'
ansible-playbook 2.9.7
  python version = 3.8.2 (default, Mar 24 2020, 03:08:36) [GCC 9.3.0]

So when I run my playbook with two host I get this

$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
      1 1597a5a9948014489de663c8fb4438db /tmp/txt
     39 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt
     60 ac34af7b876f793d09a2225e23f43088 /tmp/txt

and the next time
$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; md5sum /tmp/txt; done | sort | uniq -c | sort -n
      1 1597a5a9948014489de663c8fb4438db /tmp/txt
     43 317f53fd9236220d0ab65e4aac4b3c5a /tmp/txt
     56 ac34af7b876f793d09a2225e23f43088 /tmp/txt

As you can see it fails i 1 % of the cases.

Lets up the stake and go for 6 hosts, the playbook is the same, but since 6 hosts
make potensial a lot of combination, I sort the file before I run md5sum.
So if everything is working OK I should only get one line with one hash like this
   100 aae4eb078da77ff549495a60be1a52c2

$ for i in {1..100}; do >/tmp/txt; ansible-playbook test.yml &>/dev/null; sort /tmp/txt | md5sum; done | sort | uniq -c | sort -n
      1 2465fc0f03254f2bf915a04c762c9d49 -
      2 254316531e34aa3ce563c22cf636ea01 -
      2 31f07322fb0283ee5475e5acc74de059 -
      2 ab8ecb5209d15e9852be208b79f309eb -
      3 7abbd5c371c5e4ccb8bc965105148605 -
     90 aae4eb078da77ff549495a60be1a52c2 -

Not even close, it fails in 10% of the runs.

So lets make the machine even more busy.
To do that I shutdown the VM and reduced the cores count from 2 to 1.
Now we have 5 forks at the same time "fighting" for this 1 core and trying to
write to the same file at the same time.

This is the result.

      1 254316531e34aa3ce563c22cf636ea01 -
      1 6c5ffba74557139441576f9f6536c536 -
      1 8da4fee8643e840ae60169eac2d60afd -
      1 9a7861beaa38064dd7179cf66b90173f -
      1 aae4eb078da77ff549495a60be1a52c2 -
      1 d5fad8b2b8f31c4d201309e2261ee362 -
      2 2465fc0f03254f2bf915a04c762c9d49 -
      2 2ea77912754b2a786046ece007bfd0a7 -
      2 6503321b7879267a6a5eb06411e09b5e -
      2 9ce72c8f1a3f059db908868df22c2618 -
      2 ab8ecb5209d15e9852be208b79f309eb -
      2 adc72709b7b7cdc8aaad85c01b66e38d -
      2 c4b05365f19a397fc9b2dda817219d49 -
      2 cf2b918385c6d1f75d7a0630d8881422 -
      2 d678deb816d9433ff1c61d60ec1073f1 -
      3 83758fc71d9927b3141c325692534add -
      3 989fe3db59b18149e58657daab4721dc -
      3 a421781f0d1512c9c66e9f7714bed570 -
      4 7bc2be35864ae017608466e62dd680aa -
      5 83ba567764a618ec0a0b405d49de8fa9 -
      5 c98ae97d4888d870d9a05fe7f9b339a3 -
      5 d336136909e4067c35858990dd4f9211 -
      6 31f07322fb0283ee5475e5acc74de059 -
      6 76aabdf63a1f041f6572fb51fb496283 -
      6 cce45b22f8bc7806e3dc1128980f33f6 -
      6 cce83e09262e3efced931d81c85ac809 -
      6 decd8e81f11339ba387cb7cf61a28f34 -
      7 aed3d7850c9c5478d84bf78fdc922328 -
     11 d8ee7367ba9763647a58ab23a6f01c5f -

Yeah, it fails miserably, in 99% of the cases.
The correct md5sum is this one

$ cat sorted.txt
a1
a2
a3
a4
a5
a6

$ md5sum sorted.txt
aae4eb078da77ff549495a60be1a52c2 sorted.txt

So no combination of Ptyhon and Ansible released will alter this,
it's just how it work.
That is why we have serial, fork and throttle to tune the behavior to avoid race
conditions.

I have explained in my previous mail why this is by change/luck, and it will
fail if you controller had some more load that just 2 hosts.