lineinfile problem

I have a problem where a line that contains a certain space delimited string must exist in a file. The string can be anywhere in the line: start, middle, or end. If a line that contains that string does not exist, then a line with that string needs to be inserted into the file. I have been trying to use the lineinfile module to solve this, but so far I have been unsuccessful. Below is the code I have tried:

  • lineinfile:

dest: “/path/to/myfile”
regexp: ‘^(.)(\sthis_string_must_exist)(\s+.*)$’
line: ‘\g<1> this_string_must_exist\g<3>’
backrefs: true
state: present

I have tried many variations of the regular expression and line, but nothing I have tried totally achieves what I am looking for. I have verified that the regular expression can locate a line with the string regardless of where the string is within the line, and if such a line is located it is not modified. But I have not been successful in getting a new line inserted if it does not already exist.
Does anyone have any suggestions?
-Mark

The reason it doesn't work is explained in the documentation.

https://docs.ansible.com/ansible/lineinfile_module.html#options

In the comments about the backrefs:
"... and if the regexp doesn't match anywhere in the file, the file will be left unchanged."

So you'll need to find a way to do it without backrefs: yes if you are going to use lineinfile.

Yes, I finally spotted that immediately after I posted my initial query (when all else fails, read the documentation). Now I am trying to use lineinfile (or maybe grep) to look for the string and register the results, then use a second lineinfile to add a line if the results of the first lineinfile indicates that it is missing. This is a bit complicated because I have several strings that I need to look for and I want to use ‘with_items’ to store them.
-Mark

Hi Mark,

I found the following https://relativkreativ.at/articles/how-to-use-ansibles-lineinfile-module-in-a-bulletproof-way very useful when I needed to find a reliable way to edit lines in a configuration file with Ansible tasks.

See if it helps you understand how to use lineinfile in a bulletproof way.

Regards,
Jinesh

Your article had interesting information, but it doesn’t look like it will help. I tried using negative look-ahead, but got nowhere with it.
Now I am trying to use egrep to scan the file, and then use lineinfile to act upon the results. I put together the below snippet for testing:

  • shell: chdir=~/tmp egrep “^.\s{{item}}\s+.*$” mytestfile
    register: result
    changed_when: false
    failed_when: false
    with_items:

  • “string2”

  • “string6”

  • “string7”

  • debug: var={{item}}
    with_items:

  • result

This returns the following:

TASK [debug] *******************************************************************
ok: [localhost] => (item=result) => {
“item”: “result”,
“result”: {
“changed”: false,
“msg”: “All items completed”,
“results”: [
{
“_ansible_item_result”: true,
“_ansible_no_log”: false,
“_ansible_parsed”: true,
“changed”: false,
“cmd”: “egrep "^.\sstring2\s+.$" mytestfile",
“delta”: “0:00:00.003006”,
“end”: “2017-01-26 22:33:27.722164”,
“failed”: false,
“failed_when_result”: false,
“invocation”: {
“module_args”: {
“_raw_params”: "egrep "^.
\sstring2\s+.$" mytestfile”,
“_uses_shell”: true,
“chdir”: “/home/pdxmft/tmp”,
“creates”: null,
“executable”: null,
“removes”: null,
“warn”: true
},
“module_name”: “command”
},
“item”: “string2”,
“rc”: 0,
“start”: “2017-01-26 22:33:27.719158”,
“stderr”: “”,
“stdout”: “string1 string2 string3”,
“stdout_lines”: [
“string1 string2 string3”
],
“warnings”:
},
{
“_ansible_item_result”: true,
“_ansible_no_log”: false,
“_ansible_parsed”: true,
“changed”: false,
“cmd”: “egrep "^.\sstring6\s+.$" mytestfile",
“delta”: “0:00:00.003093”,
“end”: “2017-01-26 22:33:27.837867”,
“failed”: false,
“failed_when_result”: false,
“invocation”: {
“module_args”: {
“_raw_params”: "egrep "^.
\sstring6\s+.$" mytestfile”,
“_uses_shell”: true,
“chdir”: “/home/pdxmft/tmp”,
“creates”: null,
“executable”: null,
“removes”: null,
“warn”: true
},
“module_name”: “command”
},
“item”: “string6”,
“rc”: 1,
“start”: “2017-01-26 22:33:27.834774”,
“stderr”: “”,
“stdout”: “”,
“stdout_lines”: ,
“warnings”:
},
{
“_ansible_item_result”: true,
“_ansible_no_log”: false,
“_ansible_parsed”: true,
“changed”: false,
“cmd”: “egrep "^.\sstring7\s+.$" mytestfile",
“delta”: “0:00:00.002986”,
“end”: “2017-01-26 22:33:27.951455”,
“failed”: false,
“failed_when_result”: false,
“invocation”: {
“module_args”: {
“_raw_params”: "egrep "^.
\sstring7\s+.$" mytestfile”,
“_uses_shell”: true,
“chdir”: “/home/pdxmft/tmp”,
“creates”: null,
“executable”: null,
“removes”: null,
“warn”: true
},
“module_name”: “command”
},
“item”: “string7”,
“rc”: 0,
“start”: “2017-01-26 22:33:27.948469”,
“stderr”: “”,
“stdout”: “string7 string8 string9”,
“stdout_lines”: [
“string7 string8 string9”
],
“warnings”:
}
]
}
}

What I would like to do now is walk through the results list and extract the “rc” and “item” values, something along the line of this:

  • debug: var={{item.rc}}
    with_items:
  • result.results[*]

But of course that does not work. So how can I walk through a list in this manner? I looked at ‘with_subelements’ but was unable to put something together there.
-Mark

First guess: Drop the "[*]". Ansible knows this is a list and should
use each element as "item".

Johannes

File: example_file.txt

`

string1 string2 string3
string4 string5 string0
string7 string8 string9

`

File: run.sh

`

#!/usr/bin/env bash
ansible-playbook -vvv -i ‘localhost,’ -c local test.yml

`

File: test.yml

`

  • hosts:

  • localhost
    tasks:

  • name: “Slurp the file we wish to search”
    slurp:
    src: ./example_file.txt
    register: slurped_file

  • name: “To avoid base64 decoding multiple times, we do it once”
    set_fact: plain_file=“{{ slurped_file[‘content’] | b64decode }}”

- name: “You can use this task to test your regex”

debug: msg=“{{ plain_file | regex_search(‘string7’) is none}}”

  • name: “Insert the wanted line into the file as it doesn’t exist”
    lineinfile:
    dest: ./example_file.txt
    line: “something {{ item }} somethingelse”
    state: present
    when: plain_file | regex_search(item) is none
    with_items:
  • string6
  • string1
  • string8
  • foobar

`

Example output:

`

$ ./run.sh
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: test.yml *************************************************************
1 plays in test.yml

PLAY [localhost] ***************************************************************

TASK [setup] *******************************************************************
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/system/setup.py
ESTABLISH LOCAL CONNECTION FOR USER: algomi-deploy
EXEC /bin/sh -c ‘( umask 77 && mkdir -p “echo ~/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950” && echo ansible-tmp-1485515755.89-254326766505950=“echo ~/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950” ) && sleep 0’
PUT /tmp/tmpP7Zu66 TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/setup.py
EXEC /bin/sh -c ‘chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/setup.py && sleep 0’
EXEC /bin/sh -c ‘/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/setup.py; rm -rf “/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515755.89-254326766505950/” > /dev/null 2>&1 && sleep 0’
ok: [localhost]

TASK [Slurp the file we wish to search] ****************************************
task path: /opt/home/algomi-deploy/test.yml:4
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/network/basics/slurp.py
ESTABLISH LOCAL CONNECTION FOR USER: algomi-deploy
EXEC /bin/sh -c ‘( umask 77 && mkdir -p “echo ~/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445” && echo ansible-tmp-1485515756.18-190467128946445=“echo ~/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445” ) && sleep 0’
PUT /tmp/tmpUjpiaz TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/slurp.py
EXEC /bin/sh -c ‘chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/slurp.py && sleep 0’
EXEC /bin/sh -c ‘/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/slurp.py; rm -rf “/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.18-190467128946445/” > /dev/null 2>&1 && sleep 0’
ok: [localhost] => {
“changed”: false,
“content”: “c3RyaW5nMSBzdHJpbmcyIHN0cmluZzMKc3RyaW5nNCBzdHJpbmc1IHN0cmluZzAKc3RyaW5nNyBzdHJpbmc4IHN0cmluZzkK”,
“encoding”: “base64”,
“invocation”: {
“module_args”: {
“src”: “./example_file.txt”
},
“module_name”: “slurp”
},
“source”: “./example_file.txt”
}

TASK [To avoid base64 decoding multiple times, we do it once] ******************
task path: /opt/home/algomi-deploy/test.yml:10
ok: [localhost] => {
“ansible_facts”: {
“plain_file”: “string1 string2 string3\nstring4 string5 string0\nstring7 string8 string9\n”
},
“changed”: false,
“invocation”: {
“module_args”: {
“plain_file”: “string1 string2 string3\nstring4 string5 string0\nstring7 string8 string9\n”
},
“module_name”: “set_fact”
}
}

TASK [Insert the wanted line into the file as it doesn’t exist] ****************
task path: /opt/home/algomi-deploy/test.yml:17
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/files/lineinfile.py
ESTABLISH LOCAL CONNECTION FOR USER: algomi-deploy
EXEC /bin/sh -c ‘( umask 77 && mkdir -p “echo ~/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270” && echo ansible-tmp-1485515756.42-101048875702270=“echo ~/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270” ) && sleep 0’
PUT /tmp/tmpPv_qtT TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/lineinfile.py
EXEC /bin/sh -c ‘chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/lineinfile.py && sleep 0’
EXEC /bin/sh -c ‘/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/lineinfile.py; rm -rf “/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.42-101048875702270/” > /dev/null 2>&1 && sleep 0’
changed: [localhost] => (item=string6) => {
“backup”: “”,
“changed”: true,
“diff”: [
{
“after”: “”,
“after_header”: “./example_file.txt (content)”,
“before”: “”,
“before_header”: “./example_file.txt (content)”
},
{
“after_header”: “./example_file.txt (file attributes)”,
“before_header”: “./example_file.txt (file attributes)”
}
],
“invocation”: {
“module_args”: {
“backrefs”: false,
“backup”: false,
“content”: null,
“create”: false,
“delimiter”: null,
“dest”: “./example_file.txt”,
“directory_mode”: null,
“follow”: false,
“force”: null,
“group”: null,
“insertafter”: null,
“insertbefore”: null,
“line”: “something string6 somethingelse”,
“mode”: null,
“owner”: null,
“regexp”: null,
“remote_src”: null,
“selevel”: null,
“serole”: null,
“setype”: null,
“seuser”: null,
“src”: null,
“state”: “present”,
“unsafe_writes”: null,
“validate”: null
},
“module_name”: “lineinfile”
},
“item”: “string6”,
“msg”: “line added”
}
skipping: [localhost] => (item=string1) => {
“changed”: false,
“item”: “string1”,
“skip_reason”: “Conditional check failed”,
“skipped”: true
}
skipping: [localhost] => (item=string8) => {
“changed”: false,
“item”: “string8”,
“skip_reason”: “Conditional check failed”,
“skipped”: true
}
Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/files/lineinfile.py
EXEC /bin/sh -c ‘( umask 77 && mkdir -p “echo ~/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938” && echo ansible-tmp-1485515756.52-54649606538938=“echo ~/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938” ) && sleep 0’
PUT /tmp/tmpAvxVae TO /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/lineinfile.py
EXEC /bin/sh -c ‘chmod u+x /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/ /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/lineinfile.py && sleep 0’
EXEC /bin/sh -c ‘/usr/bin/python /opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/lineinfile.py; rm -rf “/opt/home/algomi-deploy/.ansible/tmp/ansible-tmp-1485515756.52-54649606538938/” > /dev/null 2>&1 && sleep 0’
changed: [localhost] => (item=foobar) => {
“backup”: “”,
“changed”: true,
“diff”: [
{
“after”: “”,
“after_header”: “./example_file.txt (content)”,
“before”: “”,
“before_header”: “./example_file.txt (content)”
},
{
“after_header”: “./example_file.txt (file attributes)”,
“before_header”: “./example_file.txt (file attributes)”
}
],
“invocation”: {
“module_args”: {
“backrefs”: false,
“backup”: false,
“content”: null,
“create”: false,
“delimiter”: null,
“dest”: “./example_file.txt”,
“directory_mode”: null,
“follow”: false,
“force”: null,
“group”: null,
“insertafter”: null,
“insertbefore”: null,
“line”: “something foobar somethingelse”,
“mode”: null,
“owner”: null,
“regexp”: null,
“remote_src”: null,
“selevel”: null,
“serole”: null,
“setype”: null,
“seuser”: null,
“src”: null,
“state”: “present”,
“unsafe_writes”: null,
“validate”: null
},
“module_name”: “lineinfile”
},
“item”: “foobar”,
“msg”: “line added”
}

PLAY RECAP *********************************************************************
localhost : ok=4 changed=1 unreachable=0 failed=0

`

Final contents of the example_file.txt file:

`

string1 string2 string3
string4 string5 string0
string7 string8 string9
something string6 somethingelse
something foobar somethingelse

`

  • The above slurps a file from the “remote” node.
  • It converts the base64 encoded contents of the file into plain text so that we can regex search it
  • It uses lineinfile to insert lines using with_items when the regex_search of the plain test file doesn’t match an item

The downsides of the above play are that the slurping a large file may not be possible for you due to memory restrictions. Also, you weren’t clear on what should be inserted into the file if the sought for string was not found and so the solution may not be complete.

Note: the regex_search filter is something new (i.e. not in the docs yet). I didn’t know about it until I came across this PR https://github.com/ansible/ansible/pull/14696

I ran the above on ansible 2.2.1.0.

I was not aware of the slurp module prior to this. That combined with the regex_search filter certainly opens many avenues for searching files. I can see where this could be problematic for large files, but in this case that is not an issue. I am running Ansible version 2.2.0.0 and that appears to include the regex_search filter. This solution works for me, so I will go with it.
I looked around for more information about the regex_search filter, but found very little. It is stored in the ansible/plugins/filter/core.py file if you want to look at it and see what other undocumented filters are available.
-Mark