Replace unmanaged text block with managed block?

Get Help
1

Hello everyone,

I would like to replace an unmanaged block of text from a configuration file (for Dovecot) with a managed block. The difficulty is that the two blocks have different content, so I thought of splitting it up into two steps: first find and remove the old block, then add the new block using the ansible.builtin.blockinfile module. The second part is easy, but I cannot figure out how to remove a multi-line string.

Here is what the original (unmanaged) block looks like:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    #mode = 0666
  }

  # Create inet listener only if you can't use the above UNIX socket
  #inet listener lmtp {
    # Avoid making LMTP visible for the entire internet
    #address =
    #port =
  #}
}

It would be enough to match the first (the opening ^service lmtp {) and last (the closing }) line and include everything in between. However, I have not been able to find out how to use ansible.builtin.replace to match this entire content. My idea was to use this:

- name: 'Remove default LMTP configuration'
  ansible.builtin.replace:
    path: '/etc/dovecot/conf.d/10-master.conf'
    before: '# BEGIN ANSIBLE MANAGED LMTP CONFIGURATION'
    regexp: '^service lmtp \{\n(.*\n)*\n^\}'

Since . does not match newlines I use ^service lmtp \{\n (explicit new line), followed by an arbitrary amount of of lines and ending with ^\} (a closing curly brace), making use of the fact that the closing } is not indented. However, the regex does not match and the old block remains in the file. And even if it did I would have to make the match non-greedy, otherwise I would delete up to the last unindented } of the entire file.

Is there a way to write a multiline regex for replace? Or maybe I am going at this completely the wrong way and there is a smarter solution?

2

According to https://docs.ansible.com/ansible/latest/collections/ansible/builtin/replace_module.html#parameter-regexp,

The regular expression to look for in the contents of the file.
Uses Python regular expressions; see re — Regular expression operations — Python 3.12.7 documentation.
Uses MULTILINE mode, which means ^ and $ match the beginning and end of the file, as well as the beginning and end respectively of each line of the file.
Does not use DOTALL, which means the . special character matches any character except newlines. A common mistake is to assume that a negated character set like [^#] will also not match newlines.
In order to exclude newlines, they must be added to the set like [^#\n].

Because regex: doesn’t use DOTALL, you have to use “extra stuff” to express a dot that includes newlines. It looks like you were trying to do that. I used the following expression on your data with success:

    regexp: '^service lmtp \{(.|\n)*?^}$'

Note the trailing $, which you missed.

But the real reason your tasks would never match is because of your before: parameter. before: doesn’t specify text that ansible.builtin.replace will insert before the replacement. Rather, it restricts the scope of the replacement action to the part of the file which appears before the specified string. If your file doesn’t already contain # BEGIN ANSIBLE MANAGED LMTP CONFIGURATION, then that task isn’t going to affect that file.

Here’s a complete working example:

---
# HiPhish.yml
- name: Replace example
  hosts: localhost
  gather_facts: false
  vars:
    filename: HiPhish.txt
  tasks:
    - name: Create temp file
      ansible.builtin.copy:
        dest: '{{ filename }}'
        content: |
          stuff before
          service lmtp {
            unix_listener /var/spool/postfix/private/dovecot-lmtp {
              #mode = 0666
            }

            # Create inet listener only if you can't use the above UNIX socket
            #inet listener lmtp {
              # Avoid making LMTP visible for the entire internet
              #address =
              #port =
            #}
          }
          stuff after

    - name: Use Replace
      ansible.builtin.replace:
        path: '{{ filename }}'
        regexp: '^service lmtp \{(.|\n)*?^}$'
        replace: |
          ### Start of replaced text ###
          Ha ha! Gotchya.
          ### End of replaced text ###
1 Like
3

Thank you for the regex, it works.

But the real reason your tasks would never match is because of your before: parameter. before: doesn’t specify text that ansible.builtin.replace will insert before the replacement. Rather, it restricts the scope of the replacement action to the part of the file which appears before the specified string. If your file doesn’t already contain # BEGIN ANSIBLE MANAGED LMTP CONFIGURATION, then that task isn’t going to affect that file.

The reason I was using before is for idempotency: after deleting the unmanaged block the next task will append the managed block at the end. On a subsequent run I do not want replace to delete the managed block, hence I was using before as a way to say “not after this string”.

Is there a better way of telling Ansible to not delete after the marker line?

4

Okay, but the task in your original post in the topic, named “Remove default LMTP configuration” won’t do what it says because “# BEGIN ANSIBLE MANAGED LMTP CONFIGURATION” isn’t in /etc/dovecot/conf.d/10-master.conf. So you have a chicken-and-egg problem.

If this were my problem, I’d break it into several steps.

  1. Use ansible.builtin.lineinfile to loop over '# BEGIN ANSIBLE MANAGED LMTP CONFIGURATION' and '# END ANSIBLE MANAGED LMTP CONFIGURATION', making sure that if they don’t already exist in the file that they get appended.
  2. Use the task “Remove default LMTP configuration” from your original post. Since the “# BEGIN […]” line is now there, that task’s before: parameter should do the Right Thing™.
  3. Use ansible.builtin.replace to replace everything between the “# BEGIN” and “# END” lines. You can use the before: parameter to indicate the “# END” line, and the after: parameter to indicate the “# BEGIN”. The regex: parameter is required, so (.|\n)* should get everything between the two, including nothing.

You may be tempted to omit the “# END” line since you won’t have anything after it, but that would be a mistake. If you find that later you need to replace existing or add not-yet-existing configuration and put it under Ansible management, having an explicit beginning and ending of each such section would be handy.

This whole house of cards assumes the dovecot package (or whatever created '/etc/dovecot/conf.d/10-master.conf') isn’t going to come along and replace it later with an upgrade or phase of the moon change or who knows what else. Some common config files (like /etc/passwd) are changed in well-understood ways that multiple changing entities are aware of so they generally don’t step on each other’s work. I wouldn’t comfortably make that assumption in the case of conf.d/*.conf files for random applications.

You also asked if maybe there’s a better way to go about it, or a smarter solution. I don’t know about dovecot in particular, but in general, if an application supports the …/app/conf.d/##-name.conf paradigm, you’re better off leaving installed files alone and adding/overriding config with your own conf.d/##-name.conf files. If you define another “service lmtp {…}” in a conf file that’s processed after '/etc/dovecot/conf.d/10-master.conf', it’s possible that later redefinition would override the original, in which case you need not mess with the 10-master.conf file at all. Then you could build, say, '/etc/dovecot/conf.d/99-local-ansible.conf' through an Ansible template. That would be far preferable if that works. You would need to test of course to see.

1 Like
5

Oh wow, I feel so stupid in hindsight. Yes, Dovecot does support exactly what you describe and it’s a much better solution than editing existing files. Not just easier to implement in Ansible, but also easier to maintain and undo if necessary.

2 Likes