Complex process to generate template/lineinfile data?

Archives Ansible Project
1

Hi all,

Well, it's not really that complex, but seems complex to do within the constraints of ansible/jinja.

My scenario is that I run dirvish to back up multiple containers on multiple hosts. The containers are backed up via the filesystems on the hosts.

The root filesystem of the container lives under /var/lib/lxc/<container>/rootfs.

Sometimes, extra filesystems are also used; they live under /guestfs/<container>-<name>, where 'name' is a possibly shortened version of the mountpoint.

Relevant sections of host_vars files look like this:

filesystems:
   - { index: 1, suffix: srv, size: 2, container_mountpoint: '/srv/' }
backup_paths:
   - { shortname: 'ROOT', path: '/' }
   - { shortname: 'var', path: '/var/' }
   - { shortname: 'home', path: '/home/' }
   - { shortname: 'srv', path: '/srv/' }

So when I'm setting up backups, I need to treat any backup path for this container starting with '/srv/' differently, in that I need to backup something under /guestfs/ rather than under /var/lib/lxc.

I have a template that does this for the dirvish config:

client: {{ hostvars[container_host].fqdn }}
{% set ns = namespace(extra_fs='') %}
{% for mountpoint in filesystems %}
{% if item['path'] | regex_search(mountpoint['container_mountpoint']) %}
{% set ns.extra_fs = mountpoint %}
{% endif %}
{% endfor %}
{% if ns.extra_fs %}
tree: /guestfs/{{ inventory_hostname + '-' + ns.extra_fs['suffix'] + '/' + item['path'] |regex_replace(ns.extra_fs['container_mountpoint'], '') }}
{% else %}
tree: /var/lib/lxc/{{ inventory_hostname }}/rootfs{{ item['path'] }}
{% endif %}
rsh: ssh -i /root/.ssh/id_rsa_dirvish {{ hostvars[container_host].fqdn }}

{% for backup_path in (backup_paths | select('search', item['path'] + '.*')) if not (backup_path['shortname'] == item['shortname']) %}
{% if loop.index == 1 %}
excludes:
{% endif %}
   {{ backup_path['path'] }}*
{% endfor %}

As you can see, that's quite complicated.

Then in addition, I need to add to a list of acceptable paths on the host (for my ssh forced command script to read).

That looks like this (for the same container described in the host_vars above):

...
/var/lib/lxc/example-web1/rootfs/
/var/lib/lxc/example-web1/rootfs/var/
/var/lib/lxc/example-web1/rootfs/home/
/guestfs/example-web1-srv/
...

That isn't a template, because the file contains paths for all the containers. I suppose I could modify my script to read any files in a directory instead of a single file, but that would be a separate project, and the template would be just as nasty as the one above.

I'm not even sure where to start with generating lines to add to the file.

Any suggestions?

Is this stuff reasonable to do with ansible, and a templating language?

Do I need to write a custom plugin of some kind?

If I had a dynamic inventory, then I could probably generate extra variables at that stage, but that's further away too.

Cheers,
Richard

2

I should give lots of credit to the nice people on irc who helped me get as far as I have.

I've also thought that if I could pass data (as json perhaps?) to stdin of a local script, and capture the output (a list of strings, as json again?), that would probably work ok - I could then write the script in a language I'm more familiar with, such as perl.

Is that doable?

Thanks,
Richard

3

I'm experimenting with writing a vars plugin, referring to these:

https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html
https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/vars/host_group_vars.py

I've put it in the vars_plugins directory in my role.

So far, I can create variables, and access them from my role, which is good.

What I can't do, as far as I can see, is access existing facts/vars. Do they exist at this point? Or is my plugin running at a lower level?

I was hoping they would be returned by the call to:

super(VarsModule, self).get_vars(loader, path, entities)

... but that returns None.

Is there a different kind of plugin that could do the job?

Thanks,
Richard

4

plugins in general only get the variables they need, vars_plugins are
supposed to generate them, not change them, so they get very little
access to variables in general.

For data manipulation you want to create a filter plugin, you still
need to be explicit about the data you feed it.

5

Thanks for that - this is what I've ended up with. There's probably some copy/paste boilerplate that's not needed in my case.

Does it look reasonable?
I've had a suggestion that I should avoid lineinfile, but I'm not sure what a good substitute is (other than rearranging my setup so I can template out individual files, rather than editing the existing one).

Cheers,
Richard

------task----------------
- name: update dirvish-forced-command list
   lineinfile:
     dest: "/usr/local/etc/dirvish-forced-command/default.list"
     regexp: "^{{ item }}$"
     line: "{{ item }}"
     insertafter: EOF
   delegate_to: "{{ container_host }}"
   with_items: "{{ backup_paths | gen_dfc_lines(inventory_hostname, filesystems) }}"
   tags:
     - backup

------plugin------------

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

from ansible.errors import AnsibleError, AnsibleFilterError, AnsibleFilterTypeError
from ansible.module_utils.six import string_types, integer_types, reraise, text_type

def gen_dfc_lines(backup_paths, container_name, filesystems):
     '''Generate a list of lines for dirvish_forced_command'''
     if not isinstance(filesystems, list):
         raise AnsibleFilterTypeError('filesystems should be a list')
     if not isinstance(backup_paths, list):
         raise AnsibleFilterTypeError('backup_paths should be a list')

     fixed_backup_paths =
     for path_entry in backup_paths:
         found = False
         for fs_entry in filesystems:
             path = path_entry['path']
             fs = fs_entry['container_mountpoint']
             if path.startswith(fs):
                 # remove prefix from path
                 path = '/' + path[len(fs):]
                 # use guestfs path
                 fixed_path = "/guestfs/{cname}-{suffix}{bpath}".format(
                     cname = container_name,
                     suffix = fs_entry['suffix'],
                     bpath = path
                 )
                 found = True
         if not found:
             # use rootfs path
             fixed_path = "/var/lib/lxc/{cname}/rootfs{bpath}".format(
                 cname = container_name,
                 bpath = path
             )
         fixed_backup_paths.append(fixed_path)
     return fixed_backup_paths

class FilterModule(object):
     ''' backup helper filters '''

     def filters(self):
         return {
             'gen_dfc_lines': gen_dfc_lines
         }