# Complex process to generate template/lineinfile data?

**URL:** <https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172>\
**Category:** Ansible Project\
**Created:** [May 15, 2021, 6:39am UTC](https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172 "2021-05-15T06:39:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Richard\_Hector](https://avatars.discourse-cdn.com/v4/letter/r/db5fbb/32.png) [@Richard\_Hector](https://forum.ansible.com/u/Richard_Hector)\
**Post date:** [May 15, 2021, 6:39am UTC](https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172/1 "2021-05-15T06:39:04Z")

</div>

Hi all,

Well, it's not really that complex, but seems complex to do within the constraints of ansible/jinja.

My scenario is that I run dirvish to back up multiple containers on multiple hosts. The containers are backed up via the filesystems on the hosts.

The root filesystem of the container lives under /var/lib/lxc/\<container\>/rootfs.

Sometimes, extra filesystems are also used; they live under /guestfs/\<container\>-\<name\>, where 'name' is a possibly shortened version of the mountpoint.

Relevant sections of host\_vars files look like this:

filesystems:  
&nbsp;&nbsp;&nbsp;- { index: 1, suffix: srv, size: 2, container\_mountpoint: '/srv/' }  
backup\_paths:  
&nbsp;&nbsp;&nbsp;- { shortname: 'ROOT', path: '/' }  
&nbsp;&nbsp;&nbsp;- { shortname: 'var', path: '/var/' }  
&nbsp;&nbsp;&nbsp;- { shortname: 'home', path: '/home/' }  
&nbsp;&nbsp;&nbsp;- { shortname: 'srv', path: '/srv/' }

So when I'm setting up backups, I need to treat any backup path for this container starting with '/srv/' differently, in that I need to backup something under /guestfs/ rather than under /var/lib/lxc.

I have a template that does this for the dirvish config:

client: {{ hostvars[container\_host].fqdn }}  
{% set ns = namespace(extra\_fs='') %}  
{% for mountpoint in filesystems %}  
{% if item['path'] | regex\_search(mountpoint['container\_mountpoint']) %}  
{% set ns.extra\_fs = mountpoint %}  
{% endif %}  
{% endfor %}  
{% if ns.extra\_fs %}  
tree: /guestfs/{{ inventory\_hostname + '-' + ns.extra\_fs['suffix'] + '/' + item['path'] |regex\_replace(ns.extra\_fs['container\_mountpoint'], '') }}  
{% else %}  
tree: /var/lib/lxc/{{ inventory\_hostname }}/rootfs{{ item['path'] }}  
{% endif %}  
rsh: ssh -i /root/.ssh/id\_rsa\_dirvish {{ hostvars[container\_host].fqdn }}

{% for backup\_path in (backup\_paths | select('search', item['path'] + '.\*')) if not (backup\_path['shortname'] == item['shortname']) %}  
{% if loop.index == 1 %}  
excludes:  
{% endif %}  
&nbsp;&nbsp;&nbsp;{{ backup\_path['path'] }}\*  
{% endfor %}

As you can see, that's quite complicated.

Then in addition, I need to add to a list of acceptable paths on the host (for my ssh forced command script to read).

That looks like this (for the same container described in the host\_vars above):

...  
/var/lib/lxc/example-web1/rootfs/  
/var/lib/lxc/example-web1/rootfs/var/  
/var/lib/lxc/example-web1/rootfs/home/  
/guestfs/example-web1-srv/  
...

That isn't a template, because the file contains paths for all the containers. I suppose I could modify my script to read any files in a directory instead of a single file, but that would be a separate project, and the template would be just as nasty as the one above.

I'm not even sure where to start with generating lines to add to the file.

Any suggestions?

Is this stuff reasonable to do with ansible, and a templating language?

Do I need to write a custom plugin of some kind?

If I had a dynamic inventory, then I could probably generate extra variables at that stage, but that's further away too.

Cheers,  
Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Hector](https://avatars.discourse-cdn.com/v4/letter/r/db5fbb/32.png) [@Richard\_Hector](https://forum.ansible.com/u/Richard_Hector)\
**Post date:** [May 16, 2021, 4:24am UTC](https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172/2 "2021-05-16T04:24:40Z")

</div>

I should give lots of credit to the nice people on irc who helped me get as far as I have.

I've also thought that if I could pass data (as json perhaps?) to stdin of a local script, and capture the output (a list of strings, as json again?), that would probably work ok - I could then write the script in a language I'm more familiar with, such as perl.

Is that doable?

Thanks,  
Richard

---

<div class="post-metadata">

**Author:** ![Richard\_Hector](https://avatars.discourse-cdn.com/v4/letter/r/db5fbb/32.png) [@Richard\_Hector](https://forum.ansible.com/u/Richard_Hector)\
**Post date:** [May 17, 2021, 11:09am UTC](https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172/3 "2021-05-17T11:09:42Z")

</div>

I'm experimenting with writing a vars plugin, referring to these:

[https://docs.ansible.com/ansible/latest/dev\_guide/developing\_plugins.html](https://docs.ansible.com/ansible/latest/dev_guide/developing_plugins.html)  
[https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/vars/host\_group\_vars.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/vars/host_group_vars.py)

I've put it in the vars\_plugins directory in my role.

So far, I can create variables, and access them from my role, which is good.

What I can't do, as far as I can see, is access existing facts/vars. Do they exist at this point? Or is my plugin running at a lower level?

I was hoping they would be returned by the call to:

super(VarsModule, self).get\_vars(loader, path, entities)

... but that returns None.

Is there a different kind of plugin that could do the job?

Thanks,  
Richard

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\
**Post date:** [May 17, 2021, 2:24pm UTC](https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172/4 "2021-05-17T14:24:10Z")

</div>

plugins in general only get the variables they need, vars\_plugins are  
supposed to generate them, not change them, so they get very little  
access to variables in general.

For data manipulation you want to create a filter plugin, you still  
need to be explicit about the data you feed it.

---

<div class="post-metadata">

**Author:** ![Richard\_Hector](https://avatars.discourse-cdn.com/v4/letter/r/db5fbb/32.png) [@Richard\_Hector](https://forum.ansible.com/u/Richard_Hector)\
**Post date:** [May 19, 2021, 4:56am UTC](https://forum.ansible.com/t/complex-process-to-generate-template-lineinfile-data/35172/5 "2021-05-19T04:56:43Z")

</div>

Thanks for that - this is what I've ended up with. There's probably some copy/paste boilerplate that's not needed in my case.

Does it look reasonable?  
I've had a suggestion that I should avoid lineinfile, but I'm not sure what a good substitute is (other than rearranging my setup so I can template out individual files, rather than editing the existing one).

Cheers,  
Richard

------task----------------  
- name: update dirvish-forced-command list  
&nbsp;&nbsp;&nbsp;lineinfile:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;dest: "/usr/local/etc/dirvish-forced-command/default.list"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;regexp: "^{{ item }}$"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;line: "{{ item }}"  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;insertafter: EOF  
&nbsp;&nbsp;&nbsp;delegate\_to: "{{ container\_host }}"  
&nbsp;&nbsp;&nbsp;with\_items: "{{ backup\_paths | gen\_dfc\_lines(inventory\_hostname, filesystems) }}"  
&nbsp;&nbsp;&nbsp;tags:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- backup

------plugin------------

from \_\_future\_\_ import (absolute\_import, division, print\_function)  
\_\_metaclass\_\_ = type

from ansible.errors import AnsibleError, AnsibleFilterError, AnsibleFilterTypeError  
from ansible.module\_utils.six import string\_types, integer\_types, reraise, text\_type

def gen\_dfc\_lines(backup\_paths, container\_name, filesystems):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'''Generate a list of lines for dirvish\_forced\_command'''  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if not isinstance(filesystems, list):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise AnsibleFilterTypeError('filesystems should be a list')  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if not isinstance(backup\_paths, list):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise AnsibleFilterTypeError('backup\_paths should be a list')

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fixed\_backup\_paths =   
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for path\_entry in backup\_paths:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;found = False  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for fs\_entry in filesystems:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;path = path\_entry['path']  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fs = fs\_entry['container\_mountpoint']  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if path.startswith(fs):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# remove prefix from path  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;path = '/' + path[len(fs):]  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# use guestfs path  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fixed\_path = "/guestfs/{cname}-{suffix}{bpath}".format(  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cname = container\_name,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;suffix = fs\_entry['suffix'],  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bpath = path  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;found = True  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if not found:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;# use rootfs path  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fixed\_path = "/var/lib/lxc/{cname}/rootfs{bpath}".format(  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cname = container\_name,  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;bpath = path  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fixed\_backup\_paths.append(fixed\_path)  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return fixed\_backup\_paths

class FilterModule(object):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;''' backup helper filters '''

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;def filters(self):  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'gen\_dfc\_lines': gen\_dfc\_lines  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}