Filesystem "swiss army knife": fsbuilder; Looking for feedback

Project Discussions
,
1

Many times I find the ansible.builtin methods for dealing with files a bit obtuse. In particular, they require grouping by operation (template, file, directory, etc), and also there’s a lot of boilerplate involved if you need to set up a directory, template a config file in it, and remove a file, for example.

For several years I’ve been playing with this idea of a “filesystem mega module”, having implemented it both as modules and as roles I can import. I’ve been pretty happy with the ergonomics of this module, so I wanted to wrap it up in a bow and spank it on the bottom and send it out into the world.

The primary goal is to allow the grouping of operations logically, so it’s clear that you’re creating a directory, then putting a file in it, triggering a notify, removing a file, without having to search around other tasks in the file and making sure that they stay in order (easier because of their proximity).

See Ansible Galaxy for the docs and links to the github and the docs site.

Here’s an example:

Overview: Settings you specify after the “fsbuilder:” line are the defaults. The default state is “template”, and the default “src” for a template is “basename(dest) + ‘.j2’”.

- name: Deploy myapp - comprehensive example with loop
  linsomniac.fsbuilder.fsbuilder:
    owner: root
    group: myapp
    mode: a=rX,u+w
  loop:
    - dest: /etc/myapp/conf.d
      state: directory
    - dest: /etc/myapp/config.ini
      validate: "myapp --check-config %s"
      backup: true
      notify: Restart myapp
    - dest: /etc/myapp/version.txt
      content: "version={{ app_version }}"
    - dest: /etc/myapp/static.dat
      state: copy
    - dest: /etc/myapp/current
      src: /opt/myapp/releases/v2.1
      state: link
    - dest: /etc/myapp/.last-deploy
      state: touch
    - dest: /etc/myapp/legacy.conf
      state: absent

I use this extensively in our playbooks, I’ve got around 50 usages that I run on a daily basis (I respin half of our dev and staging environment every morning). So it’s at least a little battle-tested.

Thanks for any feedback you care to give.

2

Hello, Sean

Thank you for putting this out to the world and welcome to the ansible community.

I actually find this (the ansible way) more conviniet - when one writes playbooks, with each action and module they convey some meaning. Having one module or role that does it all goes agains this and blurs the meaning of each step in the playbook or role. You should not be concerned with my opinion.

Just my point of view here - I love ansible for its explicitnes - nothing (what matters) is hidden from the user. Here there are a lot of logic is implicit - hidden from the user. For me it was not obvious at all what the code does from the playbook you provided as an example.

As I understand module (plugin) will use variables from the loop, in particular item variable to get all it’s parameters. This requires a lot of testing:

  1. In a way you are breaking standard ansible functionality. How to use this module in a loop?
  2. What if this module is already in a loop (include tasks loop)? What item variable will take precedence? This is not just for you to test, but for users to understand. There is an ansible way, but in your collection there is seems to be a different way - some user will be confused which approach will be used where (I am confused)
  3. Ansible is declarative and here result will depend on the order in which elements are in the list. In your example it is literal list, but imagine this list is dynamically calculated. There might be no control over in which order elements appear, or it might differ from time to time. This module will fail depending on the order, or provide unexpected results.

Saying all that, I still love your approach - you made ansible work for you in your own use case. And this is what ansible is all about (from my POV). Otherwise there wouldn’t be hundreds collections and roles. I would like to steer you implementation, if you do not mind.

Before running a module (for instance one from ansible.builtin) you can enhance it’s arguments in the action plugin.

Here is an example of action plugin that does exactly what ansible.builtin.file does and has all the same arguments.

from __future__ import absolute_import, division, print_function

__metaclass__ = type

from ansible.plugins.action import ActionBase


class ActionModule(ActionBase):
    def run(self, tmp=None, task_vars=None):
        super(ActionModule, self).run(tmp, task_vars)
        module_args = self._task.args.copy()
        ## --->>> HERE you can change module_args <<<---
        return self._execute_module(
            module_name="ansible.builtin.file",
            module_args=module_args,
            task_vars=task_vars,
            tmp=tmp,
        )

Before executing ansible module you can change variables - in your case add your own logic with implicit file names, etc.
With that approach: 1. you will depend on ansible standard functionality - you do not have to develop it on your own. 2. you can change the logic of the module. 3. you can call different ansible modules depending on the action plugin input variables (your case). 4. some logic might be simplified

In your code

    # Map state names to handler methods
    STATE_HANDLERS: dict[str, str] = {
        "copy": "_handle_copy",
        "template": "_handle_copy",  # Action plugin converts template -> copy
        "directory": "_handle_directory",
        "exists": "_handle_exists",
        "touch": "_handle_touch",
        "absent": "_handle_absent",
        "link": "_handle_link",
        "hard": "_handle_hard",
        "lineinfile": "_handle_lineinfile",
        "blockinfile": "_handle_blockinfile",
    }

You can remove handlers you created (and have to maintain) to ansible modules - they are already maintained.

I hope you will find this feedback useful.

1 Like
3

Just my two cents. I pretty much like this approach and have been using the similar approach for years now. My solution thou is in a form of a role, using all standard Ansible modules, no custom code.

The reason I like this approach is that it is simple to understand, simple to process in the head and very obvious. Reading trough a bunch of tasks that each deal with a single file is painful for me. I also found that users of my solutions (coworkers) quickly got a grasp of it. Much easier than writing a long list of tasks. In the end, they just want to do something on a bunch of files, create some, template some and a simple list of files and possibly some action for each item is way easier to write.

1 Like
4

Thanks for the thorough reply, I really appreciate it. I’m sure there are cases and users where the ansible.builtins are preferred, so we have that option in those cases and fsbuilder for other cases/users where that is preferred. Everyone wins. :slight_smile:

WRT using it within a loop, a previous version had a “loop_var” where you could specify an alternative name, I don’t use that anywhere but I could definitely see that being useful.

I also had a version that instead of using a task “loop:” it had a “files:” attribute of the fsbuilder module and you’d just list the objects there. ISTR that made some portions of the implementation trickier, but would fit the model of like the “apt: packages:” switching from loop to taking a list of packages. Maybe I should consider going in that direction again.

This evolved from a pattern I was using like:

template:
  src: "{{ item.src | default(basename(item.dest) + '.j2'}}"
  owner: "{{ item.owner | default('root') }}"
  group: "{{ item.group | default('root') }}"
  mode: "{{ item.mode | default('a=rX,u+w') }}"
loop:
  - dest: /etc/myapp/conf.ini
  - dest: /etc/myapp/secrets.ini
    group: secrets
    mode: "a=-,u=rw,g=r"

So that I could consolidate all my templating into a single data structure and not have to break it out into a bunch of different blocks based on some small differences like a different group or mode in otherwise similar sets of files.

I have setups where I’m dealing with 10-20 directories and files and the difference is pages and pages of boilerplate vs something that’ll fit on a single screen, which I find easier to manage. Particularly in relation to ordering, where I can see the whole ordering or I have to jump around and make sure I’m preserving ordering when I can only see only a part of it on my screen at once.

WRT the ordering of dynamic objects, yes I could see that being a problem. I’d say that this module is incompatible with having a “loop” input that has non-deterministic ordering, if the objects within that sequence have ordering dependencies.

On to your suggestions about leveraging Ansible modules to do more of the heavy lifting, that was an approach I took on one of the first versions of this module, 2-4 years ago, and I had some success at that but in the end the code was not nearly so clear as the example you showed and it was hard for me to maintain, I forget the exact reasons that I abandoned that for the “import role” method.

I’ll have to play around with that based on what you’ve said here and see if I can get something working that is more a thin layer on top of the ansible modules.

Thanks again for the feedback!