Simplicity vs. New Modules?

Archives Ansible Project
1

The general question is: how do I decide if I should make a feature request for a new module (that I may then code myself) vs. know this is functionality I should just code on a per-play basis?

The supporting data: I am coding a somewhat ambitious project[1], the goal being to be able to set up many different large database clusters in a matter of hours. For example, I might want a 30-node Web/MySQL cluster with 2 masters, 24 DB slaves, and 4 WWW nodes. On DBs: MHA software (for reconfiguring replication topology). On WWW’s: HAproxy configured to talk to the database nodes.

So far I’ve only installed MySQL master/slaves, and I’ve needed to change /etc/hostname and /etc/hosts and set ulimits. I’ve coded up the bits in scripts[2] which act an awful lot like a module, in that they’re idempotent. Should I be requesting/making “hostname” and “ulimits” modules, or would I leave it in script form?

I foresee several other “modules” that I’ll be coding up in the near future, including a MySQL replication topology configuration. Once I delve into HBase, Cassandra, and Postgres[3] systems, I’ll probably discover other modules that need to be created.

[1] https://github.com/time-palominodb/PalominoClusterTool
[2] hostname: https://github.com/time-palominodb/PalominoClusterTool/blob/master/AnsiblePlaybooks/Ubuntu/BaseSaneSystem/files/fixHostName.pl
ulimits: https://github.com/time-palominodb/PalominoClusterTool/blob/master/AnsiblePlaybooks/Ubuntu/MySQLMasterSlaves/files/fixMySQLulimits.pl

Note that my code is pretty quick/simple/dirty (I’m on a deadline) and isn’t representative of how I’d code a module. They make assumptions about the initial state that aren’t true in general.

[3] What I’d be doing with Postgres seems not to overlap with existing Postgres modules.

2

The general question is: how do I decide if I should make a feature request for a new module (that I may then code myself) vs. know this is functionality I should just code on a per-play basis?

Not all modules, fair warning, will make it into Ansible-core. I’m generally striving to include modules that at least ~15% of the user base are likely to use each.

Of course, that’s what the ansible-resources list of interesting modules is for.

Also, yeah, you’d have to write them :slight_smile:

The supporting data: I am coding a somewhat ambitious project[1], the goal being to be able to set up many different large database clusters in a matter of hours. For example, I might want a 30-node Web/MySQL cluster with 2 masters, 24 DB slaves, and 4 WWW nodes. On DBs: MHA software (for reconfiguring replication topology). On WWW’s: HAproxy configured to talk to the database nodes.

So far I’ve only installed MySQL master/slaves, and I’ve needed to change /etc/hostname and /etc/hosts and set ulimits. I’ve coded up the bits in scripts[2] which act an awful lot like a module, in that they’re idempotent. Should I be requesting/making “hostname” and “ulimits” modules, or would I leave it in script form?

Seems to me the template module would be fine for the files in /etc/, and you can do some pretty creative things in Jinja2

I foresee several other “modules” that I’ll be coding up in the near future, including a MySQL replication topology configuration. Once I delve into HBase, Cassandra, and Postgres[3] systems, I’ll probably discover other modules that need to be created.

If the configuration is of the form “create a file, run a command” or “if this file changes, restart these services”, it probably doesn’t need to be a module at all, and can be done straight in playbooks.

The command module takes a “creates=” parameter so it can be used in idempotent ways, i.e. run only if this file does not exist.

–Michael

3

1. for editing /etc/hosts - a list member wrote a fileline module which is more or less an idempotent 'sed' (which I guess sed is, too sorta)

2. for setting the hostname - a module is probably not ridiculous it would be short: run hostname and then edit the network file in /etc/sysconfig (for red hat-ish systems)

3. for ulimits a template makes sense to me

Now in all cases - since you can make a dir named 'library' in the dir where your playbooks are - and you can draw from that path for modules - you'd still be able to provide a module - but have it be custom.

I really like this feature b/c if I need a whacked out module I can make it use the ansible module features and just make sure all of its outputs
are json (and sensible :slight_smile:

-sv

4

You both recommend using templates, but I don’t know what the /etc/hosts and /etc/security/limits.conf will look like beforehand. I just want to add in my own bits somewhere, leaving whatever was there before alone, since my tool doesn’t know anything about the infrastructure it’s been tasked to modifying. If someone has already setup some ulimits for “joebobd,” I don’t want to blow those limits away with my ulimits for mysqld.

Does the template module have some capability of saying “add in these lines to the [top|middle|bottom] of existing file, leaving all other contents alone”? If not, and now that I’ve finished typing this, maybe I want to add that feature to the “template” module.

5

https://github.com/azemon/ansible/blob/lineinfile/library/lineinfile

If someone wants to port the lineinfile module to the new module
structure for 0.7 with standardized arguments (with_underscores, dest
vs destfile, etc) and defaults, I can be persuaded to include it.

Needs tests too though, to make sure existing lines aren't clobbered
and that BOF/EOF/etc work.

Further, it must not rewrite the file unless it actually NEEDS to
change, whereas write now it looks like it always rewrites the file
and then just tells you if it changed it.

6

(Keep in mind as I write this I’m pondering writing it myself, so don’t take this as a request for you to do work) The module you link is great, but “line in file” is just a subset of “blob of ASCII in file,” and that implementation you link doesn’t have any templating capability (does it?).

Here’s what I want to end up with (the extra comments help delimit the blob for future reference):

joebobd files 1024

PalominoClusterTool ulimits for MySQL delimiter BEGIN

mysql files 40000
mysql processes 120000

PalominoClusterTool ulimits for MySQL delimiter END

freddyd files 9001

I would hope to have templates/ulimits.blob contain something like:

mysql files {{ mysql[“ulimits_files”] }}
mysql processes {{ mysql[“ulimits_processes”] }}

The module is given the start/end delimiters as parameters. The blob is run through the templating engine first. Then, if the resulting blob (bracketed by the delimiters) doesn’t exist in the file, it can be placed at the location specified. If it is in the file, but is changed, then the old is removed, and the new is placed at the location specified. It seems little complexity to add the option to leave the new blob in the same place as the old blob.

I wager the change from “line in file” to “blob in file” adds little complexity (would need to handle begin/end blob marker handling - the templating capability could be left off entirely if it adds too much complexity). And it encapsulates the special case of line in file.

In summary:

module: blobinfile
parameters:
src; required=no; [file containing template data to substitute]
dest; required=yes; [file where blob substitution will be made]
line; required=no; [one line of ASCII text to substitute]
location; required=no; default=append; [“prepend”, “append”, or regexp defining where to put blob]
startdelimiter; required=no; default=\n; [the delimiter that defines start of blob for replacing existing]
enddelimiter; required=no; default=\n; [define the end of blob for replacing existing]

Unsure how to handle if they pass in >1 of “line” and “src”. Perhaps make a lineinfile module that calls blobinfile to disambiguate. Unsure how to explain that the blob won’t be moved around if there are no substantive changes (aka the existing blob won’t be moved from the middle of the file to the end of the file just because you specified “append”).

I’m struggling to think, even with the additional complexity, how this could take more than a few hours to make the initial rev. Adding in a requirement to do stream processing and leave the file untouched if no modifications doesn’t seem that hard, either.

Comments?

7

Comments?

The merging of user and centrally managed data is not something I'm a
huge fan of, so ... yeah, not really wanting to complicate it further.
  I also think you could probably just call the line-in module
multiple times.

If it were me, I would probably just ask the user to save their
preferences in /etc/foo/config.d/user.conf (some place different),
call the template module to push down the other piece of the file, and
use the assemble module to merge them back together and save the file
as /etc/foo.conf/.

It's already true that you have access to Jinja2 and the full
templating variable namespace in action lines, so you might not need
to treat things as a template on the module side of things.

--Michael