Sharing code between modules

I'm writing a set of modules that do a bunch of things with the TeamCity CI server. I have 2 such modules so far and they use some common code - currently these two modules have a lot of copied code and it's already getting a little troublesome to modify the same code in 2 places. I'd prefer to have these modules import from some common module, but ansible modules are kind of a special beast because they get transferred to the remote server.

There is some special magic that runs server-side which detects imports of special things like ansible.module_utils and actually copies the contents of the utils stuff into the module code before copying it to the remote machine. From a glance this looks to be specific to ansible.module_utils and other internal stuff like that.

I'm wondering if there has been any talk of extending this so that it could be more general and could be used to import things that don't live in ansible core?

For example, I'd have in a "library" directory:

teamcity_common.py
teamcity_project.py
teamcity_build_config.py

The latter 2 would be able to import from the first one.

I'm not sure I would choose to intercept "import" as changing the behavior of a built-in Python statement strikes me as confusing. I'd probably create some kind of pragma that gets embedded in a Python comment - e.g.:

    ##ansible: include ansible.module_utils, teamcity_common

Any suggestions would be appreciated.

Marc

gic that I referred to before is probably well-known to the Ansible core developers and people who have poked around in internals a fair amount, but for those who aren't familiar with it but are curious:
https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/module_common.py

My proposal is essentially to allow module developers a way to get some of this magic, by using an explicit directive, such as:

##ansible: include teamcity_common

which would copy the contents of teamcity_common.py into the current module at that point.

I chose this weird syntax over `import` in an attempt to make it clearer that what this does is something special and ansible-specific and it's not an actual Python `import`, as it's doing text substitution and doesn't really involve the `PYTHONPATH` and such.

Ansible can already do this, but it currently requires that the shared code be copied into the module_utils directory in your Ansible installation.

Your other option is to make it a real Python module, and install it on the remote nodes so that your Ansible modules can import it.

Yeah those are the options I considered for now and I had one other which is that I could write my code in separate files and then have a Makefile or something that concatenates them together, essentially doing what Ansible does, but doing it a "compile-time".

I guess if my modules turned out to be generally useful and good, then I could submit a PR and that could include a file that gets installed in the module_utils directory. But long-term it seems a little limiting that something has to go into core to be able to do that, so I wonder if there would be interest in a PR that would extend the magic a bit as previously mentioned?

All,

Long time listener, first time caller. What about a decorator on the module class to indicate the extra modules/assets required to be transferred with a module?

Thanks,
-John

That's an interesting idea. I'm not a core dev though so I'd wait for one of them to weigh in before playing with it myself.

I don't think the decorator would work except as something else that
overrides real python syntax to specify substitutions when the module
is constructed to send over the wire. So it wouldn't actually gain us
anything. If you think that it could work differently feel free to
expand on the idea :slight_smile:

We have talked about a separate directory to search for module_utils
type snippets of code using the same mechanism as ansible's current
replacer. Not a high priority because:

1) Most sites can already do this simply by dropping their extra files
into /usr/lib/python2.7/site-packages/ansible/module_utils/ (use the
path that is appropriate for your ansible install) Since the only
thing that makes the asible control machine special is that you have
access to credentials on that machine this covers most use cases
(there are a few that it doesn't -- where you don't have privileges to
modify that directory and are not allowed by policy from running from
a checkout/separate tarball/your own machine).

2) I personally want to deprecate replacer in favour of real imports
(from a limited directory structure).I've got some code that starts
down this path but it definitely won't make the cut for 2.0...
Possibly something for 2.1. This isn't mutually exclusive with people
having a directory that parallels module_utils but I wouldn't want
people to start adopting bad habits due to relying on what replacer
does rather than what a real import would do.

As for what tokens to use to activate module replacer... probably the
most natural is a second import that looks like the current one but
uses a different directory. For instance:
"from ansible.user_module_utils.FOO import *"

I share your dislike of overloading import but it's not worse than
what we already have. If you want to go the comment route, replacer
used to use comments before the "from ansible.module_utils[..] import
*" string was created. It used (and still does although it's
deprecated): "#<<INCLUDE_ANSIBLE_MODULE_COMMON>>" to substitute
basic.py into a module. Maybe that could be expanded to more than
basic.py or perhaps it is best consigned to the dustbin as it's better
to have just one thing that people have to know than two that do the
same thing (and at this point, we can't get rid of the present
replacer syntax for backwards compatibility reasons).

If you're interested in ziploader, I put together some pieces to prove
it could work last time I had some vacation:

https://gist.github.com/abadger/d3592c1c9ef37ca54db0

That's a standalone script that creates a file which can be executed
by a python interpreter via a pipeline or as an argument on the
command line. it's one of maybe three parts that would have to come
together to make a working proof of concept. The others are changing
modules_common modify_module code so that we detect when a module
needs to use a snippet from module_utils and then add it to the zip
file's payload (and recursively scan those files for other module
snippets from module_utils) and then plugging all this into the
existing framework so that certain things can be configured from
hostvars (compressing the zip data is likely something that needs to
be configurable in case a remote host doesn't have a python with
zlib), making it coexist with the existing replacer strategy for
backwards compatibility, and then modifying the module_utils and
module code so that it can be used by zip_loader instead of replacer
(mostly changing things so we don't rely on variables being present at
the global scope). The latter is the kind of thing I mean when I talk
about not wanting a user-expandable directory to allow bad habits.

There's a few ways to make ziploader coexist with replacer. A few
weeks ago I talked with a few other people about making replacer only
respond to "from ansible.module_utils.FOO import *" (which is what it
uses now) and letting all other things that import from
ansible.module_utils use the real import mechanism (from
ansible.module_utils import FOO and from ansible.module_utils.FOO
import BAR). Another way would be to make things explicit: "from
ansible.module_utils.FOO import *" would use replacer "from
ansible.module_utils2.FOO import *" would use real imports. I think
#2 will lead to duplication of module_utils code so I currently am
favoring #1.

Right now ziploader is on hold for me -- lots of work to do to get
ansible-2.0 out the door so I can't be spending too much time on 2.1
stuff yet.

-Toshio