Using custom module_utils-path for shared code

Get Help
,
1

Hi all,

I wrote two callback-plugins which use an identical log-method, so I would like to share this code. Doing so, I moved that code to an external file which I would like to import.
Currently, I always get an error, that this external file can not be found. Any help would be apricated.

My structure does look like this

workspace
 |- playbooks
     |- ansible.cfg
	 |- my.yml
	 |- playbooks.yml
 |- collections
 |- plugins
     |- action
	 |- callback
	     |- general
			 |- log_errors.py
			 |- log_hosts.py
	 |- filter
	 |- lookup
 |- roles
	 |- my
	 |- roles
	 |- ...

For using this structure, the ansible.cfg in the playbooks-dir has be modified:

roles_path    = ../roles
collections_paths = ../collections
action_plugins     = ../plugins/action
callback_plugins   = ../plugins/callback
lookup_plugins     = ../plugins/lookup
filter_plugins     = ../plugins/filter

The above works fine, also the custom callback-plugins work if I don’t share the code.

For using the shared code I did the following changes.

Added the file containing the shared code into a new directory:

workspace
 |- module_utils
     |- logging.py

Updated the path in the ansible.cfg:

 module_utils   = ../module_utils

In my callback-plugins I added the following line:
from ansible.module_utils.logging import write_log

If I execute a playbook, I get the following warning:

[WARNING]: Skipping plugin (<some_path>/plugins/callback/general/log_errors.py) as it seems to be
invalid: No module named ‘ansible.module_utils.logging’

If however, I copy the file from my customer folder module_utils to /usr/local/lib/python3.8/dist-packages/ansible/module_utils/logging.py it works.

Any Ideas what I am doing wrong?

Update: Also renaming my custom module_utils-folder to something else (while updating path in ansible-cfg and import) did not do the trick.

2 Likes
2

Have you tried
from …module_utils.controller_api import ControllerAPIModule

no slashes
Here is where I pulled the reference

you can also import module utils from other collections
from ansible_collections.awx.awx.plugins.module_utils.awxkit import ControllerAWXKitModule

1 Like
3

Interesting syntax, never seen before. Didn’t know that it could be valid.

If just tried

from ..module_utils.logging import write_log

and it returns

[WARNING]: Skipping plugin (<path>/plugins/callback/general/log_errors.py) as it seems to be invalid: No module named 'ansible.plugins.module_utils'

Maybe it works in the linked example because it is inside of a collection which somehow handles the scope differently. I will have a closer link to the repo and see if I can find something similar.

Additionally I also checked with dump to check if the value is set, and this seems also to be fine (also did changes to the ansible.cfg and the path gets updated accordingly):

$ ansible-config dump | grep util
DEFAULT_MODULE_UTILS_PATH(/<path>/playbooks/ansible.cfg) = ['/<path>/module_utils']
1 Like
4

A similar question was recently asked in https://groups.google.com/g/ansible-project/c/rBFuX3MCapk - the answer is basically no. It would work if you put your plugin into a collection, though.

5

Heh, that was me. (@felixfontein I still owe you some docs about shared code. I have not forgotten!)

In my case, I wrote a custom test - sch_distinct - and a custom filter - sch_uniqcomb - and I needed to use some code from the test in the filter. I eventually settled on this structure in a “local” (to my project) collection:

collections/ansible_collections/mw/tablinx/plugins
├── filter
│   └── tablinx.py
└── test
    └── tablinx.py

(And, yes, I gave all the files the same name. If we were all named Bob no-one would ever forget a name.)

So the filter uses the following import to pull in the test code’s distinct function:

from ansible_collections.mw.tablinx.plugins.test.tablinx import distinct

The nice extra is that out in user land, it’s all nicely scoped to the mw.tablinx namespace. I’m starting to think about moving the bulk of my project-local bits into this local collection so everything gets scoped.

schedules_all_uc: "{{ schedules_all | mw.tablinx.sch_uniqcomb }}"
whackamole: "{{ foo is mw.tablinx.sch_distinct }}" # The sch_distinct test uses distinct() on the back end.
5 Likes
6

Thanks a lot for your help!

So why do I then can modify the module_utils-path at all? Would it work for actual modules but not for plugins (even without being part of a collection?)

7

Local module_utils can only be used in local modules.

1 Like
8

Wha…!? After all that, moments ago I found 7.7. Keep plugin entry files to a minimal size. in which it mentions

Move reusable functions and classes, such as those for data validation or manipulation, to a module_utils/ (for Ansible modules) or plugin_utils/ (for all other plugin types) file and import them into plugins.

This is the first time I’ve heard about plugin_utils/. I’m happy with the local collection solution, but I’ve really got to investigate this plugin_utils/ shared code functionality. sigh It never ends!

1 Like
9

plugin_utils is an “officially reserved name” in collection plugins/ directories, but it has no native support from ansible-core’s side. The “officially reserved name” basically means that everyone agrees on that plugins/plugin_utils/ in collections is used for that, but you could also use plugins/random_other_name/ instead (ansible-core doesn’t care how the folder is called, as long as it doesn’t collide with the special ones it supports for modules, module_utils, and supported plugin types).

So plugin_utils/ won’t work outside collections. And if you already use a local collection, you can also use plugins/module_utils/.

(The main reason plugin_utils/ is recommended for plugin-specific code that isn’t used by modules is that you cannot import from ansible.xxx besides ansible.module_utils.xxx in Python code in plugins/module_utils/. So the idea is to put all code that needs to import other things from ansible in plugins/plugin_utils/, since that directory has no such restrictions.)

The official place where plugin_utils is specified is in the requirements document for collections included in Ansible: Ansible community package collections requirements — Ansible Documentation

2 Likes
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.