We have custom lookup plugin, which internally loads the “dig” plugin, like this:
from ansible.plugins.loader import lookup_loader
dig_module = lookup_loader.get("community.general.dig")
We also have unit tests (using the Python unittest module) that exercise this lookup plugin directly, outside of Ansible.
This all worked fine under Ansible 2.14. Under 2.16.2 it still works when run as part of an Ansible playbook, but in the unit tests the lookup_loader.get() returns None and prints the following warning:
[WARNING]: errors were encountered during the plugin load for community.general.dig: ['expected str, bytes or os.PathLike object, not NoneType']
This can be reproduced just by running the above two lines in a Python shell. (We’re using Python 3.12.1.)
I’m guessing that some extra initialization code is now necessary, which Ansible runs when it starts, so we need to replicate that in our unit tests, but what? I’m not even sure how to get a stack trace to find where exactly this error is coming from.
I don’t get that warning, but I can reproduce the problem.
Do you run the unit tests with ansible-test? Several of the lookup plugin unit tests in community.general (and likely other collections) use lookup_loader.get(), and these still pass.
I guess the problem is that you are not running your tests with ansible-test, and thus outside of Ansible. This has never been supported, so that it did work in the past is not a promise that it will still work now.
I would suggest you use ansibe-test to run the unit tests (and organize the unit tests similar to how they are organized in other collections so ansible-test is happy with them).
I tried to get that working, but without success. We don’t have a “collection” for our plugin - it’s just in the lookup_plugins directory under the playbook directory, and I really don’t want to make any major changes to this just to fix one little unit test.
Finally I managed to fool it by renaming our ansible directory to “ansible_collections” and creating a “fakenamespace/fakecollection/tests” directory.
Then it still failed with “FATAL: Command “git ls-files -z --cached --others --exclude-standard” returned exit status 128.” I don’t understand why git is involved here at all. OK, I worked around that. Installed pytest. It still fails:
$ ansible-test units -v
Configured locale: en_US.UTF-8
WARNING: Skipping unit tests on Python 2.7 because it is only supported by module/module_utils unit tests. No module/module_utils unit tests were selected.
WARNING: Skipping unit tests on Python 3.8 because it is only supported by module/module_utils unit tests. No module/module_utils unit tests were selected.
WARNING: Skipping unit tests on Python 3.9 because it is only supported by module/module_utils unit tests. No module/module_utils unit tests were selected.
Creating container database.
WARNING: Skipping unit tests on Python 3.10 because it could not be found.
WARNING: Skipping unit tests on Python 3.11 because it could not be found.
Unit test controller with Python 3.12
Initializing "/tmp/ansible-test-uj5wyphx-injector" as the temporary injector directory.
Injecting "/tmp/python-dsobpbb8-ansible/python" as a execv wrapper for the "/home/em/.direnv/python-3.12.1/bin/python" interpreter.
Stream command: pytest -r a -n auto --color yes -p no:cacheprovider -c /home/em/.direnv/python-3.12.1/lib/python3.12/site-packages/ansible_test/_data/py ...
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: -n tests/unit/test_netplan.py tests/unit/test_plugins.py
inifile: /home/em/.direnv/python-3.12.1/lib/python3.12/site-packages/ansible_test/_data/pytest/config/default.ini
rootdir: /media/sf_work/lops2/ansible_collections/fakenamespace/fakecollection
FATAL: Command "pytest -r a -n auto --color yes -p no:cacheprovider -c /home/em/.direnv/python-3.12.1/lib/python3.12/site-packages/ansible_test/_data/pytest/config/default.ini --junit-xml /media/sf_work/lops2/ansible_collections/fakenamespace/fakecollection/tests/output/junit/python3.12-controller-units.xml --strict-markers --rootdir /media/sf_work/lops2/ansible_collections/fakenamespace/fakecollection -v tests/unit/test_netplan.py tests/unit/test_plugins.py" returned exit status 4.
So I’ve given up on ansible-test now. I’m sure there is just some one-line tweak I need to add to work around the expected str, bytes or os.PathLike object, not NoneType warning. Yes, I know it’s not the supported way, but it seems that the supported way is very, very awkward for my (and apparently many other people’s) use case.
I dug into the code a bit more to see what it’s doing:
>>> lookup_loader._find_fq_plugin("community.general.dig", ".py", plugin_load_context=PluginLoadContext())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/em/.direnv/python-3.12.1/lib/python3.12/site-packages/ansible/plugins/loader.py", line 552, in _find_fq_plugin
pkg_path = os.path.dirname(pkg.__file__)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen posixpath>", line 181, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType
So now I have a stack trace. This error happens because
sys.modules.get('ansible_collections.community.general.plugins.lookup').__file__ is None
Adding an empty __init__.py file to the directory returned by .__path__ for that module (“~/.direnv/python-3.12.1/lib/python3.12/site-packages/ansible_collections/community/general/plugins/lookup” in my case) works around the problem and the unit test runs.
It’s quite a silly thing to have to do, but hey, it works! Maybe manipulating sys.path in the unit test would be a better workaround - I’m not sure.
I’m not sure if that code could just use pkg.__path__ instead of os.path.dirname(pkg.__file__). They’re the same on in my case, but I don’t know enough Python to say if they’d always be the same.
ansible-test is for collections (and ansible-core), so if you don’t have one, you cannot use it.
I would really suggest to move your plugin into a collection (you can have a local collection foo.bar local to your playbooks in a collections/ansible_collections/foo/bar subdirectory, you don’t need to keep it in a separate repository or even build and publish it).
That’s because you are not using ansible-core’s Python module (as in: Python module, not Ansible module) loader, which doesn’t need the __init__.py files to be present, but you’re (implicitly) using the plain Python module loader. I guess for older versions that one was added magically when using the plugin loader, but now it isn’t.
You should probably run init_plugin_loader() from ansible.plugins.loader before using the loader, that seems to initialize everything:
from ansible.plugins.loader import lookup_loader, init_plugin_loader
init_plugin_loader()
dig_module = lookup_loader.get("community.general.dig")
If I run that in our filter plugin I get a warning:
/home/em/.direnv/python-3.12.1/lib/python3.12/site-packages/ansible/plugins/loader.py:1479: UserWarning: AnsibleCollectionFinder has already been configured
warnings.warn(‘AnsibleCollectionFinder has already been configured’)
so I added init_plugin_loader() to the unit test only, and now it passes without the __init__.py file.
