I had a general question about the use-cases/purpose of molecule vs ansible-test. For some background , I’m on a team that has a large repo with many custom roles and a number of custom plugins and modules. We have not moved our repo to the galaxy model yet, though we do make use of some third party collections and of course the community collection.
We do not use ansible-test at all at this time. We do, however, use tox + pytest for our plugin and module unit tests and molecule (3 currently) for role unit and integration tests.
My question is this: I’ve become the molecule SME over time and the rest of my team finds it extremely cumbersome to use. Given where ansible’s development energy and attention is being used, if we were to start from a blank slate, which testing tool should we choose or what reasons would we choose one over the other?
One thing that jumps out at me is that it seems as though ansible-test really expects you to be in a collection dir or have pulled down the ansible repo and sourced the env script under the hacking dir. Perhaps that is my answer right there; galaxy (ansible-test) vs no galaxy (molecule).
I will be spending time refactoring our repo and tests and wanted to make sure I was starting off on the right foot.
I never used molecule, so I can’t say much about it. But let me write some general notes first.
For unit tests and sanity tests for modules and plugins I’d always use ansible-test. Of course that requires you to have a collection (I guess that’s what you mean with galaxy model?). I think it’s a good idea to migrate to using collections, even if you ‘only’ use local collections (i.e. a collections/ansible_collections/<namespace>/<name>/ directory structure below your playbook directory). This allows you to use all tooling that’s available for collections, like module/plugin sanity tests and docsite creation and docs validation with antsibull-docs.
I’m not sure how well molecule is working with collections though; I think there are collections that use molecule for role tests, and maybe also for module/plugin integration tests, so it can be done.
I’m personally using ansible-test also for role integration tests (for roles inside collections), though that doesn’t seem to be very common, but since I already use it for integration tests for modules and plugins, it’s easier to use it also for roles than using something else (with a different setup procedure) for roles.
The next part might be completely wrong since my knowledge of molecule is very limited and I don’t have hands-on experience.
I think molecule has a better test isolation than ansible-test, since you get a new environment for every role/module/plugin (or at least that’s how I understood it, maybe I’m also wrong ). But that also implies that ansible-test will be faster if the setup takes enough time. But you have to watch out that you don’t end up in a situation where running all tests passes, but only running some specific tests doesn’t (because other tests install required dependencies which some other tests forget to install).
I am using molecule for testing my roles. I have not moved the roles to collections yet, but you can find how to deal with roles inside collections in the molecule documentation , as it is the default now. I would say molecule is ideal to test your roles because you can create infra from scratch and test them locally, either with the help of containers or VMs. For my roles, I am using molecule’s docker driver to create the infra.
On the other hand, I have no experience with ansible-test. So I am a bit on the opposite side of @felixfontein. I guess that if you have your own modules and want to follow Ansible conventions, use ansible-test. To test your roles on fresh infra, use molecule.
Just my 2ct as someone who uses both molecule and ansible-test (and is curious about tox-ansible):
For me molecule works very well with roles (in- or outside collections), especially as I can test against different operating systems at the same time (I am using Docker as well).
However, when it comes to sanity and integration tests (we do not have unit tests as of now), I prefer ansible-test. I never troubled myself with thinking about how to use molecule for these tests, but it feels leaner to me this way and you usually do not need different operating systems there.
). But that also implies that ansible-test will be faster if the setup takes enough time. But you have to watch out that you don’t end up in a situation where running all tests passes, but only running some specific tests doesn’t (because other tests install required dependencies which some other tests forget to install).