Should we be using molecule or ansible-test for unit/integration tests?

Get Help
, ,
1

Hello all,

I had a general question about the use-cases/purpose of molecule vs ansible-test. For some background , I’m on a team that has a large repo with many custom roles and a number of custom plugins and modules. We have not moved our repo to the galaxy model yet, though we do make use of some third party collections and of course the community collection.

We do not use ansible-test at all at this time. We do, however, use tox + pytest for our plugin and module unit tests and molecule (3 currently) for role unit and integration tests.

My question is this: I’ve become the molecule SME over time and the rest of my team finds it extremely cumbersome to use. Given where ansible’s development energy and attention is being used, if we were to start from a blank slate, which testing tool should we choose or what reasons would we choose one over the other?

One thing that jumps out at me is that it seems as though ansible-test really expects you to be in a collection dir or have pulled down the ansible repo and sourced the env script under the hacking dir. Perhaps that is my answer right there; galaxy (ansible-test) vs no galaxy (molecule).

I will be spending time refactoring our repo and tests and wanted to make sure I was starting off on the right foot.

Thanks!

1 Like
Ansible Collection Testing Strategies
2

I never used molecule, so I can’t say much about it. But let me write some general notes first.

For unit tests and sanity tests for modules and plugins I’d always use ansible-test. Of course that requires you to have a collection (I guess that’s what you mean with galaxy model?). I think it’s a good idea to migrate to using collections, even if you ‘only’ use local collections (i.e. a collections/ansible_collections/<namespace>/<name>/ directory structure below your playbook directory). This allows you to use all tooling that’s available for collections, like module/plugin sanity tests and docsite creation and docs validation with antsibull-docs.

I’m not sure how well molecule is working with collections though; I think there are collections that use molecule for role tests, and maybe also for module/plugin integration tests, so it can be done.

I’m personally using ansible-test also for role integration tests (for roles inside collections), though that doesn’t seem to be very common, but since I already use it for integration tests for modules and plugins, it’s easier to use it also for roles than using something else (with a different setup procedure) for roles.

The next part might be completely wrong since my knowledge of molecule is very limited and I don’t have hands-on experience.

I think molecule has a better test isolation than ansible-test, since you get a new environment for every role/module/plugin (or at least that’s how I understood it, maybe I’m also wrong :slight_smile: ). But that also implies that ansible-test will be faster if the setup takes enough time. But you have to watch out that you don’t end up in a situation where running all tests passes, but only running some specific tests doesn’t (because other tests install required dependencies which some other tests forget to install).

1 Like
3

Hi @ryguy,

I am using molecule for testing my roles. I have not moved the roles to collections yet, but you can find how to deal with roles inside collections in the molecule documentation , as it is the default now. I would say molecule is ideal to test your roles because you can create infra from scratch and test them locally, either with the help of containers or VMs. For my roles, I am using molecule’s docker driver to create the infra.

On the other hand, I have no experience with ansible-test. So I am a bit on the opposite side of @felixfontein. I guess that if you have your own modules and want to follow Ansible conventions, use ansible-test. To test your roles on fresh infra, use molecule.

Cheers,
Rodrigo

2 Likes
4

A thing I wanted to add here some time ago, but apparently forgot to send off:

To be honest, after reading ERROR: invalid-extension: Official Ansible modules must have - #4 by sivel, I wouldn’t recommend ansible-test for integration tests anymore.

(I’m not not recommending it, either, though. Also, a related thread: Ansible Collection Testing Strategies)

5

Just my 2ct as someone who uses both molecule and ansible-test (and is curious about tox-ansible):
For me molecule works very well with roles (in- or outside collections), especially as I can test against different operating systems at the same time (I am using Docker as well).
However, when it comes to sanity and integration tests (we do not have unit tests as of now), I prefer ansible-test. I never troubled myself with thinking about how to use molecule for these tests, but it feels leaner to me this way and you usually do not need different operating systems there.

6

Hello everyone, reviving this topic because it is very relevant to the new community.openwrt collection.

The collection’s purpose is the management of OpenWrt devices, the usual testing target platforms are not useful. OTOH, there are official docker images of OpenWrt built for the x86_64 architecture, which is perfect for us.

Whilst ansible-test integration gives us the right granularity to test each module, molecule gives us the ability of easily replicating the test for the docker images supporting multiple versions of OpenWrt. Recent versions of molecule support collections, with configuration files located by default at extensions/molecule.

See more of the discussion we had at:

The outcome of that PR was integration tests using the same role-like structure, but a mandatory runme.sh script triggering the molecule test at the collection level, but using the integration test target as a role.

1 Like
7

Hi @russoz

I saw your post Integrating `ansible-test integration` (no pun intended) and `molecule` over the holidays but unfortunately I didn’t have a chance to reply until now. I’m interested in this topic and your approach to using molecule for integration tests in the community.openwrt collection because I initially looked at doing something similar for GitHub - ansible-collections/community.beszel: Ansible content to automate the configuration of Beszel Hub and Agents. The Beszel project publishes a container image for the Beszel Hub and when I was initially looking to create the integration tests, I thought molecule would be an ideal fit. However, when I checked the documentation early on in my maintainer journey found that ansible-test integration is the method used for running integration tests.

I think this topic ties in with a comment made by @tima last year: Your Feedback Wanted: The Ansible Collection Developer and Maintainer Journey - #4 by tima - speaking to ansible-test:

The tool was originally designed to test ansible-core , not collections. As occasionally mentioned here for sometime, the long-term plan was for ansible-test to remain focused on core, with a separate tool emerging for the collection ecosystem.

@tima goes on further to mention:

… With an overhauled Molecule nearing completion, it might be a good time for the community to discuss updating the documentation to recommend Molecule as the standard tool for collection testing.

Now that community.openwrt is using molecule for collection integration tests, and I could implement similar for community.beszel, it’s probably worth discussing more in depth about how molecule could be used for collection testing (not just roles but for integration tests too), and maybe creating some convention around how that could look in a collection (repository / file structure) for molecule scenarios when wanting to use molecule for integration tests and get some documentation created around it.

Interested in your thoughts @russoz.

8

Watching this topic with interest as we have a collection that we should implement better testing on, i know the older versions of molecule could be a little cumbersome? Looking forward to re-visiting molecule to see how you have implemented the testing in those collections mentioned i.e. community.openwrt and community.beszel.

1 Like
9

Hi @dbrennand

Happy 2026! :slight_smile: I think that is not a straightforward comparison/replacement that can be made, at least not in the way things are now.

molecule is more like a specialised tool (using very large quotes, a “lower level” kind of tool) compared with ansible-test. It has its degree of complexity, of course, but it is largely restricted to the “start container - run tests - destroy container” scenario.

Whereas in ansible-test integration you have things like: the fact that tests are roles, and the fact that all targets being roles, they can be easily reused by other tests (in community.general, see test/integration/targets/setup_*). Also, the aliases file, allowing some degree of control on how the CI interacts with that particular test. And ansible-test is aware of the distinction between Ansible controller and Ansible target nodes. Amongst other things.

Also, molecule has come caveats:

  • the documentation in the official website, whilst covering most (if not all) of the switches and knobs individually, it is poor in examples of real life usage of its features, especially when testing collections. For role testing the documentation is better and there are many examples available by googling or directly searching withing github, but …
  • … the framework has gone under a significant amount of changes in the recent couple of years. There is now this “ansible-native” way of expressing the configuration, which seems to be the new preferred way of doing so. However, most of the examples in the internet are based on the “old” configuration structure. Official docs fall short of providing real-life examples for this new model as well.

All that being said, in:

We ended up creating a mixed solution: ansible-test integrated with molecule, using pytest-ansible as well. That solution has a lot of room for improvement as it is now, but it does allow us to have the best of both worlds. Better see the code in that PR than trying to explain it here.

Some things that we may change on that:

  • parameterize the OpenWrt version, so that the matrix of versions can be controlled from GH workflows
  • Maybe remove the pytest-ansible in the middle from the mix
  • Try pull-request-change-detection: true in the GH action, so that not all tests are executed for every change
  • Some way to run the tests locally against a live router (I am planning buying a new one for home, so the old one would become lab rat). Possibly an env var.
1 Like
11

@russoz Thanks for your detailed response on this :slightly_smiling_face:

the fact that all targets being roles, they can be easily reused by other tests (in community.general, see test/integration/targets/setup_* ).

Yes, in community.beszel we have one called setup_hub which is re-used across different integration tests. I don’t think there is any reason from a technical standpoint for why this kind of setup_* role target for the collection’s integration tests couldn’t be implemented in a molecule scenario instead. :thinking:

the aliases file, allowing some degree of control on how the CI interacts with that particular test. And ansible-test is aware of the distinction between Ansible controller and Ansible target nodes.

AFAIK I don’t think molecule can do any of that, so if those are required for the collection integration tests then this is definitely a gap with molecule right now. Are you using any of that functionality from ansible-test integration for your collection integration tests in community.openwrt right now?

Now that I think of it, community.beszel is using antsibull-nox sessions.ansible_test_integration_w_default_container.core_python_versions which I believe under the hood runs ansible-test integration with specific Python versions.

It appears to me that maybe a wider community conversation about whether ansible-test integration is still the best tool / approach for collection integration testing, given that molecule has the new “ansible-native” design (as you mentioned). But that being said, there are some gaps which molecule might not be able to fill at this time.

the framework has gone under a significant amount of changes in the recent couple of years. There is now this “ansible-native” way of expressing the configuration, which seems to be the new preferred way of doing so

The new “ansible-native” design with molecule IMO provides greater flexibility as you can have any Ansible tasks you want in the create or prepare sequences to be ran before the actual testing sequence (converge). Basically the equivalent of the integration setup_* targets. This includes having task files or roles (using include_role) which can be re-used across molecule scenarios. Please see below examples.

However, most of the examples in the internet are based on the “old” configuration structure. Official docs fall short of providing real-life examples for this new model as well.

Completely agree with you here, the real-world examples are lacking right now using the new design but I’m thinking we could work together to improve the documentation around this? I’m assuming you have some examples in community.openwrt we could also reference in documentation?

I have some real-world examples in community.beszel:

  1. extensions/molecule/common/create.yml - The common create stage tasks. Deploys a Docker Compose stack and creates a container using community.docker collection modules.
  2. extensions/molecule/common/beszel_public_key.yml - Common tasks to get the Beszel Hub public key which is re-used in different molecule scenarios.

These are both re-used in the agent_default molecule scenario.

  • Maybe remove the pytest-ansible in the middle from the mix

Yes, I can see from a maintenance perspective that adding another dependency / tool to the collection could increase complexity and maintenance burden etc.

Hopefully I’ll have some time this week to look at your implementation in detail tying together ansible-test, molecule and pytest-ansible in community.openwrt - very cool! I’d also be interested in discussing whether that implementation could be further developed to become a newer / alternate method for collection integration testing, or we feedback tooling improvements so molecule can be used without the need for pytest-ansible being in the mix.

Appreciate your thoughts on this topic @russoz.

12

Could you please provide more details what molecule is missing? From this description I could not understand what is meant here.

How one can use molecule with pytest-ansible? In the docs pytest is only mentioned in the context of parallel molecule scenario runs - which is not good idea if you cannot isolate molecule managed instances. I am using molecule quite extensively and I do not use nor pytest nor pytest-ansible,that is why I am wondering if I am missing something.

13

Hi @kks

Could you please provide more details what molecule is missing? From this description I could not understand what is meant here.

Sure, I was referring to @russoz comment about ansible-test being able to do all the things below and I don’t think molecule can do all of of those:

the aliases file, allowing some degree of control on how the CI interacts with that particular test. And ansible-test is aware of the distinction between Ansible controller and Ansible target nodes.

Although, thinking about this again now, I think molecule can make the distinction between the Ansible controller (which would be the host running molecule i.e localhost), and the managed node(s) would be the molecule managed instance(s) in the molecule managed inventory.

How one can use molecule with pytest-ansible ? In the docs pytest is only mentioned in the context of parallel molecule scenario runs - which is not good idea if you cannot isolate molecule managed instances. I am using molecule quite extensively and I do not use nor pytest nor pytest-ansible ,that is why I am wondering if I am missing something.

Sorry for any confusion. This was referring to what @russoz and the other community.openwrt maintainers have implemented in PR 49 - Integration test with molecule. From my understanding of it, ansible-test integration sees the runme.sh in the integration target and executes it, that installs molecule and the Docker driver for molecule, it then uses pytest to run a fixture defined in conftest.py which finally triggers the molecule scenario defined here: community.openwrt/extensions/molecule/integration_test at main · ansible-collections/community.openwrt · GitHub

Do I understand that flow right @russoz?

14

One quite important feature of ansible-test integration that molecule doesn’t seem to support either is change detection, so that a PR only runs the integration tests that affect changed content. While this doesn’t matter for smaller collections, it is a huge problem for larger collections, or collections with slow test suites.

15

Did you mean to say that GitHub action for ansible-test supports change detection, but for molecule “community” (or “official”) GitHub action is simply absent?

Or do you refer to ansibull-nox change detection feature?

https://docs.ansible.com/projects/antsibull-nox/change-detection/

And it should not support change detection. Detecting change could be done with github, git and probably other tools. For instance I archived that only some molecule test scenarios are executed by having github action that adds labels to pr for each role, module or plugin changed in that PR and another sets of actions - one for each role, module, plugin that run respective molecule scenarios if label is set (they can also be executed manually).

16

No, I mean the change detection that is part of ansible-test (and has been there for many many years). It is also supported by the GitHub action for ansible-test, but the main driver is ansible-test itself.

17

Is there documentation how it works?

does not have anything related to change detection. Google only gives the link I provided to antsibull-nox.

18

I don’t think there is any documentation, I only learned about how it worked by looking at the source and how it is used in the AZP CI scripts used in ansible/ansible and some collections.

19

Yes. I am fully aware this is very convoluted, and this might get simplified later on.

21

I’m not sure what you’re replying to. What you wrote seems completely unrelated to my post.

Your GHA workflow also doesn’t look like it will work.

(Post that this replied to was removed.)