Hey Ansible Community,
As Don shared in his Ansible community roadmap 2026 post, and as mentioned in Clarifying the Role of ansible-test in Collection Testing a key area for the Community & Partner Engineering team is Developing a unified strategy for collection testing.
In this post I want to kick things off by outlining some of the motivations and goals driving this initiative. I also want to share an overview of the current collection testing landscape to establish a shared understanding for further discussion.
Why are we doing this?
A large motivation for this initiative is direct community feedback from CfgMgmtCamp 2026 during the contributor summit and other posts such as:
- CfgMgmtCamp 2026 discussion (7/12): Collection testing is still a mess
- Should we be using molecule or ansible-test for unit/integration tests? - Get Help
- Integrating
ansible-test integration(no pun intended) andmolecule- Project Discussions
Another reason we want to create a unified collection testing strategy is to reduce fragmentation across the ecosystem. You might recall Greg’s Ansible community strategy from 2023 (and the relevant XKCD comic).
While Greg’s strategy document focused mainly on fragmentation in the community in areas such as communication and decision making, his analogy about communities of projects being like houses in a neighborhood could also extend to Ansible collections.
When you look around the collection ecosystem, you can find most collections use different tools for testing. Going back to Greg’s analogy, it’s fine for projects to do their own thing and make their own decisions inside their own “house” but that can often lead to difficulties across the whole neighborhood. In the case of collections, this often makes it challenging for contributors to work across them.
What do we want to achieve?
With this initiative we want to establish a single, authoritative collection testing strategy that reduces fragmentation caused by multiple overlapping tools both in the Ansible community and in Red Hat.
As an outcome of that strategy, we hope to reduce maintenance burden, add consistency for community and Red Hat partner collection maintainers, resulting in content quality improvements across the Ansible collection ecosystem.
Some additional benefits that we hope the effort to establish a strategy will achieve:
-
Identify collection testing areas which need clarity.
-
Improve documentation of the unified collection testing tooling.
-
Create migration guides based on the unified collection testing tooling.
-
Create “AI” skills to assist collection maintainers with the migration to the agreed collection testing tooling.
Call to Action
This initiative to develop a unified collection testing strategy requires participation and collaboration. To get to the best end result, we need to hear as many voices from the community as possible so that we can consider a broad set of views. We ask that you join us in this initiative and help us shape the strategy.
As things come into focus and the strategy evolves, we’ll share updates on the forum. We ask that everyone keep an eye on the discussions, get involved, and have your say.
For now, here are some ways you can get started in this initiative and participate:
- Read through the landscape overview in the below section of this post. Are we overlooking anything? Are there any technical inaccuracies? Do you have any general comments, questions, or suggestions? If so, please reply to this post and let us know.
- Review the pull request in Ansible documentation that clarifies the role of ansible-test as a collection test tool.
- Watch the collection-test-strat tag on the forum to follow the discussion.
Landscape Overview
The collection testing landscape as it stands at the time of writing this post has several tools. Let’s start by breaking it down into four areas: unit, integration, sanity and static analysis. Please note that it’s out of scope for this post to explain these areas. There is extensive documentation in the Ansible documentation which you can refer to:
- Testing Ansible - Types of tests
- Testing Ansible and Collections - Testing an Ansible Collection
- Testing Ansible and Collections - Commands
- Ansible Lint Documentation - Homepage
Ansible Test
The de facto collection testing tool since Ansible 2.9 and the initial breakout of collections from ansible-core, known back then as ansible-base. It is used across all three test types. For collection testing ansible-test has some known limitations which make the maintainer experience for testing collections with it challenging. Some examples include:
- The requirement for the
ansible_collectionsdirectory so that ansible-test can discover the collection for testing. You can see an example in the Ansible documentation and was initially discussed in ansible/ansible/issues/60215 back in 2019. - No ability to disable certain sanity tests via the
ansible-testconfiguration file: ansible/ansible/pull/74989 - Allow licenses other than GPLv3+: ansible/ansible/issues/67032
- No ability to ignore the shebangs sanity test for
files/andtemplates/directories in collection roles and integration test targets: ansible/ansible/pull/79700
One feature to note with ansible-test is its support for running unit and integration tests via the --docker flag, utilizing the container images maintained by the Ansible core team:
- Quay - base-test-container
- Quay - default-test-container
- Quay - alpine-test-container
- Quay - fedora-test-container
- Quay - ubuntu-test-container
Another aspect closely related to ansible-test is the Azure Pipelines (AZP) provided by Red Hat. Certain collections (such as community.general) use AZP for running integration tests on FreeBSD, MacOS and Linux VMs.
As mentioned in Clarifying the Role of ansible-test in Collection Testing, there have been several posts from the Ansible Core team that ansible-test is no longer receiving feature enhancements for collection testing. We are in the process of updating the Ansible documentation to clarify this: Add note on Collection Testing with ansible-test.
ansible-test is one of particular interest to our strategy for unifying collection testing tools. It seems like a good starting point because it no longer receives feature enhancements for collection testing. This alone seems like a strong reason to consider alternatives for collection testing. It is also common ground that affects both the Ansible community and Red Hat equally.
Unit Testing Tools
For Ansible collection content, unit testing is applicable to modules and plugins. Below are the tools for this area in the current ecosystem:
| Name | Description | Location in Collection | Documentation / Project Link |
|---|---|---|---|
| ansible-test units | Invokes unit tests via pytest. | tests/unit | Testing collections — Ansible Community Documentation |
| pytest-ansible | A pytest plugin that enables the use of ansible in tests, enables the use of pytest as a collection unit test runner, and exposes molecule scenarios through a pytest fixture. | tests/unit | home - Ansible Pytest Documentation |
pytest-ansible
pytest-ansible has several pytest fixtures (ansible_module, ansible_adhoc, ansible_facts) for interacting with Ansible via pytest. Furthermore, there is a pytest fixture provided by pytest-ansible for triggering molecule scenarios.
Integration Testing Tools
For Ansible collection content, integration testing is applicable to modules, plugins, roles and playbooks. Below are the tools for this area used in the current ecosystem:
| Name | Description | Location in Collection | Documentation / Project Link |
|---|---|---|---|
| ansible-test integration | Invoke integration tests via ansible-test sanity. | tests/integration | Testing collections — Ansible Community Documentation |
| molecule | An ansible-native testing framework for collections, playbooks, and roles with configurable workflows for testing any system or service. | extensions/molecule | Ansible Molecule GitHub - ansible/molecule Collection Testing - Ansible Molecule |
Molecule
Molecule has gone through some significant changes since v25.9.0 with the introduction of:
- An ‘ansible-native’ approach to testing collections: “Treats Molecule as orchestration around “plain Ansible” artifacts: inventories, create.yml/destroy.yml/converge.yml playbooks.”
- The ‘ansible-native’ approach means that the molecule-plugins project utilized in prior molecule versions is less relevant as the drivers which used to take care of provisioning resources can now be done via collection modules in the
create.ymland other test sequence playbooks. - Molecule is now more ‘collection aware’ with automatic scenario discovery:
$ uv run molecule list
INFO Collection 'community.beszel' detected.
INFO Scenarios will be used from 'extensions/molecule'
INFO agent_airgap ➜ list: Executing
INFO agent_airgap ➜ list: Executed: Successful
INFO agent_default ➜ list: Executing
INFO agent_default ➜ list: Executed: Successful
INFO agent_token ➜ list: Executing
INFO agent_token ➜ list: Executed: Successful
INFO hub_default ➜ list: Executing
INFO hub_default ➜ list: Executed: Successful
Instance Name │ Driver Name │ Provisioner Name │ Scenario Name │ Created │ Converged
╶───────────────┼─────────────┼──────────────────┼───────────────┼─────────┼───────────╴
instance │ default │ ansible │ agent_airgap │ false │ false
instance │ default │ ansible │ agent_default │ false │ false
instance │ default │ ansible │ agent_token │ false │ false
instance │ default │ ansible │ hub_default │ false │ false
Sanity Testing Tools
For Ansible collection content, sanity testing is applicable to modules and plugins.
There is currently only one tool in this area which is ansible-test and the sanity command. At the time of writing this post there are no alternatives. Related documentation: Testing collections — Ansible Community Documentation.
Static Analysis
For Ansible collection content, static analysis is applicable to modules, plugins, roles and playbooks.
Below are the tools related to this area used in the current ecosystem:
| Name | Description | Documentation / Project Link |
|---|---|---|
| pycodestyle | PEP8 compliance. Used by ansible-test sanity. |
pycodestyle’s documentation — pycodestyle 2.14.0 documentation |
| pylint | Static code analyzer. Used by ansible-test sanity. |
Pylint 4.1.0-dev0 documentation |
| yamllint | YAML linter. Used by ansible-test sanity and ansible-lint. |
yamllint documentation |
| ansible-lint | Linter for Ansible playbooks and roles. | home - Ansible Lint Documentation |
| ruff | Python linter and formatter. Available in antsibull-nox. |
Ruff |
| black | Python formatter. Available in antsibull-nox. |
Black |
| flake8 | Python linter. Available in antsibull-nox. |
Flake8 |
| mypy | Static type checker. Available in antsibull-nox. |
mypy Documentation |
Other Tooling
The tooling listed below is used in the ecosystem for collection testing.
| Name | Description | Documentation / Project Link |
|---|---|---|
| ansible-dev-tools (ADT) | A metapackage which installs other developer tooling for running Ansible content and testing collections. | Ansible Development Tools Documentation |
| ansible-dev-environment (ade) | A pip-like installer for Ansible collections. Used by tox-ansible. |
Ansible Development Environment Documentation |
| andebox | A script to assist Ansible developers by encapsulating some boilerplate tasks, especially the ability of running tests from the root directory of the collection. | GitHub - russoz-ansible/andebox: Ansible Developer’s (tool)Box |
| ansible-doc | Used during ansible-test sanity to validate plugin docs. |
Ansible documentation |
| antsibull-docs | Provides tooling to lint Ansible collection documentation. Used by antsibull-nox.Using subcommand lint-collection-docs. |
antsibull-docs – Ansible Documentation Build Scripts |
| galaxy-importer | Builds and imports Ansible collections to Galaxy and Automation Hub. Used by Partner Engineering and antsibull-nox. |
GitHub - ansible/galaxy-importer |
| antsibull-nox | A nox helper library for Ansible collections. | Antsibull Nox Helper |
| tox-ansible | A tox plugin for testing Ansible collections. | Home - Tox Ansible Documentation |
Existing Collection Testing Pathways
During my research I mapped ansible-test and the two primary collection testing orchestration tools (antsibull-nox and tox-ansible) into two pathways: Community and Red Hat supported. Red Hat supported contains tooling maintained by the Red Hat Ansible DevTools team.
The below pathway headings contain diagrams with each area, their tools and how it relates to the content in a collection. I hope that this helps visualise the pathways for those of us that are visual learners. Furthermore, as part of my research I implemented both orchestration tools in the community.beszel collection:
| Pathway | Links |
|---|---|
Community (antsibull-nox) |
Tag: 1.0.0 GitHub Actions Workflow |
Red Hat Supported (tox-ansible) |
Branch: tox-ansible PR: #40 GitHub Actions Workflow |
Community Pathway (Traditional)
The diagram below shows the ‘traditional’ approach to collection testing, using ansible-test without any orchestration tooling. This is essentially the de facto standard in the ecosystem with many collections using the ansible-test-gh-action. At the time of writing, any new collections created from the collection_template uses this GitHub action.
Community Pathway (antsibull-nox)
The diagram below shows the antsibull-nox collection testing pathway. Maintained by Felix Fontain, antsibull-nox provides nox sessions to run different tools in the ecosystem, some of the sessions are enabled by default, others are opt-in. Generally, it provides a lot of flexibility to collection maintainers.
Some key areas to highlight are that:
- It relies on
ansible-testfor the sanity, unit and integration sessions. - The sanity, unit and integration sessions use the
--dockerflag with ansible-test which requires a container engine such as Docker or Podman to be installed. antsibull-noxtakes care of configuring theansible_collectionspath foransible-testand generating the matrices of Python versions andansible-coreversions supported by a collection.- Support was recently added to run
moleculeas an alternative integration session. - It has support for linting collection documentation and several static analysis linters.
Red Hat Supported (tox-ansible)
The diagram below shows the tox-ansible collection testing pathway. This pathway uses Red Hat supported tooling from the Red Hat Ansible DevTools team. tox-ansible provides tox environments to run unit, integration, sanity tests and test validity of the collection artifact via galaxy-importer.
Some key areas to highlight are:
- The tool for integration testing is
molecule. - The tool for unit testing is
pytestand thepytest-ansibleplugin. - The tool for sanity testing is
ansible-test sanity.- During discussions with the DevTools team it was identified that this is the area which has been the most difficult to break away from due to how closely coupled some of the
ansible-test sanitychecks (such asvalidate-modules) are toansible-coreinternals.
- During discussions with the DevTools team it was identified that this is the area which has been the most difficult to break away from due to how closely coupled some of the
ansible-dev-environmentis used to install all Python and Ansible collection dependencies required for the environments.tox-ansibleandansible-dev-environmentfavor uv for creating the Python virtual environment used for testing, downloading the required Python version for the matrices of tests and installing dependencies.
During my implementation of this pathway in the community.beszel collection I identified several points of friction with this pathway and raised GitHub issues for these:
Documentation: User Guide References Non Existent GitHub Action Workflow · Issue #548- Bug: Molecule package missing from Tox Integration Environment install_deps · Issue #549
- Bug: Ansible Integration and Unit Test Collection dependencies are not discovered automatically · Issue #550
Patching ansible.cfg causes exit status 2 · Issue #349
I also created these issues downstream for tracking and I plan to submit bugfixes and documentation updates for these.
