# Unified Collection Testing Strategy - Kick off & Landscape Overview **URL:** \ **Category:** Collection Development\ **Tags:** azure, fedora, ubuntu, collection-test-strat, ansible-lint, galaxy-ng\ **Created:** [May 14, 2026, 4:23pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799 "2026-05-14T16:23:45Z")\ **Posts on this page:** 18\ **Page:** 2
**Author:** ![jborean](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jborean/32/1525_2.png) [@jborean](https://forum.ansible.com/u/jborean)\ **Post date:** [May 18, 2026, 10:03am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/21 "2026-05-18T10:03:09Z")
> [@dbrennand](#): > > RE: The `pslint` target in `ansible-test sanity` - It runs PSScriptAnalyzer only, right? Yep that’s correct, the files you’ve listed are pretty much the general gist. We run `Invoke-PSScriptAnalyzer` with a custom ruleset and use `pwsh` on the `ansible-test` “controller” host. Our docker images come with all this pre-configured but you can definitely still run it locally if you have the requirements all set up. > [@dbrennand](#): > > With the latest changes with `molecule` and being `ansible-native’ - You can roll your own inventory and in theory be able to provision a Windows VM on a public cloud I think at a minimum being able to provide your own inventory for test targets is a must have. But the core-ci provided Windows hosts are pretty critical for all the `ansible-collections` based Windows collections. Without that we will be stuck with either offline CI and manual testing which is not ideal. While I don’t think the tool should hook into the core-ci API and provision these hosts I do think there is a conversation to be had to either expose a specific action in `ansible-test` or some other tool that uses the `core-ci` API to provision these hosts for the collections in `ansible-collections`. The core-ci API is also used to provision the RHEL, FreeBSD, and macOS targets so this functionality would certainly be useful there. > [@dbrennand](#): > > Wouldn’t the above scenario be applicable more to testing modules on PowerShell 5 (Desktop) right? As that requires Windows, but for PowerShell 6+ could a container with `pwsh` binary installed such as ones provided by Microsoft be sufficient? Have you ever tested that? Yep and historically `powershell.exe` on Windows has been the only workable PowerShell target using Ansible. Ansible 2.21 has added the ability to target `pwsh.exe` on both Windows and Linux which will help cover a few of those gaps but testing will be very limited for a lot of the collections if you can’t run on Windows as a test target. While I’m certainly a fan of PowerShell based modules I’m sure most people will probably only really care about them when it comes to Windows and Window specific scenarios. ---
**Author:** ![mattclay](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mattclay/32/268_2.png) [@mattclay](https://forum.ansible.com/u/mattclay)\ **Post date:** [May 18, 2026, 4:32pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/22 "2026-05-18T16:32:32Z")
> [@dbrennand](#): > > Are you referring to [GitHub - ansible-community/ansible-test-gh-action: A composite GitHub Action encapsulating the GitHub Actions CI/CD workflows setup necessary for testing Ansible collection repositories on GitHub · GitHub](https://github.com/ansible-community/ansible-test-gh-action) ? No, I was referring to the feature in ansible-test that allows it to request ephemeral cloud resources (virtual machines, cloud credentials) from various cloud providers, including AWS and Azure, without the need for any credentials. This feature is supported for CI jobs on both Azure Pipelines and GitHub Actions. When not run in a CI pipeline on one of those systems, credentials are required. ---
**Author:** ![mattclay](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mattclay/32/268_2.png) [@mattclay](https://forum.ansible.com/u/mattclay)\ **Post date:** [May 18, 2026, 4:43pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/23 "2026-05-18T16:43:41Z")
> [@dbrennand](#): > > Regarding code coverage, I wanted to see if I could get `pytest-cov` working with `tox-ansible` and I have managed to with the use of `.coveragerc` and slight modification to the `tox-ansible.ini` file: [Implement tox-ansible for collection testing by dbrennand · Pull Request #40 · ansible-collections/community.beszel · GitHub](https://github.com/ansible-collections/community.beszel/pull/40/changes/a7013204a68e9fff2b1a2b01d002bb7ad33c0982) Keep in mind that this only covers what are referred to as unit tests in ansible-test, which are typically a tiny fraction of the tests used for collections, if they exist at all. The majority of tests tend to be integration tests, which under ansible-test also support code coverage. That means tests written using roles, playbooks and ad-hoc commands are able to collect code coverage for both controller-side plugins and modules, even when run on a different host from the controller. In addition to supporting modules written in Python, coverage is also supported for PowerShell modules. I’m not aware of any tools other than ansible-test that support code coverage in any of these scenarios. ---
**Author:** ![russoz](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/russoz/32/127_2.png) [@russoz](https://forum.ansible.com/u/russoz)\ **Post date:** [May 19, 2026, 6:25am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/24 "2026-05-19T06:25:55Z")
As mentioned in the chat recently, in `community.openwrt` we are using `molecule` tests for: - generic tests: our `extensions/molecule/default` scenario does not aim to thoroughly test any particular component, it just cherry picks from a bunch of different ones - it gives us a quick response on whether the collection is working (we have a custom machinery to handle the fact that our modules are in shell script instead of Python) - integration tests: per module/plugin, using the same directory structure prescribed by `ansible-test integration` - we can use it to test, but CI relies on `molecule` - roles tests: despite being placed within a collection, each role (only one right now) may have its own `molecule` directory, and our CI iterates over those cases and run them separately **Issues** The requirement for the collections to test against `devel` or `milestone` comes with a couple of glitches: - As consequence of the ansible-core release process: when a new release branch is created off `devel`, that version is now in a “gap” state. It was happening until a couple of days ago: 2.21 was branched off, but it had not been released yet. The CI workflow did not have that `rc1` version in it, nor the `b1`, `b2`, `b3` before it. In the meanwhile, `devel` was already numbered as 2.22. - ansible-core `devel` is supporting Python 3.15, which has not been released yet. In community.general (and I suppose most of others) the tests use `ansible-test integration`, which uses our own custom docker/VM images, for both controller and target. Specifiying the Python version is done in `ansible-test`’s CLI. In `community.openwrt` we run molecule in the standard images provided by GHA, and those do not support Python 3.15 yet. I have tried to set the CI to use one of the beta versions, but some dependency used by Sphinx do not support that Python version yet. Therefore, testing `devel` is incomplete and not directly in my hands. I reckon these issues are not blockers to anything, we can (probably should) just ignore them as they are solved as soon as releases come out. But I felt it would be important to mention them nonetheless. ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [May 19, 2026, 7:14am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/25 "2026-05-19T07:14:00Z")
> [@mattclay](#): > > The majority of tests tend to be integration tests, which under ansible-test also support code coverage. That means tests written using roles, playbooks and ad-hoc commands are able to collect code coverage for both controller-side plugins and modules, even when run on a different host from the controller. In addition to supporting modules written in Python, coverage is also supported for PowerShell modules. I’m not aware of any tools other than ansible-test that support code coverage in any of these scenarios. Thanks @mattclay - I’ve personally not used this functionality in `ansible-test integration` for code coverage. @felixfontein do you have an example of this for `community.general`? I took a look in Azure DevOps but couldn’t find a coverage report. ---
**Author:** ![jborean](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jborean/32/1525_2.png) [@jborean](https://forum.ansible.com/u/jborean)\ **Post date:** [May 19, 2026, 6:57pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/26 "2026-05-19T18:57:52Z")
> [@dbrennand](#): > > do you have an example of this for `community.general`? I took a look in Azure DevOps but couldn’t find a coverage report. For `community.windows` we run with the `--coverage` option so ```bash ansible-test windows-integration --coverage ... ``` The final coverage job collects all the coverage data per test matrix, combines it into one then uploads it to `codecov.io` > Doing it manually is like ```bash # Clear out any existing coverage data ansible-test coverage erase # Run test(s) with the coverage collector, can also # be done with windows-integration, units ansible-test integration --coverage ping --docker # View the console report, has a few extra args ansible-test coverage report # Combine all the various reports into one for publishing # to codecov.io or other areas. File is created under # tests/results/coverage/coverage ansible-test coverage combine ``` ---
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [May 19, 2026, 7:24pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/27 "2026-05-19T19:24:02Z")
Coverage is only computed during the nightly CI runs. It’s uploaded to [Codecov](https://app.codecov.io/gh/ansible-collections/community.general). ---
**Author:** ![mattclay](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/mattclay/32/268_2.png) [@mattclay](https://forum.ansible.com/u/mattclay)\ **Post date:** [May 19, 2026, 8:28pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/28 "2026-05-19T20:28:38Z")
Collections tested on Azure Pipelines typically run code coverage as part of their scheduled CI runs, uploading to [Codecov](https://app.codecov.io/gh/ansible-collections). Code coverage isn’t collected and reported for PRs on these collections unless requested (`ci_coverage` and `ci_complete` on the last commit message). Reporting on code coverage is really only effective when all tests are run, which is generally limited to schedule runs. For anyone unfamiliar with this setup, it can give the appearance that code coverage isn’t used at all. Another common misconception is that only unit tests contribute to code coverage, which is something I briefly addressed in a previous comment. ---
**Author:** ![Jeff\_Pullen](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeff_pullen/32/5318_2.png) [@Jeff\_Pullen](https://forum.ansible.com/u/Jeff_Pullen)\ **Post date:** [May 20, 2026, 12:40pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/29 "2026-05-20T12:40:32Z")
This initiative really makes me happy, thank you! I want to follow up on something I believe @kks was touching on earlier. There are two major groups of Ansible users that are testing collections. Those are: 1. vendors/communities focused on the python 2. users/organizations that are focused on roles and playbooks There are a lot of challenges with the testing for group one that I believe have been summarized already. The problem is even worse for the second group though. The process is not intuitive and there isn’t a good “right” way. The biggest challenges for group two are: 1. Lack of relevant documentation and examples 2. Information that is available assumes python knowledge or outdated (prior to collections) 3. Lack of emphasis on molecule-plugins that provide an easy-button for testing 4. Multiple layers of abstraction (/extensions/molecule/utils + /extensions/molecule/role + /tests/integration/X) 5. Layers of helper tools that each have their own learning curve I’d ask that we not lose sight of the second group of users and their needs in this effort. Over the years it feels like the testing tools have moved away from some of the principals laid out in [The Zen of Ansible](https://www.redhat.com/en/blog/the-zen-of-ansible)". - **Ansible is not Python.** - YAML sucks for coding. - Playbooks are not for programming. - **Ansible users are (most likely) not programmers.** ---
**Author:** ![kks](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kks/32/10352_2.png) [@kks](https://forum.ansible.com/u/kks)\ **Post date:** [May 20, 2026, 1:31pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/30 "2026-05-20T13:31:21Z")
This is a very good point, indeed. ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [May 26, 2026, 8:47am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/31 "2026-05-26T08:47:55Z")
Hi @Jeff_Pullen - Thanks for your feedback here. Wanted to ask some follow up questions. With group two as you explained it, are you referring to somebody testing Ansible content (such as a role or playbook) outside of a collection structure? E.g., a standalone role in a GitHub repo? > [@Jeff\_Pullen](#): > > Lack of relevant documentation and examples Lack of documentation and examples for what exactly? Do you have specific tooling or scenarios in mind here? > [@Jeff\_Pullen](#): > > Information that is available assumes python knowledge or outdated (prior to collections) Is there a particular guide on [docs.ansible.com](http://docs.ansible.com) that you’re referring to here? > [@Jeff\_Pullen](#): > > Lack of emphasis on molecule-plugins that provide an easy-button for testing I was discussing with @konstruktoid about [`molecule-plugins`](https://github.com/ansible-community/molecule-plugins) recently on [Matrix](https://matrix.to/#/!ZiHKxiCHRtflvQEbRt:libera.chat/$lXFVGzckoSdT1mFgzShuoUkXSHfE1HcUWg4siEFq_vc?via=ansible.com&via=matrix.org&via=ansible.im). We were talking about how the `docker` driver with newer molecule versions doesn’t need to be installed anymore. The `community.docker` collection can be used to create a container for the scenario in the `create` sequence. See the [example](https://github.com/dbrennand/ansible-collection-hardening/blob/molecule-docker-scenario/extensions/molecule/delete_users_docker/create.yml#L12-L25) here which parses [ansible-collection-hardening/extensions/molecule/delete\_users\_docker/molecule.yml at molecule-docker-scenario · dbrennand/ansible-collection-hardening · GitHub](https://github.com/dbrennand/ansible-collection-hardening/blob/molecule-docker-scenario/extensions/molecule/delete_users_docker/molecule.yml) to create the containers. This is what this point from the original post is alluding to: > [@dbrennand](#): > > The ‘ansible-native’ approach means that the [molecule-plugins](https://github.com/ansible-community/molecule-plugins) project utilized in prior molecule versions is less relevant as the drivers which used to take care of provisioning resources can now be done via collection modules in the `create.yml` and other test sequence playbooks. ---
**Author:** ![Jeff\_Pullen](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeff_pullen/32/5318_2.png) [@Jeff\_Pullen](https://forum.ansible.com/u/Jeff_Pullen)\ **Post date:** [May 26, 2026, 5:26pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/32 "2026-05-26T17:26:00Z")
@dbrennand I appreciate the follow up. Lots of great questions, but these are going to get confusing smashing it all together. I’ll try my best. **Question 1:** > With group two as you explained it, are you referring to somebody testing Ansible content (such as a role or playbook) outside of a collection structure? E.g., a standalone role in a GitHub repo? I’m focused on testing ansible content as part of a collection. The process for testing stand-alone roles has extensive documentation and examples and the process has generally stayed the same for a very long time. **Question 2:** > Lack of documentation and examples for what exactly? Do you have specific tooling or scenarios in mind here? How to test ansible content (YAML) as part of a collection. The most typical scenario that has to be out-of-the-box is running local podman/docker containers that include systemd for an automation developer to test as they’re writing the code and validating ansible-lint won’t fail a pipeline. **Question 3:** > Information that is available assumes python knowledge or outdated (prior to collections) The challenge is mostly that there is nearly a decade of content covering how to do this (books, videos, blogs, etc) that sort-of work, don’t work at all anymore, or take the person down some really frustrating paths. I just reviewed the molecule documentation and see there have been some major reworks in the past year that I wasn’t tracking. This is a huge step in the right direction thanks to folks like @cidrblock . There is now enough information there for someone to get started, which is a big improvement. Assuming people are aware, we likely see more examples of it that can be used as a reference. So much of the the information related to collection testing assumes that it is module focused. We don’t do a great job of differentiating role only collections from collections with modules. So users will encounter a long list of tools that seem required but may not be needed for their use-case. Question / Comment 4: > I was discussing with @konstruktoid about [`molecule-plugins`](https://github.com/ansible-community/molecule-plugins) recently on [Matrix](https://matrix.to/#/!ZiHKxiCHRtflvQEbRt:libera.chat/$lXFVGzckoSdT1mFgzShuoUkXSHfE1HcUWg4siEFq_vc?via=ansible.com&via=matrix.org&via=ansible.im). We were talking about how the `docker` driver with newer molecule versions doesn’t need to be installed anymore. The `community.docker` collection can be used to create a container for the scenario in the `create` sequence. See the [example](https://github.com/dbrennand/ansible-collection-hardening/blob/molecule-docker-scenario/extensions/molecule/delete_users_docker/create.yml#L12-L25) here which parses [ansible-collection-hardening/extensions/molecule/delete\_users\_docker/molecule.yml at molecule-docker-scenario · dbrennand/ansible-collection-hardening · GitHub](https://github.com/dbrennand/ansible-collection-hardening/blob/molecule-docker-scenario/extensions/molecule/delete_users_docker/molecule.yml) to create the containers. This is my major point of contention with the direction of ansible content testing. It appears that flexibility is being prioritized over simplicity. The ‘ansible-native’ approach provides a tremendous amount of capability but adds a lot of complexity. Enough complexity that even having done it many times I still end up having to relearn and troubleshoot it every time to get things working (for example fighting all the relative paths in the molecule configs). It also takes a non-opinionated stance on the implementation so there are many ways to get it working that have the same effect but are very unique. This makes troubleshooting and maintenance more difficult. The old way: ```auto # molecule.yml driver: name: docker ``` The new Ansible Native alternative: ```auto # create.yml --- - name: Create hosts: localhost gather_facts: false vars: molecule_inventory: all: children: molecule: hosts: {} tasks: - name: Create containers community.docker.docker_container: name: "{{ item.name }}" hostname: "{{ item.name }}" image: "{{ item.image }}" state: started command: "{{ item.command | default('sleep 1d') }}" privileged: "{{ item.privileged | default(false) }}" volumes: "{{ item.volumes | default(omit) }}" log_driver: json-file register: result loop: "{{ molecule_yml.platforms }}" loop_control: label: "{{ item.name }}" - name: Print container details ansible.builtin.debug: msg: "{{ result.results }}" - name: Fail if container is not running when: > item.container.State.ExitCode != 0 or not item.container.State.Running ansible.builtin.fail: msg: >- Container {{ item.container.Name }} failed to start properly. Exit Code: {{ item.container.State.ExitCode }}. Running: {{ item.container.State.Running }}. loop: "{{ result.results }}" loop_control: label: "{{ item.container.Name }}" - name: Add containers to molecule inventory vars: inventory_partial_yaml: | all: children: molecule: hosts: "{{ item.name }}": ansible_connection: community.docker.docker ansible_python_interpreter: /usr/bin/python3 ansible.builtin.set_fact: molecule_inventory: >- {{ molecule_inventory | combine(inventory_partial_yaml | from_yaml, recursive=true) }} loop: "{{ molecule_yml.platforms }}" loop_control: label: "{{ item.name }}" - name: Write molecule inventory ansible.builtin.copy: content: "{{ molecule_inventory | to_yaml }}" dest: "{{ molecule_ephemeral_directory }}/inventory/molecule_inventory.yml" mode: "0600" - name: Force inventory refresh ansible.builtin.meta: refresh_inventory - name: Fail if molecule group is missing ansible.builtin.assert: that: "'molecule' in groups" fail_msg: "molecule group was not found inside inventory groups: {{ groups }}" run_once: true - name: Validate inventory hosts: molecule gather_facts: false tasks: - name: Check container access ansible.builtin.raw: /bin/true changed_when: false ``` There may be valid reasons to move away from molecule drivers, but I think its worth considering how we can still live up to the goal of “radically **simple** IT automation”. ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [June 3, 2026, 7:46am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/33 "2026-06-03T07:46:20Z")
Hi @Jeff_Pullen Thank you for your detailed answers 🙂 > [@Jeff\_Pullen](#): > > The most typical scenario that has to be out-of-the-box is running local podman/docker containers that include systemd for an automation developer to test as they’re writing the code From my own experience, molecule is well suited to this scenario. I was reviewing the molecule documentation and there is [Systemd container - Ansible Molecule](https://docs.ansible.com/projects/molecule/guides/systemd-container/#systemd-container). Do you think this guide would benefit from being a full user guide? For example, showing the `create.yml` playbook example and the configuration of the container under `platforms` in `molecule.yml` to support `systemd` inside a container using `docker` or `podman` as the container engine? > [@Jeff\_Pullen](#): > > So much of the the information related to collection testing assumes that it is module focused. We don’t do a great job of differentiating role only collections from collections with modules. So users will encounter a long list of tools that seem required but may not be needed for their use-case. Thank you for your feedback here. I think there is an opportunity here for us to add documentation in the future specifically around role testing. Right now, when you navigate to [Developers | Ansible documentation | Ansible documentation](https://docs.ansible.com/developers.html) there is no guide about role testing. ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [June 3, 2026, 7:47am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/34 "2026-06-03T07:47:34Z")
Another short update that [Bug: Molecule package missing from Tox Integration Environment install\_deps · Issue #549 · ansible/tox-ansible · GitHub](https://github.com/ansible/tox-ansible/issues/549) is now closed. 🙂 ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [June 15, 2026, 8:10am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/35 "2026-06-15T08:10:34Z")
Issue [Documentation: User Guide References Non Existent GitHub Action Workflow · Issue #548 · ansible/tox-ansible · GitHub](https://github.com/ansible/tox-ansible/issues/548) is now closed. Fixed in [chore: fix CI/CD pipeline section and add skip functionality docs by dbrennand · Pull Request #564 · ansible/tox-ansible · GitHub](https://github.com/ansible/tox-ansible/pull/564) 🙂 ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [June 15, 2026, 2:21pm UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/36 "2026-06-15T14:21:10Z")
[feat: auto-discover and install collection requirements for unit/integration tests by dbrennand · Pull Request #567 · ansible/tox-ansible · GitHub](https://github.com/ansible/tox-ansible/pull/567) also merged and fixes [Bug: Ansible Integration and Unit Test Collection dependencies are not discovered automatically · Issue #550 · ansible/tox-ansible · GitHub](https://github.com/ansible/tox-ansible/issues/550) 🚀 🎉 ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [September 1, 2026, 10:39am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/37 "2026-09-01T10:39:33Z")
Split this topic on September 1, 2026, 10:39am UTC 9 posts were split to a new topic: [Ansible Role Testing](https://forum.ansible.com/t/ansible-role-testing/46251) ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [August 28, 2026, 11:16am UTC](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/44 "2026-08-28T11:16:17Z")
## Red Hat Supported Pathway updates Following up on my previous update from 15 June, I wanted to share some changes that have merged into `tox-ansible` and `pytest-ansible`. These updates apply to the **Red Hat Supported Pathway** discussed in my original post. They close several gaps identified in this post and when I implemented the pathway in the `community.beszel` collection. ## tox-ansible The relevant changes are included in [tox-ansible v26.7.1](https://github.com/ansible/tox-ansible/releases/tag/v26.7.1). ### Collection and Python dependency handling Building on the collection requirements work from [#567](https://github.com/ansible/tox-ansible/pull/567), [#580](https://github.com/ansible/tox-ansible/pull/580) expands Python dependency discovery. Collection `requirements.yml` files are automatically discovered and installed with `ade install -r` for unit and integration environments. Python dependency files are also searched for in common collection locations, including: - `test-requirements.txt` - `requirements-test.txt` - `requirements.txt` - `tests/unit/requirements.txt` - `tests/integration/requirements.txt` - `meta/ee-requirements.txt` This reduces the amount of collection specific configuration required and makes `tox-ansible` work with a wider range of existing collection layouts. ### Unit test coverage [#581](https://github.com/ansible/tox-ansible/pull/581) adds opt-in unit test coverage reporting. Coverage can be enabled with `--coverage` or configured through `pyproject.toml` or `tox-ansible.ini`. tox-ansible now: - Installs pytest-cov into the tox environment only when coverage is enabled. - Reports eligible plugin files that were not imported by the tests as 0% covered. - Keeps coverage data isolated between tox environments. - Avoids requiring each collection to maintain its own `.coveragerc` or custom test commands previously mentioned in [18](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799/18). This closes an important gap for Python unit tests and makes coverage reporting much easier to enable consistently for collections using `tox-ansible`. ### Matrix selection and configuration I also submitted two PRs that improve the local and CI experience: - [#582](https://github.com/ansible/tox-ansible/pull/582) fixes an incorrect warning when a project uses `[tool.tox-ansible]` in `pyproject.toml`. - [#583](https://github.com/ansible/tox-ansible/pull/583) makes `--matrix-scope` apply to normal tox execution as well as GitHub Actions matrix generation. For example: `tox --ansible --matrix-scope unit` ### First class Molecule test type [@cidrblock](https://github.com/cidrblock)’s [#585](https://github.com/ansible/tox-ansible/pull/585) adds a first class molecule test type to `tox-ansible`. The new functionality: - Automatically discovers scenarios under `extensions/molecule/`. - Runs Molecule across the same Python × ansible-core matrix as the other test types. - Uses `python3 -m molecule test --all` by default. - Supports additional arguments through `molecule_append` or full command replacement through `molecule_commands` via `[tool.tox-ansible]` in `pyproject.toml`. - Installs Molecule collection dependencies through ADE. - Supports `--matrix-scope molecule`. - Automatically omits `integration-*` environments when there are no remaining integration targets or pytest integration fixtures. This gives Molecule a first class path through the same tox based orchestration layer as unit and sanity tests in `tox-ansible`. I’ve been using these changes in the community.beszel collection on the [main branch](https://github.com/ansible-collections/community.beszel/tree/main). The CI uses the latest tox-ansible release and runs the first class Molecule environments. The [latest scheduled CI run](https://github.com/ansible-collections/community.beszel/actions/runs/33064612668) completed successfully after running the generated `tox-ansible` matrix. ## pytest-ansible There are also two significant changes in pytest-ansible from [@cidrblock](https://github.com/cidrblock): - [#602](https://github.com/ansible/pytest-ansible/pull/602) deprecates Molecule scenario execution through pytest. - [#603](https://github.com/ansible/pytest-ansible/pull/603) documents the planned retirement of pytest-ansible, with project EOL planned for December 2026. The Molecule via pytest integration remains available temporarily, but now emits deprecation warnings. The documentation now points users towards: - `molecule test --all` for Molecule scenario execution. - `tox-ansible` for collection testing matrices. - ADE plus plain pytest for collection unit tests. The reason for moving Molecule out of pytest is also important: the pytest integration cannot correctly support Molecule’s `shared_state` orchestration or Molecule’s native `--workers` handling. This changes how I view `pytest-ansible` in this pathway. The deprecation and changes in `tox-ansible` for first class molecule support means we have a clear indication to remove the `pytest-ansible` molecule integration fixture going forward. I have implemented this in [community.beszel - #48](https://github.com/ansible-collections/community.beszel/pull/48). [Previous page](https://forum.ansible.com/t/unified-collection-testing-strategy-kick-off-landscape-overview/45799.md?page=1)