# CfgMgmtCamp 2026 discussion (7/12): Collection testing is still a mess **URL:** \ **Category:** Collection Development\ **Tags:** cfgmgmgtcamp, windows, testing, ansible-test, molecule, collections, ansible-lint\ **Created:** [February 7, 2026, 9:01pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294 "2026-02-07T21:01:00Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [February 7, 2026, 9:01pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/1 "2026-02-07T21:01:00Z")
# As a collection developer/maintainer **Collection testing is still a mess** # Latest developments None that I know of. This wasn’t discussed at CfgMgmtCamp 2026. # Original text The main tool for collection testing is still ansible-test. Unfortunately ansible-test doesn’t wants to be a tool for collection testing, it wants to be a tool for ansible-core testing. There has been talk about handing ansible-test off to another team to improve its collection testing parts for years. So far: nothing happened. Well, ansible-test is basically three different programs in one: 1. `ansible-test sanity` does sanity checks. There is no alternative for that. 2. `ansible-test units` does unit testing. While there is `pytest-ansible` ([documentation](https://docs.ansible.com/projects/pytest-ansible/)), it’s not clear whether it is meant to replace `ansible-test units` or not, and there aren’t really instructions for that. Also `ansible-test units` does a lot more, like: 3. `ansible-test integration` and `ansible-test windows-integration` and `ansible-test network-integration`. (I’ve only ever used the first, so no idea how different the other two are.) This is a quite polished and versatile integration testing tool using Ansible roles for tests (though you can also customize that by adding a `runme.sh`). A possible replacement is Molecule, but that is a) working somewhat different, and b) has quite a few features missing that ansible-test has. Also, again, there is no documentation on how to migrate. Over the years there are more and more things that are very annoying to collection developers and that aren’t getting fixed (despite PRs existing). Examples: - Allow to disable certain sanity tests via ansible-test’s config ([ansible-test: allow to enable/disable sanity tests with config by felixfontein · Pull Request #74989 · ansible/ansible · GitHub](https://github.com/ansible/ansible/pull/74989)). - Allow more licenses for modules, resp. remove the “modules must be GPLv3+ licensed” requirement `ansible-test sanity` imposes ([ansible-test sanity expects all modules to be licensed under the GPLv3 · Issue #67032 · ansible/ansible · GitHub](https://github.com/ansible/ansible/issues/67032)). - Ignore shebangs in `files/` and `templates/` in collection roles and integration test targets ([Sanity tests: ignore shebangs in files/ and templates/ in collection roles and integration test targets by felixfontein · Pull Request #79700 · ansible/ansible · GitHub](https://github.com/ansible/ansible/pull/79700)). - Avoid restrictions on path structure, at least when run with container/VM isolation (when everything is copied around anyway) ([Support running ansible-test on collections outside a collection root · Issue #60215 · ansible/ansible · GitHub](https://github.com/ansible/ansible/issues/60215)). ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [February 9, 2026, 9:49am UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/2 "2026-02-09T09:49:26Z")
Related discussion: [Should we be using molecule or ansible-test for unit/integration tests?](https://forum.ansible.com/t/should-we-be-using-molecule-or-ansible-test-for-unit-integration-tests/5539) ---
**Author:** ![sivel](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/sivel/32/1198_2.png) [@sivel](https://forum.ansible.com/u/sivel)\ **Post date:** [February 9, 2026, 4:07pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/3 "2026-02-09T16:07:42Z")
> [@felixfontein](#): > > Over the years there are more and more things that are very annoying to collection developers and that aren’t getting fixed (despite PRs existing). For the record, and I [stated this](https://forum.ansible.com/t/error-invalid-extension-official-ansible-modules-must-have/5719/4) in another forum post, there are no plans to further extend ansible-test for collection testing. What it is, is basically what it is. (ansible-test is what it is for collection testing) ---
**Author:** ![gundalow](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/gundalow/32/99_2.png) [@gundalow](https://forum.ansible.com/u/gundalow)\ **Post date:** [February 9, 2026, 4:17pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/4 "2026-02-09T16:17:11Z")
> [@sivel](#): > > For the record, and I [stated this](https://forum.ansible.com/t/error-invalid-extension-official-ansible-modules-must-have/5719/4) in another forum post, there are no plans to further extend ansible-test for collection testing. What it is, is basically what it is. (ansible-test is what it is for collection testing) +1 to @sivel’s point here The discussions over the past few months, and reinforce by last weeks in-person discussion at CfgMgmtCamp are strongly leaning towards having `ansible-nox` as the unified approach ([Repo](https://github.com/ansible-community/antsibull-nox) & [Docs](https://docs.ansible.com/projects/antsibull-nox/introduction/)) for testing, which I personally really like the sound of. ---
**Author:** ![dbrennand](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dbrennand/32/7830_2.png) [@dbrennand](https://forum.ansible.com/u/dbrennand)\ **Post date:** [February 9, 2026, 4:19pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/5 "2026-02-09T16:19:27Z")
We’re making good progress on adding a nox session for running [`molecule`](https://docs.ansible.com/projects/molecule/) via `antsibull-nox` too. PR in progress: [Add Molecule Session Support by dbrennand · Pull Request #187 · ansible-community/antsibull-nox · GitHub](https://github.com/ansible-community/antsibull-nox/pull/187) 🙂 ---
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [February 11, 2026, 7:24am UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/6 "2026-02-11T07:24:31Z")
The main problem is that antsibull-nox can’t fix some of the issues ansible-test has. It would be better if ansible-test would be continued to also be developed for collections. Also I’m pretty convinced that forking ansible-test (as it is apparently discussed) is not a good idea, since it’s mainly a waste of time and a huge and unnecessary duplication of code. ---
**Author:** ![Shaps](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/shaps/32/1356_2.png) [@Shaps](https://forum.ansible.com/u/Shaps)\ **Post date:** [February 11, 2026, 3:33pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/7 "2026-02-11T15:33:04Z")
Couldn’t the parts that are useful to collection testing be “moved out” of ansible-test itself into either a shared tool or a library. I also don’t think forking ansible-test entirely makes sense, but separating the bits that are useful might make it easier for everyone to maintain/use? ---
**Author:** ![tima](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tima/32/854_2.png) [@tima](https://forum.ansible.com/u/tima)\ **Post date:** [March 21, 2026, 11:43pm UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/8 "2026-03-21T23:43:56Z")
> [@felixfontein](#): > > The main tool for collection testing is still ansible-test. Unfortunately ansible-test doesn’t wants to be a tool for collection testing, it wants to be a tool for ansible-core testing. There has been talk about handing ansible-test off to another team to improve its collection testing parts for years. So far: nothing happened. Stating that nothing has happened is not accurate. Three years ago we started building a comprehensive toolset specifically for collection development. That effort resulted in [the release of Ansible Development Tools (ADT)](https://forum.ansible.com/t/introducing-the-new-ansible-development-tools/6362). To highlight just a few pieces, [we introduced ansible-creator](https://docs.ansible.com/projects/creator/installing/) and [modernized Molecule to more natively support collections](https://docs.ansible.com/projects/molecule/philosophy/). There is a lot more in ADT, including dedicated [pytest-ansible](https://github.com/ansible/pytest-ansible) and [tox-ansible](https://github.com/ansible/tox-ansible) plugins. The only outstanding feature we might be missing is “sanity” (a problematic term we need to discard) testing. Honestly, for me and some others, it is hard to gauge how much of that type of testing collections actually need versus ansible-core. ADT is not perfect and we always welcome feedback. However, the tools are out there and actively maintained, so asserting that “nothing happened” is not accurate. ---
**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\ **Post date:** [May 1, 2026, 11:43am UTC](https://forum.ansible.com/t/cfgmgmtcamp-2026-discussion-7-12-collection-testing-is-still-a-mess/45294/9 "2026-05-01T11:43:00Z")
I still haven’t seen any documentation on how to migrate unit and integration tests from ansible-test to anything else. Also from the [documentation of pytest-ansible](https://docs.ansible.com/projects/pytest-ansible/) (I’ve read it multiple times), I’m still absolutely clueless on what it actually does and how it relates to regular collection unit tests run by ansible-test. tox-ansible simply runs ansible-test IIRC (and the test matrixes it creates especially for `ansible-test sanity` are OK for quick local testing, but definitely not for CI). So no, I don’t see that the tools are really there yet.