CfgMgmtCamp 2026 discussion (7/12): Collection testing is still a mess

[felixfontein]

As a collection developer/maintainer

Collection testing is still a mess

Latest developments

None that I know of.
This wasn’t discussed at CfgMgmtCamp 2026.

Original text

The main tool for collection testing is still ansible-test. Unfortunately ansible-test doesn’t wants to be a tool for collection testing, it wants to be a tool for ansible-core testing. There has been talk about handing ansible-test off to another team to improve its collection testing parts for years. So far: nothing happened.

Well, ansible-test is basically three different programs in one:

1. ansible-test sanity does sanity checks. There is no alternative for that.

2. ansible-test units does unit testing. While there is pytest-ansible (documentation), it’s not clear whether it is meant to replace ansible-test units or not, and there aren’t really instructions for that. Also ansible-test units does a lot more, like:

   • installing Python requirements;
   • using different Python versions (through the default Docker image) appropriate to what a specific ansible-core version supports on controller and target;
   • doing change detection.

3. ansible-test integration and ansible-test windows-integration and ansible-test network-integration. (I’ve only ever used the first, so no idea how different the other two are.) This is a quite polished and versatile integration testing tool using Ansible roles for tests (though you can also customize that by adding a runme.sh). A possible replacement is Molecule, but that is a) working somewhat different, and b) has quite a few features missing that ansible-test has. Also, again, there is no documentation on how to migrate.

Over the years there are more and more things that are very annoying to collection developers and that aren’t getting fixed (despite PRs existing).

Examples:

• Allow to disable certain sanity tests via ansible-test’s config (ansible-test: allow to enable/disable sanity tests with config by felixfontein · Pull Request #74989 · ansible/ansible · GitHub).
• Allow more licenses for modules, resp. remove the “modules must be GPLv3+ licensed” requirement ansible-test sanity imposes (ansible-test sanity expects all modules to be licensed under the GPLv3 · Issue #67032 · ansible/ansible · GitHub).
• Ignore shebangs in files/ and templates/ in collection roles and integration test targets (Sanity tests: ignore shebangs in files/ and templates/ in collection roles and integration test targets by felixfontein · Pull Request #79700 · ansible/ansible · GitHub).
• Avoid restrictions on path structure, at least when run with container/VM isolation (when everything is copied around anyway) (Support running ansible-test on collections outside a collection root · Issue #60215 · ansible/ansible · GitHub).