CfgMgmtCamp 2026 discussion (7/12): Collection testing is still a mess

Project Discussions Collection Development
, , , , , ,
1

As a collection developer/maintainer

Collection testing is still a mess

Latest developments

None that I know of.
This wasn’t discussed at CfgMgmtCamp 2026.

Original text

The main tool for collection testing is still ansible-test. Unfortunately ansible-test doesn’t wants to be a tool for collection testing, it wants to be a tool for ansible-core testing. There has been talk about handing ansible-test off to another team to improve its collection testing parts for years. So far: nothing happened.

Well, ansible-test is basically three different programs in one:

  1. ansible-test sanity does sanity checks. There is no alternative for that.

  2. ansible-test units does unit testing. While there is pytest-ansible (documentation), it’s not clear whether it is meant to replace ansible-test units or not, and there aren’t really instructions for that. Also ansible-test units does a lot more, like:

    • installing Python requirements;
    • using different Python versions (through the default Docker image) appropriate to what a specific ansible-core version supports on controller and target;
    • doing change detection.

  3. ansible-test integration and ansible-test windows-integration and ansible-test network-integration. (I’ve only ever used the first, so no idea how different the other two are.) This is a quite polished and versatile integration testing tool using Ansible roles for tests (though you can also customize that by adding a runme.sh). A possible replacement is Molecule, but that is a) working somewhat different, and b) has quite a few features missing that ansible-test has. Also, again, there is no documentation on how to migrate.

Over the years there are more and more things that are very annoying to collection developers and that aren’t getting fixed (despite PRs existing).

Examples:

4 Likes
CfgMgmtCamp 2026 discussion (0/12): Problems in the Ansible world and how to improve on them
CfgMgmtCamp 2026: Community and Contributor Summit
The Bullhorn #218
2

Related discussion: Should we be using molecule or ansible-test for unit/integration tests?

1 Like
3

For the record, and I stated this in another forum post, there are no plans to further extend ansible-test for collection testing. What it is, is basically what it is. (ansible-test is what it is for collection testing)

4

+1 to @sivel’s point here

The discussions over the past few months, and reinforce by last weeks in-person discussion at CfgMgmtCamp are strongly leaning towards having ansible-nox as the unified approach (Repo & Docs) for testing, which I personally really like the sound of.

5

We’re making good progress on adding a nox session for running molecule via antsibull-nox too. PR in progress: Add Molecule Session Support by dbrennand · Pull Request #187 · ansible-community/antsibull-nox · GitHub :slightly_smiling_face:

1 Like
6

The main problem is that antsibull-nox can’t fix some of the issues ansible-test has. It would be better if ansible-test would be continued to also be developed for collections.

Also I’m pretty convinced that forking ansible-test (as it is apparently discussed) is not a good idea, since it’s mainly a waste of time and a huge and unnecessary duplication of code.

7

Couldn’t the parts that are useful to collection testing be “moved out” of ansible-test itself into either a shared tool or a library.
I also don’t think forking ansible-test entirely makes sense, but separating the bits that are useful might make it easier for everyone to maintain/use?