It has been mentioned a few times by the Ansible Core team here on the Forum [1] [2] that there are no plans for new testing functionality for collections in ansible-test or to extend ansible-test to be more flexible for collection testing. Furthermore, as Sivel mentioned in his post, ansible-test was never actually designed for collection testing:
"The original reason that ansible-test became the standard for testing collections was largely because nothing else existed, so collections inherited the same rules as ansible-core.”
Right now the Ansible documentation contradicts this messaging, so our first item is to update the documentation to explicitly state that ansible-test is not a tool for collection testing. We want to stress that the purpose of this is not to say collection maintainers should stop using ansible-test for testing collections at this time, we are updating the documentation to clarify the purpose of ansible-test.
In the next two weeks we will be sharing another update to kick off the unified strategy for collection testing, outlining some goals of the initiative and providing an overview of the current collection testing landscape so we’re all on the same page and in a good place to begin further discussions on this area. Please keep an eye out for this upcoming post and the collection-test-strat tag.
Hey @felixfontein - Yes, right now ansible-test is the collection testing tool. Nothing is being deprecated at this time.
I know things seem right now a bit vague . When this post I mentioned above comes about the unified strategy for collection testing initiative, I hope things will become clearer and we all can discuss what comes next.
Saying that a tool is not for a purpose it was previously used for is deprecation (there are other things—like warning messages—that are also deprecation, but that doesn’t make this not deprecation.) This feels very strange as a first step.
Whether it was intended or not is somewhat moot. The reality is that it was modified to meet the needs for the 2.10 release, and has been the defacto standard. So it is a tool for collection testing. It’s just not being extended further for this need. What it delivers is still useful, and largely mandated or required for collections.
From a testing perspective ansible-test is critical for the Windows collections as it allows access to the core-ci API to spin up short lived AWS/Azure instances. I’m not sure if this could ever be replaced without the core-ci infrastructure being made a public API and I don’t think that would ever happen.
From a sanity perspective then yea we could easily replace a lot of the functionality outside of Ansible if that is truly desired.
. When this post I mentioned above comes about the unified strategy for collection testing initiative, I hope things will become clearer and we all can discuss what comes next.