Ansible Collection Testing Strategies

[felixfontein]

A somewhat related discussion: Should we be using molecule or ansible-test for unit/integration tests?
And another important piece: ERROR: invalid-extension: Official Ansible modules must have - #4 by sivel

So basically right now I would definitely recommend to run ansible-test sanity --docker, but I’m not so sure anymore about unit and integration tests. I’ve (traditionally) used ansible-test for both of them (and am still doing that in all my collections), but especially after reading ERROR: invalid-extension: Official Ansible modules must have - #4 by sivel I’m wondering where this will go. (Also ansible-test has been somewhat stale in the last few years, it’s next to impossible to even get very simple improvements in, see for example Sign in to GitHub · GitHub)

@thorian93

But first: Can anyone comment, if the ansible-test $MODE --docker is future-proof, or are we all moving towards Tox Ansible ultimately?

I’d definitely use ansible-test for sanity tests. If Tox Ansible runs ansible-test sanity --docker, then I guess that’s also fine.

For unit tests, I’d also still use ansible-test units --docker. (Though I guess Tox Ansible also works here, but I guess you need to install the Python versions yourself, while ansible-test units --docker comes with them out of the box. That’s generally a lot less hassle )

For integration tests, I’m not sure.

@thorian93

Moving on: Just yesterday my integration tests started failing, as the new quay.io/ansible/default-test-container:10.0.0 container was released, which seems to be built on Ubuntu 24.04. The reason for the failures is, that the oldstable version of my software does not support Ubuntu 24.04 anymore. I assume the older version of the container was based on Ubuntu 22.04 which all versions of my software support. Now I am looking for solutions/workarounds, that do not involve creating VMs at some cloud provider.

Here it sounds like you are using ansible-test with the default Docker container from ansible-core devel, which can randomly break due to changing things. Generally I would avoid using the default Docker container for tests that require specific setup (like installing software that’s not available everywhere). Better use one of the OS containers for that, for example ubuntu2204: ansible-test integration --docker ubuntu2204; you can find the list of all supported containers here: ansible/test/lib/ansible_test/_data/completion/docker.txt at devel · ansible/ansible · GitHub (or look at the ansible-test --help output). (Put in the right branch into the URL to see the supported OS containers for a specific ansible-core 2.x version; the list changes from time to time.)

Likely using --docker ubuntu2204 instead of --docker default (which is the long form of simply --docker) should fix your CI At least until the ubuntu2204 container is removed (which I hope won’t happen for some time).