Ansible Collection Testing Strategies

Project Discussions
, , , ,
1

I have been at this point several times now, and I am starting to wonder if I am alone with this struggle, or if I am just the last one, who did not find the holy grail yet. This is probably not a simple technical question, but maybe more about philosophies, so brace yourselves before reading on.

I am maintaining the checkmk.general collection, which contains both roles and modules.

All the below runs both on GitHub workflows and locally.

Now for roles, I am pretty happy with molecule, which to this day seems to be the go-to testing approach for Ansible roles, if one does not want to build something elaborate themselves. Any objections so far?

Now when it comes to modules, it gets more interesting: Sanity testing is one thing, that I do with ansible-test sanity --docker. Easy-peasy.

I have no unit tests yet, as we do not have time to create them currently.

But the real beast are integration tests. As the collection targets a piece of software, I need to install the software first. Currently, I am using the aforementioned ansible-test integration --docker approach, installing the software into the container and then running my module(s) against it within the container. This way, I have no external dependencies and the tests are mostly stable and reliable. This worked just fine until yesterday. But first: Can anyone comment, if the ansible-test $MODE --docker is future-proof, or are we all moving towards Tox Ansible ultimately?
Moving on: Just yesterday my integration tests started failing, as the new quay.io/ansible/default-test-container:10.0.0 container was released, which seems to be built on Ubuntu 24.04. The reason for the failures is, that the oldstable version of my software does not support Ubuntu 24.04 anymore. I assume the older version of the container was based on Ubuntu 22.04 which all versions of my software support. Now I am looking for solutions/workarounds, that do not involve creating VMs at some cloud provider.

After drawing this rather complex picture, let me try to generalize the question that keeps haunting me: What is the best way to integration-test an Ansible collection against a software product, that needs to be installed locally, without using a complex environment with a lot of dependencies and a ton of resources at a cloud provider for VMs and such?

I am looking for all kinds of comments, no matter if they confirm my approach, show me a better strategy or tell me what a stupid dingo I am doing the things I do the way I do them.

Thanks everyone! :v:

2

A somewhat related discussion: Should we be using molecule or ansible-test for unit/integration tests?
And another important piece: ERROR: invalid-extension: Official Ansible modules must have - #4 by sivel

So basically right now I would definitely recommend to run ansible-test sanity --docker, but I’m not so sure anymore about unit and integration tests. I’ve (traditionally) used ansible-test for both of them (and am still doing that in all my collections), but especially after reading ERROR: invalid-extension: Official Ansible modules must have - #4 by sivel I’m wondering where this will go. (Also ansible-test has been somewhat stale in the last few years, it’s next to impossible to even get very simple improvements in, see for example Sign in to GitHub · GitHub)

I’d definitely use ansible-test for sanity tests. If Tox Ansible runs ansible-test sanity --docker, then I guess that’s also fine.

For unit tests, I’d also still use ansible-test units --docker. (Though I guess Tox Ansible also works here, but I guess you need to install the Python versions yourself, while ansible-test units --docker comes with them out of the box. That’s generally a lot less hassle :slight_smile: )

For integration tests, I’m not sure.

Here it sounds like you are using ansible-test with the default Docker container from ansible-core devel, which can randomly break due to changing things. Generally I would avoid using the default Docker container for tests that require specific setup (like installing software that’s not available everywhere). Better use one of the OS containers for that, for example ubuntu2204: ansible-test integration --docker ubuntu2204; you can find the list of all supported containers here: ansible/test/lib/ansible_test/_data/completion/docker.txt at devel · ansible/ansible · GitHub (or look at the ansible-test --help output). (Put in the right branch into the URL to see the supported OS containers for a specific ansible-core 2.x version; the list changes from time to time.)

Likely using --docker ubuntu2204 instead of --docker default (which is the long form of simply --docker) should fix your CI :slight_smile: At least until the ubuntu2204 container is removed (which I hope won’t happen for some time).

3 Likes
3

Thanks a ton for your extensive answer @felixfontein! :heart: If I could give two or more likes, I would.

It is very relieving for me to read, that for the most part I am still on a good trajectory. At least that is what I hear, when esteemed community members do what I am doing. :white_check_mark:

Now for the integration tests: Your suggestion was actually among my first thoughts. But the OS containers only support their native Python, right? That would limit my testing scope considerably.

I guess my hard-to-swallow-pill here is, that we would need unit tests to make sure our modules work with several Python versions and use the integration tests with a limited set of Python to only verify successful interaction with the software. Does that make sense?

4

Yes, that’s (unfortunately) the case. But on the other hand, OS containers allow you to test more closely actual environments that your users might be using, as opposed to the ‘generic’ default Docker container. For collections that use OS containers I usually use a bunch of different OS containers (with different ansible-core versions, and including the ones here: images/ansible-test at main · ansible-community/images · GitHub).

That would help, but generally you can already cover quite a few Python versions with OS containers, and except some extremes like Python 2.7 that’s often enough.

What you can also do is have some things that only run in CI (because they’re more annoying to get to run locally), where you start the software that you need in a container that you connect to the container used by ansible-test; then you can use ansible-test’s default container without having to worry to install the software in it. Obviously that only works with services that can be accessed over the network, which happens to be the case for your collection (if I understand it correctly). I think the mysql or postgresql collections do something similar (to make it easier to test against specific versions of the DB).

Then you can use some OS containers for things that you can also easily try locally, and the special setup in CI to really cover all Python versions.

2 Likes
5

I sure do appreciate all the input, thanks a lot! :pray:

I will discuss the different approaches with my team and maybe update here for everyone reading along, which path we went down. :motorway:

1 Like