Up until now, much of our work on the ansible package has gone towards reviewing collections for inclusion and releasing the package. I’d like to do more to test and improve the collections in the ansible package. To that end, I’ve been working on changes to antsibull and the release playbook to allow us to more effectively test the package. In addition to @mariolenz’s work on removing unmaintained collections, this should help improve the quality of the ansible community package.
As part of the release playbook, we now run a test to ensure that Python files in the ansible package are syntactically correct and can (at least with the Python interpreter used to build the release). This was added in build-release: ensure Python files in the package compile (#552) · ansible-community/antsibull@27754fb · GitHub.
I have a WIP patch that clones collections and ensures that its contents roughly match the Galaxy collection artifact. We cannot reliably run tests from the Galaxy collection artifacts, as they may be missing files needed to run tests, so it’s important that the contents in the git repositories are actually what’s being uploaded to Galaxy.
I’ll extend that PR to support running a subset of sanity tests across the cloned collections. I’m proposing we enforce the following sanity tests:
I want to avoid running sanity tests that are cosmetic in nature or take a particularly long time to run or that overlap with other tests. There are over 100 collections in the ansible package, so it’s important that each individual collection’s tests don’t take too long. I mainly want to ensure that code is syntactically valid  and we can build docs without errors. We might also consider running
These tests would ideally be run on a regularly scheduled basis and not as part of the release process; we don’t need to make that any longer/more complicated.
The initial results are… not super promising. I ran the four tests above over all the collections from 9.0.0b1 using ansible-core 2.16.0rc1. It took a little over an hour on my laptop. 40.20% (41 / 102) of the collections failed at least one of these tests. There are a smaller number of collections with file differences between tagged releases and Galaxy artifacts.
We can always go through the collections and file issues. We can remove collections that don’t respond after a certain amount through the regular process for nonresponsive collection maintainers. I’m open to ideas here.
Thanks to @rfc2549 for his previous work on running sanity tests for collections in the ansible package.
We’ve had multiple problems with syntax errors that break packages. ↩︎