During the Ansible Contributor Summit at CfgMgmtCamp, we (including SC members) discussed improving the stability of the community package.
Current collection requirements only mandate testing against stable ansible-core versions. While many maintainers voluntarily test against devel, failing to catch regressions early leads to:
User friction: Problems are only discovered after a core release.
Cascading failures: Broken collections often break their dependents, impacting the wider ecosystem.
To ensure content stability, I propose requiring collections to run tests against the devel or milestone branches of ansible-core.
I am not from any commitee, but if you mandate to test agaist devel you have to provide tools to isolate tests. As testing agains moving branch will just lead to flaky tests.
Let me explain. If I run tests against devel every day - even if I do not change anything in my collection / content tests might fail just because devel has an issue or just in a middle of refactoring.
But it I test agains specific commit in devel - every time tests run if collection has not been changed result will be the same (devel frozen to specific commit).
Question is how to keep this commit I am running againts moving together with devel. That might be some tool (or its extension) like dependabot or renovate. Renovate seems to have an option to follow git references - Git Refs - Renovate Docs . This update to git reference (commit) to devel should run every week or month (depends on maintainer and how ofter they relese) and create PRs. This will allow to fix all test depedencies.
So when there will be failure because of devel update it will be evident what change (from what ref to what ref caused it).
IMO I think this requirement would be good for the wider ecosystem and allow maintainers to catch changes in ansible-core early. If this kind of requirement was in place before the ansible-core 2.19 release then possibly the whole saga with ansible.netcommon may have been caught earlier as the CI would have been failing.
to me (a maintainer of a few collections) it doesn’t feel like a big problem:
doesn’t happen too often
you can ignore failures in PRs until you fix them - yes, it doesn’t look very good, but one failure from typically tens of others green checks we run in collections doesn’t look too scary
It’s my personal view ^, though I see your concern and situations can be different
! Anyway, for people who wants more stability, there’s an option to test against the milestone branch instead of devel as PR suggests
We’ve generally said that if you test against devel, it should be “non voting”, meaning it should not impact normal operations if it causes issues. Depending on the setup, a nightly or weekly job could still report failures against devel.
The milestone branch was created to give an alternative that was closer to devel, but had predictable updates and was consistent between those updates. As of the 2.21 release schedule, we’ve switched from 3 dev phases (when the milestone branch was advanced) to 5 or 6 dev phases, and as a result the advancements are more frequent (4 weeks instead of 8).