During the Ansible Contributor Summit at CfgMgmtCamp, we (including SC members) discussed improving the stability of the community package.
Current collection requirements only mandate testing against stable ansible-core versions. While many maintainers voluntarily test against devel, failing to catch regressions early leads to:
User friction: Problems are only discovered after a core release.
Cascading failures: Broken collections often break their dependents, impacting the wider ecosystem.
To ensure content stability, I propose requiring collections to run tests against the devel or milestone branches of ansible-core.
I am not from any commitee, but if you mandate to test agaist devel you have to provide tools to isolate tests. As testing agains moving branch will just lead to flaky tests.
Let me explain. If I run tests against devel every day - even if I do not change anything in my collection / content tests might fail just because devel has an issue or just in a middle of refactoring.
But it I test agains specific commit in devel - every time tests run if collection has not been changed result will be the same (devel frozen to specific commit).
Question is how to keep this commit I am running againts moving together with devel. That might be some tool (or its extension) like dependabot or renovate. Renovate seems to have an option to follow git references - Git Refs - Renovate Docs . This update to git reference (commit) to devel should run every week or month (depends on maintainer and how ofter they relese) and create PRs. This will allow to fix all test depedencies.
So when there will be failure because of devel update it will be evident what change (from what ref to what ref caused it).
IMO I think this requirement would be good for the wider ecosystem and allow maintainers to catch changes in ansible-core early. If this kind of requirement was in place before the ansible-core 2.19 release then possibly the whole saga with ansible.netcommon may have been caught earlier as the CI would have been failing.
to me (a maintainer of a few collections) it doesn’t feel like a big problem:
doesn’t happen too often
you can ignore failures in PRs until you fix them - yes, it doesn’t look very good, but one failure from typically tens of others green checks we run in collections doesn’t look too scary
It’s my personal view ^, though I see your concern and situations can be different
! Anyway, for people who wants more stability, there’s an option to test against the milestone branch instead of devel as PR suggests
We’ve generally said that if you test against devel, it should be “non voting”, meaning it should not impact normal operations if it causes issues. Depending on the setup, a nightly or weekly job could still report failures against devel.
The milestone branch was created to give an alternative that was closer to devel, but had predictable updates and was consistent between those updates. As of the 2.21 release schedule, we’ve switched from 3 dev phases (when the milestone branch was advanced) to 5 or 6 dev phases, and as a result the advancements are more frequent (4 weeks instead of 8).
I personally prefer devel over milestone, for mainly two reasons:
milestone changes on certain dates, and then you get a month worth of changes. If you can plan to invest enough time in your collection every time milestone is bumped, this should be fine. But if you can’t plan that much time in ahead of time for these fixed dates, this isn’t great.
It can be that something in milestone is broken. This won’t get fixed until milestone is updated again. (Bugs in devel might already get fixed a few days or even just hours later.)
Also you won’t be able to test bigger changes when they get merged, only some weeks later. For example, there are some bigger PRs open in ansible/ansible right now (projections + setting vars, support for old Python versions, …) that haven’t been merged yet or were merged after the start of the last development cycle (March 2nd), so you will only encounter them in CI when milestone is bumped again - basically when ansible-core 2.21.0b1 is released (April 6th). This reduces the time you have to react to these big changes, and potentially find bugs in these changes that affect your code (and getting them fixed before the release candidates start). As opposed to the large changes of ansible-core 2.19 the changes that are waiting to get merged now likely don’t affect many collections, so this likely isn’t a problem, but for some this might be.
I want to emphasize again that the above points are my reasons to prefer devel over milestone, and that for other people these points could be less relevant or totally irrelevant. For that reason requiring that at least one of devel and milestone is used sounds good to me.
The scenario @kks describes is a perfect mix of devel and milestone IMO, since the maintainers control when the branch is advanced, and first run CI to see whether it breaks something before bumping devel. This might be the best solution for most maintainers for whom simply using devel is too much work / too dangerous, and who also share the concers I have for milestone. (You could even set up something so the update PRs are automatically merge if CI passes. That way you only have to take a look when something fails, and then you don’t have that much pressure as with devel or milestone to fix it.) We definitely have to make sure that our formulation of the rule allows to proceed this way as well. (I.e. fix a commit on devel and update it at least ~once per month to the latest.)
I basically agree that this is a good requirement. But since this is “just” about finding out about new linting requirements early, should we mention that it’s OK if those tests are non-voting? Although there’s always the risk that people ignore failures when the tests are non-voting.
Should we also explain the pros and cons of testing with devel and milestone here, or somewhere else and link to this? Or does there already exist such an explanation somewhere that we could link to?
All in all, sounds like a good idea to me. Let’s just clarify some details and the small print.
That’s not true; it’s also about integration or unit tests failing early because of changes in ansible-core that affect something used by the collection.
That’s a good question. I am tempted to say that non-voting is fine, but I’m a bit worried that it’s taken as an excuse to simply ignore the results, which is contrary to what we want to achieve by requiring to test against these branches. On the other hand, it’s better to have some “always red” as a remainder that something isn’t fully right.
(I know because I have a CI job in antsibull-nox which tells me when this happens, so I can adjust milestone’s supported Python versions in case they change.)
You MUST run the ansible-test sanity command against the devel or milestone branch of ansible-core so that you find out about new linting requirements earlier.
That’s why I said it’s just about finding out about new linting requirements. Maybe we should drop the “linting”, then, and only mention “requirements”? Or maybe change this to
so that you find out about issues with the next ansible-core release early
Sorry, I didn’t had time to look at the PR yet. I was assuming that this is about generally testing the collection, and not just about sanity tests.
The PR only requires that sanity tests are run with ansible-core devel or milestone, and actually drops the requirement that all of CI runs with all supported ansible-core versions (I wrote a comment on that).
I think we should rather require that all of CI is also run with devel or milestone (or “something inbetween”, i.e. what @kks sugested).
At first glance, this looks good to me. I’ve just found a (probably very minor) wording issue. And I’m not even sure there, because I’m not a native speaker of English.
When is it expected for collections to comply with this change? There will be a certain period of time for collections to update their CI for this new requirement? How long would that be?
Is this already documented somewhere generally for new inclusion requirements?
Disclaimer: I have a personal interest that they are
I think it would be easier for me to understand what this means if I could see an example. And / or to see a possible implementation of this in the collection_template.
Or is the collection template already OK and we just have to make things clearer in the documentation? Otherwise, I think we also have to update the template.
@mariolenz Yes, I think it’s a good example to discuss
It looks like the collection’s CI is already compliant with this requirement as milestone is being tested in sanity and unit tests. It’s also running once a day which is far more frequent than the once per week requirement.
EDIT: As for the collection template, the workflow is already set up to test against devel and runs daily:
@SteeringCommittee and community, please vote. The poll will end (if no objections) 2026-03-26.
If the proposal is accepted:
We’ll announce the change via Bullhorn and under news-for-maintainers tag
New inclusion candidates will have to abide; the existing collections will have at least a couple of months. Anyway, Mario always creates an issue in problematic collections first before initiating removal, so maintainers will have plenty of time to adjust.
Assuming the proposal is merged , let’s ensure that the GitHub Actions and in-line documentation in those workflows is updated. Even for non-included collections, we can still guide people in the right direction. Having a weekly scheduled job against devel would help all collection maintainers.
