Collection inclusion: vote on collections submitted for inclusion before starting the review process

Hi everyone.

I still have a feeling that we shouldn’t include all submitted collections in the package even if they satisfy the requirements.
And I’m not alone here as a few Steering Committee people already raised similar concerns.
In this case, the context is these two requests made by same maintainer:

I think, first, Steering Committee should decide whether the submitted stuff is worth having in the package.
It’s like in most of the projects: maintainers decide whether to include a feature or not. Maintainers usually don’t commit changes automatically just because the changes satisfy all formal requirements.
Like, say, in case of collections we don’t merge stuff just because a PR have tests and a changelog fragment…

My proposal is to discuss if it’s good to introduce a requirement that the Steering Committee will discuss and necessarily vote on every inclusion request before starting the review process.
The addition to the process could roughly consist of the following steps:

  • After a request is submitted, a topic is created with a default vote day set (say in a week or two)
  • During this period, the participants can take a brief look at the collection, express their opinions, ask submitters for clarifications, etc.
  • Then the vote will happen, if it’s approved, from this point the inclusion process will continue as is now

Ideas? FYI I’m gonna start vote at some point (maybe on a policy change PR depending how the discussion will go)

1 Like

This immediately brings to mind this discussion, which I think is relevant: Should we keep the fortinet collections (fortinet.fortimanager, fortinet.fortios) in the community Ansible package?

I feel strongly against the idea that we should reject collections that meet the requirements based on a purely subjective “worthiness” metric.

It’s very subject to bias, more so than other criteria.


I would like to understand what’s at the root of this idea. It feels like we’re starting with the conclusion (“some collections are not worthy of being in the package”) without defining why we think that.

If the motivation is, “There’s too many reviews to do” then that’s how we should center the discussion. If the motivation is, “The package size is too large”, then we can talk about that, if it’s “I don’t see the use for certain collections and don’t like seeing them” we can even discuss that! If this comes from the opinion that the community package should simply not exist (something that has also been raised), then we can revisit that too.

Whatever it is, I think we ought to first be discussing the problem statement before we propose solutions.

3 Likes

@briantist thanks for the fair input!

Not on my side but can be fair for others

Neither this one can be fair for others

Neither this one (for me) but could be revisited if there’s enough support.

I see the following reasons:

  • Just the quality of stuff: even if formally the code satisfy, some people might just not like the way things implemented on another level. And you can’t reject it because say “there’s a lot of boilerplate code” because it’s not a requirement not to have it written that way. It’s also subjective.
  • If i wrote several services that only i use locally or 10-100 people in my company/customers, is it fine if i submit collections for them whatever reason, say if just because i don’t want to create and maintain my own website for my collections docs - I want to delegate it to ansible doc folks :joy: I.e. the audience or motivation behind requesting the inclusion don’t fit the package scale.

Yes, both the reasons are subjective but there’s a dozen of people in the committee, not a single person who decides.

Thoughts?

1 Like

Enforcing quality standards is the job of the inclusion requirements. Disliking implementation choices is not a good reason to exclude something that otherwise meets the requirements. The committee is currently allowed to exclude things if “presence of the collection will significantly deteriorate the Ansible package user experience or the package build process”, which a small collection where you dislike some of the code choices is not doing.

Again, if this is a new requirement you want to impose it should be objective measurement(s) listed as a requirement (e.g. “Collections must have a minimum level of interest from the community, so new inclusion requests will not be reviewed until they have received at least 17 upvotes or the Galaxy collection has been downloaded 1700 times.”)

The Steering Committee cannot be expected to intuit the scale of possible interest for every inclusion. For example, since they’re a Chinese company that you’ve never had occasion to deal with it may surprise you to learn that IEIT Systems had $3.25 billion in revenue for Q3 2023. (Whether that means any actual interest from end users in using Ansible with a particular product of theirs is more nebulous, but it’s not unreasonable to believe that some of their customers would be well served by its inclusion in the community package.)

2 Likes

Exactly this.

I’m also glad you mentioned that one of the inclusion requests called out specifically that the collection was for a Chinese company, because I felt uncomfortable about the way that was presented, as though it should matter what country’s users a collection may serve.

As you pointed out, while perhaps none of us have had occasion to work with that company, it doesn’t mean it’s unworthy of being in the package. This is precisely the type of bias I am worried about.


@Andersson007 thank you very much for those explanations and hypotheticals, those are very helpful to understanding the root of the discussion!

some people might just not like the way things implemented on another level. And you can’t reject it because say “there’s a lot of boilerplate code” because it’s not a requirement not to have it written that way. It’s also subjective.

This may simply be an irreconcilable difference of opinion between us, but for me someone “just not liking it” without being able to give a concrete criteria that can be applied evenly is not a good reason to reject.

  • If i wrote several services that only i use locally or 10-100 people in my company/customers, is it fine if i submit collections for them whatever reason, say if just because i don’t want to create and maintain my own website for my collections docs - I want to delegate it to ansible doc folks :joy: I.e. the audience or motivation behind requesting the inclusion don’t fit the package scale.

@flowerysong already said what I would have (in better words), but I do want to briefly call out the docs thing, since we literally have had inclusion requests that were motivated more by documentation than their code being included (at least on the surface it seemed that way).

I still don’t think that motivation is a reason to reject it; presumably they think enough people are going to use it that they think there’s a point to having the documentation available.

But I think if we’re seeing this, then as a community we could put more effort into making it easy to publish your own docsite since that’s something people seem to want. Totally different discussion though :slight_smile:


Finally, I want to propose a slight change to your proposal. As @flowerysong pointed out:

The committee is currently allowed to exclude things if “presence of the collection will significantly deteriorate the Ansible package user experience or the package build process”

This could be expanded to allow the committee to veto inclusion for any reason as long as it’s put to a vote. I’m not thrilled about the idea, but I would suggest these differences from your current proposal:

  • The default action is to accept if inclusion criteria are met.
  • Having a vote to include is an exception that must be initiated by quorum of N members (could be as few as 1, that’s a detail).
  • The vote may be initiated at any time during the inclusion process, in order to accommodate a wide range of concerns, but depending on the nature of the concerns, voting may be extended until after other inclusion criteria are evaluated.
  • The member(s) who called the vote can explain their case and what they don’t like about the collection, and we can discuss and decide.
  • The vote is a vote to include not exclude. I think it’s important to word it this way because inclusion is the default.
  • During discussion, we may want to figure out if we think the issues raised are reconcilable by the maintainer (code style?), or if they are something we think could never be reconciled (???)

I am worried that if we have to vote to include on every collection, before the criteria process, we will tend toward not including. I have no evidence for that, it’s just a feeling, and I think that’s a negative outcome (my opinion).

tl;dr: if we make any change, I think it’s as a sort of escape hatch, and it’s an exception, but I think I’d prefer not doing this and sticking to criteria that are as objective and measurable as possible.

1 Like

@flowerysong @briantist thanks for the great discussion!

Not sure if it would work as a good measurement as it can be easily achieved by using a script:)

That’s a very good point, worth a topic, would you like to start it?

I like it, looks much better then the initial

Sounds fair

@flowerysong @briantist I’d say I feel persuaded by you folks:) Let’s wait and see what others think.
The question about docs topic is relevant

1 Like

The discussion on what to include (and what not to include) is probably almost as old as the ansible package itself (even before the collection split) and I’m very sure this won’t be the last time we discuss this :wink:

Every time this comes up I go through some extreme ideas (like abolishing the package completely, or removing everything that’s too vendor-specific) and quickly throw each of them away again, since they all kind of suck. (Abolishing the package is a valid discussion, I don’t want to say that the idea itself sucks, just that starting that discussion in this context is not a good idea. Having that as a separate discussion is totally fine, in case someone wants to reboot that discussion.)

I’d like to have some criteria which allow to weight size of the included content vs. usefulness to the community at large, which I think would be a good metric (assuming the general quality of a collection is good enough, for which we already have criteria). But I don’t think it’s possible to come up with a objective way to judge this. Just measuring “usefulness to the community at large” is something that I think is impossible to judge - @flowerysong’s argument above is a very good indication for that. Looking at “objective” stats such as upvotes on the GH inclusion request or Galaxy download counts is a proxy that’s IMO not very close to the truth (and can easily be faked). Just restricting upvotes to a “trusted” set of community members (I’m not going to say how to find such a set, in fact I have no clue how to do that, but let’s assume we have it for some reason) isn’t that much more helpful either, since then we are building a package specific to the needs of this explicit set, and we might miss large parts of the community which happend to be not represented by this set. So I’m pretty sure it’s not possible to come up with truly objective and useful metrics.

So yeah, what we are left to do is probably things like (most of them already mentioned above):

  • make it easier to create your own collection docsite (BTW, antsibull-docs now has some docs: antsibull-docs – Ansible Documentation Build Scripts);
  • maybe even make it possible to have your collection being documented on docs.ansible.com without being part of the Ansible package (that comes with its own can of worms);
  • improve documentation rendering on galaxy.ansible.com (so most of the information is already there and you don’t need another place for the docs in many cases? there are a lot of things left now, like extra docs, role argument specs, or docs fragments from other collections which break rendering completely);
  • make it easier to not use the ansible package, but simply use the collections you need (EEs? your own specialized ansible package build? …?);
  • … more ideas?
4 Likes

Thanks, folks, for sharing the great thoughts on the topic! Really appreciated.
All the stuff mentioned sounds fair.
Once again feeling persuaded that keeping the true level playing field is important here.
I’ll put the resolved tag. Later if needed, we can refer to this discussion if similar ones pop up.
I think Galaxy rendering improvement is a great idea, so any PRs to improve it will be much appreciated.
Another topic to discuss is that we should make the inclusion process more effective. Please take a look.

Thanks!