Right now all shared code (and docs fragments, which I’ll silently include here) for collections lives in ansible-core itself: https://github.com/ansible/ansible/tree/devel/lib/ansible/module_utils contains code that can be used by both modules and plugins, and all other parts of ansible-core (that aren’t marked as private) can be used by plugins.
The problem with adding new shared code to ansible-core is that it takes some time (potentially years) until all collections can use it, since they usually support multiple ansible-core versions (some even still ansible-base 2.10 and Ansible 2.9). Depending on the kind of shared code (for example new module utils coming with docs fragments), collections have to wait until they dropped support for the last ansible-core version that did not yet have this shared code, which can take many years.
This would be a lot easier if there exists a collection that contains such new code. Other collections can depend on it; bumping a minor version for such a collection (and getting end-users to upgrade to the new version) is a lot easier than getting end-users to upgrade their ansible-core version (which always comes with breaking changes as well when you get new features).
To avoid this being too theoretical, let’s consider an example: some plugin_utils and docs fragment which allows inventory plugins to explicitly include/exclude specific hosts by Jinja2 conditions. I’ve started writing code for that in community.docker (see the PR), but that’s something that could be used in many other inventory plugins in many other collections as well. Since depending a random collection on community.docker just for that piece of code sounds like a bad idea, the best place right now for adding such a feature would be ansible-core. But then this could land earliest in ansible-core 2.17 (2.16 already had its feature freeze), and community.docker supports multiple older ansible-core versions (2.11+ right now), so it would take many years until community.docker would be able to use such functionality from ansible-core itself. So right now the only “good” choice is adding that code to community.docker, and the only “good” choice for other collections who want to use that code is to copy (!) it so they don’t have to depend on community.docker.
There are some choices for such a generic collection that contains shared code. For example, there’s already ansible.utils. But ansible.utils already contains multiple end-user facing tools (plugins and modules). When quickly looking at this idea on IRC (#ansible-community) / Matrix (#community:ansible.com), @briantist and me agreed that such a shared-code collection should better not contain anything for end-users (i.e. no plugins, modules, and roles). The basic idea is to avoid complications due to needs of end-user things, like removals of features that turned out to be not so good - these require a major version bump, which quickly causes problems.
The major problem with having a common collection is versioning. If there are only ever new 1.x.y releases and the collection sticks to semantic versioning, all collections depending on it can depend on
>=1.x.y,<2.0.0 (for some specific minimum
1.x.y depending on the collection). So for any set of collections that depends on the shared code collection, you can always pick a version of the shared code collection that makes all of them happy. But as soon as we decide to bump the major version, dependency hell can easily break loose. foo.bar needs 1.x.y of our common collection, baz.bam needs 2.x.y, and foo.bam is happy with both 1.x.y and 2.x.y. With such dependencies there will be no way to install all three of foo.bar, baz.bam, and foo.bam at the same time… So avoiding new major releases is the best strategy, combined with forcing users of this collection to be as nice as possible to all other users. For collections included in Ansible we already have some rules which cover this pretty well: Ansible community package collections requirements — Ansible Documentation (Basically they say that if you screw up and do not support the latest major release when a new major Ansible release comes up, your collection will be thrown out. This is currently mainly aimed at ansible.netcommon, a shared requirement of most network collections, that actually caused such problems in the past.)
One other topic is: what to add to this collections? If every random feature someone wants to have is added, we end up with a huge mess that grows too large and sooner or later causes problems (even if it “just” means that part of the collection are basically unmaintained). So IMO some strict rules for adding new content (or in other word: features) is needed.
I don’t want this post to end up as a large discussion of all possible things, so I’m going to stop here. Let’s discuss what everyone thinks of:
- having such a collection (or multiple such collections?),
- which rules should apply (for adding features, who should maintain it, …),
- … anything else you can come up with?