Should automation developers be starting with role-only collections?

Project Discussions
, ,
1

Should automation developers be starting with role-only collections?

I genuinely cannot answer that, and following up on my earlier posts about group two (the folks testing roles and playbooks rather than Python modules), what surprised me digging in is that there does not seem to be an official answer either. The guidance across docs.ansible.com is not consistent about it, and I could not find an authoritative position anywhere on whether a role or playbook author should be working inside a collection at all.

That unanswered question matters because the tooling has quietly answered it for us. The modern, supported, CI-ready testing path (ansible-creator scaffolding, pytest-ansible, tox-ansible, antsibull-nox) has converged on the extensions/molecule/ layout, which only exists inside a collection. So the moment you want supported testing, you are in collection territory whether or not you ever decided a collection was the right unit for your work.

Getting reliable information along the way is harder than it should be too. In an earlier post I said standalone role testing was stable and well documented. Having gone through the current state, I have to walk that back. The examples are abundant, but most of them predate Molecule’s “ansible-native” rework (v25.9, September 2025), which changed how scenarios are configured, and they teach a model the project now treats as legacy. Those older resources are still what you find first: a search for how to test a role surfaces the old driver:/platforms: style, some of it very recent (I found a tutorial dated April 2026, tested against Molecule 26.4.0, still teaching the deprecated config). Meanwhile Molecule’s current docs no longer include a getting-started guide specifically for a role, only playbooks and collections, and the official guidance is not consistent about which shape it assumes (the Dev Tools testing page documents only a role within a collection, while the Sample Ansible setup page assumes bare roles and playbooks). To be clear, the recent Molecule docs rework by @cidrblock and team is a genuine and much-needed improvement. The problem now is discoverability: the good current material is not what a search turns up.

Even past the outdated information, the structural complexity is real, and it spikes as soon as content moves into a collection. This is where I think we under-serve role-only users:

  • A wave of additional tools arrives at once. Beyond Molecule you are now looking at ansible-creator, ansible-test, pytest-ansible, and one of tox-ansible or antsibull-nox, plus the galaxy.yml and namespace and versioning that come with the collection format itself. Each looks optional in isolation, but they interlock.
  • Documentation and examples for role-only collections are thin. Most collection-testing material assumes there are modules in the collection, so a role-only author has to work out which of that long list actually applies to them.
  • Each of those tools brings its own configuration, and that is where the hidden complexity lives:
    • Where the config even lives. Molecule alone splits across a shared extensions/molecule/config.yml, a per-scenario molecule.yml, and an inventory.yml, and then tox-ansible or antsibull-nox adds a tox.ini or a noxfile.py plus antsibull-nox.toml on top.
    • Redirection and inheritance between those files. The shared base config is inherited and overridden per scenario, and one tool invokes another (for example tox-ansible runs Molecule through pytest-ansible), so tracing what actually ran, and why, means chasing config across several files.
    • The config is python-centric. A noxfile.py is literally Python, and the pytest integration test is a Python file. Someone whose whole world is YAML is now editing Python to run a test.
    • None of it is familiar. nox, tox, and pytest are conventions from the Python developer world, not something a YAML automation author has any reason to already know.

So I think we need to definitively answer this question first, should a role or playbook author be starting in a collection? If the answer is yes, then the role-only, content-focused case needs to be tailored to automation developers writing YAML rather than python. It needs to be first-class and out of the box, with its own documentation and examples and not something an author reverse-engineers from module-oriented tooling.

Originally posted in Unified Collection Testing Strategy - Kick off & Landscape Overview - Split into separate post.

3 Likes
Unified Collection Testing Strategy - Kick off & Landscape Overview
2

Proposal: A standard arrangement for Molecule scenarios

Molecule can already test a role, a playbook, or a collection. What it lacks is a standard arrangement for how a scenario is put together, so that is left to each author, and there is nothing shared to lint or validate against. The current lack of a unifying standard has resulted in a learning cliff and conflicting examples. The point of a standard is a path an ordinary role author can follow without building that machinery. I want to propose one.

A scenario is three separable things:

  • Machinery: how a test host is created and destroyed, plus the boilerplate that wires the run together.
  • Inventory: which images and providers you test against.
  • Content: the converge that runs your code and the verify that checks it.

Only the content should differ between scenarios. Today Molecule makes you copy all three into every one. The standard I am proposing is that machinery and inventory live once, and a scenario holds only its content. Nothing here needs new tooling; every piece already exists, and we need to agree on how they are arranged.

Machinery: the platform collection owns create/destroy. A platform collection ships its provisioning once, as ansible-native playbooks callable by fully-qualified name:

containers.podman/
└── playbooks/
   ├── molecule_create.yml
   └── molecule_destroy.yml

Every scenario then looks the same, whether it tests a role or a collection:

# role:        <role>/molecule/default/
# collection:  <collection>/extensions/molecule/default/
# identical below the parent either way:
    ├── molecule.yml
    ├── create.yml     # import_playbook: containers.podman.molecule_create
    ├── destroy.yml    # import_playbook: containers.podman.molecule_destroy
    ├── converge.yml   # yours
    └── verify.yml     # yours

containers.podman owns the podman create/destroy, amazon.aws owns the EC2 one. You import it by name instead of copying it. (david_igou.molecule_provisioners already implements this pattern; it needs to be a standard, not one person’s collection.) The create.yml/destroy.yml files are one-line imports; they are separate files only because Molecule resolves these by filename, and lifting that so molecule.yml can name the playbook directly is a small enhancement worth making later.

Inventory: which images you test is data, not config. Today the image and provider set lives inside each scenario’s molecule.yml, or in shared config that authors inherit through non-standard tricks. The first means editing every scenario to change the set; the second is the kind of unofficial inheritance that has to be excluded from ansible-lint and schema validation, because there is no standard shape for it.

The set should instead be data the create playbook consumes. This falls out of the create/destroy contract below. Updates to what images to test can be centrally maintained in a single inventory. It is flat data you can open and read, not config merged from up the tree.

Content: converge and verify are all that differ. Once machinery and inventory are standardized, a scenario carries only the code that runs the thing under test and the assertions that check it.

What the standard removes.

  • One scenario format across roles, playbooks, and collections, all under the molecule directory. No separate tests/integration/targets/ tree to keep in sync.
  • The image inventory lives in one data file, instead of being repeated in every molecule.yml or inherited through non-standard config.
  • Shared files can be linted and schema-validated, because they have a standard shape instead of being ad hoc.
  • The silent zero-host success (a run that matches no hosts still reports pass) is closed by the create contract.
  • There is a blessed pattern to point people at, instead of the deprecated driver:/platforms: examples that are currently the most discoverable.

What I am asking this group to ratify. Four things:

  1. The location and names of a collection’s provisioning playbooks and the inventory: playbooks/molecule_<stage>.yml.
  2. One scenario format shared by a role, a playbook, and a collection.
  3. The convention that the image and provider inventory is data the create playbook consumes, not config repeated per scenario.
  4. A small create/destroy input/output contract: instances come from inventory, destroy loops the same group, and create fails loudly instead of reporting success over zero hosts. That contract is what makes provisioners interchangeable across platforms and closes the silent-failure gap. I have worked it out and can bring it as a starting point.

This is a light lift for collections. A collection that manages a thing generally already starts and stops it to test itself, so shipping molecule_create/molecule_destroy is mostly naming work, given clear standards and copy-ready examples. It stays optional: a collection that ships nothing loses nothing. Users / teams are also empowered to create and share their own “providers” using their existing collection distribution methods.

What deserves its own thread. A real standard still has to settle: whether create/destroy ships as a playbook or a role and how it surfaces in Galaxy alongside modules and roles; how inputs are expressed and validated (argument specs, defaults); and who owns the contract and how it is versioned without breaking consumers. I will open that thread and bring the fuller contract if there is appetite here.

3

… plugins and modules.

That was proposed by me (among other things) in 2023 https://github.com/ansible/molecule/issues/3919
This was before the molecule “redesign” and issue is closed now. So I suppose this is rejected. But I strongly believe that having explicit playbooks that have to be copied from one molecule scenario to another is a bad thing (and molecule still forces one to do that)

See my proposal in the link above - molecule playbooks to provision/delete/etc should be in separate collection, not just separate files.

Why it should be a convention? There are plenty of scenarios where this kind of complexity is not required. But idea is good. What is stopping you from removing platform section from molecule.yml today - nothing. Just make necessary adjustments to your create / destroy / etc playbooks so they get inventory from any place you like.

I completely agree that there is a lot of complexity around how to use molecule for running tests and this complexity has been created in part by the fact that proper examples and documentation are missing. But that would be very hard to resolve this situation by adding more complexity and creating any standard is a complex endeavour.

1 Like
4

@kks Great feedback and points.

I just want to address your final point to clarify my position. If the specifics of my proposal are adding more complexity than that is not intentional and something that I would want to iron out / eliminate.

My goal isn’t really to add on, but rather standardize around a “happy path” that will meet the needs of most automation devs (YAML). Except for the molecule enhancement to skip the one task import_playbook, everything else works “as-is” today. Its just leaning into a particular standard that users can reference and defend their decisions with. In fact the majority of the specifics are exactly what we have for examples in different places but never assembled into a single comprehensive package that automation devs can easily find. Worse, some of the docs conflict because they are addressing the other Ansible Dev (Python) group (for example using /tests/integration/targets).

Your suggestion from 2023 of just having stand alone collections for the molecule playbooks/roles for importing may be the better answer. I checked last night and only 20 out of 100 collections bundled with ‘ansible’ have any roles included. Making the create/destroy content may not be all that difficult, but it is still extra work the maintainers of those existing collections may not want.

5

FWIW I use symlinks for this.

6

I use ansible collections for this. Playbook can be part of a collection. Molecule relies on playbooks. So I have separate molecule.* collections with just molecule playbooks, that are invoked from create.yml, destroy.yml, converge.yml, etc. This allows for version management and portability.

1 Like
7

Sorry if my contribution to this thread has been off-topic, I’ve just reread the thread title and I guess it doesn’t have relevance for roles, I haven’t started using collections yet.

8

Thanks @Jeff_Pullen, @kks and @chris for your replies in this thread. I want to provide my thoughts on the areas raised above.

Scope: A dedicated post to split the role only discussion

This post was to focus specifically on collection testing. Role only testing is a critical part of the Ansible content ecosystem, but it is a different problem with a different persona, YAML first rather than Python first. Mixing the two into one discussion makes the discussions in this post about collections more confusing.

I’d like to propose that we move the role testing replies above into a separate post, dedicated to that topic. This will help ensure collection testing decisions do not taint role only concerns and vice versa.

Molecule: Playbook copying between scenarios

On the point about copying playbooks across scenarios, you don’t need too. A common playbook can live once and be imported into each scenario with the ansible.builtin.import_playbook module. Please see the examples below:

The same goes for molecule’s shared_state: true. It’s not collection only, molecule/default at the root of a role repository works the same way as extensions/molecule/default inside a collection. extensions/molecule is a collection convention but not a hard requirement for this functionality. So the point in #37 that you end up in collection territory the moment you want supported testing is not strictly true IMO.

Lack of documentation is the causing factor for several issues

The more I look at these replies, the more I think a lot of this comes back to documentation, and the lack of it for role testing specifically. For example, the playbook copying problem above is likely a symptom of this documentation gap. The import_playbook pattern and shared_state in role mode are not documented anywhere (that I know of) currently, so when documented examples show a full copy of create/destroy per scenario, authors copy. When search returns the driver:/platforms: style, authors follow that.

So, I think a real action item here for us is to focus on documentation for Molecule and on docs.ansible.com too for role testing. I feel like this would go a long way and help a lot. What do you think?

1 Like
9

+1

Yes, splitting these two use-cases / personas would help a lot.

The documentation is also a big part of it. Even when it does exist, there isn’t a good name or distinction between developing on the Python side vs the YAML side. Developing Ansible means different things to different people. If you go searching the internet for answers you’re more likely to land on the Python side of the documentation, even though I would wager there are many many times more people looking for the YAML docs.

1 Like
10

I’ve moved the original post and related responses into this new post for continued discussion.

I think the three Molecule documentation areas we should focus on is:

The contents in Using reads like a guide to me, as it’s using ansible-creator - Maybe we could re-organize / consolidate these areas so it doesn’t feel so fragmented / scattered?

Furthermore, we could look to add a “Role Testing” section to the Using area too. :thinking: