List options in modules: "add" or "set"? Let's talk about terminology

Project Discussions Collection Development
,
1

Many modules manage something that is a collection of things: group members, tags, DNS records, repositories, SSH keys, interfaces. The usual interface is a type: list option, and sooner or later every author of such a module has to answer the same question:

With state: present and things: [a, b, c], should the target end up with exactly [a, b, c], or should a, b, and c simply be added to whatever is already there?

There is no single right answer: both behaviors are legitimate and both are needed. What is missing, as far as I can tell, is a shared vocabulary for them. Each module answers the question in its own way, with its own option name, its own polarity, and its own default. Users have to read the docs of every module to find out which of the possible behaviors they get (and sometimes the docs do not say).

This post is not a proposal for a “right way” to handle lists. It is an invitation to compare notes and, if there is interest, to agree on names for the different strategies that already exist, so that modules choosing the same strategy can describe it with the same words.

More than one question

“Add or set” is the most visible part of the problem, but there are a few related questions hiding behind it:

  1. Items not listed: are they kept, or removed?
  2. Option omitted vs. empty list: does leaving things out mean “do not touch”, and does things: [] mean “remove everything”? Or are they the same?
  3. Duplicates and order: is the target a set, an ordered sequence, or a sequence that allows repetitions? Is [a, b] the same as [b, a]? What happens with [a, a]?
  4. Scope: when “removing what is not listed”, removing it from where? Everything on the target, or only within some subset (an access level, a group of interfaces)?
  5. Item identity: when the list contains dictionaries (an interface with its settings, a member with its access level), is an existing item updated, replaced wholesale, or left alone?

What community.general does today

A survey of community.general modules found at least these strategies. Module names are examples, not an exhaustive list.

Strategy How it is expressed Examples (default)
Always exact, no switch Not expressed: passing [] clears the list Most ipa_* modules (ipa_role, ipa_sudorule, ipa_hbacrule, …), consul_token, consul_role, influxdb_user
Boolean, true = add append, array_add, namespace_append ipa_group, ipa_hostgroup (false, so exact by default), osx_defaults, pmem
Boolean, true = exact exclusive ini_file (true), ipa_dnsrecord (true)
solo cloudflare_dns, dnsimple, netcup_dns (false)
purge gitlab_label, gitlab_milestone, gitlab_*_variable, rhsm_repository (false)
purge_tags ali_instance (false)
remove_others apt_repo (false)
remove_extra_pvs lvg (true)
Choice option add / replace-all git_config (replace-all)
merge / replace manageiq_group (replace)
replace / add osx_defaults for dictionaries (replace)
Exact, limited to a scope purge_users: [<access levels>] gitlab_project_members, gitlab_group_members (no purge when omitted)
A value of state state: present / absent / exact ldap_attrs (present)
Add/remove only state: present / absent applied to the listed items, no exact mode manageiq_tags, manageiq_policies, package managers
Not documented Plain type: list without saying what happens to existing items Several modules that handle tags, labels, profiles, DNS server lists

A few things stand out:

  • The same concept has many names. “Make the target match the list exactly” is spelled exclusive, solo, purge, purge_tags, remove_others, remove_extra_pvs, replace, replace-all, exact, and append: false.
  • The same name means different things. purge is “remove unlisted labels” in gitlab_label, but “delete untracked files” in hg. replace is “replace all values” in git_config, but “recreate the vault” in ipa_vault.
  • Polarity is not consistent. For append and array_add, true means “add”. For exclusive, solo, and purge, true means “exact”. ipa_group with append: false and ini_file with exclusive: true do the same thing.
  • Defaults are not consistent. Some modules default to the exact behavior (lvg, ini_file, ipa_dnsrecord, ipa_group), others to the additive one (apt_repo, rhsm_repository, cloudflare_dns).
  • Duplicates and order are rarely documented. Some modules compare as sets, some deduplicate the input, some deliberately allow repeated values (git_config with add_mode: add, ini_file with exclusive: false).
  • Loops do not combine well with the exact strategies. rhsm_repository and ansible.posix.authorized_key both warn that their exact mode is applied per loop iteration.

Elsewhere in the ecosystem

Other projects have faced the same question, with different answers:

  • ansible.builtin.user has append (default false) for groups. The documentation of the development branch states that from ansible-core 2.21, append is required whenever groups is specified. In other words, the user must always state which behavior they want.
  • ansible.posix.authorized_key has exclusive (default false).
  • amazon.aws uses purge_<thing> booleans across the collection, for example purge_tags (default true). The docs make the “omitted vs. empty” distinction explicit: if tags is not set, tags are not modified, even with purge_tags: true.
  • Network resource modules (for example cisco.ios.ios_vlans) express the strategy through state, with a vocabulary shared across many collections: merged (add or update the listed items), replaced (replace the configuration of the listed items, leave the others alone), overridden (the listed items become the entire configuration), and deleted. This vocabulary also separates two things that are often mixed up: replacing an item versus replacing the whole collection.
  • community.docker.docker_container has comparisons, which can be set per option to strict, ignore, or allow_more_present. This is about comparing rather than changing, but it names the same three ideas: exact, not managed, and additive.

A different shape: each item carries its own state

When the list contains dictionaries, a module could take yet another route. Consider a hypothetical module that manages network interfaces:

- some_interface_module:
    state: present          # default for items that do not set their own
    interfaces:
      - name: link1         # created or updated
        listen_port: 51821
      - name: old_link
        state: absent       # removed
  • Each item has its own state, and the module-wide state is the default for the items that do not set one.
  • Interfaces that are not listed are never touched. Removing one means listing it with state: absent.
  • A single task can add some items and remove others.
  • A separate option, say managed_interfaces, could declare the full set of interfaces the module considers its own, independently of which ones are written in this run. That would make the exact strategy safer: unlisted items would only be removed within that declared scope, similar to what purge_users does in the GitLab member modules.

Lists nested inside each item (say, addresses or dns) would still need their own answer to all the questions above, and it is not necessarily the same answer as for the outer list.

Concepts that seem to need names

Without picking any of them, these are the distinct behaviors that show up above, with the words already in use for each:

Behavior Words already in use
Listed items are ensured, other items are kept append, add, merge, merged, allow_more_present, exclusive: false, solo: false, purge: false
The target ends up with exactly the listed items exact, exclusive, solo, purge, purge_*, remove_others, remove_extra_*, replace, replace-all, overridden, strict, append: false
Listed items are removed, other items are kept state: absent, deleted
Exact, but only within a declared subset purge_users: [<levels>]
Each listed item declares its own desired state no common name (the hypothetical <list>[].state above)
An existing item is replaced as a whole, not merged replaced (network resource modules)
The list is not managed at all option omitted, ignore

Separately from those, it might help to have standard phrasing for how values are compared: as a set, as an ordered sequence, or as a sequence that allows repetitions.

Dictionaries have the same problem

This post focuses on lists, but type: dict options raise very similar questions: are keys that are not specified kept or removed, does an omitted option differ from {}, and are nested values merged or replaced? The same variety shows up there too, for example merge_profile in community.general.lxd_profile, dict_mode in community.general.osx_defaults, purge_tags for the tags dictionaries in amazon.aws, and allow_more_present for dictionaries in community.docker. Dictionaries also add a question of their own: how deep a merge goes. Whatever terminology comes out of this discussion would probably need to work for dictionaries too, but it seems easier to start with lists.

Questions for discussion

  • Is a shared terminology for these strategies useful, or is the variety a feature, given how different the managed resources are?
  • If it is useful, should it cover only names (so that, say, “exact” means the same thing everywhere it appears), or also how the choice is exposed (boolean vs. choice option vs. state value)?
  • Is the network resource vocabulary (merged, replaced, overridden, deleted) a good base to build on outside network modules, or is it too tied to state and to device configuration?
  • Should “option omitted means not managed” and “empty list means clear it” be the expected default meanings, as amazon.aws documents for tags?
  • Should the expected way to handle duplicates and ordering be stated in the docs every time?
  • Where should such a convention live: in the Ansible developer guide, or in each collection’s contributor docs?
  • Are there other strategies or names in your collections that are not covered here?

Any experience from maintaining modules like these, including what users found confusing, would be very welcome.

AI was used to survey the modules and to draft this post.

2

Isn’t this basically “add or set”?

There’s also another twist: omitted can also mean “ignore for idempotency, and use default (usually empty list) when re-creating the object”. This is for example something the community.docker.docker_container module does. (But doing it the other way would also be possible. It again depends on what the user wants which behavior is preferred.)

I guess this also depends on whether objects can actually be updated, or only re-created. In the latter case, there’s also the question on what should happen with attributes the module is unaware of. Should these not be set on the newly created object, or should the module try to copy these (which can cause a lot of trouble)?

Generally no matter which choice you make, there will be users filing bug reports about the behavior. Making the behavior configurable allows users to get whatever they want, but also makes the module’s UX more complex, especially if you have a configuration per option (expect if you hide it away behind one big option like community.docker.docker_container’s comparison option).

In many cases, this depends (and should depend) on what the option is about. There are few cases (I’m aware of) where making this configurable could make sense.

I think it would be good to have a shared terminology. (I’m aware that it will take years to reach it for a large subset, and will never happen for everything, but I still think it’s a good thing.)

I think both.

This is one of the aspects of resource modules I really like (there are others I don’t). I’m not sure whether having this as state is the best idea, and it can be very useful to have it more fine-grained - especially in cases where objects need to be completely re-created and can’t be updated, like Docker containers.

It would be good to have something that works for all kind of modules, and not only for modules that can update state without having to re-create the underlying objects.

As I mentioned above, there’s a third option (which is mostly relevant for modules that need to re-create the underlying objects and can’t edit them). Generally I expect non-specified options (for object state) to be ignored during idempotency checks. Whether an explicit empty list means nothing or “replace” should depend on the “add vs. set” configuration.

That would be best, I think.

I’d prefer the Ansible developer guide if it’s a community-wide convention. Collections can have more specific guidelines, but the general thing should be in the Ansible docs.

3
  1. Should “option omitted means not managed” and “empty list means clear it” be the expected default meanings, as amazon.aws documents for tags?

As one of the people involved in the AWS collection discussions, this is my preferred behaviour. To me “omitted” means “you’re not responsible for this.”

Generic reasons

  • This is consistent with a number of core modules like ansible.builtin.file, ansible.builtin.user, ansible.builtin.cron.
  • When you have more complex resources, it’s easier to tune just the piece you want rather than the needing to describe the whole thing (And the alternative to this is an explosion in modules)
  • When you add new features to the module, you don’t risk suddenly nuking configuration that someone had to manually set, or set through a command because it wasn’t supported.

More “AWS specific” reasons:

  • Amazon add features pretty regularly, to avoid forcing people to aggressively upgrade the SDK we need to have the distinction between “I’m trying to change something and my SDK doesn’t support it” and “The module was updated to support a feature I’m not using, I don’t care if you can’t read/write it”.
  • The AWS permissions model is flexible. It’s not uncommon for someone to be managing a resource and only being permitted to touch specific parameters, where you need to make a similar distinction.
  1. Should the expected way to handle duplicates and ordering be stated in the docs every time?

In my opinion, until there’s a well adopted convention it needs to be documented in the module. We shouldn’t expect users to look in lots of different place to figure out

  • Does this module have a convention? (not a given)
  • Does the collection have a convention? (not a given, we took the deliberate step in the AWS collections to try and clean this up) This gets more complex if you’re using modules across multiple collections.
  • Does the community as a whole have a convention? (Not today)
  1. Where should such a convention live: in the Ansible developer guide, or in each collection’s contributor docs?

My personal preference would be the developer guide, but collections should be free to duplicate the documentation, especially during transition phases.

  1. Are there other strategies or names in your collections that are not covered here?

I personally like the “purge_THING” standard, where this gets messy is with ordered lists rather than sets/dicts. The examples I can think of where this comes into play in AWS, we either say “This is an ordered list, items not in the list will be removed”, or AWS has already provided some form of “priority” on the sub-objects and we just expose that.

1 Like
4

A common convention I’ve used in some of the Windows collections when dealing with a list of values is:

option_to_edit:
  add: ~
  set: ~
  remove: ~

The values themselves would either be a string identifier or a dictionary itself if more complex but essentially entries in add are added if not already present, remove removes if present, and set will replace the entire value with what was specified. It might reorder depending on the resource as well.

This allows people to add, remove, or idempotently set the value based on their requirements. If they wanted to clear anything then set: [] can be used. It also has the flexibility to add further options outside of add/set/remove if needed.

An example of this is microsoft.ad.object attributes. The microsoft.ad.user delegates option shows how you can expand the behaviour of such an option through lookup_failure_action same with microsoft.ad.user groups. Finally microsoft.ad.user spn is just a simple list of strings instead of a list of complex values for more simpler options.

This certainly isn’t perfect but it seemed quite flexible over things I’ve used in the past like dependencies and dependency_action for ansible.windows.win_service. The downside of this approach is I can’t combine add/remove operations together but the upside is a flatter option declaration.

1 Like