This is another discussion about the new Secret masking feature of ansible-core 2.22. I started working on some PRs in community.general to make lookup plugins that retrieve secrets automatically register sensitive values they look up as secrets (PR 1, PR 2, PR 3, PR 4, PR 5, PR 6). While working on them, I started having doubts on whether this is actually a good idea (for some plugins in community.sops, I changed the default to “do not register anything as secrets, unless the user explicitly asks for it”). This is more critical now that ansible-core 2.22 will likely also register and mask secrets that are less than 4 letters long, and mask 4 and 5 letter long secrets also inside words and not just at word boundaries: accidentally registering non-secret values can cause big pain, especially for short and generic texts (like 0, 5, test, user, localhost, …). Think of a random playbook output where all these texts (even the mentioned individual digits!) are replaced by $REDACTED$.
Right now, I only merged code that automatically registers secrets (I explicitly exclude values to options that are marked with secret: true here to restrict scope of the discussion) in case the values are obviously secrets, such as GitHub tokens or Docker Swarm tokens. Everything that retrieves values created by humans, such as passwords stored in password/secret/vault managers (see most of the PRs linked above) IMO have a certain chance of containing non-sensitive, trivial strings (see examples above) that are only stored in such places for convenience reasons, and would be dangerous to register.
I think it would be better (instead of registering such values) to make it easier for users to register output of plugins as secrets. Right now, there’s the ansible.builtin.register_secret filter which registers individual strings. If you have a data structure where some parts are secrets, it’s pretty painful to register these secrets, especially if you want to do it very close to the lookup to prevent secrets to show up in logs (think of verbose output which prints the results of a task).
IMO it would be best if you can directly register parts of the output of a lookup / filter plugin or a module as secrets.
Think of a lookup / filter which returns a data structure:
- foo: this is a secret # this is sensitive
bar: hello
- foo: another secret # this is sensitive
bar: world
- foo: true # if you feed this to ansible.builtin.register_secret, it will barf
members:
- user: foo
password: bar # this is sensitive
- user: baz
token: asdlkjfa98321329 # this is sensitive
Try to come up with some Jinja2 that takes this result - as a first step, assume it has already been stored in a variable - and registers all the secrets in it with ansible.builtin.register_secret, without accidentally passing true to it. That’s definitely not fun. (I’d likely use community.general.json_query as a building block.)
And then, as the next level, try to do this as a list of filters directly after lookup(...) / query(...) / | filter so that you actually get the unmodified values, while at the same time registering the secrets. (I don’t think this is possible without creating a new filter plugin.) Otherwise, if you do:
- name: Retrieve data
ansible.builtin.set_fact:
my_value: '{{ lookup("foo.bar.lookup", whatever) }}'-
- name: Register secrets:
ansible.builtin.set_fact:
_secrets: >-
{{ my_value | community.general.json_query(...) | filter(...) | map("ansible.builtin.register_secret") }}
and run the playbook with -v, the secret values are printed in clear during the first task. So you really want to write:
- name: Retrieve data and register secrets
ansible.builtin.set_fact:
my_value: '{{ lookup("foo.bar.lookup", whatever) | ...magic happens here... }}'-
so that only the secrets are registered, but you end up with the same (complex) data structure as my_value.
I think we need some more flexible filter plugins that allow to register specific parts of their input as secrets, while ignoring things that shouldn’t be registered as secrets (like booleans, and maybe also too short strings or specific strings), while returning the original output.
For module results, this is likely more complex. What you could do is use register projections together with such filters to also register the secrets before the task is actually done; I haven’t used register projections yet (since I had to support older ansible-core versions as well) so I’m not sure how exactly they interact with callbacks etc., and whether this can actually be used to properly remove secret values returned by modules (that don’t themselves register these values as secrets)…
I’m starting this thread so we can collect ideas of how to do this - what kind of filters / other kind of plugins would be needed/userful, how to write tasks/… where secrets can appear in that should be hidden, etc.
There’s always a chance we will revisit this decision and at least put the floor at 2 or 3 but only time will tell if these short secrets become an actual problem. Ultimately caution won out over what we deemed to be practical.