New Secret Masking Feature - 2.22

Project Discussions
, ,
1

New Ansible 2.22 Feature - Secret Masking

A new feature has just been merged in devel for ansible-core, a secret registration and masking system. The initial implementation is in ansible/ansible#87457 and the documentation is in review at ansible/ansible-documentation#3934. We would love feedback from anyone who can try it on devel before release, especially collection maintainers with custom modules or callback plugins.

Big shout out to @pkingstonxyz and @mkrizek for bringing this feature over the time.

What it is

Ansible keeps a registry of secret values for the life of the process. Any text that leaves Ansible through display output (screen and log_path), callback plugin results, or module logging on the managed node has registered secrets replaced with $REDACTED$.

Unlike the old VALUE_SPECIFIED_IN_NO_LOG_PARAMETER on module values with no_log=True, masking is now non-destructive. Variables, module arguments, and task results keep the real value which can be used in registered variables. Only the rendered output through callbacks of display calls will redact those value.

Values are registered automatically from various sources; vault content and passwords, no_log module options, vars_prompt and --ask-* prompts, connection and become passwords, and the password/unvault lookups and vault/unvault filters. Secrets can also be manually registered with the new register_secret filter plugin.

What it looks like

Two new filters, register_secret and mask_secrets, cover anything Ansible does not register for you:

- name: Generate a database password
  ansible.builtin.command: openssl rand -base64 32
  register:
    db_password: _task.result.stdout | register_secret
    password_result: _task.result
  no_log: true

- name: Usable, but masked in output
  ansible.builtin.debug:
    msg: "Setting database password to {{ db_password }}"

The last task prints Setting database password to $REDACTED$, and db_password still holds the real value for later tasks. Use mask_secrets when writing output somewhere Ansible does not control, such as a file on a target.

What it means for end users

  • no_log module options no longer return VALUE_SPECIFIED_IN_NO_LOG_PARAMETER in results. The real value is kept and masked only in output. Update any playbooks or tests that check for that placeholder.
  • The no_log task keyword is unchanged and still censors the whole task result. It does not register anything, so pair it with register_secret if a later task uses the value.
  • Secrets shorter than 4 characters are never masked to avoid collisions in normal out, and 4 to 6 character secrets are only masked as whole words. Only exact matches are masked, so hashed or base64 copies are not.
  • This is best effort and a safety net, not a replacement for Vault and no_log if you encounter one of the known limitations

What it means for module and plugin developers

  • A new public API in ansible.module_utils.secrets works in modules and all controller plugins:
from ansible.module_utils.secrets import register_secret, mask_secrets

token = register_secret(session['token'])  # returned unchanged, masked in output

mask_secrets(f"API Token Result {token}")  # API Token Result $REDACTED$

PowerShell modules get the same via the Ansible.Secrets C# util with [Ansible.Secrets.SecretMasker]::RegisterSecret() and ::MaskString().

  • Plugin options can be marked secret: true so the value is registered regardless of which source set it. Use this for any password or token option.
  • Callback plugins should set ANSIBLE_SUPPORTS_MASKING = True on the class and mask the results themselves. Callbacks that set it receive unmasked results and must mask anything written outside Display with mask_secrets(). Callbacks that do not set it keep receiving pre-masked results through a compatibility shim that will be deprecated and removed in a future release.
  • heuristic_log_sanitize(), remove_values(), and sanitize_keys() are deprecated for removal in 2.25. AnsibleModule.log() and run_command() no longer apply the old password heuristics, so register any secret passed on a command line that is not a no_log option.

Full details, including the length rules and limitations, are in the documentation PR Add documentation for secret masking by jborean93 · Pull Request #3934 · ansible/ansible-documentation · GitHub that is not yet live.

Known Problems

This is a list of known problems in the current implementation that we are aiming to either fix or explicitly document as a known limitation:

  • Module journalctl/syslog of invocation args do not redact module options with a value less than 4 characters in suboptions Edit: Merged into devel
  • A failure in register_secrets filter that contains the literal value leaks in the Ansible error origin statement - {{ 12345678 | register_secret }} (fails due to being non-str)
    • Still trying to find the best way forward for this
  • register_secrets filter only fails for invalid types, shorter secrets are silently ignore: Edit: Merged into devel

I’ll be editing this section so it should stay up to date.

9 Likes
2

Nice work on this, and the non-destructive part especially.

A question about the length floor. A value under four characters is discarded at registration with no warning and no error, and register_secret returns it unchanged either way, so nothing tells the author it did not take effect. The value then prints in full.

The floor assumes the threat is guessing the value, where three characters is no keyspace. Plenty of short values are not credentials at all: an environment code, a customer or org short name, a subdomain. Nobody needs to guess those. What matters is that the string shows up in a log that goes into a support ticket or a public CI job, and no longer value is available, because it is the author’s org name.

So the behavior is fail open, silently, for everyone. Over-redacting costs a log that is hard to read. Under-redacting publishes something explicitly marked as not for publication.

Is fail open the right thing to hard code here, or should it be a policy with a fail safe default? A warning would be the minimum. _log_invocation already takes the other path for no_log params, blanking them rather than trusting the masker, which seems like the right instinct.

3

The main problem with redacting short strings is that they tend to easily show up randomly in other strings. If you end up seeing $REDACTED$ in some Base64 encoded output because that 2- or 3-letter “secret” sequence randomly shows up in data, this is at least annoying and sometimes dangerous (since you can’t use the Base64 output anymore since you don’t know its actual value).

(This is also a reason why I would have preferred to use tagging over value redaction, since then you can also tag a one-letter string as secret without it resulting in that letter being redacted everywhere, but the problem with this approach is that it misses many places where this secret value shows up in, like composed URLs which contain username + password.)

Other services that redact strings from output, like GitLab CI redacting secrets from CI log, also only redact strings that have a certain length (and format, I think it also requires you to not use spaces IIRC).

(I’m waiting for the first person to use four spaces as a sensitive value somewhere, and then wondering why they end up having $REDACTED$ all over the place in output where something is indented by at least four spaces… I saw some issues where people complained that parts of URLs/data/… got censored in module output when they used apparently trivial passwords for no_log inputs, that happened to be similar to other strings that showed up. Like password test and URL test.example.com, where “obviously” the second test is totally unrelated to the first test and why on earth got the second one redacted?!)

4

I’m waiting for the first person to use four spaces as a sensitive value somewhere, and then wondering why they end up having $REDACTED$ all over the place in output where something is indented by at least four spaces…

After reading the proposed docs again (Add documentation for secret masking by jborean93 · Pull Request #3934 · ansible/ansible-documentation · GitHub), I noticed it explicitly says “Leading and trailing whitespace is not part of a secret.”, so this won’t be a problem :slight_smile:

There’s another thing I’ve been wondering about. Assume that there’s a module which returns a secret (for example, some token). Until now, a user might have chosen to run the module, register its result, and show the result using the ansible.builtin.debug action (and then they copy’n’paste it to some secret store), instead of using a module like ansible.builtin.copy/ansible.builtin.template or others to write it to a file (unencrypted or encrypted) or directly send it to a secrets store (OpenBao, Bitwarden, HC Vault, …).

Until now, this worked fine. But if now the module is changed to register that secret as a secret, suddenly the playbook will no longer show that secret to stdout, but instead show $REDACTED$. And there doesn’t seem to be a way to get Ansible to still print this secret unredacted.

Maybe it would be a good idea to explicitly mention this in the porting guide, in case someone’s workflow relies on this.

5

The floor here was a difficult choice to come to but it was decided to avoid false positive redactions on shorter strings that may not be a secret/sensitive value. IT is full of acronyms so 3 characters really does increase the chance for this happening. 4 was mostly chosen because pin codes usually are around 4 characters and the word boundary check we added was found to be a good middle ground but we can certainly understand the pros and cons behind lowering the floor.

The primary goal was more for redacting credentials and other private/secret values. I can see the desire for trying to redact personal and identifying information but I think this feature isn’t designed for this purpose.

@felixfontein has certainly hit the nail on the head though. We’ve found that shorter secrets were commonly conflicting with plaintext values which makes it trivial for someone to deduce what is a secret from what they expect to be in that plaintext.

We debated this but as the warning cannot contain the value itself the warning would essentially be generic. Generic warnings are de-duplicated in Ansible so subsequent triggers would be ignored breaking the purpose of the warning in the first place. There’s also a lot of secret registrations that are done implicitly that the user cannot control so a warning without that secret value in the text would be close to useless for tracking down and just extra noise for end users.

Turning this into an error is also not viable due to that implicit registration. Playbooks that work today would fail with such errors in the case where a secret that is too short is attempted to be registered. Even worse is that there would be no way to selectively turn it off for individual modules of registration points.

I do think we could at least error in the register_secret filter which is a new plugin so has no backwards compatibility concerns and is always a user defined registration rather than an implicit one done by the plugin author. I’m still unsure whether the default behaviour would be to error (would be my pick right now), warn, or ignore but having a kwarg to control that is probably the best choice.

This was chosen to ensure the masking did not regress on the logging side which had access to the no_log information and was scoped in a way that we could still redact the information.

It’s certainly not ideal but this was ultimately the reasoning behind why we don’t fail or warn.

Thank you for bringing this up, we should definitely point this out in the porting guide and I’ll update the docs PR sometime shortly with this point. It might be nice for an extra option to the debug actions to tell it not to go through the redaction for such scenarios but from a technical perspective the masking happens beyond the debug action so would require a lot more thought on how to actually work. We’ve also hit the feature freeze for 2.22 so would be hard to sell, especially if the solution isn’t a simple one.

6

maybe new debug_secret (or print_secret) action not just and option to make it explicit?

7

Maybe it is worth mentioning what ansible api (what functions) garanties what secrets will be redacted.

I understand that all secret registration is happening at runtime. But sometimes I know in advance that certain information is a secret (env vars). Is there planned way to register secrets or secret env vars before runtime (e.g. in ansible.cfg)

8

The issue is the redaction happens on the main Ansible process and there is no mechanism to call a method on display to bypass redaction. It would require a new method or mechanism to tell the callback not to redact the secrets in the result and even then there’s no way selectively disable redaction, it would have to be on the whole result possibly exposing other secret values.

The documentation PR Add documentation for secret masking by jborean93 · Pull Request #3934 · ansible/ansible-documentation · GitHub should hopefully cover what you are looking for here. TLDR anything on display will go through the redaction call. Anything outside of builtin can also call this API but we’ve marked display and the module logging as the two egress points in Ansible itself.

There is technically an undocumented API we’ve added that may be used by ansible-runner. You can see it in the PR itself if you are interested but it is undocumented and we reserve the right to remove or change it at any time.

Nothing is stopping you from creating a custom callback or other plugin that runs at startup to register whatever secrets in any way you want.

9

That is a nice idea I did not think of. Thanks for pointing that out

I did not get this one. Are you saying that

can change any moment? Or what do you mean by non public API?

10

I’m referring to _SECRETS_INPUT_FILES which is internal and could be subject to change. The module_utils is a public API and won’t change without the usual deprecation process.

11

If I pass a secret marked with the ansible.builtin.register_secret filter (or any other mean) to a “generic” module argument like description (which isn’t marked as no_log=True), does the module also censor the secret value in its log output? And if it does: does ansible-core makes sure to only provide the module with secrets that are actually passed to the module, and not information on other secrets that are not passed?

12

Yes it will.

Yes that’s correct, when we build the module arguments we scan the module options that will be sent to the module for any registered secrets. Only secrets present in there will be provided to the module.

For example this playbook will result in the following in the syslog/journal

- hosts: localhost
  gather_facts: false
  tasks:
  - ping:
      data: '{{ "testme" | register_secret }}'

Sep 01 00:00:00 jborean-dev ansible-ping[24798]: Invoked with data=$REDACTED$

What is not covered if when a secret is registered on the controller but not provided as any input to the module. The module won’t know that it is a secret and thus won’t be redacted in the logging side. We’ve added this already to the known limitations ansible-documentation/docs/docsite/rst/playbook_guide/playbooks_secret_masking.rst at 394c69fc765114e063418fbbcca975199b67af31 · jborean93/ansible-documentation · GitHub. This only applies to the module logging though, the results when it comes back to Ansible will still be redacted because Ansible knows about that secret.

13

Great, thanks a lot for confirming that! I was expecting this answer, but having it confirmed is a lot better than hoping for it (and takes less time than figuring this out myself, which I will eventually do when I have enough time, but not now :wink: ).