`!vault` encrypted strings no longer decrypting through `to_yaml` after 2.19 upgrade

Get Help
1

Hello! I’ve been making extensive use of vault encrypted strings like this

test: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          32313565333766366161623238333866356334363961326362336537666336643564383363316664
          6365663430383564623464386338623536363465393962320a386130306665653734316432316265
          62393235353362393334383862393538646134393735393434613862323139333732353333366136
          3563613331393237310a626362373031646265386461616537386232336234363361363361333533
          363

I provide the password file via the ANSIBLE_VAULT_PASSWORD_FILE environment variable.
On 2.18.8 this works all fine, however on 2.19 the strings are now placed as they are into the templates and are no longer being decrypted.
I could find nothing in the changelog that would indicate that this feature was changed.

For completeness I use this script to encrypt

#! /bin/bash

# Switch to script dir
cd "$(dirname "${BASH_SOURCE[0]}")" || exit 1

if [[ ! ( -n "$ANSIBLE_VAULT_PASSWORD_FILE" && -f "$ANSIBLE_VAULT_PASSWORD_FILE" ) ]]; then
	echo "Vault Password File not found."
	echo "Make sure the environment variable ANSIBLE_VAULT_PASSWORD_FILE is set to the password file"

	exit 1
fi

# Ansible is fussy about that
chmod 600 "$ANSIBLE_VAULT_PASSWORD_FILE"

variable="${1:-variable}"

if [ $# -ge 2 ]; then
	value="$2"
else
	read -r -p "Enter value for the variable \"$variable\": " value
fi

ansible-vault \
	encrypt_string \
	"--vault-password-file=$ANSIBLE_VAULT_PASSWORD_FILE" \
	--encrypt-vault-id default \
	--name "$variable" \
	"$value"

and no flags related to the vault on the actual ansible CLI.

As an important note, I do need this to continue to work on both versions unfortunately.

2

You have not provided enough information to reproduce this issue. How are you using this variable?

$ cat group_vars/all.yml
test: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          39396431376337393065666330653666646134396166663637326464633637366236396661663964
          6131386237643066633763656265643961386431653533300a303032356432643337623462633033
          64356437623761376463316161616461356430326263303032646233646363313739616334356232
          6262633637376539380a326364363437636432666136316539613964326432323161666564663130
          3137

$ cat test.yml
- hosts: localhost
  gather_facts: false
  tasks:
    - template:
        src: test.j2
        dest: /tmp/atest

$ cat test.j2
The value of test is {{ test }}

$ ansible-playbook test.yml
PLAY [localhost] ***************************************************************

TASK [template] ****************************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

$ cat /tmp/atest
The value of test is foo
3

Ah interesting, it appears that the filter to_yaml is causing the issues here:

user@host $ cat test.j2 
The value of test is {{ test | string | to_yaml }}
The value of test is {{ test | string }}
The value of test is {{ test }}
user@host $ cat atest 
The value of test is !vault |
  $ANSIBLE_VAULT;1.1;AES256
  34323132663762633735366366623966626462663238633934323735303332323836316263373662
  3437323133343466343262643233333030613238396530650a306261633464393033663831306464
  38643465346130636533376230613362626433326331626533333736376331303936396264333862
  3738633631336430610a653432383261663864303736343136346265353536633261353532333137
  6536

The value of test is test
The value of test is test

On Ansible 2.18.8 this works fine

4

This might be an intended change. Making it harder to accidentally lose metadata like “this value is from Vault” was one of the goals of the data tagging project, and to_yaml already behaved like this for values it knew were vaulted.

5

I’m explicitly casting it to a string though. How else are you supposed to yaml escape a vault value if even explicitly casting it to a string does nothing?

6

This might be an intended change.

Yep- especially < 2.19, accidental loss of wrappers/tags on strings usually == CVE-worthy RCE or disclosure, so we usually try pretty hard to preserve them. In this case, it was basically an accident that it worked prior to 2.19, because the underlying type used for vaulted values was a non-string-derived sequence that the string filter converted to a plain string, where now it’s stored as an actual string subclass with a tag containing the source ciphertext (which the serializer always uses if it encounters it) - the string filter sees that it’s already a string, so just passes it through.

There were originally some serializer knobs exposed to the filters to control this behavior in 2.19, but they caused more problems than they solved and would’ve complicated/blocked some future changes we want to make, so they were mostly removed by the final release.

Even though the behavior you’re relying on < 2.19 was an unintentional side-effect of the old implementation, I can see why you want it, and we’ve got a couple of ideas to mess around with that could get us there- more to come…

7

There is another way to do this that should work the same way on all versions: force it to make a new string by concatenating an empty string to it:

{{ test ~ '' | to_yaml }}
8

Another idea would be converting it to JSON and back. That should hopefully get rid of the vault tag, and also works for more complex data structures than strings, as long as they aren’t too complex for JSON :slight_smile:

9

Am I taking crazy pills or isn’t this exactly what | string is supposed to do?

10

It converts something to a string. But a vaulted string already is a string (just with some extra tag that it’s vaulted), so in this case its output is its input.

The concatenation, or mapping to JSON and then back are both ways that remove this ‘tag’. As long as the tag is present, to_yaml will print the encrypted string instead of the decrypted string. Once the tag is gone, you get what you want.

1 Like
11

isn’t this exactly what | string is supposed to do?

|string stringifies non-strings… Purely from Ansible’s data type perspective, vaulted values have always been (and continue to be) strings, but from a Python (implementation detail) data type perspective, prior to 2.19, they weren’t. That discrepancy has been the cause of countless bugs and whac-a-mole over the 10 years or so that it’s been true- anything in ansible-core or user plugin code that might encounter a vaulted value in a context where it should act like a string had to be guarded with something like str(value) if isinstance(value, AnsibleVaultEncryptedUnicode) else value to ensure it behaved like a string. Changing them to be proper strings in 2.19 eliminates all that mess.

We could just force the string filter to strip vault tags in the name of absolute backward compatibility, but most of the people I’ve informally polled on that particular question agree that the previous behavior is a case of “unintentional side effect/implementation detail” rather than “feature/behavior that should be preserved” - looking at it purely from an Ansible data type perspective, it’s already a string, so having the string filter make a copy and stripping some/all of its metadata is counterintuitive.

12

But they’re clearly not proper strings if they carry metadata? And if they display different behavior based on where they are fed into.

Because if they were just strings, then why does feeding a string into to_yaml not give me a representation of said string and instead give a representation of the encrypted string.

It may be just me, but a string to me is a string of characters. Nothing more, nothing less. And apparently the !vault syntax doesn’t create a string. Fair. After all it’s not a plain string. But what absolutely makes no sense is that a filter that is supposed to turn anything that’s not a string into a string. And when I pipe a non-string into it, I get a non-string.

So either the data model is broken or the filter. But you’re not telling me that both of the things you mentioned are reasonable assumptions together.

Simply put, if a vault value is a string, then to_yaml is broken because it doesn’t output a representation of the string. Or it is an object with attached metadata and string is broken because it doesn’t erase said metadata to produce a plain string.
I’d argue that string is the broken function here, as I believe that the result of adding an empty string to an object should return the same result as piping said object into string.

And simply speaking, the Python internal representation is irrelevant here. It’s an implementation detail, I as an user of the software, shouldn’t have to care about.
to_json apparently doesn’t care about that metadata, so why does to_yaml care about it? Printing the string in a template also doesn’t care about that metadata.

Essentially we’re having Schrödinger’s encrypted string here that behaves differently depending on how you look at it, and I find that at best inconsistent and unintuitive.

This change to me sounds like a code purist changing things to make the code “clean” but forgot that the code doesn’t matter to end users.
And frankly I find it shocking that this change is getting defended here like it’s the most natural thing in the world.

You see this string? It’s not actually a string, but also is actually a string. And under these sets of filters and uses it behaves like a string, but under these other sets of filters and uses it doesn’t behave like one. Perfectly normal and expected behavior. What are you upset about? Just use these silly workarounds. Like this one where we serialize and deserialize it to loose the metadata, which any proper serialization should keep, but this one doesn’t for reasons.

Simply put: Pick one. Either it’s a proper string or it isn’t. Don’t pretend it’s both and neither.

Edit: Also I checked the change logs and couldn’t find any mention of it there. For a breaking change!

13

I can’t disagree with any of your points. I would have expected it to show the decrypted value pretty much everywhere except in the context of a callback, and then I wouldn’t expect to get the encrypted value; rather I would have thought you’d get either a string of 8 asterisks or the contents of ansible_vaulted_value_alternative or something like that.

I wouldn’t expect {{ secret ~ '' }} to result in removal of the vault tag. I would expect any new string would inherit the vault tag if any of the strings it was created from had it. Then it would be up to some context to determine when vaulted strings get exposed.

Way back, when I thought (wrongly) that I understood how Ansible worked, I looked at hacking in an alternative to no_log: true. The idea was that when any vaulted value was read in, it would be added to an initially empty list of “Strings We Must Never Log”. Then it would be up to callbacks to do the Right Thing. That naive approach requires some exceptions for things like “True” and “Changed”, but it still accidentally exposes info if, for example, you try to log “The Quick Brown Fox” and you get “The ******** Brown Fox”. Then you know one of the vaulted values is “Quick”.

I’m not sure which is more problematic: apparently arbitrary rules for when you get vaulted values vs. accidental indirect exposure potential of not-very-good secrets. Here’s hoping the “couple of ideas to mess around with” that @nitzmahone alluded to in comment #6 above pans out.

1 Like