`!vault` encrypted strings no longer decrypting through `to_yaml` after 2.19 upgrade

Get Help
1

Hello! I’ve been making extensive use of vault encrypted strings like this

test: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          32313565333766366161623238333866356334363961326362336537666336643564383363316664
          6365663430383564623464386338623536363465393962320a386130306665653734316432316265
          62393235353362393334383862393538646134393735393434613862323139333732353333366136
          3563613331393237310a626362373031646265386461616537386232336234363361363361333533
          363

I provide the password file via the ANSIBLE_VAULT_PASSWORD_FILE environment variable.
On 2.18.8 this works all fine, however on 2.19 the strings are now placed as they are into the templates and are no longer being decrypted.
I could find nothing in the changelog that would indicate that this feature was changed.

For completeness I use this script to encrypt

#! /bin/bash

# Switch to script dir
cd "$(dirname "${BASH_SOURCE[0]}")" || exit 1

if [[ ! ( -n "$ANSIBLE_VAULT_PASSWORD_FILE" && -f "$ANSIBLE_VAULT_PASSWORD_FILE" ) ]]; then
	echo "Vault Password File not found."
	echo "Make sure the environment variable ANSIBLE_VAULT_PASSWORD_FILE is set to the password file"

	exit 1
fi

# Ansible is fussy about that
chmod 600 "$ANSIBLE_VAULT_PASSWORD_FILE"

variable="${1:-variable}"

if [ $# -ge 2 ]; then
	value="$2"
else
	read -r -p "Enter value for the variable \"$variable\": " value
fi

ansible-vault \
	encrypt_string \
	"--vault-password-file=$ANSIBLE_VAULT_PASSWORD_FILE" \
	--encrypt-vault-id default \
	--name "$variable" \
	"$value"

and no flags related to the vault on the actual ansible CLI.

As an important note, I do need this to continue to work on both versions unfortunately.

2

You have not provided enough information to reproduce this issue. How are you using this variable?

$ cat group_vars/all.yml
test: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          39396431376337393065666330653666646134396166663637326464633637366236396661663964
          6131386237643066633763656265643961386431653533300a303032356432643337623462633033
          64356437623761376463316161616461356430326263303032646233646363313739616334356232
          6262633637376539380a326364363437636432666136316539613964326432323161666564663130
          3137

$ cat test.yml
- hosts: localhost
  gather_facts: false
  tasks:
    - template:
        src: test.j2
        dest: /tmp/atest

$ cat test.j2
The value of test is {{ test }}

$ ansible-playbook test.yml
PLAY [localhost] ***************************************************************

TASK [template] ****************************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

$ cat /tmp/atest
The value of test is foo
3

Ah interesting, it appears that the filter to_yaml is causing the issues here:

user@host $ cat test.j2 
The value of test is {{ test | string | to_yaml }}
The value of test is {{ test | string }}
The value of test is {{ test }}
user@host $ cat atest 
The value of test is !vault |
  $ANSIBLE_VAULT;1.1;AES256
  34323132663762633735366366623966626462663238633934323735303332323836316263373662
  3437323133343466343262643233333030613238396530650a306261633464393033663831306464
  38643465346130636533376230613362626433326331626533333736376331303936396264333862
  3738633631336430610a653432383261663864303736343136346265353536633261353532333137
  6536

The value of test is test
The value of test is test

On Ansible 2.18.8 this works fine

4

This might be an intended change. Making it harder to accidentally lose metadata like “this value is from Vault” was one of the goals of the data tagging project, and to_yaml already behaved like this for values it knew were vaulted.

5

I’m explicitly casting it to a string though. How else are you supposed to yaml escape a vault value if even explicitly casting it to a string does nothing?

6

This might be an intended change.

Yep- especially < 2.19, accidental loss of wrappers/tags on strings usually == CVE-worthy RCE or disclosure, so we usually try pretty hard to preserve them. In this case, it was basically an accident that it worked prior to 2.19, because the underlying type used for vaulted values was a non-string-derived sequence that the string filter converted to a plain string, where now it’s stored as an actual string subclass with a tag containing the source ciphertext (which the serializer always uses if it encounters it) - the string filter sees that it’s already a string, so just passes it through.

There were originally some serializer knobs exposed to the filters to control this behavior in 2.19, but they caused more problems than they solved and would’ve complicated/blocked some future changes we want to make, so they were mostly removed by the final release.

Even though the behavior you’re relying on < 2.19 was an unintentional side-effect of the old implementation, I can see why you want it, and we’ve got a couple of ideas to mess around with that could get us there- more to come…

7

There is another way to do this that should work the same way on all versions: force it to make a new string by concatenating an empty string to it:

{{ test ~ '' | to_yaml }}
8

Another idea would be converting it to JSON and back. That should hopefully get rid of the vault tag, and also works for more complex data structures than strings, as long as they aren’t too complex for JSON :slight_smile:

9

Am I taking crazy pills or isn’t this exactly what | string is supposed to do?

10

It converts something to a string. But a vaulted string already is a string (just with some extra tag that it’s vaulted), so in this case its output is its input.

The concatenation, or mapping to JSON and then back are both ways that remove this ‘tag’. As long as the tag is present, to_yaml will print the encrypted string instead of the decrypted string. Once the tag is gone, you get what you want.

1 Like
11

isn’t this exactly what | string is supposed to do?

|string stringifies non-strings… Purely from Ansible’s data type perspective, vaulted values have always been (and continue to be) strings, but from a Python (implementation detail) data type perspective, prior to 2.19, they weren’t. That discrepancy has been the cause of countless bugs and whac-a-mole over the 10 years or so that it’s been true- anything in ansible-core or user plugin code that might encounter a vaulted value in a context where it should act like a string had to be guarded with something like str(value) if isinstance(value, AnsibleVaultEncryptedUnicode) else value to ensure it behaved like a string. Changing them to be proper strings in 2.19 eliminates all that mess.

We could just force the string filter to strip vault tags in the name of absolute backward compatibility, but most of the people I’ve informally polled on that particular question agree that the previous behavior is a case of “unintentional side effect/implementation detail” rather than “feature/behavior that should be preserved” - looking at it purely from an Ansible data type perspective, it’s already a string, so having the string filter make a copy and stripping some/all of its metadata is counterintuitive.

12

But they’re clearly not proper strings if they carry metadata? And if they display different behavior based on where they are fed into.

Because if they were just strings, then why does feeding a string into to_yaml not give me a representation of said string and instead give a representation of the encrypted string.

It may be just me, but a string to me is a string of characters. Nothing more, nothing less. And apparently the !vault syntax doesn’t create a string. Fair. After all it’s not a plain string. But what absolutely makes no sense is that a filter that is supposed to turn anything that’s not a string into a string. And when I pipe a non-string into it, I get a non-string.

So either the data model is broken or the filter. But you’re not telling me that both of the things you mentioned are reasonable assumptions together.

Simply put, if a vault value is a string, then to_yaml is broken because it doesn’t output a representation of the string. Or it is an object with attached metadata and string is broken because it doesn’t erase said metadata to produce a plain string.
I’d argue that string is the broken function here, as I believe that the result of adding an empty string to an object should return the same result as piping said object into string.

And simply speaking, the Python internal representation is irrelevant here. It’s an implementation detail, I as an user of the software, shouldn’t have to care about.
to_json apparently doesn’t care about that metadata, so why does to_yaml care about it? Printing the string in a template also doesn’t care about that metadata.

Essentially we’re having Schrödinger’s encrypted string here that behaves differently depending on how you look at it, and I find that at best inconsistent and unintuitive.

This change to me sounds like a code purist changing things to make the code “clean” but forgot that the code doesn’t matter to end users.
And frankly I find it shocking that this change is getting defended here like it’s the most natural thing in the world.

You see this string? It’s not actually a string, but also is actually a string. And under these sets of filters and uses it behaves like a string, but under these other sets of filters and uses it doesn’t behave like one. Perfectly normal and expected behavior. What are you upset about? Just use these silly workarounds. Like this one where we serialize and deserialize it to loose the metadata, which any proper serialization should keep, but this one doesn’t for reasons.

Simply put: Pick one. Either it’s a proper string or it isn’t. Don’t pretend it’s both and neither.

Edit: Also I checked the change logs and couldn’t find any mention of it there. For a breaking change!

13

I can’t disagree with any of your points. I would have expected it to show the decrypted value pretty much everywhere except in the context of a callback, and then I wouldn’t expect to get the encrypted value; rather I would have thought you’d get either a string of 8 asterisks or the contents of ansible_vaulted_value_alternative or something like that.

I wouldn’t expect {{ secret ~ '' }} to result in removal of the vault tag. I would expect any new string would inherit the vault tag if any of the strings it was created from had it. Then it would be up to some context to determine when vaulted strings get exposed.

Way back, when I thought (wrongly) that I understood how Ansible worked, I looked at hacking in an alternative to no_log: true. The idea was that when any vaulted value was read in, it would be added to an initially empty list of “Strings We Must Never Log”. Then it would be up to callbacks to do the Right Thing. That naive approach requires some exceptions for things like “True” and “Changed”, but it still accidentally exposes info if, for example, you try to log “The Quick Brown Fox” and you get “The ******** Brown Fox”. Then you know one of the vaulted values is “Quick”.

I’m not sure which is more problematic: apparently arbitrary rules for when you get vaulted values vs. accidental indirect exposure potential of not-very-good secrets. Here’s hoping the “couple of ideas to mess around with” that @nitzmahone alluded to in comment #6 above pans out.

1 Like
14

This make variables substitution difficult to predict.

Here is an example from a real world situation.

gitlab_ldap_password: !vault |
  $ANSIBLE_VAULT;1.1;AES256
  ********
  ********
  ********
gitlab_ldap:
  main:
    label: LDAP
    host: ldap
    port: 636
    encryption: simple_tls
    verify_certificates: true
    bind_dn: cn=chiche,ou=donne,ou=nous,o=tout
    password: "{{ gitlab_ldap_password }}"

The rendering is different for the same variable when it’s addressed directly or by the filter to_nice_yaml.

# In the following line, gitlab_ldap_password is rendered unciphered.
gitlab_rails['ldap_password'] = "{{ gitlab_ldap_password }}"
# => gitlab_rails['ldap_password'] = "123456"


# In the following line, gitlab_ldap_password is rendered ciphered.
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
{{ gitlab_ldap|to_nice_yaml }}
EOS
# => gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
#    main:
#        label: LDAP
#        host: ldap
#        port: 636
#        encryption: simple_tls
#        verify_certificates: true
#        bind_dn: cn=chiche,ou=donne,ou=nous,o=tout
#        password: !vault |
#              $ANSIBLE_VAULT;1.1;AES256
#               ********
#               ********
#               ********

The not explicit ~ "" trick restore the expected behavior. Difficult to predict when a user will use a vaulted variable.

gitlab_ldap:
  main:
    password: "{{ gitlab_ldap_password ~ '' }}"
2 Likes
15

I think one big problem of this discussion is that it’s about two very different aspects of ansible-core 2.19, and in many posts it’s not clear to me at all for/against which of these two aspects the post arguments.

So we have:

  1. The data tagging feature, which adds to every value handled in Ansible some tags that carry information. These are things like where the value comes from (file, line, column), whether it is deprecated (reason, removal version/date, …), whether it is a template or not (for strings only), and whether it comes from a vault (this information was there already before data tagging). (There might be more tags, I’m not that familiar with the implementation, and more tags will come in later versions, especially the “sensitive data” tag.)

  2. The behavior of the to_yaml and to_nice_yaml filters. These now preserve the vault tag, and did not do that before.

And then maybe a third point, related to 1):

  1. Whether operations on tagged data should keep some of the tags. More specific to the above posts: If you concat two strings, one of them is vaulted, what should the result be? Should it depend on whether the other string is empty or not, or should it always be the same result?

I really think 1) is the best thing that happened in Ansible land for a very long time. This extra information is extremely useful. Ever looked at the error messages ansible-core 2.19 produces, how they tell you where something comes from? Or being told that a module’s return value you’re using at some point is deprecated and will be eventually removed? That’s impossible without data tagging.

Then let me get to 3) first. This is debatable, but if you like things being predictable, do you really want a ~ b to be a vaulted string depending whether one of a and b were vaulted and the other one is an empty string, but the result not being a vaulted string if both a and b are not empty strings? I think that makes results a lot harder to predict. (Also please note that vaulted != sensitive. For the sensitive tag this argument is different. Vaulted means that the string appeared in an Ansible vault as-is, which makes it also sensitive, but not vice-versa.) So "{{ gitlab_ldap_password ~ '' }}" not being a vaulted string makes very much sense in my opinion (if gitlab_ldap_password is a vaulted string), since you’ve taken a vaulted string and modified it (even if it is a modification that doesn’t change its value as a string).

And then there’s 2). I guess the main question here is what folks use to_yaml and to_nice_yaml for. If they use it to generate YAML files for consumption by Ansible itself, then keeping vaulted strings vaulted is the right behavior. If they use it to generate YAML files for consumption by other programs, then that behavior is absolutely useless since other programs do not know Ansible vaulted strings. I think that most people use Ansible to manage other programs than Ansible itself, so preserving vaulted strings is something they have absolutely no use for. Unfortunately, the filter as it is now is only useful when creating YAML files for Ansible itself, but not for any other program, since it perserves vaulted strings, and does not allow to configure that behavior. (If the behavior would be configurable, one could now debate what the default should be.)

16

This is the first time I’ve been made aware of the existence of a “sensitive” tag. Your explanation of its relation to the vaulted tag at least explains the (in my opinion ill-considered) change in default behavior of the to_yaml filter.

Let me drop my current thoughts here:

  • Changing the default behavior of transformative text filters is at best questionable. Rude even. Certainly annoying. Although I could imagine a scenario where that might be called for, this isn’t it.
  • Changing the default behavior of a subset of text filters smacks of inconsistencies that will haunt the Ansible ecosystem forever.
  • to_yaml and to_json make sense to users. Breaking one of them in this way will not gain any friends.

I’ve never considered the problem of producing partially vaulted yaml from within Ansible. So naturally I consider that to be a tiny corner case, one that could easily be solved with a combination of tagged data and some sort of “preserve vaulted” parameter to whatever text filters it might be useful in. But breaking every extant use of to_yaml for the sake of a minuscule set of cases – in fact currently non-existant – would be unfortunate and not well received among users.

Will there be a comprehensive set of filters and tests which allow users to deal with setting and querying various tags? For example, if I use some magic code to generate a password, I’d want to tag that as sensitive. Or one might want to invoke different behaviors depending on whether some data contained sensitive or vaulted values. Would such plugins also handle user-defined tags?

17

That assumes many things which are not givens, like the future invocation of Ansible will have access to the same vault key.

If the intent is to avoid incurring CVEs, then make to_yaml fail if you throw vaulted values into it without specifying whether to preserve or decrypt vaulted data. But don’t silently break current uses. Either preserve current behavior, or break current behavior fail accompanied with appropriate messaging.

18

This is not a change in behaviour of the to_yaml filter. It has always behaved this way.

19

Oh! I’m sorry, I completely missed that. I thought this was something new with 2.19. Too bad, too, because if it were it’s not to late to fix it. [backing away slowly, shaking head]

20

It doesn’t exist yet. It is planned, but there’s no ETA yet (AFAIK) since it’s another very large change (similar to data tagging itself).

If you’re talking about the string filter: AFAIK it was not intended that it removes the vault tag. That it did this in previous ansible-core versions is likely an unintended behavior. In many cases this behavior is clearly a bug: if you use | string to ensure that you’re converting non-strings (like ints, floats, booleans, lists, dicts) to strings, you usually do not want vaulted strings to loose their vaulted tag.

Using | string | to_yaml as a “work-around” to | to_yaml’s behavior was one of the few uses where this behavior was useful - as opposed to many where it isn’t useful, and in some cases actually a CVE-worthy bug. (And bug-fixes tend to always break some folks’ use-cases, just saying space-bar heater as an extreme exaggeration of this.)

We definitely need a way to dump YAML that does not preserve vaulted strings, and that does not involve hacks / exploits bugs.

Right now there are some filters and tests in ansible._protomatter for this:

ansible-doc --list --type filter ansible._protomatter
ansible-doc --list --type test ansible._protomatter

These are currently hightly experimental and can change at any point of time. I guess the idea is to have some more stable forms of these eventually, but right now there aren’t any yet. (The core team wants to avoid adding something now that’s too hasty and will require breaking changes to fix it later on, which again would annoy users…)