# Secret variables / data: encryption

**URL:** <https://forum.ansible.com/t/secret-variables-data-encryption/13042>\
**Category:** Ansible Project\
**Created:** [June 29, 2013, 10:18pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042 "2013-06-29T22:18:30Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![Ernest0x](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@Ernest0x](https://forum.ansible.com/u/Ernest0x)\
**Post date:** [July 1, 2013, 1:01pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/21 "2013-07-01T13:01:04Z")

</div>

Very nice. Thanks!  
Are (or can they be) jinja filters also available inside playbooks?

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 1, 2013, 1:10pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/22 "2013-07-01T13:10:45Z")

</div>

> Are (or can they be) jinja filters also available inside playbooks?

To a certain extent, I believe. For example, this works:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- action: shell echo '{{ mypasswd | vault }}' \> /tmp/00

Regards,

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-JP

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\
**Post date:** [July 1, 2013, 1:57pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/23 "2013-07-01T13:57:48Z")

</div>

“encrypted values are contained in vars files and are  
decrypted on use.”

Where do they get the inputs to decrypt them from?

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\
**Post date:** [July 1, 2013, 2:01pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/24 "2013-07-01T14:01:00Z")

</div>

First off, thanks for the initiative.

This is too early for a core pull request IMHO as there’s still a pretty good discussion on the list on how this might work.

I’m also finding the interface somewhat confusing because it’s still using lookup plugins (which I asked that it not), and it requires using the keyczar CLI.

If we can dumb things down, I’d like to have this as a core feature, but right now, there’s a lot of reading/understanding required for it’s use, and it seems to require a bit too much thinking.

Not trying to be negative here, I just think it needs to be really smooth to fit in with the rest of Ansible.

–Michael

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 1, 2013, 2:19pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/25 "2013-07-01T14:19:45Z")

</div>

> I'm also finding the interface somewhat confusing because it's still using  
> lookup plugins (which I asked that it not),

I don't understand. It's a Jinja2 filter. Where's the lookup plugin?  
(Maybe I'm confusing the terminology.)

> and it requires using the  
> keyczar CLI.

For creating a key (or more than one key) initially, yes. I see  
absolutely no point in re-inventing the wheel, particularly as crypto  
stuff is concerned. People much more clever than I have done that.

After a key is initially created, that's it; no more keyczar CLI. That's  
the reason I submitted the ansible-vault utility. (Of course, I could  
add key creation to that, which would avoid using the keyczar CLI  
entirely.)

> If we can dumb things down, I'd like to have this as a core feature, but  
> right now, there's a lot of reading/understanding required for it's use,  
> and it seems to require a bit too much thinking.

Pardon me, but I disagree.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1. Create a key.  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2. Use it.

I've been verbose in my doc (on the PR page) so as to be as clear as  
possible.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-JP

---

<div class="post-metadata">

**Author:** ![Ernest0x](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@Ernest0x](https://forum.ansible.com/u/Ernest0x)\
**Post date:** [July 1, 2013, 2:24pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/26 "2013-07-01T14:24:10Z")

</div>

Great.  
Now what is needed is a way to make sure that decrypted values are not  
being leaked in ansible logs, though I am not sure what is the correct  
place to do the checks.  
Maybe the easiest thing to do would be to add a global "log" attribute  
that could be also overridden by tasks. Then it could be set to "log:  
no" so that no logs are produced (quite intrusive), set to "log: debug"  
to log everything or set to other levels such as "critical", "warning",  
"notice", etc with each module having to decide what to log based on the  
active log level.

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan](https://avatars.discourse-cdn.com/v4/letter/m/c4cdca/32.png) [@Michael\_DeHaan](https://forum.ansible.com/u/Michael_DeHaan)\
**Post date:** [July 1, 2013, 2:56pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/27 "2013-07-01T14:56:25Z")

</div>

I guess I’m saying “too many steps” here.

I just want it to be as smooth as Possible.

I think it could be done so you didn’t even need the Jinja2 filter, you could just use all the vars files unencrypted during Ansible playbook runtime, but they would be locked when you try to edit them, and could have a full vars\_files or two full of content that you could edit plaintext when needed.

Sort of like how ssh-agent doesn’t require re-entering your SSH key password, right?

---

<div class="post-metadata">

**Author:** ![Ernest0x](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@Ernest0x](https://forum.ansible.com/u/Ernest0x)\
**Post date:** [July 1, 2013, 5:36pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/28 "2013-07-01T17:36:02Z")

</div>

The Jinja2 filter is nice. It allows to store mixed encrypted and  
unencrypted values and it can be used both in playbooks and templates.  
You don't have to encrypt/lock the whole variable file just because you  
have some password vars inside it (the usual case). In addition, it  
allows to have different encryption keys for different vars without  
having to create separate vault files for each key.

keyczar is a good choice too. It has a nice abstraction over handling  
key stores and keys, it is tested, and keyczart's code could be  
incorporated in the ansible-vault utility in order to have one utility  
for all vault tasks, which could be also taught to edit encrypted values  
directly inside variable files (e.g. "ansible-vault --key ckey --set  
mysecretvar='plaintext' varsfile").

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 1, 2013, 5:46pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/29 "2013-07-01T17:46:35Z")

</div>

> I just want it to be as smooth as Possible.

It is. 🙂 [Admittedly, with the exception that ansible-vault utility  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;needs more love.]

> I think it could be done so you didn't even need the Jinja2 filter, you  
> could just use all the vars files unencrypted during Ansible playbook  
> runtime, but they would be locked when you try to edit them, and could have  
> a full vars\_files or two full of content that you could edit plaintext when  
> needed.

I'm (very slowly) starting to understand what you want: everything is  
encrypted and gets magically decrypted on Ansible run. This is way  
overkill, and poses a number of problems respectively has a number of  
inconveniences:

1. How to detect whether a particular vars file is encrypted (by file  
&nbsp;&nbsp;&nbsp;name extension, by content)?  
2. Completely disables things like printing/sending a vars file  
3. Makes version control near to meaningless -- having all the  
&nbsp;&nbsp;&nbsp;crypto-stuff in the files.  
4. Requires some 'magic' (like the SSH agent you mention below) which is  
&nbsp;&nbsp;&nbsp;able to feed Ansible with keys. Even if we pass a single key to  
&nbsp;&nbsp;&nbsp;Ansible, the next point:  
5. Makes using more than a key terribly difficult. We discussed using  
&nbsp;&nbsp;&nbsp;some form of PKI in Antwerp, but more or less chucked that due to  
&nbsp;&nbsp;&nbsp;'what do we encrypt to?' The current user? The current host? The  
&nbsp;&nbsp;&nbsp;target host? How do teams share keys, etc. Most complicated, I think.

The original intention was to encrypt really secret stuff, like  
passwords for MySQL (my.cnf) and things like that. Most variables do not  
need to be encrypted, and I believe the current PR solves that rather  
neatly: encrypt only what absolutely needs to be encrypted.

> Sort of like how ssh-agent doesn't require re-entering your SSH key  
> password, right?

I'm hearing you, but I don't think this is the way to go.

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-JP

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 1, 2013, 5:49pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/30 "2013-07-01T17:49:57Z")

</div>

> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;In addition, it  
> allows to have different encryption keys for different vars without  
> having to create separate vault files for each key.

Bingo!

> keyczar is a good choice too. It has a nice abstraction over handling  
> key stores and keys, it is tested, and keyczart's code could be  
> incorporated in the ansible-vault utility in order to have one utility  
> for all vault tasks, which could be also taught to edit encrypted values  
> directly inside variable files (e.g. "ansible-vault --key ckey --set  
> mysecretvar='plaintext' varsfile").

Indeed. I would like to augment ansible-vault like so:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ansible-vault -N -k mynewkey # create new key w/ all requirements  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;echo secret | ansible-vault -k mynewkey - # encrypt  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;  
Hide the keyczar CLI utility and making of all directories, etc. Add a  
`-l' option to list keys, maybe `-r' to remove, and that's about it.

I like your idea of an --edit function. #feature

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-JP

---

<div class="post-metadata">

**Author:** ![GW1](https://avatars.discourse-cdn.com/v4/letter/g/c89c15/32.png) [@GW1](https://forum.ansible.com/u/GW1)\
**Post date:** [July 1, 2013, 6:59pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/31 "2013-07-01T18:59:16Z")

</div>

```auto
But it could probably be simply rewritten to be a lookup plugin, right?

```

```auto

```

```auto
To me the most obvious approach that doesn't add new concepts is to use it as a lookup plugin and storing passwords in individual files, exactly how the password lookup plugin does it. So you would put in your inventory things like:

```

```auto
mysql_password: "{{ lookup('decrypt', 'credentials/path/to') }}"

```

---

<div class="post-metadata">

**Author:** ![Michael\_DeHaan2](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@Michael\_DeHaan2](https://forum.ansible.com/u/Michael_DeHaan2)\
**Post date:** [July 1, 2013, 7:04pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/32 "2013-07-01T19:04:14Z")

</div>

Somewhat OT, as I have not had a time to read the details on the vault stuff yet, but I would _LOVE_ all lookup plugins to work as Jinja2 filters, because I dislike the whole {{ lookup(…) }} thing for being too dense.

Would love a better way.

---

<div class="post-metadata">

**Author:** ![Dag\_Wieers](https://avatars.discourse-cdn.com/v4/letter/d/45deac/32.png) [@Dag\_Wieers](https://forum.ansible.com/u/Dag_Wieers)\
**Post date:** [July 1, 2013, 9:37pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/33 "2013-07-01T21:37:23Z")

</div>

Problem we discovered for the 'mandatory' filter is that filters only get the values passed, which basically means errors from filters are vague since they have no context (no filename, no line number, no variable name, only filter name and value).

This means that if you have a template/playbook with more than one filter, the error will not tell you which occurence of the filter failed.

I opened a feature request for jinja2, but that's only going to be useful in the distant future...

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[https://github.com/mitsuhiko/jinja2/issues/245](https://github.com/mitsuhiko/jinja2/issues/245)

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 2, 2013, 9:44am UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/34 "2013-07-02T09:44:19Z")

</div>

> [1] [https://github.com/ansible/ansible/pull/3393](https://github.com/ansible/ansible/pull/3393)

We've had quite a bit of positive feedback on the idea, as well as some  
loud doubts, and I want to recap how I think this should be done.

Intention is to protect certain variables in vars files.

---

<div class="post-metadata">

**Author:** ![Vincent\_Van\_der\_Kuss](https://avatars.discourse-cdn.com/v4/letter/v/7c8e57/32.png) [@Vincent\_Van\_der\_Kuss](https://forum.ansible.com/u/Vincent_Van_der_Kuss)\
**Post date:** [July 2, 2013, 10:20am UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/35 "2013-07-02T10:20:04Z")

</div>

That sums it up pretty nicely.

+1000 !

VIncent

---

<div class="post-metadata">

**Author:** ![Serge\_van\_Ginderacht](https://avatars.discourse-cdn.com/v4/letter/s/b782af/32.png) [@Serge\_van\_Ginderacht](https://forum.ansible.com/u/Serge_van_Ginderacht)\
**Post date:** [July 2, 2013, 10:26am UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/36 "2013-07-02T10:26:25Z")

</div>

> Most vars will not be encrypted. This is meant as a protection to vars  
> which contain sensitive data -- basically clear-text passwords that  
> nobody wants to have floating around in files that get added to  
> repositories, etc.
> 
> I don't think encrypting vars files completely is useful: makes things  
> like `grep', and `diff' impossible to use.

+1

> On the other hand, allowing more than a single key is useful in  
> multi-tenant situations, but more than a single key is not required.

Does this mean mulltiple keys which each can decrypt all passwords? So  
different users get different keys, and a key can be revoked without  
impacting other users?  
If yes, +1

Thanks,

Serge

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 2, 2013, 10:36am UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/37 "2013-07-02T10:36:11Z")

</div>

> Does this mean mulltiple keys which each can decrypt all passwords? So  
> different users get different keys, and a key can be revoked without  
> impacting other users?

Not to start with, no; you have to choose a key to encrypt a particular  
value with. If you revoke (i.e. delete) the key, the value can no longer  
be decrypted.

The way I envision this is having a key for cust1's vars, and another  
for cust2's vars, and I use vault on a per-customer basis.

Keep it simple. 🙂

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-JP

---

<div class="post-metadata">

**Author:** ![Ernest0x](https://avatars.discourse-cdn.com/v4/letter/e/ed655f/32.png) [@Ernest0x](https://forum.ansible.com/u/Ernest0x)\
**Post date:** [July 2, 2013, 11:36am UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/38 "2013-07-02T11:36:46Z")

</div>

This could be worked around like this: And then: 1) ansible-vault: Create a copy of to edit to a temporary file 2) ansible-vault: Spawn editor on temporary file 3) human: Edit as you wish. If you want a value to be encrypted, use some ‘template magic’ like this: “myval: {{ plaintext | vault\_encrypt(‘keyname’) }}” 4) human: Save and exit from editor 5) ansible-vault: Parse temporary file line by line, process occurrences of “{{ plaintext | vault\_encrypt(‘keyname’) }}” and replace them with the processed output. 6) ansible-vault: Move temporary file in place of the original file. What do you think?

---

<div class="post-metadata">

**Author:** ![jpmens1](https://avatars.discourse-cdn.com/v4/letter/j/e9c0ed/32.png) [@jpmens1](https://forum.ansible.com/u/jpmens1)\
**Post date:** [July 2, 2013, 12:39pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/39 "2013-07-02T12:39:38Z")

</div>

> 3) human: Edit as you wish. If you want a value to be encrypted, use  
> some 'template magic' like this: "myval: {{ plaintext |  
> vault\_encrypt('keyname') }}"

Basically yes, but:

For a new variable, human enters, say,

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;password: vault(verysecret)

that's easy to parse upon saving: grab what's inside vault(), encrypt,  
and store as

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;password: "Axkkwkekke..="

And on the second round, i.e. when human edits that file a second time?  
How do we find the encrypted values? 😉 We can't just go guessing that  
anything that looks like base64 is a "vault" variable....

I'm thinking along the lines of having the encrypted values stored in  
the vars file as

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;password: "vault:Axkkwkekke..="

in other words, prefixed with "vault:", which would a) give human who  
looks at file an indication that it's not just base64-encoded but  
belongs to vault and shouldn't be touched, and b) would allow for  
parsing when editing, as you suggest. (Obviously the Jinja2 vault()  
filter would strip that off before trying to decrypt it.)

Would that be viable?

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;-JP

---

<div class="post-metadata">

**Author:** ![Brian\_Coca1](https://avatars.discourse-cdn.com/v4/letter/b/278dde/32.png) [@Brian\_Coca1](https://forum.ansible.com/u/Brian_Coca1)\
**Post date:** [July 2, 2013, 1:44pm UTC](https://forum.ansible.com/t/secret-variables-data-encryption/13042/40 "2013-07-02T13:44:22Z")

</div>

I know this is a BIG ask, but for environments with multiple Admins an implementation of Shamir’s secret would rock.

[Previous page](https://forum.ansible.com/t/secret-variables-data-encryption/13042.md?page=1)

[Next page](https://forum.ansible.com/t/secret-variables-data-encryption/13042.md?page=3)
