Very nice. Thanks!
Are (or can they be) jinja filters also available inside playbooks?
Are (or can they be) jinja filters also available inside playbooks?
To a certain extent, I believe. For example, this works:
- action: shell echo '{{ mypasswd | vault }}' > /tmp/00
Regards,
-JP
“encrypted values are contained in vars files and are
decrypted on use.”
Where do they get the inputs to decrypt them from?
First off, thanks for the initiative.
This is too early for a core pull request IMHO as there’s still a pretty good discussion on the list on how this might work.
I’m also finding the interface somewhat confusing because it’s still using lookup plugins (which I asked that it not), and it requires using the keyczar CLI.
If we can dumb things down, I’d like to have this as a core feature, but right now, there’s a lot of reading/understanding required for it’s use, and it seems to require a bit too much thinking.
Not trying to be negative here, I just think it needs to be really smooth to fit in with the rest of Ansible.
–Michael
I'm also finding the interface somewhat confusing because it's still using
lookup plugins (which I asked that it not),
I don't understand. It's a Jinja2 filter. Where's the lookup plugin?
(Maybe I'm confusing the terminology.)
and it requires using the
keyczar CLI.
For creating a key (or more than one key) initially, yes. I see
absolutely no point in re-inventing the wheel, particularly as crypto
stuff is concerned. People much more clever than I have done that.
After a key is initially created, that's it; no more keyczar CLI. That's
the reason I submitted the ansible-vault utility. (Of course, I could
add key creation to that, which would avoid using the keyczar CLI
entirely.)
If we can dumb things down, I'd like to have this as a core feature, but
right now, there's a lot of reading/understanding required for it's use,
and it seems to require a bit too much thinking.
Pardon me, but I disagree.
1. Create a key.
2. Use it.
I've been verbose in my doc (on the PR page) so as to be as clear as
possible.
-JP
Great.
Now what is needed is a way to make sure that decrypted values are not
being leaked in ansible logs, though I am not sure what is the correct
place to do the checks.
Maybe the easiest thing to do would be to add a global "log" attribute
that could be also overridden by tasks. Then it could be set to "log:
no" so that no logs are produced (quite intrusive), set to "log: debug"
to log everything or set to other levels such as "critical", "warning",
"notice", etc with each module having to decide what to log based on the
active log level.
I guess I’m saying “too many steps” here.
I just want it to be as smooth as Possible.
I think it could be done so you didn’t even need the Jinja2 filter, you could just use all the vars files unencrypted during Ansible playbook runtime, but they would be locked when you try to edit them, and could have a full vars_files or two full of content that you could edit plaintext when needed.
Sort of like how ssh-agent doesn’t require re-entering your SSH key password, right?
The Jinja2 filter is nice. It allows to store mixed encrypted and
unencrypted values and it can be used both in playbooks and templates.
You don't have to encrypt/lock the whole variable file just because you
have some password vars inside it (the usual case). In addition, it
allows to have different encryption keys for different vars without
having to create separate vault files for each key.
keyczar is a good choice too. It has a nice abstraction over handling
key stores and keys, it is tested, and keyczart's code could be
incorporated in the ansible-vault utility in order to have one utility
for all vault tasks, which could be also taught to edit encrypted values
directly inside variable files (e.g. "ansible-vault --key ckey --set
mysecretvar='plaintext' varsfile").
I just want it to be as smooth as Possible.
It is. 
[Admittedly, with the exception that ansible-vault utility
needs more love.]
I think it could be done so you didn't even need the Jinja2 filter, you
could just use all the vars files unencrypted during Ansible playbook
runtime, but they would be locked when you try to edit them, and could have
a full vars_files or two full of content that you could edit plaintext when
needed.
I'm (very slowly) starting to understand what you want: everything is
encrypted and gets magically decrypted on Ansible run. This is way
overkill, and poses a number of problems respectively has a number of
inconveniences:
1. How to detect whether a particular vars file is encrypted (by file
name extension, by content)?
2. Completely disables things like printing/sending a vars file
3. Makes version control near to meaningless -- having all the
crypto-stuff in the files.
4. Requires some 'magic' (like the SSH agent you mention below) which is
able to feed Ansible with keys. Even if we pass a single key to
Ansible, the next point:
5. Makes using more than a key terribly difficult. We discussed using
some form of PKI in Antwerp, but more or less chucked that due to
'what do we encrypt to?' The current user? The current host? The
target host? How do teams share keys, etc. Most complicated, I think.
The original intention was to encrypt really secret stuff, like
passwords for MySQL (my.cnf) and things like that. Most variables do not
need to be encrypted, and I believe the current PR solves that rather
neatly: encrypt only what absolutely needs to be encrypted.
Sort of like how ssh-agent doesn't require re-entering your SSH key
password, right?
I'm hearing you, but I don't think this is the way to go.
-JP
In addition, it
allows to have different encryption keys for different vars without
having to create separate vault files for each key.
Bingo!
keyczar is a good choice too. It has a nice abstraction over handling
key stores and keys, it is tested, and keyczart's code could be
incorporated in the ansible-vault utility in order to have one utility
for all vault tasks, which could be also taught to edit encrypted values
directly inside variable files (e.g. "ansible-vault --key ckey --set
mysecretvar='plaintext' varsfile").
Indeed. I would like to augment ansible-vault like so:
ansible-vault -N -k mynewkey # create new key w/ all requirements
echo secret | ansible-vault -k mynewkey - # encrypt
Hide the keyczar CLI utility and making of all directories, etc. Add a
`-l' option to list keys, maybe `-r' to remove, and that's about it.
I like your idea of an --edit function. #feature
-JP
But it could probably be simply rewritten to be a lookup plugin, right?
To me the most obvious approach that doesn't add new concepts is to use it as a lookup plugin and storing passwords in individual files, exactly how the password lookup plugin does it. So you would put in your inventory things like:
mysql_password: "{{ lookup('decrypt', 'credentials/path/to') }}"
Somewhat OT, as I have not had a time to read the details on the vault stuff yet, but I would LOVE all lookup plugins to work as Jinja2 filters, because I dislike the whole {{ lookup(…) }} thing for being too dense.
Would love a better way.
Problem we discovered for the 'mandatory' filter is that filters only get the values passed, which basically means errors from filters are vague since they have no context (no filename, no line number, no variable name, only filter name and value).
This means that if you have a template/playbook with more than one filter, the error will not tell you which occurence of the filter failed.
I opened a feature request for jinja2, but that's only going to be useful in the distant future...
We've had quite a bit of positive feedback on the idea, as well as some
loud doubts, and I want to recap how I think this should be done.
Intention is to protect certain variables in vars files.
That sums it up pretty nicely.
+1000 !
VIncent
Most vars will not be encrypted. This is meant as a protection to vars
which contain sensitive data -- basically clear-text passwords that
nobody wants to have floating around in files that get added to
repositories, etc.I don't think encrypting vars files completely is useful: makes things
like `grep', and `diff' impossible to use.
+1
On the other hand, allowing more than a single key is useful in
multi-tenant situations, but more than a single key is not required.
Does this mean mulltiple keys which each can decrypt all passwords? So
different users get different keys, and a key can be revoked without
impacting other users?
If yes, +1
Thanks,
Serge
Does this mean mulltiple keys which each can decrypt all passwords? So
different users get different keys, and a key can be revoked without
impacting other users?
Not to start with, no; you have to choose a key to encrypt a particular
value with. If you revoke (i.e. delete) the key, the value can no longer
be decrypted.
The way I envision this is having a key for cust1's vars, and another
for cust2's vars, and I use vault on a per-customer basis.
Keep it simple.
-JP
This could be worked around like this: And then: 1) ansible-vault: Create a copy of to edit to a temporary file 2) ansible-vault: Spawn editor on temporary file 3) human: Edit as you wish. If you want a value to be encrypted, use some ‘template magic’ like this: “myval: {{ plaintext | vault_encrypt(‘keyname’) }}” 4) human: Save and exit from editor 5) ansible-vault: Parse temporary file line by line, process occurrences of “{{ plaintext | vault_encrypt(‘keyname’) }}” and replace them with the processed output. 6) ansible-vault: Move temporary file in place of the original file. What do you think?
3) human: Edit as you wish. If you want a value to be encrypted, use
some 'template magic' like this: "myval: {{ plaintext |
vault_encrypt('keyname') }}"
Basically yes, but:
For a new variable, human enters, say,
password: vault(verysecret)
that's easy to parse upon saving: grab what's inside vault(), encrypt,
and store as
password: "Axkkwkekke..="
And on the second round, i.e. when human edits that file a second time?
How do we find the encrypted values? 
We can't just go guessing that
anything that looks like base64 is a "vault" variable....
I'm thinking along the lines of having the encrypted values stored in
the vars file as
password: "vault:Axkkwkekke..="
in other words, prefixed with "vault:", which would a) give human who
looks at file an indication that it's not just base64-encoded but
belongs to vault and shouldn't be touched, and b) would allow for
parsing when editing, as you suggest. (Obviously the Jinja2 vault()
filter would strip that off before trying to decrypt it.)
Would that be viable?
-JP
I know this is a BIG ask, but for environments with multiple Admins an implementation of Shamir’s secret would rock.