Parsing and vaulting yaml the Ansible way with ruamel.yaml

I tend to favor vaulted scalar values in vars files over entire vaulted files, partly because I can grep for variable names successfully. (I grep a lot!) But creating and managing vaulted scalars is a bit clunky.

To make it less clunky, I wrote a simple filter in Python that can be invoked from within an editor. Select the lines of yaml containing your keys and values, pass that block of yaml through the filter’s stdin (most text editors have a way to do that), and it will use ruamel.yaml to parse the yaml, process it, and return the result through its stdout. It will vault for you any unvaulted scalars it finds, and unvault any named vaulted data it finds, all while preserving your indentation. You can give it the optional parameter of the vault identity you’d like it to use for vaulting; otherwise it will use whatever your ANSIBLE_VAULT_IDENTITY environment variable is set to. Underneath of course it’s calling ansible-vault to do the actual vault work.

“Sounds great! What’s the problem?” Yeah, well, it isn’t particularly kind to comments, for one thing. It would be really cool to make it “vault→unvault round-trip idempotent”, which it mostly is if comments aren’t involved.

Another thing, which will be obvious if you look at the code, particularly around lines 252 and following, is that I totally failed at making ruamel.yaml vault-aware. My Python skills are of the cargo-culture copy-n-paste variety. And my staring at Ansible’s own code and the ruamel.yaml docs has yielded no enlightenment. The working code as it is feels really hacky.

Even so, I think this is a very useful tool, it’s dead simple to use, and it basically works. So here’s my “ask”, which I realize is perhaps bigger than I could expect anyone to volunteer for, in easy to “don’t even think about it” order:

  • I’d like someone familiar with Python to give this a once-over (it’s right at 300 lines) and point out the most embarrassing non-Pythonic warts.
  • I’d like someone who understands ruamel.yaml well enough to make it parse and emit !vault values like it ought to.
  • If possible, I’d like to know how to use the components of the installed Ansible suite to do this as more of an extension of Ansible rather than the awkward imitation work-alike that it currently is. (It already requires having ansible-vault installed. Requiring ansible-lint or whatever else is fair.)

Finally, after any or all of this is done, I’d like it to still be as dead simple to use as it currently is.

2 Likes

Thank you for this excellent script. I forked and modified it significantly to allow for things like encrypting all values and preserving quotes and embedded control characters. I don’t know if it’s more “Pythony” as I’m no more a Python guy than you were two years ago. But you may find some of the changes interesting.

Your idea to add it as an ansible extension is interesting and I may look into it, later. For now, it’s doing what I need as a CLI tool.

Thanks, again, and please share any thoughts you may have.

UPDATE:

I was assuming Ansible reparsed the YAML input after decryption but found that’s not the case–as you had already learned, I’m sure. So, I modified the script to warn when there is a possibility a file encrypted with this filter may not be restored accurately when decrypted.

3 Likes

Hi guys, that looks good! Would you mind if I incorporate that into andebox?

Fine by me. @sthames42 ?

1 Like

I was considering rolling back the gist to the previous revision which includes the handling of non-string values as well as quoted strings. I was then considering creating a from_av filter that could be used to resolve this values when they are non-quoted strings. But, I haven’t decided if it’s worth it. Maybe you guys have an opinion. If you clone the gist, you can see the previous revision. You can’t in gist.github.

But, yes, of course. Do whatever you want with it.
Thanks for asking.

1 Like

Thanks! I am keeping this in the backburner for now, prioritizing the openwrt stuff these days.

I have just created new feature: vault/unvault specific values within YAML files · Issue #166 · russoz-ansible/andebox · GitHub for it.

1 Like