How to use a secret in place of a Credential

I have a successful job template in a workflow, but it worked because it always needed the same Credential no matter the server it was updating. That is changing. Every server is going to have its own credential now, and that credential is going to be stored in a CyberArk Password Vault accessed via Central Credential Provider. I can no longer use an AWX Credential, but have to use CyberArk secrets. I see the documents on how to pull the secret from the CCP, but I don’t know how to use it after I’ve got it.

I’m not sure how to ask the question. My job template currently does an ansible.builtin.copy that connects to the target server using the Ansible Credential. It was easy. But now how do I supply the results of the CyberArk lookup to the same ansible.builtin.copy without a credential attached to the play?

Maybe pulling the credential magically sets the same variables as attaching a Credential object did? and I don’t have to do anything? Maybe I set an environmental variable with the CyberArk results? and then the ansible.builtin.copy connects using that variable? Maybe I attach a credential to the play, then use awx.awx.credential to change the password in that credential immediately prior to the ansible.builtin.copy? Has anyone done this?

Thanks,

Kevin

Okay. Still no answer, since I need input from the Password Vault team before I can take the next step, but I’m going to record this documentation link here and move on to another task. This gives some insight into how the plugin is built and seems to indicate the CyberArk plugin should magically populate the necessary fields after a successful lookup. This may be easy:
awx/docs/credentials/credential_plugins.md at devel · ansible/awx · GitHub

So, the main problem you’re facing is the fact that every server will require a unique password for the username account you’re currently accessing them with. What server OS, and what account name is it? Linux’s root user? Windows’ local administrator? These are good accounts to have the password rotated frequently, but they’re not usually a good idea to use directly for automation through things like Ansible.

If it is the root user, can you set up an ssh key to use instead of a password? Not only is it more secure than a password, but it would enable you to disable ssh password auth for root altogether (can still login from a tty or via su).

In either OS case, it’s generally common practice to have computers/servers joined to an identity management system, whether that’s Active Directory, RADIUS, IPA, or just plain old LDAP. If your servers are joined to such a management system, then you should be able to use your own user/admin account (or a service account) to access these servers and use become to elevate your privileges as needed without ever touching the local superuser accounts.


With all that said, I haven’t even touched on what to do with CyberArk and AWX. AWX has both a CyberArk Central Credential Provider Lookup and CyberArk Conjur Secrets Manager Lookup credential types. These are “lookup” credentials that enable you to configure other Credentials in AWX that hook into external vaults to fill in secrets data.

Here is an example of my own using TSS:

This isn’t magically looking up those secrets, I specifically told AWX what ID and Field fills out each credential property.

AWX also only allows a single Machine credential per job for authenticating to hosts with. So, even if you integrate CyberArk here, only one unique password will be available to use for all hosts in a given play. It won’t magically/dynamically lookup credentials per host.

It might be possible to use cyberark_credential in a custom role that dynamically looks up the password for your inventory hosts.

Something like:

# tasks file in fictional role: checkout_cyberark_password
- name: Retrieve CyberArk Credential
  cyberark_credential:
    api_base_url: "https://cyberark.company.com"
    app_id: "TestID"
    query: "Safe=test;UserName=admin;HostName={{ inventory_hostname }}"
    fail_request_on_password_change: true
  register: cyberark
  delegate_to: localhost

- name: Set password to retrieved credential
  set_fact:
    ansible_password: "{{ cyberark.result.content }}"
  no_log: true

# for convenience, add a skippable gather_facts step to the role
- name: Gather Facts
  gather_facts:
  when:
    - gather_facts is defined
    - gather_facts | bool

Then modify your playbooks to look somewhat like this:

- hosts: all
  gather_facts: false # can't authenticate until credentials retrieved
  roles:
    - role: checkout_cyberark_password # must be first role in list, and there must be no 'pre_tasks:' tasks that requires authenticating to hosts
      vars:
        gather_facts: true
  tasks:
    - debug:
        vars: ansible_facts

Take the above with a grain of salt. I haven’t used CyberArk myself, nor have I been forced to work with unique/randomized ansible_password’s in AWX. Using strictly Ansible cli, if you are using CyberArk’s conjur and Ansible’s lookup plugin for it, you could dynamically lookup the ansible_password as an inventory variable, or populate them ahead of time with a script that encrypts them with ansible-vault. However, AWX’s inventory sync doesn’t (yet) support vault encrypted variables (without embedding the vault password in the EE), and you will probably have a lot of chicken and egg problems with trying to set the password to something like ansible_password: "{{ lookup('cyberark.conjur.conjur_variable') }}".

TL;DR: I would try to get some kind of domain account for administrative duties and/or otherwise use an ssh key instead of password auth.

1 Like

Thank you so much for that well-thought-out reply. I really appreciate it.

The target servers are Linux, and all accounts involved are non-privileged. Of all your answers, the best is probably to let CyberArk give me the user’s own password, and have them elevated to the service account in question. The account that needs to do the work is a non-privileged user account under which an application server is running.

Let’s call the account tomcat_account. Once upon a time, tomcat_account had the same password across every Linux server. In a future over which I have no control, tomcat_account is going to get a new and unique password on every server every night. I immediately decided to get around this with ssh private/public key pairs, but without some root access support (and we will get none), rotating those keys (I could get away with rotating weekly or monthly) becomes a problem of its own. I’d have to go with a Blue/Green pairing of keys, and my company wants everything to flow without maintenance. Blue/Green key rotation is a job they won’t let me take on, so I have to find something easier. Since laziness is a virtue, I cannot really argue the point.

CyberArk’s cyberark.pas.cyberark_credential allows me to query something like:
"Safe=automation;UserName=tomcat_account;servername=server1"
The reason I was hoping for magic is that this query allows me to retrieve a unique and specific credential from the pool of managed credentials. I can certainly retrieve today’s password for tomcat_account on server1. The question becomes how do I get the automation node to use that credential on server1. Looking at the plugin docs, CyberArk may actually do that by magic, and may do it separately for each host. I don’t know. I trust your word on the probability that bit of magic will fail. I’m guessing hostvars are not used for credentials, so trying this on multiple servers will just muddle everything. (I’ll trust and verify, but assume it’s going to fail.)

When that fails, your idea of grabbing admin_john’s password from CyberArk and logging him on to the box, then elevating him to tomcat_account is gold. It’s a more robust solution than the direct call CyberArk may enable. This is probably where I will put my energy.

Thank you, again!

Rather than argue why you need to login as what sounds like a daemon that should be set to /sbin/nologin, requiring you to need some sudo permissions to modify its files (since apparently you will never get any kind of root permissions)…

For once in my life, I will actually argue for ssh certificates (via CyberArk).

Just in Time access with short-lived SSH certificates | CyberArk Docs

The idea here, is that you will never need to rotate tomcat’s authorized public keys, because all keys will be signed by a trusted CA. Then, you just need to checkout a short lived private key from the CA managed by CyberArk.

Additionally, CyberArk appears to have a very interesting entry for ansible:
Secure DevOps using Ansible | CyberArk Docs

That looks like you’d have to find a way to configure PSMP in such a way that Ansible (and then in turn, AWX) can use it as part of ssh.

1 Like

And a solid argument it is. :slight_smile:

I will wander back to the 3 sets of admins I need to coordinate to make anything happen, and see whether this is the thing that will make everyone happy!

I’m liking the sound of it.

Thank you again,

Kevin