Handle Vault credentials type in AWX

Get Help
,
1

Hi

In order to improve my AWX management for my team, i’m currently trying to configure Hashicorp Vault Credentials but i’m having some errors and doubt.

Usually in my Ansible project I run from a Debian server, I’m used to set up Vault secrets using lookup like this (works in var file or within vars:)

### Vault Configuration
ansible_hashi_vault_token="{{ lookup('env','VAULT_ANSIBLE') }}"

my_secret="{{ lookup('community.hashi_vault.vault_kv2_get', 'my_secret', engine_mount_point='kv/', token=ansible_hashi_vault_token) }}"

#### Credentials (compte de service ansible) ####
ansible_user="{{ my_secret.secret.ansible_user_from_vault}}"

Ansible will seek for an environment variable which is called ‘VAULT_ANSIBLE’ that is the result token of an AppRole call to my vault server. It allows to run playbook without writing any token in the code.

Now after adding project in AWX, i’m trying to make the Vault part working =>

  1. I created Credentials with HashiCorp Vault Secret Lookup as you can see here. Simply add Server URL + Token for testing.

    VAULT_AWX1
    VAULT_AWX11645×855 31.3 KB

  2. When I click on Test, no matter what secrets I want to retrieve I always have this error

    VAULT_AWX2
    VAULT_AWX2796×679 21.9 KB

Here’s my questions :

  • which log file can give me more information about the credentials/api call to Vault?
  • Is there any thing I miss for this peculiar configuration ? should i try to add my root CA or intermediate CA while configuring credentials ? if yes which cert format should it be ?
  • is there a proper way/another way to use Vault lookup within AWX ?

I saw on many forum that people are creating custom Credential Types, then use an Injector to use vault lookup.

Thanks a lot for your help
Gael

EDIT 12 12 2023
Found the logs (using kubectl logs my-pod) corresponding to the Test credential. Seems to be a django Bad Request but don’t know how to investigate that;

2023-12-12 14:40:40,628 WARNING  [4fdf7cfc36804499b34fa50a28369fb7] awx.api.generics status 400 received by user admin attempting to access /api/v2/credentials/5/test/ from 10.10.10.10
2023-12-12 14:40:40,635 WARNING  [4fdf7cfc36804499b34fa50a28369fb7] django.request Bad Request: /api/v2/credentials/5/test/
2023-12-12 14:40:40,635 WARNING  [4fdf7cfc36804499b34fa50a28369fb7] django.request Bad Request: /api/v2/credentials/5/test/
10.10.10.10 - - [12/Dec/2023:14:40:40 +0000] "POST /api/v2/credentials/5/test/ HTTP/1.1" 400 21 "https://my-awx.net/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0" "10.230.173.133"
[pid: 23|app: 0|req: 228/1089] 172.20.245.206 () {72 vars in 2440 bytes} [Tue Dec 12 14:40:40 2023] POST /api/v2/credentials/5/test/ => generated 21 bytes in 76 msecs (HTTP/1.1 400) 14 headers in 598 bytes (1 switches on core 0)
2

After many tests and troubleshoot, finally ends up with a working solution :

  1. I provided the root + intermediate certificate as a pem file in the credentials settings (be careful to have the correct cert, in the correct order, in pem format : intermediate + root
  • From this, it moves from SSLError to HTTP 404
  • After some tests the perfect setting to make it work was this one

enter image description here

I perform some other test without certificate and I have SSLError everytime.

  1. Don’t forget to select v2 for the API version in credentials settings
    enter image description here

Conclusion : you should provide your custom CA content or as a file in the credentials settings.

3

Hi,

I am also facing similar errors in the AAP + Hashicorp vault integration. How to get custom CA in my setup. I am running Vault as a Docker container, which is hosted inside a VM. I gave rootCA of vault server in the AAP UI —> Credentials —> CA cert. I am using token-based authentication for testing credentials, always getting ‘Bad Request’, not able to see any other logs related to this issue.

Do you have any idea how to fix this issue?

4

My last solution post doesn’t work for you ?

5

No, it didn’t work for me. Always getting below error

'Something went wrong with the request to test this credential'.

Bad Request

Not able to move forward to next stage

6

are you sure about your certificate content and format ?
what if you do a

openssl x509 -in your_cert.cer -text -noout

Does the file/content is valid ?

7

My certificate is valid. Here you mean AAP certificate, right?

I can retrieve secrets from Hashicorp vault via terminal server using curl, where I am running AAP. I am facing an issue with UI; when I checked certificate details in the web browser, it says valid only. When I exported cert details and saved them on the local server, there I checked to run openssl cmd. Cert valid till Jan 2035.

Is there anything else do I need to verify there

8

I don’t mean AAP certificate, I mean your root CA certificate you put in AAP :slight_smile:

9

Hi,

Yes, I am also talking about the same. I am thinking now that my AAP is running as a container in a separate VM; hashicorp Vault also runs as a container in a different VM. I am accessing AAP UI via browser; that is a different machine. In this scenario, do you think I missed any certificates to add?

I have checked the AAP cert and the Vault server cert; both are valid.

10

I got the same architecture as yours (except I use AWX instead of AAP).
Which version of AAP are you running ?

11

I am using AAP 2.5. controller - 4.6.3, hub - 4.10.1

12

Can you tell me how you managed to see logs from the pod? I mean, which container do we need to look into?

13

As mentionned in my original post i’m using kubectl logs my-pod to get logs.
Concerned pod should be the one responsible for “web” and/or the one responsible for “task”

14

oh ok, let me check it there

15

I am also getting exactly the same logs; how do you manage to create a custom CA to resolve this error?

My scenario is really strange: both gitlab and hashicorp are running as a container on top of the same VM, and AAP is running on a different VM. I was able to establish connections to gitlab instances without an issue, but when I tried to access the vault server, I got a bad request. How is it possible that both Gitlab and Vault use the same certificates and one works while the other does not?

16

Sorry I was busy last days.
If you’re 100% sure of your certificate format (check both of them with the openssl cli i provided in the older comment) so it’s probably an AAP error / misfunction.

Could you provide a screenshot from your config using private messages ? (like my post in dec 2023 at " Don’t forget to select v2 for the API version in credentials settings")

17

Hi, Now, I am able to test the credentials after changing the cert in .pem format and uploading it there. It is working now. But do we need to use AppRole to use vault credentials inside playbook or its fine to use vault token as well?

I mean, how shall we test vault collection in the AAP running via playbook.

18

Right, so until here this is what I wrote on Dec 2023 in this thread…
In my case I prefer to use AppRole cause we have to restrict some access/actions but if you prefer token it’s up to you.

I mean, how shall we test vault collection in the AAP running via playbook.

As I still don’t understand what you try to achieve, just basically read these documentation to know how to use Hashicorp Vault with Ansible/AWX/AAP :

https://galaxy.ansible.com/ui/repo/published/community/hashi_vault/

Good luck.

19

What is the purpose of using the AAP’s HashiCorp Vault secret lookup credentials? When we pass “HashiCorp Vault credentials” inside other credentials of the “Machine” type, does it have the ability to retrieve secrets from the vault server?

A playbook that uses a vault collection will therefore retrieve the VAULT_ADDR & VAULT_TOKEN values from the credentials (in this case, machine credentials) selected in the templates when I wish to run it. If not, do we have to make our own credentials plugins to accomplish the same thing?

Because in my situation, I can easily obtain the value of secrets from the vault server if I construct a custom credential plugin. However, the VAULT_ADDR & VAULT_TOKEN values are not injecting correctly if I use the Hashicorp Vault secret lookup credentials.

Is there a constraint preventing us from using the HashiCorp Vault secret lookup for certain types of secrets?

How did you manage AAP automatically injects value for secret lookup plugins? Did you explicitly configure injected value in the environment variables?

20

I can only speak about AWX but in my case I had to create custom credentials type to be able to use AppRole.
For AAP and token use i don’t know sorry.

Have a read to this documentation.

(Also, if you run AAP i think you have official support isn’t it ? maybe they will be more skilled for this ?)