# Private CAs/custom certs, LDAPS Code 21 Unable to verify first certificate

**URL:** <https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119>\
**Category:** Get Help\
**Tags:** documentation, awx, ee\
**Created:** [February 28, 2024, 7:39pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119 "2024-02-28T19:39:20Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [February 28, 2024, 7:39pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/1 "2024-02-28T19:39:20Z")

</div>

I have been reading documentation -  
[[Trusting a custom certificate authority - Ansible AWX Operator Documentation](https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/trusting-a-custom-certificate-authority.html)]

[[Add private CAs to the containers · Issue #376 · ansible/awx-operator · GitHub](https://github.com/ansible/awx-operator/issues/376)]

I have not been able to get this to work. I exec into the container and run:  
`openssl s_client -connect gsil-domain-server:636 -showcert`  
The error message I get is:  
`Verify return code: 21 (unable to verify the first certificate)`

My environment is air gapped. I am using a helm chart and the chart is modified to pull from my private repo. I used this set of instructions to modify my helm chart:  
[local registry helm chart](https://www.dbi-services.com/blog/deploy-awx-operator-with-helm-using-images-from-a-local-registry/)

All images are running without issue and I can login to the web GUI. I need to add certs to AWX for LDAP authentication and jobs that connect to our Gitlab server for source code.

#1 I have created my certs. Certs are listed as:  
`cat chain.crt`  
root ca cert  
…base64 something…  
intermediate cert  
…base64 something…

#2 I created a secret in the awx namespace.

```auto
kubectl create secret generic awx-custom-certs \
    --from-file=ldap-ca.crt=chain.crt \
    --from-file=bundle-ca.crt=chain.crt -n awx

```

**Note** : for this purpose ldap-ca.crt and bundle ca.crt are the same as our environment uses a common (and private) root-ca.

#3 My spec file/myvalues.yaml is getting deployed with the helm chart  
`helm install -n awx --create-namespace gsil-awx /tmp/awx-operator/ -f myvalues.yam`l

This is the contents of myvalues.yaml:

```auto
AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13" 
  
customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi
 
  customSecrets:
    enabled: true
    ldap_cacert_secret: awx-custom-certs
    bundle_cacert_secret: awx-custom-certs
    ldap_password_secret: awx-ldap-password

```

I am hoping @kurokobo or another developer can chime in.

Issue #376 from Github seems like it won’t readily work for this situation:

#1 tchellomello solution relies on pulling a custom container and presumes you have internet access if you follow the code.

#2 abcmiller solution is basically the same as mine AFAIK

Can anyone spot what I am doing wrong?

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [February 29, 2024, 11:50am UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/2 "2024-02-29T11:50:16Z")

</div>

Hi,

> [@jeremytourville](#):
>
> This is the contents of myvalues.yaml:

I didn’t test your `values.yaml` at all, but I can find some errors:

- Invalid indentation: `customSecrets` should not be under `customVolumes`
- Incorrect keys: `customSecrets` doesn’t have such child keys like `ldap_cacert_secret`, `bundle_cacert_secret`, nor `ldap_password_secret`.

Refer to the Helm deployment docs and ensure your `values.yaml` is correctly defined: [https://github.com/ansible/awx-operator/tree/devel/.helm/starter](https://github.com/ansible/awx-operator/tree/devel/.helm/starter)

Also the [Troubleshooting section in my guide](https://github.com/kurokobo/awx-on-k3s/blob/main/tips/trust-custom-ca.md#troubleshooting) may also helpful for you.

---

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [February 29, 2024, 1:39pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/3 "2024-02-29T13:39:38Z")

</div>

@kurokobo  
That advice is very helpful. Thanks! Can you clarify one area of confusion for me?

I have modified my myvalues.yaml file, it now looks like this:

```auto
AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13" 
    customSecrets:
      enabled: true
      ldapCacert:
        enabled: true
        crt: <contentofmybundlecacrt>
      ldap:
        enabled: true
        password: yourldapdnpassword
      bundleCacert:
        enabled: true
        crt: <contentofmybundlecacrt>
  
customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi

```

How do I correctly specify the crt field? I tried specifying` ---start of cert--- <some base64 stuff> --end of cert --`- but that failed when trying to deploy my helm chart with that.

My co-worker has suggest that I try `cat server.crt | base64 | tr -d '\n'` and paste the resulting value into the yaml file. That appears to work but I am still getting error 21. I’ll review the troubleshooting section you posted for me and see if I can figure anything out but I wanted to post this just to be sure I’m on the correct path now.

I did exec into the container again and I don’t see that the certificate I specified in myvalues.yaml is getting mounted. 🤔 Not quite sure where to look next. I did check the path` /etc/openldap/certs/` and` /etc/pki/ca-trust/source/anchors/` Thoughts?

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [February 29, 2024, 3:09pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/4 "2024-02-29T15:09:41Z")

</div>

@jeremytourville  
In the first place, for Helm deployment, there are two ways to pass custom CA certs: using `AWX.spec.*_secret`, or using `customSecrets.*Cacert`.

#### Using `AWX.spec.*_secret`

This is a way to define secrets through AWX CR, as described in [the Operator doc](https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/trusting-a-custom-certificate-authority.html)

If you already have secrets that contains your CA certs, I recommend you to use this way.

```yaml
AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    ...
    ldap_cacert_secret: awx-custom-certs ✅
    bundle_cacert_secret: awx-custom-certs ✅
    ldap_password_secret: awx-ldap-password ✅
  
customVolumes:
  ...

```

#### Using `customSecrets.*Cacert`

This is more Helm-specific configuration. It’s documented on only [the Helm doc](https://github.com/ansible/awx-operator/tree/devel/.helm/starter).

```yaml
AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    ...
  
customVolumes:
  ...

customSecrets: ✅
  enabled: true ✅
  ldapCacert: ✅
    enabled: true ✅
    crt: <contentofmybundlecacrt> ✅
  ldap: ✅
    enabled: true ✅
    password: yourldapdnpassword ✅
  bundleCacert: ✅
    enabled: true ✅
    crt: <contentofmybundlecacrt> ✅

```

If you want to use this way, the value for `crt` can be multi-line strings:

```yaml
customSecrets:
  enabled: true
  ldapCacert:
    enabled: true
    crt: |
      -----BEGIN CERTIFICATE-----
      MIIEHzCCAwegAwIBAgIJAP0/Akw/aozeMA0GCSqGSIb3DQEBCwUAMIGaMQswCQYD
      ...
      wa6cH2F8/cFvAJOE3yzIKUIMi/7HAMZfyT2DrrjmUZmRShHs+IgQNYakiMuWHzdb
      5PZ/
      -----END CERTIFICATE-----
  ldap:
    enabled: true
    password: yourldapdnpassword
  bundleCacert:
    ...

```

---

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [February 29, 2024, 3:40pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/5 "2024-02-29T15:40:04Z")

</div>

OK, so two different methods but both should achieve the same outcome. Got it.  
I am still very new to learning Kubernetes (less than 6 mos) but I am comfortable with creating the secrets needed. I’ll go that route.

I have a fair bit of Docker experience and I’m pretty comfortable there. Learning to translate those techniques and methods to Kubernetes has been challenging for me at times but I feel like I am getting there. I describe kubernetes as “Docker on steroids”

I feel like my biggest challenge overall is really understanding templating, CRDs, variables used for the AWX project. It’s a work in progress for me. I greatly appreciate the feedback. You have really helped clarify some of my confusion.

I’ll keep plugging away on this and see where I get. 🤞

---

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [February 29, 2024, 7:36pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/7 "2024-02-29T19:36:20Z")

</div>

I am making great progress my certs are working now! 😀  
Now I am just working through the process for `ldap_password_secret` from here: [Trusting a custom certificate authority](https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/trusting-a-custom-certificate-authority.html)

---

<div class="post-metadata">

**Author:** ![utoddl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/utoddl/32/249_2.png) [@utoddl](https://forum.ansible.com/u/utoddl)\
**Post date:** [February 29, 2024, 7:54pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/8 "2024-02-29T19:54:40Z")

</div>

@jeremytourville, if you could drop a note showing what you changed to make it work and mark it as a solution, or mark one of the previous posts as a solution, that will help future us. Thanks.

---

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [February 29, 2024, 8:20pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/9 "2024-02-29T20:20:16Z")

</div>

My problem was with yaml syntax and differentiating between Using `AWX.spec.*_secret` and Using `customSecrets.*Cacert`. I was getting them mixed up. Thanks to @kurokobo for pointing that out. It really took me a minute to study that and see where the differences were. My yaml had indents at the wrong spots or under the wrong headings and so the values weren’t flowing where they needed to. My final values.yaml file is like this:

```auto

AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13"
    ldap_cacert_secret: awx-custom-cert
    ldap_password_secret: awx-ldap-unpw
    bundle_cacert_secret: awx-custom-cert    

customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi

```

As a side note here- I struggled greatly with the custom volumes for a while because I was following the documentation and it had it listed wrong in the example [Custom Volumes](https://github.com/ansible/awx-operator/tree/devel/.helm/starter)

As many know and understand yaml is very particular about indents.

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [March 1, 2024, 2:24am UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/10 "2024-03-01T02:24:36Z")

</div>

Hi, did `ldap_password_secret` also work as expected? Good to hear that you’ve solved your issues.

> [@jeremytourville](#):
>
> As a side note here- I struggled greatly with the custom volumes for a while because I was following the documentation and it had it listed wrong in the example [Custom Volumes](https://github.com/ansible/awx-operator/tree/devel/.helm/starter)

Good catch, indeed the indentation for example is incorrect. If you have interested, this is a good chance to contribute AWX Operator by filling issue and sending PR 😃

---

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [March 1, 2024, 1:20pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/11 "2024-03-01T13:20:32Z")

</div>

> [@kurokobo](#):
>
> Hi, did `ldap_password_secret` also work as expected?

Eventually after doing some more troubleshooting. I had to use `customSecrets.*Cacert` instead of `AWX.spec.*_secret`

I tried to use these resources. [LDAP password secret documentation](https://github.com/ansible/awx-operator/issues/1520)

This documentation was the most clear to me though. It’s in the section  
**LDAP BIND DN Password secret**

[trusting-a-custom-certificate-authority](https://github.com/ansible/awx-operator/blob/devel/docs/user-guide/advanced-configuration/trusting-a-custom-certificate-authority.md)

Presently my secret contains only the ldap dn password. If I try to add `ldap_password_secret` (using the `AWX.spec.*_secret` syntax) to myvalues.yaml I can see that only two of my four pods spin up. (the operator and db) If I remove that field all four pods spin up. (as expected- operator, db, awx-task, awx-web). Running` kubectl describe` show me this:

```auto
Name: awx-ldap-unpw
Namespace: awx
Labels: <none>
Annotations: <none>

Type: Opaque

Data
====
password 14 bytes

```

That certainly seems correct AFAIK. Is there another value that needs to be included in my secret besides the password?

I ultimately ended up with a myvalues yaml file that looked like this:

```auto
AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13"
    ldap_cacert_secret: awx-custom-cert
    #ldap_password_secret: awx-ldap-unpw
    bundle_cacert_secret: awx-custom-cert
    extra_settings:
    - setting: AUTH_LDAP_SERVER_URI
      value: >-
        "ldaps://ad01.abc.com:636 ldaps://ad02.abc.com:636"

    - setting: AUTH_LDAP_BIND_DN
      value: >-
        "CN=LDAP User,OU=Service Accounts,DC=abc,DC=com"    
   
customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi
 
customSecrets: 
  enabled: true
  ldap:  
    enabled: true   
    password: yourldapdnpassword

```

Maybe not what I would have preferred but it does work. My admin team can login using their AD credentials after setting all the parameters for LDAP in the GUI.

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [March 2, 2024, 4:50am UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/12 "2024-03-02T04:50:16Z")

</div>

@jeremytourville  
Hi,

> [@jeremytourville](#):
>
> Running` kubectl describe` show me this:
> 
> ```auto
> Name: awx-ldap-unpw
> Namespace: awx
> Labels: <none>
> Annotations: <none>
> 
> Type: Opaque
> 
> Data
> ====
> password 14 bytes
> 
> ```
> 
> That certainly seems correct AFAIK. Is there another value that needs to be included in my secret besides the password?

See [the example command in documentation](https://ansible.readthedocs.io/projects/awx-operator/en/latest/user-guide/advanced-configuration/trusting-a-custom-certificate-authority.html):

```bash
kubectl create secret generic <resourcename>-ldap-password \
    --from-literal=ldap-password=<your_ldap_dn_password>

```

The KEY for the data is important.

As this example, Operator expects that there is a password with key `ldap-password`, instead of just `password`.

> <https://github.com/ansible/awx-operator/blob/d0827ba42680f42ed7d0ddbe1c751e376dd3e4c4/roles/installer/tasks/load_ldap_password_secret.yml#L12>

So you have to have secret like:

```auto
Name: awx-ldap-unpw
Namespace: awx
Labels: <none>
Annotations: <none>

Type: Opaque

Data
====
ldap-password 14 bytes ✅

```

---

<div class="post-metadata">

**Author:** ![jeremytourville](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/jeremytourville/32/2448_2.png) [@jeremytourville](https://forum.ansible.com/u/jeremytourville)\
**Post date:** [March 4, 2024, 12:48pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/13 "2024-03-04T12:48:51Z")

</div>

OH!!! How did I miss that? So simple. 😊

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\
**Post date:** [April 3, 2024, 12:49pm UTC](https://forum.ansible.com/t/private-cas-custom-certs-ldaps-code-21-unable-to-verify-first-certificate/4119/14 "2024-04-03T12:49:08Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.