Private CAs/custom certs, LDAPS Code 21 Unable to verify first certificate

jeremytourville · February 28, 2024, 7:39pm

I have been reading documentation -
[Trusting a custom certificate authority - Ansible AWX Operator Documentation]

[Add private CAs to the containers · Issue #376 · ansible/awx-operator · GitHub]

I have not been able to get this to work. I exec into the container and run:
openssl s_client -connect gsil-domain-server:636 -showcert
The error message I get is:
Verify return code: 21 (unable to verify the first certificate)

My environment is air gapped. I am using a helm chart and the chart is modified to pull from my private repo. I used this set of instructions to modify my helm chart:
local registry helm chart

All images are running without issue and I can login to the web GUI. I need to add certs to AWX for LDAP authentication and jobs that connect to our Gitlab server for source code.

#1 I have created my certs. Certs are listed as:
cat chain.crt
root ca cert
…base64 something…
intermediate cert
…base64 something…

#2 I created a secret in the awx namespace.

kubectl create secret generic awx-custom-certs \
    --from-file=ldap-ca.crt=chain.crt  \
    --from-file=bundle-ca.crt=chain.crt -n awx

Note: for this purpose ldap-ca.crt and bundle ca.crt are the same as our environment uses a common (and private) root-ca.

#3 My spec file/myvalues.yaml is getting deployed with the helm chart
helm install -n awx --create-namespace gsil-awx /tmp/awx-operator/ -f myvalues.yaml

This is the contents of myvalues.yaml:

AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13" 
  
customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi
 
  customSecrets:
    enabled: true
    ldap_cacert_secret: awx-custom-certs
    bundle_cacert_secret: awx-custom-certs
    ldap_password_secret: awx-ldap-password

I am hoping @kurokobo or another developer can chime in.

Issue #376 from Github seems like it won’t readily work for this situation:

#1 tchellomello solution relies on pulling a custom container and presumes you have internet access if you follow the code.

#2 abcmiller solution is basically the same as mine AFAIK

Can anyone spot what I am doing wrong?

kurokobo · February 29, 2024, 11:50am

Hi,

I didn’t test your values.yaml at all, but I can find some errors:

Invalid indentation: customSecrets should not be under customVolumes
Incorrect keys: customSecrets doesn’t have such child keys like ldap_cacert_secret, bundle_cacert_secret, nor ldap_password_secret.

Refer to the Helm deployment docs and ensure your values.yaml is correctly defined: https://github.com/ansible/awx-operator/tree/devel/.helm/starter

Also the Troubleshooting section in my guide may also helpful for you.

jeremytourville · February 29, 2024, 1:39pm

@kurokobo
That advice is very helpful. Thanks! Can you clarify one area of confusion for me?

I have modified my myvalues.yaml file, it now looks like this:

AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13" 
    customSecrets:
      enabled: true
      ldapCacert:
        enabled: true
        crt: <contentofmybundlecacrt>
      ldap:
        enabled: true
        password: yourldapdnpassword
      bundleCacert:
        enabled: true
        crt: <contentofmybundlecacrt>
  
customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi

How do I correctly specify the crt field? I tried specifying ---start of cert--- <some base64 stuff> --end of cert --- but that failed when trying to deploy my helm chart with that.

My co-worker has suggest that I try cat server.crt | base64 | tr -d '\n' and paste the resulting value into the yaml file. That appears to work but I am still getting error 21. I’ll review the troubleshooting section you posted for me and see if I can figure anything out but I wanted to post this just to be sure I’m on the correct path now.

I did exec into the container again and I don’t see that the certificate I specified in myvalues.yaml is getting mounted. Not quite sure where to look next. I did check the path /etc/openldap/certs/ and /etc/pki/ca-trust/source/anchors/ Thoughts?

kurokobo · February 29, 2024, 3:09pm

@jeremytourville
In the first place, for Helm deployment, there are two ways to pass custom CA certs: using AWX.spec.*_secret, or using customSecrets.*Cacert.

Using `AWX.spec.*_secret`

This is a way to define secrets through AWX CR, as described in the Operator doc

If you already have secrets that contains your CA certs, I recommend you to use this way.

AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    ...
    ldap_cacert_secret: awx-custom-certs   ✅
    bundle_cacert_secret: awx-custom-certs   ✅
    ldap_password_secret: awx-ldap-password   ✅
  
customVolumes:
  ...

Using `customSecrets.*Cacert`

This is more Helm-specific configuration. It’s documented on only the Helm doc.

AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    ...
  
customVolumes:
  ...

customSecrets:   ✅
  enabled: true   ✅
  ldapCacert:   ✅
    enabled: true   ✅
    crt: <contentofmybundlecacrt>   ✅
  ldap:   ✅
    enabled: true   ✅
    password: yourldapdnpassword   ✅
  bundleCacert:   ✅
    enabled: true   ✅
    crt: <contentofmybundlecacrt>   ✅

If you want to use this way, the value for crt can be multi-line strings:

customSecrets:
  enabled: true
  ldapCacert:
    enabled: true
    crt: |
      -----BEGIN CERTIFICATE-----
      MIIEHzCCAwegAwIBAgIJAP0/Akw/aozeMA0GCSqGSIb3DQEBCwUAMIGaMQswCQYD
      ...
      wa6cH2F8/cFvAJOE3yzIKUIMi/7HAMZfyT2DrrjmUZmRShHs+IgQNYakiMuWHzdb
      5PZ/
      -----END CERTIFICATE-----
  ldap:
    enabled: true
    password: yourldapdnpassword
  bundleCacert:
    ...

jeremytourville · February 29, 2024, 3:40pm

OK, so two different methods but both should achieve the same outcome. Got it.
I am still very new to learning Kubernetes (less than 6 mos) but I am comfortable with creating the secrets needed. I’ll go that route.

I have a fair bit of Docker experience and I’m pretty comfortable there. Learning to translate those techniques and methods to Kubernetes has been challenging for me at times but I feel like I am getting there. I describe kubernetes as “Docker on steroids”

I feel like my biggest challenge overall is really understanding templating, CRDs, variables used for the AWX project. It’s a work in progress for me. I greatly appreciate the feedback. You have really helped clarify some of my confusion.

I’ll keep plugging away on this and see where I get.

jeremytourville · February 29, 2024, 7:36pm

I am making great progress my certs are working now!
Now I am just working through the process for ldap_password_secret from here: Trusting a custom certificate authority

utoddl · February 29, 2024, 7:54pm

@jeremytourville, if you could drop a note showing what you changed to make it work and mark it as a solution, or mark one of the previous posts as a solution, that will help future us. Thanks.

jeremytourville · February 29, 2024, 8:20pm

My problem was with yaml syntax and differentiating between Using AWX.spec.*_secret and Using customSecrets.*Cacert. I was getting them mixed up. Thanks to @kurokobo for pointing that out. It really took me a minute to study that and see where the differences were. My yaml had indents at the wrong spots or under the wrong headings and so the values weren’t flowing where they needed to. My final values.yaml file is like this:


AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13"
    ldap_cacert_secret: awx-custom-cert
    ldap_password_secret: awx-ldap-unpw
    bundle_cacert_secret: awx-custom-cert    

customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi

As a side note here- I struggled greatly with the custom volumes for a while because I was following the documentation and it had it listed wrong in the example Custom Volumes

As many know and understand yaml is very particular about indents.

kurokobo · March 1, 2024, 2:24am

Hi, did ldap_password_secret also work as expected? Good to hear that you’ve solved your issues.

Good catch, indeed the indentation for example is incorrect. If you have interested, this is a good chance to contribute AWX Operator by filling issue and sending PR

jeremytourville · March 1, 2024, 1:20pm

Eventually after doing some more troubleshooting. I had to use customSecrets.*Cacert instead of AWX.spec.*_secret

I tried to use these resources. LDAP password secret documentation

This documentation was the most clear to me though. It’s in the section
LDAP BIND DN Password secret

trusting-a-custom-certificate-authority

Presently my secret contains only the ldap dn password. If I try to add ldap_password_secret (using the AWX.spec.*_secret syntax) to myvalues.yaml I can see that only two of my four pods spin up. (the operator and db) If I remove that field all four pods spin up. (as expected- operator, db, awx-task, awx-web). Running kubectl describe show me this:

Name:               awx-ldap-unpw
Namespace:          awx
Labels:             <none>
Annotations:        <none>

Type:               Opaque

Data
====
password        14 bytes

That certainly seems correct AFAIK. Is there another value that needs to be included in my secret besides the password?

I ultimately ended up with a myvalues yaml file that looked like this:

AWX:
  # enable use of awx-deploy template
  enabled: true
  name: awx
  spec:
    admin_user: admin
    hostname: awx.idm.gsil.org
    image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx:23.7.0
    image_version: 23.7.0
    init_container_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee
    init_container_image_version: latest
    ee_images:
    - name: AWX EE (latest)
      image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    redis_image: gsil-docker1.idm.gsil.org:5001/redis
    redis_image_version: "7"
    control_plane_ee_image: gsil-docker1.idm.gsil.org:5001/quay.io/ansible/awx-ee:latest
    postgres_image: gsil-docker1.idm.gsil.org:5001/postgres
    postgres_image_version: "13"
    ldap_cacert_secret: awx-custom-cert
    #ldap_password_secret: awx-ldap-unpw
    bundle_cacert_secret: awx-custom-cert
    extra_settings:
    - setting: AUTH_LDAP_SERVER_URI
      value: >-
        "ldaps://ad01.abc.com:636 ldaps://ad02.abc.com:636"

    - setting: AUTH_LDAP_BIND_DN
      value: >-
        "CN=LDAP User,OU=Service Accounts,DC=abc,DC=com"    
   
customVolumes:
  postgres:
    enabled: true
    hostPath: /var/lib/postgresql/data
    size: 2Gi
    storageClassName: local-storage
  projects:
    enabled: true
    hostPath: /opt/projects/data
    size: 5Gi
 
customSecrets: 
  enabled: true
  ldap:  
    enabled: true   
    password: yourldapdnpassword

Maybe not what I would have preferred but it does work. My admin team can login using their AD credentials after setting all the parameters for LDAP in the GUI.

kurokobo · March 2, 2024, 4:50am

@jeremytourville
Hi,

jeremytourville:

Running kubectl describe show me this:
Name:               awx-ldap-unpw
Namespace:          awx
Labels:             <none>
Annotations:        <none>

Type:               Opaque

Data
====
password        14 bytes
That certainly seems correct AFAIK. Is there another value that needs to be included in my secret besides the password?

See the example command in documentation:

kubectl create secret generic <resourcename>-ldap-password \
    --from-literal=ldap-password=<your_ldap_dn_password>

The KEY for the data is important.

As this example, Operator expects that there is a password with key ldap-password, instead of just password.

github.com

ansible/awx-operator/blob/d0827ba42680f42ed7d0ddbe1c751e376dd3e4c4/roles/installer/tasks/load_ldap_password_secret.yml#L12


      
          - name: Retrieve LDAP bind password Secret
            k8s_info:
              kind: Secret
              namespace: '{{ ansible_operator_meta.namespace }}'
              name: '{{ ldap_password_secret }}'
            register: ldap_password
            no_log: "{{ no_log }}"
          
          - name: Load LDAP bind password Secret content
            set_fact:
              ldap_bind_password: '{{ ldap_password["resources"][0]["data"]["ldap-password"] | b64decode }}'
            no_log: "{{ no_log }}"
            when: '"ldap-password" in ldap_password["resources"][0]["data"]'

So you have to have secret like:

Name:               awx-ldap-unpw
Namespace:          awx
Labels:             <none>
Annotations:        <none>

Type:               Opaque

Data
====
ldap-password        14 bytes   ✅

jeremytourville · March 4, 2024, 12:48pm

OH!!! How did I miss that? So simple.

system · April 3, 2024, 12:49pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
AWX ldap spec/myvalues.yaml file Get Help awx , collections , kubernetes , ee	3	394	March 1, 2024
Trusting a Custom Certificate Authority does not work Get Help awx , collections , kubernetes	0	282	December 21, 2023
SSL Issue with new AWX install Get Help awx	4	121	July 25, 2024
AWX failing to clone repo from Azure Devops Server 2019 Get Help awx , playbook , azure	3	31	October 5, 2024
Awx postgress setup via tls certs Get Help awx , postgresql	3	509	October 9, 2023

Private CAs/custom certs, LDAPS Code 21 Unable to verify first certificate

Using AWX.spec.*_secret

Using customSecrets.*Cacert

Related topics

Using `AWX.spec.*_secret`

Using `customSecrets.*Cacert`