How to add new certificate chain in AWX?

Get Help
, , ,
1

Hi
My team has just migrated GitLab instance from one to another domain (so fqdn change from git.domain1.com to git.domain2.com) and perform AWX projects migration.

By default, gitlab works in http, AWX was able to pull project using deploy tokens credentials.
However, since I set up SSL certificate on the new instance, every project pulling ends with :
Failed to download remote objects and refs: fatal: unable to access 'https://git.domain2.com/awx/test-linux-ping.git/': SSL certificate problem: self-signed certificate in certificate chain

For what I know, this error come from git client (so in our case, git command called by AWX) but here’s my questions :

  1. What should I update to enable AWX to trust the new certificate chain of domain2.com ?
  2. Which pods are responsible for git pull actions ?

(Moreover:
I’ll update my custom Execution Environment image by adding the cert chain in my dockerfile if some projects have to deal with *.domain2.com ) but in my case, the EE isn’t used when git pull any projects.

Thanks a lot for your help
Gael

2

Hi @motorbass

but in my case, the EE isn’t used when git pull any projects.

That’s right, I can’t 100% recall right now but I think the awx-task pod is used for Project updates, but don’t quote me on that :sweat_smile:

You will need to make sure that AWX is aware of your new CA chain. You can do this using the Trusting a Custom Certificate Authority - Ansible AWX Operator Documentation ):

bundle_cacert_secret
1 Like
3

Hi ! Thanks , I just add my certs in my EE then configure awx as it’s mentionned on the guide your provided.

but there’s something wrong :

  1. I created a generic secret kubectl create secret generic myca-custom-certs --from-file=bundle-ca.crt="bundle-ca.crt" bundle-ca.crt is a file that contains intermediate CA then root CA.

  2. According to the link you provide, I just edited my spec: adding :
    bundle_cacert_secret: myca-custom-certs

  3. To double check it was fine after pods restart, I use this to check if certificate is ok or not trying to reach my private registry :
    kubectl exec -it awx-task-644478848d-pvm7x -- openssl s_client -connect git.domain2.com -servername git.domain2.com

Result show :

subject=C = FR, ST = XX, L = XXXXX, O = XXXXX, OU = XXX, CN = git.domain2.com
subject=C = FR, ST = XX, L = XXXXX, O = XXXXX, OU = XXX, CN = intermediate CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4538 bytes and written 411 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
  • So it means that AWX pods now have their certificate list updated and are able to check cert of my gitlab instance.
  • Also, i got the same result if i manually start a container and launch the same command with my custom EE.

However, from GUI, when I launch a project it fails and temp pods (automation-XXX) show :

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  75s                default-scheduler  Successfully assigned my-ns/automation-job-6262-f5nl9 to worker-11a2ba-worker-1
  Normal   Pulling    33s (x3 over 75s)  kubelet            Pulling image "git.domain2.com:5050/infrastructures/custom-ee:1.1.2"
  Warning  Failed     33s (x3 over 75s)  kubelet            Failed to pull image "git.domain2.com:5050/infrastructures/custom-ee:1.1.2": rpc error: code = Unknown desc = failed to pull and unpack image "git.domain2.com:5050/infrastructures/custom-ee:1.1.2": failed to resolve reference "git.domain2.com:5050/infrastructures/custom-ee:1.1.2": failed to do request: Head "git.domain2.com:5050/infrastructures/custom-ee:1.1.2": 
x509: certificate signed by unknown authority
  Warning  Failed     33s (x3 over 75s)  kubelet            Error: ErrImagePull
  Normal   BackOff    7s (x5 over 75s)   kubelet            Back-off pulling image "git.domain2.com:5050/infrastructures/custom-ee:1.1.2"
  Warning  Failed     7s (x5 over 75s)   kubelet            Error: ImagePullBackOff

Kind of weird…

4

@kurokobo i know you’re watching :smiley:

5

That’s because the steps in the referenced documentation above were just for AWX itself to be able to pull projects from your GitLab instance with a self-signed certificate. What you’ve showed in your last output is your K8s cluster node trying to pull from GitLab’s container registry. You need to trust your Intermediate and Root CA on the Kubernetes cluster node(s).

6

In fact my project (from AWX perspective) pull GitLab project and has to use a dedicated EE which is hosted on the same GitLab instance, but it’s a private registry.

@dbrennand why certs should be trusted from the nodes in that case ? i mean, depends on the action, certs should be updated on nodes and/or pods ?

EDIT: ok i just discovered that private registries should be configured through nodes according to many docs :smiley:

7

I’m not 100% sure on what component creates the automation-xxx pods, it may be the AWX Operator, but the deployment on your Kubernetes cluster is created for it, at that point your Kubernetes cluster node will pull the image in the deployment from your registry to run the job, in this case it’s GitLab.

In summary, you need to deploy your Root and Intermediate CA certificates on your Kubernetes cluster nodes to resolve this. If you’d like me to help further, please provide the Kubernetes distribution (OpenShift, K3s etc) and the OS of your cluster nodes.

Good luck :slightly_smiling_face:

8

To close this thread, the solution was to add thr private registry and certs on nodes, then everything run smoothly :wink:

Thanks @dbrennand

1 Like