Custom Execution Environment image from a registry w/ custom CA-signed HTTPS certiticate

Archives AWX Project
, ,
1

Greeting everyone,

I am attempting to reconfigure our AWX instance so that it can pull our custom Execution Environment image from our internal OCI registry instead of RH Quay.

This registry (oci-registry.lorient.iot) uses an HTTPS certificate signed by our custom authority so all my jobs currently fail with the following in the automation-job-XXX pod events:
Events:
Type Reason Age From Message

Normal Scheduled 4s default-scheduler Successfully assigned awx-vincent/automation-job-5857-h4svm to awx
Normal Pulling 4s kubelet Pulling image “oci-registry.lorient.iot/awx/redpesk-ee”
Warning Failed 4s kubelet Error: ErrImagePull
Warning Failed 4s kubelet Failed to pull image “oci-registry.lorient.iot/awx/redpesk-ee”: rpc error: code = Unknown desc = failed to pull and unpack image “oci-registry.lorient.iot/awx/redpesk-ee:latest”: failed to resolve reference “oci-registry.lorient.iot/awx/redpesk-ee:latest”: failed to do request: Head “https://oci-registry.lorient.iot/v2/awx/redpesk-ee/manifests/latest”: x509: certificate signed by unknown authority
Normal BackOff 3s kubelet Back-off pulling image “oci-registry.lorient.iot/awx/redpesk-ee”
Warning Failed 3s kubelet Error: ImagePullBackOff

I have followed the documentation[1] to add our CA certificate to the pods environment and can see it is correctly present there. For instance, in the awx-ee pod, I have the associated new mounts:
[…]
Mounts:
/etc/pki/ca-trust/extracted from ca-trust-extracted (rw)
/etc/pki/ca-trust/source/anchors/bundle-ca.crt from awx-bundle-cacert (ro,path=“bundle-ca.crt”)

Our CA certificate also appears to be taken into account:
(vrubiolo)$ k exec awx-7b98b865-xdmb7 -ti -c awx-ee – /bin/bash
bash-4.4$ trust list |head
pkcs11:id=%71%30%BF%E2%FE%C3%DC%7C%9B%C2%39%E1%7F%44%71%B4%71%11%CD%51;type=cert
type: certificate
label: IoT.bzh_Internal_Certification_Authority
trust: anchor
category: authority
[…]

From what I see here, the issue is that the certificate needs to be present in the K8S cluster environment (where the image pull occurs) and not in one of the AWX containers.

Could sb shed more light on the process of using an EE from a custom registry whose certificate is signed by a custom CA?

Thanks for your help,

Vincent

[1] https://github.com/ansible/awx-operator#trusting-a-custom-certificate-authority

2

Hi again,

Would someone have more insight on this issue?

Based on my research, it looks like the issue is the unability to specify custom CA certificates for the registry when one creates registry credentials in AWX (this is reflected in the UI). When adding such credentials, the associated imagePullSecret is created but the certificates are not added to the system.

I have found https://github.com/ansible/awx/issues/10252 which is exactly about this issue.

I am currently deploying over K3S so this can be managed via custom registry settings in K3S itself which I am going to look at: https://rancher.com/docs/k3s/latest/en/installation/private-registry/#with-tls

I was wondering what was the best practice on the AWX side for you this until the abovementioned ticket is addressed.

Thanks for your help,

Vincent