Trust custom Certificate Authority setup failed with "The Secret "awx-custom-certs" is invalid: metadata.annotations: Too long: must have at most 262144 bytes" Error

Get Help
,
1

**I am trying to setup Trust custom Certificate Authority - Using “awx-on-k3s/tips/trust-custom-ca.md at main · kurokobo/awx-on-k3s · GitHub” but getting The Secret “awx-custom-certs” is invalid: metadata.annotations: Too long: must have at most 262144 bytes" Error **.

  • My awx setup, already enabled with LDAP CA cert, and now trying to add additional CA trust, i have followed above url but getting error while kubectl apply -k base. I have already copied by pem file to /awx-on-k3s/base folder.
  • kubectl version
    Client Version: v1.29.6+k3s2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.6+k3s2
    AWX 24.6.1
  • Logs
    Error :- The Secret “awx-custom-certs” is invalid: metadata.annotations: Too long: must have at most 262144 bytes
  • my code:
Under awx.yml file
spec:
  ingress_annotations: |
    traefik.ingress.kubernetes.io/router.middlewares: kube-system-hsts@kubernetescrd

  ldap_cacert_secret: awx-custom-certs
  bundle_cacert_secret: awx-custom-certs


Under kustomize.yml file
secretGenerator:
  - name: awx-cert
    type: kubernetes.io/tls
    files:
      - server.crt
      - server.key

  - name: awx-custom-certs
    type: Opaque
    files:
      - ldap-ca.crt=ldaps.crt
      - bundle-ca.crt=ca-bundle.trust.crt

@kurokobo

2

Try kubectl apply --server-side -k base instead.

3

Thank you @kurokobo,

I have tried above command, it got failed with below error, but Command is successful with force conflict option.

But still certificates is not reflecting inside AWX task/web. Kindly advise.

Note: i am already using https and ldap certificate already.

[root@server  awx-on-k3s]# kubectl apply --server-side -k base
secret/awx-admin-password serverside-applied
secret/awx-postgres-configuration serverside-applied
persistentvolume/awx-postgres-15-volume serverside-applied
persistentvolume/awx-projects-volume serverside-applied
persistentvolumeclaim/awx-projects-claim serverside-applied
awx.awx.ansible.com/awx serverside-applied
Apply failed with 2 conflicts: conflicts with "kubectl-client-side-apply" using v1:
- .data.server .crt
- .data.server .key
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts
Apply failed with 1 conflict: conflict with "kubectl-client-side-apply" using v1: .data.ldap-ca.crt
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts
[root@server  awx-on-k3s]# 



[root@server awx-on-k3s]# kubectl apply --server-side --force-conflicts -k base
secret/awx-admin-password serverside-applied
secret/awx-cert serverside-applied
secret/awx-custom-certs serverside-applied
secret/awx-postgres-configuration serverside-applied
persistentvolume/awx-postgres-15-volume serverside-applied
persistentvolume/awx-projects-volume serverside-applied
persistentvolumeclaim/awx-projects-claim serverside-applied
awx.awx.ansible.com/awx serverside-applied
[root@server awx-on-k3s]#

[root@server ~]# kubectl exec -it awx-web-55948575db-vnb9c -n awx  -- bash
bash-5.1$ head /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
# ACCVRAIZ1
-----BEGIN CERTIFICATE-----
MIIH0zCCBbugAwIBAgIIXsO3pkN/pOAwDQYJKoZIhvcNAQEFBQAwQjESMBAGA1UE
AwwJQUNDVlJBSVoxMRAwDgYDVQQLDAdQS0lBQ0NWMQ0wCwYDVQQKDARBQ0NWMQsw
CQYDVQQGEwJFUzAeFw0xMTA1MDUwOTM3MzdaFw0zMDEyMzEwOTM3MzdaMEIxEjAQ
BgNVBAMMCUFDQ1ZSQUlaMTEQMA4GA1UECwwHUEtJQUNDVjENMAsGA1UECgwEQUND
VjELMAkGA1UEBhMCRVMwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCb
qau/YUqXry+XZpp0X9DZlv3P4uRm7x8fRzPCRKPfmt4ftVTdFXxpNRFvu8gMjmoY
HtiP2Ra8EEg2XPBjs5BaXCQ316PWywlxufEBcoSwfdtNgM3802/J+Nq2DoLSRYWo
G2ioPej0RGy9ocLLA76MPhMAhN9KSMDjIgro6TenGEyxCQ0jVn8ETdkXhBilyNpA
bash-5.1$
4

The certificate is updated by the update-ca-trust command in the init container for the web pod.
In other words, just running kubectl apply won’t reflect the changes. Please try restarting the web pod.

5

Thank you very much @kurokobo

Issue got fixed and CA certificate has mounted inside pods. :slight_smile: :slight_smile: