Questions about notifications in AWX 24.6.1

Get Help
,
1

I have a few questions about notifications in AWX 24.6.1

  • How can I debug failed notifications? Which log do I need to check?
  • I have a playbook that runs on EE ‘MyCompanyEE’, which can connect succesfully to website https://my.server/. But if I set this URL in an AWX webhook notification, I need to disable certificate checking. I already set the global default EE in the miscellanous system settings to ‘MyCompanyEE’ but the error remains. So what component in AWX runs the notifications, and how can I insert the root CA (which is apparently absent from that component)?

The root CA in question is ‘CN = Sectigo Public Server Authentication Root R46’

2

I’m a little lost, are you saying you want to send a notification to your Execution Environment? What on the EE is going to accept it?

For example, if you want to send a JSON payload to the EE host over HTTP/S, you can set the notification type to Webhook, which would then give you the option to “Disable SSL verification”.

3

You can define notifications in AWX to run when a job finishes.

I have a notification that creates a ticket in our ticketing system when a job fails, using a webhook POST. The team responsible for the ticketing system replaced the https-certificate, and suddenly no new tickets were created.
Apparently it uses a root CA that is not present in the CA store of whatever AWX component is responsible for running notifications.

So at the moment, for a quick fix, I turned ‘Disable SSL verification’ on on the notification, but I would like to know

  • what component is responsible for these notifications?
  • How do I inject the new root CA into that component’s root CA list?
4

I believe what you may be after is bundle_cacert_secret, a parameter you can set on the AWX custom resource definition.

I believe this dictates what CAs AWX itself trusts. Since notifications don’t use an EE, I assume they run their operations directly from one of the AWX containers.

5

Do I need to get the current root CA list from somewhere and add the Sectigo one to that list, or do I create a CA bundle with only the Sectigo certificate and AWX will add it to the existing list itself?

6

Dunno, the latter makes more sense as the list of CAs in the world is huge and I imagine the most sensible design for AWX to use would be to add the CAs in the secret in addition to its pre-existing ones. Best to test first though.

7

This seems to have worked.

What I did:

  • Download the crt of the root CA and save as c:\temp\rootca.crt
  • Create a secret in the awxtest namespace: kubectl create secret generic awx-custom-certs --from-file=bundle-ca.crt=c:\temp\rootca.crt -n awxtest
  • Update the awx.yaml file by adding bundle_cacert_secret: awx-custom-certs to the spec.
  • Redeploy: kubectl apply -f awx.yaml -n awxtest
  • Restart the task and web pods: kubectl rollout restart deployment awxtest-web -n awxtest (and the same for deployment awxtest-task)

As for logging: I found out that the API (https://awx.host/api/v2/notifications/) showed the error.