Callback provisioning no matching hosts. Already HTTP_X_FORWARDED_FOR

I’m trying to get callback provisioning working to AWX. But keep getting {“msg”:“No matching host could be found!”}. I’ve had this working before, but not in my current environment. So I’m not sure if something has changed, or there was something that was in place by default before, that isn’t where I am now. ansible_host is set to the IP address that is routable to AWX server. What are the DNS requirements, if any?

AWX Version: 19.5.0
AWX Operator: 0.15.0
Ubuntu 20.04 running AWX through K3s using this: https://github.com/kurokobo/awx-on-k3s. Ingress resource included and forwarding 80 and 443.

I can run the job from the AWX server.

Remote Host Headers:
[
“HTTP_X_FORWARDED_FOR”,
“REMOTE_ADDR”,
“REMOTE_HOST”
]

When I run the Invoke-WebRequest (powershell equivalent of curl) the AWX server starts it’s inventory that I have set to update on launch, but then doesn’t show the callback job. So I’m not sure what host it’s actually sending. Is there a log somewhere that might show this? I haven’t found one in /var/log.

ubuntu@ip-10-50-0-171:~$ kubectl -n awx get awx,all,ingress,secrets
NAME AGE
awx.awx.ansible.com/awx 5h34m

NAME READY STATUS RESTARTS AGE
pod/awx-postgres-0 1/1 Running 1 (51m ago) 5h34m
pod/awx-7d74496d7d-fvbqd 4/4 Running 4 (51m ago) 5h34m
pod/awx-operator-controller-manager-647f4c5c7d-9gfxm 2/2 Running 2 (51m ago) 5h36m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-operator-controller-manager-metrics-service ClusterIP 10.43.238.189 8443/TCP 5h36m
service/awx-postgres ClusterIP None 5432/TCP 5h34m
service/awx-service ClusterIP 10.43.39.192 80/TCP 5h34m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/awx 1/1 1 1 5h34m
deployment.apps/awx-operator-controller-manager 1/1 1 1 5h36m

NAME DESIRED CURRENT READY AGE
replicaset.apps/awx-7d74496d7d 1 1 1 5h34m
replicaset.apps/awx-operator-controller-manager-647f4c5c7d 1 1 1 5h36m

NAME READY AGE
statefulset.apps/awx-postgres 1/1 5h34m

NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/awx-ingress awx.mydomain.com 10.50.0.171 80, 443 5h34m

NAME TYPE DATA AGE
secret/default-token-ts4qg kubernetes.io/service-account-token 3 5h36m
secret/awx-operator-controller-manager-token-n9js4 kubernetes.io/service-account-token 3 5h36m
secret/awx-admin-password Opaque 1 5h34m
secret/awx-postgres-configuration Opaque 6 5h34m
secret/awx-secret-tls kubernetes.io/tls 2 5h34m
secret/awx-app-credentials Opaque 3 5h34m
secret/awx-token-rsfdt kubernetes.io/service-account-token 3 5h34m
secret/awx-secret-key Opaque 1 5h34m
secret/awx-broadcast-websocket Opaque 1 5h34m