AWX, ansbile-callback not working

Archives AWX Project
,
1

Hello,

I’am using AWX 22.7.0 on a kubernetes environment. Kubernetes is using MetalLB and Ingress. AWX is configured to use ingress in with type loadbalancer.

Everything is working fine, expect the ansible-callback. As seen many time:
/usr/bin/curl -k -s -H ‘Content-Type: application/json’ --data ‘{“host_config_key”:“#######”}’ https://awx.example.com/api/v2/job_templates/11/callback/
{“msg”:“No matching host could be found!”}

Solutions provided are the setting of: HTTP_X_FORWARDED_FOR, but this does not work for me.

Can anybody help me out?

Kind regards
Jurgen Ponds

2

can you confirm that you added “HTTP_X_FORWARDED_FOR” in settings > Miscellaneous System Settings > Remote Host Headers?

Can you also confirm that the value you sent for HTTP_X_FORWARDED_FOR matches a host in your inventory?

you can read more here https://docs.ansible.com/ansible-tower/latest/html/installandreference/proxy-support.html

AWX Team

3

Jurgen Ponds
ongelezen,
11:07 (nu)

aan awx-p...@googlegroups.com

Hello

Yes I can confirm the setting is done.
image.png

I can not find any relation between the hostname/ipaddress of the server i the logging of the awx-web container

[pid: 39|app: 0|req: 3/34] 10.244.0.1 () {54 vars in 783 bytes} [Tue Aug 29 09:00:23 2023] POST /api/v2/job_templates/11/callback/ => generated 42 bytes in 38915 msecs (HTTP/1.1 400) 11 headers in 394 bytes (1 switches on core 0)

2023-08-29 09:03:19,442 INFO [78848aea4228483189dcd833c35bb3fe] awx.analytics.job_lifecycle inventoryupdate-439 created

2023-08-29 09:03:43,660 WARNING [78848aea4228483189dcd833c35bb3fe] awx.api.generics status 400 received by user AnonymousUser attempting to access /api/v2/job_templates/11/callback/ from 10.244.0.1

2023-08-29 09:03:43,662 WARNING [78848aea4228483189dcd833c35bb3fe] django.request Bad Request: /api/v2/job_templates/11/callback/

2023-08-29 09:03:43,662 WARNING [78848aea4228483189dcd833c35bb3fe] django.request Bad Request: /api/v2/job_templates/11/callback/

[pid: 35|app: 0|req: 15/35] 10.244.0.1 () {54 vars in 783 bytes} [Tue Aug 29 09:03:18 2023] POST /api/v2/job_templates/11/callback/ => generated 42 bytes in 24683 msecs (HTTP/1.1 400) 11 headers in 394 bytes (1 switches on core 0)

10.244.0.1 - - [29/Aug/2023:09:03:43 +0000] “POST /api/v2/job_templates/11/callback/ HTTP/1.1” 400 42 “-” “curl/7.76.1” “10.244.0.1”

10.244.0.1 - - [29/Aug/2023:09:04:12 +0000] “GET /websocket/ HTTP/1.1” 101 1153 “-” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 OPR/102.0.0.0” “10.244.0.1”

[pid: 38|app: 0|req: 8/36] 10.244.0.1 () {74 vars in 1337 bytes} [Tue Aug 29 09:04:12 2023] OPTIONS /api/v2/settings/all/ => generated 330064 bytes in 773 msecs (HTTP/1.1 200) 14 headers in 601 bytes (1 switches on core 0)

10.244.0.1 - - [29/Aug/2023:09:04:13 +0000] “OPTIONS /api/v2/settings/all/ HTTP/1.1” 200 330064 “https://awx.jpp.nl/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 OPR/102.0.0.0” “10.244.0.1”

10.244.0.1 - - [29/Aug/2023:09:04:13 +0000] “GET /static/media/default.strings.json HTTP/1.1” 200 53 “https://awx.jpp.nl/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 OPR/102.0.0.0” “10.244.0.1”

10.244.0.1 - - [29/Aug/2023:09:04:13 +0000] “GET /static/media/default.strings.json HTTP/1.1” 200 53 “https://awx.jpp.nl/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 OPR/102.0.0.0” “10.244.0.1”

10.244.0.1 - - [29/Aug/2023:09:04:14 +0000] “GET /api/v2/settings/system/ HTTP/1.1” 200 1022 “https://awx.jpp.nl/” “Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 OPR/102.0.0.0” “10.244.0.1”

[pid: 35|app: 0|req: 16/37] 10.244.0.1 () {72 vars in 1306 bytes} [Tue Aug 29 09:04:14 2023] GET /api/v2/settings/system/ => generated 1022 bytes in 121 msecs (HTTP/1.1 200) 14 headers in 599 bytes (1 switches on core 0)

The IP Address should be 10.0.0.11, hostname lightning.jpp.nl

The callback command is:/usr/bin/curl -k -s --data “host_config_key=14061972” https://awx.jpp.nl/api/v2/job_templates/11/callback/

Kind regards

Jurgen

4

image.png
image.png434×159 6.79 KB

5

POST /api/v2/job_templates/11/callback/ => generated 42 bytes in 24683 msecs (HTTP/1.1 400) 11 headers in 394 bytes (1 switches on core 0)

So what are the headers that are being sent to this endpoint? were you able to grab those?

AWX Team

6

What I sent is the curl command.
/usr/bin/curl -k -s --data “host_config_key=14061972” https://awx.jpp.nl/api/v2/job_templates/11/callback/
This is what is listed the kickstart file generated by foreman/katello

kind regards
Jurgen

7

so that curl command is not sending HTTP_X_FORWARDED_FOR header at all? is there a way to customize what foreman/katello sends?

8

Yes, that is possible.
What do I need to add? I’m not so familiar with this stuff

kind regards
Jurgen

9

Ok, I already found out. And that works fine.

/usr/bin/curl -k -s --header “X-Forwarded-For: ” --data “host_config_key=” https://foreman/api/v2/job_templates/11/callback/

10

mmm, I’m using the same configuration somewhere else in a kubernetes cluster and there the x-forwarded-for header does not work. :frowning:
On a single node kubernetes it does work.

In both cases I’m using ingress, but in the awx-web container I don’t see any external ip addresses, only internal kubernetes IP addresses

11

Hello

I was a bit mislead, because it looked like the callback was working, but it turned out it was starting the jobtemplate for the kubernetes node in stead of the node which did the callback action.

So it still does not work with the X-FORWARDED_FOR in the curl header.