AWX 22.0.0 External Logging Settings to ELK Stack

Hello AWX Project Team, To whom it may concern,

I would like to seek your expertise on how to setup the external logging settings of Ansible AWX.

AWX Environment:
OS: CentOS Stream 8
K3S:
Client Version: v1.26.3+k3s1
Kustomize Version: v4.5.7
Server Version: v1.26.3+k3s1
AWX-Operator: 2.0.0
Static IP: 192.168.XXX.11

ELK Enviroment:
OS: CentOS Stream 8
ELK Version: 7.17.9
Static IP: 192.168.XXX.12

I configured the ELK Stack as per usual below are my configs.

Kibana config: /etc/kibana/kibana.yml
server.host: “0.0.0.0”
server.publicBaseUrl: “http://kibana.local.com

Elasticsearch config: /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Logstash config: /etc/logstash/logstash.yml
path.data: /var/lib/logstash
path.logs: /var/log/logstash

Logstash Pipeline config: /etc/logstash/conf.d/awx_pipeline.conf
input {
http {
port => 5055
codec => “json”
}
}

filter {
mutate {
remove_field => [ “[headers]” ]
}
}

output {
elasticsearch {
hosts => [“http://localhost:9200”]
index => “awx-logs-%{+YYYY.MM.dd}”
manage_template => false
}
}

Then next, i configure AWX Settings > Logging Settings to the following values:
Logging Aggregator: 192.168.XXX.12
Logging Aggregator Port: 5055
Logging Aggregator Type: logstash
Logging Aggregator Protocol: HTTPS/HTTP
Logging Aggregator Level Threshold: INFO

With this configuration, I wasn’t able to collect any logs or output from AWX but i did a curl test if the http://192.168.XXX.12:5055 wasn’t reachable but it posted and Kibana was able to display the logs under index awx-logs-*.

curl -XPOST -H “Content-Type: application/json” http://192.168.XXX.12:5055/ -d ‘{“message”: “Hello, World!”}’
curl -XPOST -H “Content-Type: application/json” http://192.168.XXX.12:5055/ -d ‘{“message”: “Hello, World2!”}’

using this test, I was able to conclude that my ELK Stack is working.
Am i missing something on logging settings or a configuration on AWX side?
Thank you in advance for your response.

I also had the similar issue in the past.Glad that it was just not me.
Some thing is off with this log forwarding from the AWX to ELK Stack.

Thanks,
Rakesh

Hello Rakesh,

I figured it out.
Using the boilerplate and configuration of the ELK stack above. I’ve made an adjustment on AWX GUI. See below.
I configure AWX Settings > Logging Settings to the following values:
Logging Aggregator: http://192.168.XXX.12:5055
Logging Aggregator Port: null
Logging Aggregator Type: logstash
Logging Aggregator Protocol: HTTPS/HTTP
Logging Aggregator Level Threshold: INFO
Loggers Sending Data to Log Aggregator Form:
[
“job_events”
]

logstash was able to collect the data from the execution of every jobs/templates on my AWX. go try it. cheers mate!

Jerome