Event Driven Ansible x Alertmanager. Any ideas?

Get Help
, , ,
1

Hello guys.

Problem

I have been trying to use EDA rulebooks as a medium between Alertmanager alerts and Ansible playbooks within my Kubernetes cluster.
I am using Ansible from my local machine, and I am using K8S Dynamic Inventory to access my Kubernetes cluster hosted on AWS.
I have tried a multitude of things, but due to the lack of documentation, online discussions/examples, and my limited knowledge, I was not able to make it work.

Code

  • This is my test-rulebook.yaml :
---
- name: Automatic Remediation of a webserver
hosts: localhost
sources:
- name: listen for alerts
ansible.eda.alertmanager:
# host: '123456.us-east-1.elb.amazonaws.com' #Tried external IP of service (using load balancer)
# host: 'prometheus-stack-kube-prom-alertmanager.monitoring.svc.cluster.local' #Tried internal IP of service
host: 0.0.0.0
port: 9093
rules:
- name: server down
condition: event.alert.labels.job == "node-exporter" and event.alert.status == "firing"
action:
run_playbook:
name: test-playbook.yaml # No need to share this for the moment since I am not reaching this step yet
  • This is my inventory.k8s.yaml file:
plugin: kubernetes.core.k8s
connections:
- namespaces:
- monitoring # The namespace that has Alertmanager deployed within it
  • This is my ansible.cfg file:
[defaults]
deprecation_warnings = False
[inventory]
enable_plugins = kubernetes.core.k8s

Attached Screenshots

  • Screenshot 1: When host=ExternalIP → Error.
  • Screenshot 2: When host=0.0.0.0 → Listening (but to nothing).
  • Screenshot 3: The Alertmanager service that is working within my cluster.

Versions

  • Ansible: 2.16.2
  • Kubernetes: Client (1.25.9) | Server (1.27.7)
  • Python: 3.10.13

Extra Notes

(I can provide extra screenshots for the below notes, if needed).

  • I pinged the External IP of the service using curl -v, and it works completely fine.
  • Alertmanager service has been working fine, and I can see the alerts on my Grafana Alerts section.
  • The alert I specified in the condition has been firing all the time of the tests.
  • I was able to run an Ansible playbook (independently from EDA), and it worked completely fine.
2

Hi Mohamed, i’m also new to EDA and i’m not trying to be rude, but if i understood based on what you said i think you only need to keep EDA running by running the CLI as explained here Usage — Ansible Rulebook Documentation Did you try this ? And here on example on how to use EDA https://youtu.be/Bb51DftLbPE