I am running a very very simple test with the ansible.builtin.uri and it is failing.
This is in a development setup and the terraform infra and the apps are all made to be up and running before this simple test.
I verify that the apps are running on EC2 with a browser hit to each of the ip addresses (I know is its redirecting because url changes in browser)
There are 2 EC2 nodes up each running an ansible installed Granfana and prometheus app on a terraform infra.
I also verified from my IDE (Cloud9 instance) that a wget on each of the Urls is returning the 302.
The access rules on the EC2 nodes are pretty much open
When I run the ansible playbook I am seeing these errors (I replaced public IPs with XXXX and YYYY)
I am running the ansible playbook from the same Cloud9 IDE where I tested the wget that works with a returned 302.
The url looks to be correctly formed. The correct ip addresses are pulled from inventory
My simple playbook that I wanted to append to the main playbook is below.
I am not getting any syntax errors.
I see a mix of terraform, “ip addresses”, browsers, EC2 nodes, “apps”, Granfana, Granana, prometheus, Cloud9, “pretty much open” access rules, and a little Jenkins sprinkled on top as well.
From your story, I get that you are to be able to access publicly accessible URLs from the internet.
And the ansible playbook fails. The uri tasks appear to be running on the host that runs the service.
So you cannot access the URLs from the server itself, a timeout means that there is something blocking the HTTP request and/or response.
That can be caused by any of the mentioned infrastructure bits…
Essentially, the uri module just reports that it can’t reach a URL from the host that you’re targeting, which isn’t an ansible problem, but rather a network/filtering/etc problem.
The rule that worked is source ip= public ip address of the service node (looking at the tcpdump you can understand why; pubic ip is the 18 address)
The EC2 node has one virtual interface and that is the private ip address so the source = private ip address is not needed.
Only the source ip=public ip address is needed.
Edit is below in red:
"Yet with the ansible traffic (being generated from my Cloud9 IDE where Jenkins is installed) in the tcpdump I see the traffic from ansible.builtin.uri coming in to the private ip address and I assume the ubuntu internally loopbacks that to the public ip address (?). I don’t really know, but I do know that this access list resolved the issue of ansible.builtin.uri being sent to the public ip address of the service nodes:
Source = public ip address of the service node
Allow
" The uri tasks appear to be running on the host that runs the service."
The uri tasks are running from Jenkins which is co- located on the Cloud9 IDE EC2 instance.
There are 2 separate EC2 service nodes (separate from the Cloud9 IDE EC2 instance) that are part of the terraform/ansible jenkins deployment.
So there is a total of 3 EC2 instances with Jenkins running from Cloud9 IDE EC2 instance.
This is with the edit (in red below):
The wget from the terminal on Cloud9 to the public ip of the service nodes works fine and the traffic from PC to public ip service nodes works fine.
There are access lists on amazon security group for the service nodes for both of these traffic flows and they work great.
I would have assumed that ansible running on the Jenkins Cloud9 IDE would also work as well (ansible is sending traffic to the public ip as well based upon url: http://{{ inventory_hostname }} and the inventory_hostname list of ip addresses are the public ip addresses and the url looks to be constructed properly in the Jenkins logs )
I ended up running a tcpdump on the EC2 nodes and saw something that was very odd and added the relevant access list to the security group on amazon for the nodes and it is working now. I had originally opened up everything as a test with the ansible uri traffic, and that worked, so I knew that it was an access list issue.
But why with just the ansible.builtin.uri? the access lists that I had to create based upon the tcpdump do not really make sense to me at all and it is only necessary with the ansible generated request.
As stated in original post a wget (also tried curl as well) from the Cloud9 IDE works fine and hitting the url from the browser of the PC works fine. Perhaps there is a default setting in the ansible.builtin.uri that is causing this (something to do with proxy address embedded header?)
A tcpdump on the EC2 service node for PC to EC2 service node traffic shows the original ip address of my PC as source and the destination as the public ip address of the node. (this makes perfect sense).
I had an access list rule to allow this originally so this explains why this worked fine.
Yet with the ansible traffic (being generated from my Cloud9 IDE where Jenkins is installed) in the tcpdump I see the traffic from ansible.builtin.uri coming in to the private ip address and I assume the ubuntu internally loopbacks that to the public ip address (?). I don’t really know, but I do know that this access list resolved the issue of ansible.builtin.uri being sent to the public ip address of the service nodes:
Source = public ip address of the service node Allow
Why the traffic internally on amazon is being routed to the private ip address is a mystery.
I don’t have a tcpdump decoder on the ubuntu EC2 instance or Cloud9 IDE so I can’t look at the headers in the ansible request, but perhaps it is using an embedded proxy ip of the private ip address in the request and amazon internally routes the request as such when it comes in. I don’t know.
I did run curl as well from the Cloud9 IDE and that worked as expected and did not exhibit this strange behavior.
It is working now but not sure why the traffic is routed as such for ansible.builtin.uri traffic only.
The tcpdump is below
(sorry for long response but this is a strange issue)