Unable to connect to a specific subnet from AWX tower plus DNS resolution not working.

Archives AWX Project
,
1

Hi,
We recently had an AWX Operator deployed in Kubernetes Cluster. Please find the details below:

AWX Version installed: 21.12

AWX operator version: 1.2.0

Ansible version: 2.9.14

However, I have run into a strange problem. Hosts in specific subnets are not pingable whereas other subnets ping. I am unable to connect to my LDAP server as it resides in the subnet that’s inaccessible. Could you kindly let me know what steps I need to follow in order to overcome this issue. I am new to Kubernetes. I could see a whole lot of iptables rules created post deployment, but I am hesitant to touch them as that could break the cluster communication.

Secondly DNS name resolution doesn’t work. If I provide hostnames in Host Inventory of AWX it simply fails to connect however with ip address, it works. Once again for specific subnets only.

I am unable to figure out which parameters I need to add in order for DNS resolution to work.

This is what I see currently in my awx-web container:

#kubectl exec -it awx-5bb7bdb785-zn6pw -n awx -c awx-web – bash
bash-5.1$ cat /etc/resolv.conf
search awx.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

Looking forward to your guidance and inputs to resolve these 2 outstanding issues. I need to get the testing done before I demo this to my management.

Rgds,

Kunal

2

Can someone please guide here!!

Rgds,
Kunal

3

Hi Kunal,

Sounds like the issue in question is an issue with Kubernetes networking in conflict with existing host or on-premise networks. I would suggest reaching out to a forum specific to Kubernetes (Slack, mailing list, etc) to see if that is case or something else.

  • Rilindo
4

Hi Kunal,
this looks very like a problem that I have, can you please answer the following.

Are you able to connect to those subnets from the container’s host?
If you are not then the problem is not related to AWX/Kubernetes.
If you can, can you connect using the LDAP server’s IP address?

Regards,
Michael.

5

Hi Michael,
Thanks for reaching out firstly!! Please find my replies in line:

Are you able to connect to those subnets from the container’s host?
There is no ping command on the awx-web container firstly. When I login to the container I am logged in as an awx user. I cannot seem to sudo root as it prompts for the password and I don’t know what that password is.
I ran a curl https://www.google.com and it connects
I ran a curl https://github.com and it connects

From the container host, I am able to ping all subnets just fine. However from AWX Web UI when I run a run a ping playbook to the host ip’s they fail

If you are not then the problem is not related to AWX/Kubernetes.

I am unsure where the problem lies as out of 10 subnets, only 2 subnets are accessible. Rest 8 subnets are inaccessible from AWX web UI

If you can, can you connect using the LDAP server’s IP address?
From the container, I ran the following command: echo | openssl s_client -connect <LDAP_IP_ADDRESS>:636 and it shows me the certificate contents

Since name resolution is not working from AWX Web UI, I tried filling in LDAP settings details with ip address. However from container logs it shows cannot to LDAP server

6

Hi Kunal,
is it the case that the fqdns AWX cannot resolve, are in a domain that cannot be resolved from outside your company?

7

Hi Michael,
The fqdns are in the company domain. The AWX controller resides on a Kuberenetes cluster which is in the company n/w and domain. So it’s all internal. From the host, all subnets are accessible however Kubernetes setup has created so many firewall rules that I myself cant figure out what is blocking and what its not. There is no firewalld or selinux running on the host machine. So I am unsure where the problem lies here.

Is there a core DNS or N/W Config file that Kubernetes creates in yaml form or some form that can be checked? There is nothing under AWX Operator yaml at least.
The service_type selected is nodeport.

Rgds,
Kunal

8

Hi Kunal,
go to https://www.dnsqueries.com/en/dns_lookup.php and see if you can resolve any of the fqdns that are failing.

9

Thanks for the update Michael. Will check!!

Rgds,
Kunal