Problem with WinRM Connections

I am having a problem running WinRM connections with both basic and kerberos auth.

My Ansible is deployed with RedHat AAP 4.2.0 on RHEL 9.

I setup a test Windows 2019 machine and ran the ConfigureRemotingForAnsible.ps1 script against the host. Rebooted the host for good measure. Tested from the Ansible server that I can telnet to 5985 and 5986 (confirmed) but I cannot run a Windows test playbook against the host.

Skipping callback ‘oneline’, as we already have a stdout callback.

18

19
PLAYBOOK: test.yml *************************************************************

20
2 plays in windows/test.yml

21

22
PLAY [test raw module] *********************************************************18:47:21

23

24
TASK [Gathering Facts] *********************************************************18:47:21

25
task path: /runner/project/windows/test.yml:2

26
[WARNING]: The “winrm” connection plugin has an improperly configured remote

27
target value, forcing “inventory_hostname” templated value instead of the

28
string

29
redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup

30
Using module file /usr/share/ansible/collections/ansible_collections/ansible/windows/plugins/modules/setup.ps1

31
Pipelining is enabled.

32
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: administrator on PORT 5986 TO inventory_hostname

33
fatal: [192.168.12.52]: UNREACHABLE! => {

34
“changed”: false,

35
“msg”: “ssl: HTTPSConnectionPool(host=‘inventory_hostname’, port=5986): Max retries exceeded with url: /wsman (Caused by NewConnectionError(‘<urllib3.connection.HTTPSConnection object at 0x7fa35f010f10>: Failed to establish a new connection: [Errno -2] Name or service not known’))”,

36
“unreachable”: true

37
}

Running a Windows test against a domain joined machine produces a different error that I also cannot resolve.

PLAY [Ping] ********************************************************************18:53:19

3

4
TASK [Gathering Facts] *********************************************************18:53:19

5
[WARNING]: The “winrm” connection plugin has an improperly configured remote

6
target value, forcing “inventory_hostname” templated value instead of the

7
string

8
fatal: [srvrds04]: UNREACHABLE! => {“changed”: false, “msg”: “kerberos: authGSSClientStep() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (‘Server not found in Kerberos database’, -1765328377))”, “unreachable”: true}

I am getting this error despite me confirming the SPN is fine. Confirming I can connect to the host with WinRM from a different domain joined host. Confirmed my Kerb ticket with kinit and list.

Here is some more troubleshooting information. Also as a side note. I am running an partner NFR self-support only version of AAP and have zero support from RedHat on this. It also doesn’t seem to matter if I run the job through command line ansible or through AAP. The error is the same.

My Group Vars

Hi Michael,

A stab in the dark winrm or Kerberos specifically depend on a fully working DNS.
Is your tower cluster properly looking up the windows domain DCs?
I had tried a kludge myself with hard coded names in Hosts files but that gave me same error you are getting.
Oddly the kinit and list cmd works fine, but the actual ansible connections trying to use winrm or kerberos transport failed.
And as soon as I had added my local or internal dns zone to the cluster dns things started working for me.

HTH

Hi Urs,

Yes, dealing with Active Directory, DNS was the first place I looked. I have eliminated it being a problem with DNS. My resolv.conf file is setup correctly. DNS is correct, and rDNS is correct. I have also eliminated other low hanging fruit such as, NTP, Firewalls, Windows Firewall, Ethernet Adapter zones.

[root@ansible ~]# ping mikes-wintest
PING mikes-wintest.domain.ca (192.168.12.52) 56(84) bytes of data.
64 bytes from Mikes-WinTest.domain.ca (192.168.12.52): icmp_seq=1 ttl=123 time=14.1 ms
^C
mikes-wintest.domain.ca ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 14.127/14.127/14.127/0.000 ms
[root@ansible ~]# nslookup mikes-wintest
Server: 192.168.8.11
Address: 192.168.8.11#53

Name: mikes-wintest.domain.ca
Address: 192.168.12.52

[root@ansible ~]# nslookup domain.ca
Server: 192.168.8.11
Address: 192.168.8.11#53

Name: domain.ca
Address: 192.168.15.202
Name: domain.ca
Address: 192.168.12.153
Name: domain.ca
Address: 192.168.12.20
Name: domain.ca
Address: 192.168.8.11
Name: domain.ca
Address: 192.168.15.201
Name: domain.ca
Address: 192.168.8.44
Name: domain.ca
Address: 192.168.8.21
Name: domain.ca
Address: 192.168.12.201
Name: domain.ca
Address: 192.168.9.150

[root@ansible ~]#

I setup the inventory from CLI with a yml file and have run the win_ping test against the hosts. One thing that stands out to me is this.

<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman

Should it not be populating the real FQDN of the machine here?

ansible [core 2.13.0]
config file = /etc/ansible/ansible.cfg
configured module search path = [‘/root/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.9.10 (main, Feb 9 2022, 00:00:00) [GCC 11.2.1 20220127 (Red Hat 11.2.1-9)]
jinja version = 3.0.3
libyaml = True
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /etc/ansible/projects/inventory/inventory.yml as it did not pass its verify_file() method
script declined parsing /etc/ansible/projects/inventory/inventory.yml as it did not pass its verify_file() method
Parsed /etc/ansible/projects/inventory/inventory.yml inventory source with ini plugin
Loading callback plugin minimal of type stdout, v2.0 from /usr/lib/python3.9/site-packages/ansible/plugins/callback/minimal.py
Attempting to use ‘default’ callback.
Skipping callback ‘default’, as we already have a stdout callback.
Attempting to use ‘junit’ callback.
Attempting to use ‘minimal’ callback.
Skipping callback ‘minimal’, as we already have a stdout callback.
Attempting to use ‘oneline’ callback.
Skipping callback ‘oneline’, as we already have a stdout callback.
Attempting to use ‘tree’ callback.
META: ran handlers
[WARNING]: The “winrm” connection plugin has an improperly configured remote target value, forcing “inventory_hostname” templated value instead of the string
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: ubermike@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmpnx950wor
calling kinit with pexpect for principal ubermike@DOMAIN.CA
[WARNING]: The “winrm” connection plugin has an improperly configured remote target value, forcing “inventory_hostname” templated value instead of the string
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: ubermike@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmpoh0zue5y
calling kinit with pexpect for principal ubermike@DOMAIN.CA
[WARNING]: The “winrm” connection plugin has an improperly configured remote target value, forcing “inventory_hostname” templated value instead of the string
kinit succeeded for principal ubermike@DOMAIN.CA
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: ubermike@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmp1d3m70sc
calling kinit with pexpect for principal ubermike@DOMAIN.CA
kinit succeeded for principal ubermike@DOMAIN.CA
<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman
kinit succeeded for principal ubermike@DOMAIN.CA
<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman
<inventory_hostname> WINRM CONNECTION ERROR: authGSSClientStep() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (‘Server not found in Kerberos database’, -1765328377))
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos_.py”, line 245, in generate_request_header
result = kerberos.authGSSClientStep(self.context[host],
kerberos.GSSError: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (‘Server not found in Kerberos database’, -1765328377))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3.9/site-packages/ansible/plugins/connection/winrm.py”, line 448, in winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File “/usr/local/lib/python3.9/site-packages/winrm/protocol.py”, line 166, in open_shell
res = self.send_message(xmltodict.unparse(req))
File “/usr/local/lib/python3.9/site-packages/winrm/protocol.py”, line 243, in send_message
resp = self.transport.send_message(message)
File “/usr/local/lib/python3.9/site-packages/winrm/transport.py”, line 320, in send_message
prepared_request = self.session.prepare_request(request)
File “/usr/lib/python3.9/site-packages/requests/sessions.py”, line 456, in prepare_request
p.prepare(
File “/usr/lib/python3.9/site-packages/requests/models.py”, line 320, in prepare
self.prepare_auth(auth, url)
File “/usr/lib/python3.9/site-packages/requests/models.py”, line 551, in prepare_auth
r = auth(self)
File "/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos
.py", line 453, in call
auth_header = self.generate_request_header(None, host, is_preemptive=True)
File “/usr/local/lib/python3.9/site-packages/winrm/vendor/requests_kerberos/kerberos_.py”, line 260, in generate_request_header
raise KerberosExchangeError(“%s failed: %s” % (kerb_stage, str(error.args)))
winrm.vendor.requests_kerberos.exceptions.KerberosExchangeError: authGSSClientStep() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (‘Server not found in Kerberos database’, -1765328377))
Mikes-WinTest.domain.ca | UNREACHABLE! => {
“changed”: false,
“msg”: “kerberos: authGSSClientStep() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (‘Server not found in Kerberos database’, -1765328377))”,
“unreachable”: true
}

Hi Michael,

OK, I should have said so earlier but as well as kinit and klist ping also worked, but ansible still would not connect … and gave me exactly the error message you are getting, I am not sure but I suspect the python kerberos transport is making another over the wire query off of the DC or another container is involved to open the kerberos winrm session.

I have messed with the resolvers.conf myself and cheated by adding my local ones but that alone did not resolve it. I think there is another container that tower uses .

I have posted today on this group what I did with getting the local dns info injected into the awx cluster. I am using minikube but maybe my way works on yours too?

Awx or tower uses a number of containers …

HTH

Urs

I could add the machines to my hosts file directly but I think I actually have a bug here. The winRM plugin is actually trying to connect to https://inventory_hostname:5986/wsman instead of https://mikes-wintest.domain.ca:5986/wsman.

WARNING]: The “winrm” connection plugin has an improperly configured remote target value, forcing “inventory_hostname” templated value instead of the string
kinit succeeded for principal uber…@DOMAIN.CA
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /root/.ansible/collections/ansible_collections/ansible/windows
Using module file /root/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<inventory_hostname> ESTABLISH WINRM CONNECTION FOR USER: uber…@DOMAIN.CA on PORT 5986 TO inventory_hostname
creating Kerberos CC at /tmp/tmp1d3m70sc
calling kinit with pexpect for principal uber…@DOMAIN.CA
kinit succeeded for principal uber…@DOMAIN.CA
<inventory_hostname> WINRM CONNECT: transport=kerberos endpoint=https://inventory_hostname:5986/wsman
kinit succeeded for principal uber…@DOMAIN.CA

Have a look at the warning Ansible is giving you, it is telling you it was unable to use your configured remote target value and forced inventory_hostname. Solve that and you will probably solve this problem.

That bug was introduced in 2.13.0 and recently resolved: https://github.com/ansible/ansible/pull/77894

It’ll be included in 2.13.1 on June 20.

Matt,

Thank you for that. Yes, it seems my hunch that it was a bug was correct if its already confirmed. Guess I will need to wait and patch Ansible before I can run it against any windows hosts.