WinRM and Kerberos. WINRM CONNECTION ERROR: Bad HTTP response returned from server. Code 500

[Iwan]

A few months ago, I was able to set up an ansible environment in a Virtual environment on a Ubuntu 24.04 server.
I use it mainly to manage Windows Servers through WinRM with a kerberos authentication.
I communicate on TCP/5985 port, and everything works fine.
Lately, I tried to reproduce the same configuration. I mean setting up an ansible environment in a venv to manage Windows Servers.
But it became impossible to properly communicate with Windows Servers using WinRM and kerberos. A simple win_ping shows the error.

On the working machine:

On the latest ansible environment installed:

If I execute the same thing in very verbose mode on both machines:

In text mode, the full error message:

Citation
user@semaphoreui:~$ source ~/venv/ansible2.17_python3.12/bin/activate;ansible -m win_ping -i ~/windows_hosts SERVER -k -vvvvvv;deactivate
ansible [core 2.18.0]
config file = None
configured module search path = [‘/home/user/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/ansible
ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
executable location = /home/user/venv/ansible2.17_python3.12/bin/ansible
python version = 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] (/home/user/venv/ansible2.17_python3.12/bin/python3.12)
jinja version = 3.1.4
libyaml = True
No config file found; using defaults
SSH password:
setting up inventory plugins
Loading collection ansible.builtin from
host_list declined parsing /home/user/windows_hosts as it did not pass its verify_file() method
script declined parsing /home/user/windows_hosts as it did not pass its verify_file() method
auto declined parsing /home/user/windows_hosts as it did not pass its verify_file() method
Parsed /home/user/windows_hosts inventory source with ini plugin
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Loading collection ansible.windows from /home/user/.ansible/collections/ansible_collections/ansible/windows
Loading callback plugin minimal of type stdout, v2.0 from /home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/ansible/plugins/callback/minimal.py
Attempting to use ‘default’ callback.
Skipping callback ‘default’, as we already have a stdout callback.
Attempting to use ‘junit’ callback.
Attempting to use ‘minimal’ callback.
Skipping callback ‘minimal’, as we already have a stdout callback.
Attempting to use ‘oneline’ callback.
Skipping callback ‘oneline’, as we already have a stdout callback.
Attempting to use ‘tree’ callback.
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
redirecting (type: modules) ansible.builtin.win_ping to ansible.windows.win_ping
Using module file /home/user/.ansible/collections/ansible_collections/ansible/windows/plugins/modules/win_ping.ps1
Pipelining is enabled.
<SERVER.ENTERPRISE.COM> ESTABLISH WINRM CONNECTION FOR USER: THE-WINUSER@ENTERPRISE.COM on PORT 5985 TO SERVER.ENTERPRISE.COM
creating Kerberos CC at /tmp/tmpfeohiy61
calling kinit with subprocess for principal THE-WINUSER@ENTERPRISE.COM
kinit succeeded for principal THE-WINUSER@ENTERPRISE.COM
<SERVER.ENTERPRISE.COM> WINRM CONNECT: transport=kerberos endpoint=http://SERVER.ENTERPRISE.COM:5985/wsman
<SERVER.ENTERPRISE.COM> WINRM CONNECTION ERROR: Bad HTTP response returned from server. Code 500
Traceback (most recent call last):
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 268, in send_message
root = ET.fromstring(ex.response_text)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.12/xml/etree/ElementTree.py”, line 1336, in XML
return parser.close()
^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/ansible/plugins/connection/winrm.py”, line 483, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 193, in open_shell
res = self.send_message(xmltodict.unparse(req))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 271, in send_message
raise ex
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 263, in send_message
resp = self.transport.send_message(message)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/transport.py”, line 336, in send_message
response = self._send_message_request(session, prepared_request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/transport.py”, line 352, in _send_message_request
raise WinRMTransportError(“http”, ex.response.status_code, response_text.decode())
winrm.exceptions.WinRMTransportError: Bad HTTP response returned from server. Code 500
SERVER | UNREACHABLE! => {
“changed”: false,
“msg”: “kerberos: Bad HTTP response returned from server. Code 500”,
“unreachable”: true
}

Here is the environment details on the running machine:

Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble

pip freeze
ansible-core==2.17.3
certifi==2024.8.30
cffi==1.17.0
charset-normalizer==3.3.2
cryptography==43.0.0
decorator==5.1.1
gssapi==1.8.3
idna==3.8
Jinja2==3.1.4
jmespath==1.0.1
kerberos==1.3.1
krb5==0.6.0
MarkupSafe==2.1.5
packaging==24.1
pycparser==2.22
pykerberos==1.2.4
pyspnego==0.11.1
pywinrm==0.5.0
PyYAML==6.0.2
requests==2.32.3
requests-kerberos==0.15.0
requests_ntlm==1.3.0
resolvelib==1.0.1
urllib3==2.2.2
xmltodict==0.13.0

Collection Version
ansible.posix 1.5.4
ansible.utils 5.1.1
ansible.windows 2.4.0
chocolatey.chocolatey 1.5.1
community.crypto 2.21.1
community.dns 3.0.2
community.general 9.1.0
community.library_inventory_filtering_v1 1.0.1
community.mysql 3.10.3
community.vmware 4.5.0
community.windows 2.2.0
microsoft.ad 1.6.0
vmware.vmware 1.3.0

On the non-working machines, I’ve tried to install exactly the same pip packages’ version or the latest version. Whatever I try, I always get this:

kerberos: Bad HTTP response returned from server. Code 500

error message.

As we can see, the WinRM connection is clearly established:

ESTABLISH WINRM CONNECTION FOR USER: USER@ENTERPRISE.COM on PORT 5985 TO SERVER.ENTERPRISE.COM
creating Kerberos CC at /tmp/tmpfeohiy61

and the kerberos authentication is OK:

calling kinit with subprocess for principal USER@ENTERPRISE.COM
kinit succeeded for principal USER@ENTERPRISE.COM
<SERVER.ENTERPRISE.COM> WINRM CONNECT: transport=kerberos endpoint=http://SERVER.ENTERPRISE.COM:5985/wsman

And then:

<SERVER.ENTERPRISE.COM> WINRM CONNECTION ERROR: Bad HTTP response returned from server. Code 500
Traceback (most recent call last):
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 268, in send_message
root = ET.fromstring(ex.response_text)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib/python3.12/xml/etree/ElementTree.py”, line 1336, in XML
return parser.close()
^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/ansible/plugins/connection/winrm.py”, line 483, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 193, in open_shell
res = self.send_message(xmltodict.unparse(req))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 271, in send_message
raise ex
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/protocol.py”, line 263, in send_message
resp = self.transport.send_message(message)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/transport.py”, line 336, in send_message
response = self._send_message_request(session, prepared_request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/user/venv/ansible2.17_python3.12/lib/python3.12/site-packages/winrm/transport.py”, line 352, in _send_message_request
raise WinRMTransportError(“http”, ex.response.status_code, response_text.decode())
winrm.exceptions.WinRMTransportError: Bad HTTP response returned from server. Code 500
SERVER | UNREACHABLE! => {
“changed”: false,
“msg”: “kerberos: Bad HTTP response returned from server. Code 500”,
“unreachable”: true
}

I’ve tried many things, and I do know how to fix this very annoying issue.

I really need you help.

[motorbass (Gael)]

@Iwan

Bad HTTP response returned from server. Code 500

Hi For testing purpose, are you able to allow unencrypted connection with something like this :

Set-Item -Path WSMan:\localhost\Service\AllowUnencrypted -Value true

And then re-run your playbook to check if it behaves the same or not. (my idea is to check if your venv and collections may be impacted by this setting)

[mcen1]

Working with Kerberos in Linux is extremely frustrating and the error messages are rarely useful. I feel your pain, man.

I encountered this relatively recently and my issue was that I had one too many “kerberos”-related pip packages installed and it somehow lead to winrm connectivity not working.

My currently working requirements.txt is

ansible==8.7.0
ansible-runner==2.4.0
pywinrm==0.4.3
pywinrm[kerberos]
receptorctl

I can’t account for why it used to work for you but no longer does, but it’s worth a shot.

[Iwan]

@motorbass

AllowUnencrypted

Before changing anything:

Then:

Indeed, it seems the encryption is clearly the issue:

As I want to keep everything encrypted, I restore the encryption:

So I retry the win_ping on the server that does not support the encryption:

My questions:

• As I do not want to allow unencrypted transaction on thousands of servers, how can I fix this issue ?
• Why the server installed and configured a few months ago has no issues with encryption ?

Anyway, thank you for your help @motorbass

[motorbass (Gael)]

In addition to your test, @Iwan could you share your vars.yml or vars statement that configure ansible winRM settings please ?

[Iwan]

For testing purposes, I’m using a simple hosts file:

[motorbass (Gael)]

Ok i was 99% sure you were using 5985.
5985 is for HTTP (so unencrypted) and 5986 is for HTTPS (so encrypted communication)

Could you give a try using ansible_port=5986 ?

[Iwan]

The WinRM remains with the default configuration and listens on port 5985 only.

so:

My historic server works without any issue on port 5985 and mandatory encryption.

[mariolenz]

Why did you add the vmware tag? This doesn’t look like a VMware related issue at all, or am I wrong?

[motorbass (Gael)]

Ok, so as Microsoft said “AllowUnencrypted : Allows the client computer to request unencrypted traffic” so in our case, client requests only encrypted traffic.

So probably the error code 500 means : the windows target received the request but doesn’t process it well and/or ansible doesn’t understand it as it should.

So probably we should have a look at the Ansible venv.

I had a look to my windows vars for winrm, could you give a try using only :

ansible_connection: winrm
ansible_port: 5985
ansible_winrm_transport: kerberos

Depending on the result we’ll have a look to the ansible collection version, pip version, pip collections version.

[Iwan]

Indeed, it is clearly a mistake. It is not VMware related.

[Iwan]

As requested:

It is still not working

[Iwan]

On the running server:

ansible-core 2.17.3
certifi 2024.8.30
cffi 1.17.0
charset-normalizer 3.3.2
cryptography 43.0.0
decorator 5.1.1
gssapi 1.8.3
idna 3.8
Jinja2 3.1.4
jmespath 1.0.1
kerberos 1.3.1
krb5 0.6.0
MarkupSafe 2.1.5
packaging 24.1
pip 24.2
pycparser 2.22
pykerberos 1.2.4
pyspnego 0.11.1
pywinrm 0.5.0
PyYAML 6.0.2
requests 2.32.3
requests-kerberos 0.15.0
requests_ntlm 1.3.0
resolvelib 1.0.1
urllib3 2.2.2
xmltodict 0.13.0

On the NOT running server:

ansible-core 2.18.0
certifi 2024.8.30
cffi 1.17.1
charset-normalizer 3.4.0
cryptography 43.0.3
decorator 5.1.1
gssapi 1.9.0
idna 3.10
Jinja2 3.1.4
kerberos 1.3.1
krb5 0.7.0
MarkupSafe 3.0.2
packaging 24.2
pip 24.3.1
pycparser 2.22
pyspnego 0.11.2
pywinrm 0.5.0
PyYAML 6.0.2
requests 2.32.3
requests-kerberos 0.15.0
requests_ntlm 1.3.0
resolvelib 1.0.1
urllib3 2.2.3
xmltodict 0.14.2

Not so different.

I’ve tried to install the pip packages from a ‘pip freeze’ on the running server, and install the very same version on the newly created server.
It did not fix the issue…

My questions:

• Can it be related to the gcc version ? Some packages are built, and on the running server, they were built in July.
• Should I try to reinstall everything on an Ubuntu 22.04 so the GCC version is older ?
• Any other idea ? Suggestion ?

[jborean (Jordan Borean)]

I see you have both kerberos and pykerberos installed. Both of these packages use the same Python namespace kerberos but the kerberos package does not have the extra bits to enable message encryption. Uninstall both kerberos and pykerberos if they are present and just do pip install pywinrm[kerberos]. By mixing and matching you are disabling message encryption over HTTP causing the connection to fail.

[Iwan]

@jborean

pywinrm[kerberos]

Big big thank you !

Here’d what I did:
Before uninstalling anything

What i executed:

• pip uninstall kerberos pywinrm
• pip uninstall requests_ntlm
  and then:
• pip install pywinrm[kerberos]
  
  

And the final result:

You made my day !
Thank you again !