winrm delegate_to on AWX gives 401

Archives Ansible Project
,
1

Hi all

I try to delegate a few tasks to another server during my playbook run which is executed on a Windows server, and the delegated tasks are also executed on (another) windows server.
In AWX I have set up machine credentials, which are passed on to the playbook, and those credentials should be used on both those windows servers. And it seems like ansible indeed tries to do so, but fails…?

When I run (another) simpler playbook on both those Windows servers, it succeeds on both without problems using the same credentials set.
But when I run this playbook that delegates a few tasks to the other server, it fails the delegation with error 401: kerberos: the specified credentials were rejected by the server. While those exact same credentials are accepted when that host is the “main” host for the playbook.

This is what I see in the logging:

TASK [tsm-client-win : Check if Client is registered to TSM] *******************
task path: /tmp/awx_609_b2y0mq2m/requirements_roles/tsm-client-win/tasks/register-client.yml:3
Using module file /var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/ansible/modules/windows/win_command.ps1
Pipelining is enabled.
ESTABLISH WINRM CONNECTION FOR USER: user@DOMAIN on PORT 5985 TO winhost2
creating Kerberos CC at /tmp/tmpkcp5yccl
calling kinit with subprocess for principal user@DOMAIN
kinit succeeded for principal user@DOMAIN
WINRM CONNECT: transport=kerberos endpoint=http://winhost2:5985/wsman
WINRM CONNECTION ERROR: the specified credentials were rejected by the server
Traceback (most recent call last):
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/transport.py”, line 262, in _send_message_request
response.raise_for_status()
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/requests/models.py”, line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: for url: http://winhost2:5985/wsman

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/ansible/plugins/connection/winrm.py”, line 400, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/protocol.py”, line 157, in open_shell
res = self.send_message(xmltodict.unparse(req))
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/protocol.py”, line 234, in send_message
resp = self.transport.send_message(message)
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/transport.py”, line 243, in send_message
self.build_session()
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/transport.py”, line 232, in build_session
self.setup_encryption()
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/transport.py”, line 238, in setup_encryption
self._send_message_request(prepared_request, ‘’)
File “/var/lib/awx/custom-venv/windows/lib/python3.6/site-packages/winrm/transport.py”, line 266, in _send_message_request
raise InvalidCredentialsError(“the specified credentials were rejected by the server”)
winrm.exceptions.InvalidCredentialsError: the specified credentials were rejected by the server

Can anyone shed some light onto this ? As I don’t see why authentication fails on this host when it is addressed using delegate_to and succeeds when the host is the main target of the playbook. What am I missing?

Thanks

Robin

2

Meanwhile I’m still no step closer to any solution.
I enabled Windows Remote Management Analytics and Debug logging. But absolutely nothing is logged in there when the aforementioned delegation is performed. Also the Windows audit logging shows no trace of any failed login…

How should I proceed in debugging this?

Thanks

3

Having a invalid credentials error means WinRM returns a 401 HTTP code with means the request was unauthorized. In WinRM terms this can mean a few different things;

  • The request is not encrypted

  • Using https is always encrypted

  • Using http is only encrypted if using NTLM, Kerberos, or CredSSP (depending on the versions of pywinrm and requests-kerberos installed)

  • I see you are using Kerberos over HTTP so it should be encrypted but try setting ‘ansible_winrm_message_encryption=always’ to 100% check it doesn’t fail on you.

  • The user is not a local Administrator of the remote host

  • By default only local Administrators can connect over WinRM

  • This can be changed with ‘winrm configSDDL default’ and giving the user/group Execute rights- For non domain accounts you need to make sure LocalAccountTokenFilterPolicy is set to 1

  • If not defined or set to 0, any network connections are authenticated with a limited token so the user is not seen as Administrator (see above)

  • This does not apply to domain accounts so not actually something that is relevant for you to check

  • The authentication choice is not supported

  • By default WinRM only enables Negotiate/Kerberos (NTLM + Kerberos)

  • Shouldn’t affect you because you are using Kerberos

  • Run ‘winrm get winrm/config/service/auth’ to see what’s enabled

  • CbtHardeningLevel is Strict and the required Python packages are not up to date

  • This has been fixed in the newer requests-kerberos/requests-ntlm versions but you might be on an older one

  • If ‘winrm get winrm/config/service/auth’ ‘CbtHardeningLevel’ is Strict then make sure you have a newer version of the requests-kerberos package installed

  • Probably something else I’m not thinking off

  • Kerberos auth can be troublesome to set up, one big thing you may want to verify is whether the time is in sync on your servers

Thanks

Jordan

4

Thanks for your answer Jordan…
I checked your list and indeed i’m using kerberos over HTTP. I tried setting ‘ansible_winrm_message_encryption’ to ‘always’, but the error remains the same.
The user that is used for the delegation is exactly the same user that I use to connect to the primary machine and this user is a domain admin hence implicit local admin of both machines.

I also checked winrm/config/service/auth settings and indeed only negotiate/kerberos is enabled but ‘CbtHardeningLevel’ is set to ‘Relaxed’ so requests-kerberos/requests-ntlm shouldn’t have problems with it.

The strange thing is that when I target the machine, which I here try to delegate to, directly using another simpler playbook winrm authentication completely works as expected. So winrm then doesn’t have any authentication problems on that same machine with that same user and the same ansible settings. Only when that machine is used for delegation, suddenly winrm fails to authenticate on that machine, while authentication on the actual target machine of the playbook, with the same credentials does work…

Are there other settings/variables in ansible that make delegate_to authenticate differently?

Thanks

Robin

5

I tried setting ‘ansible_winrm_message_encryption’ to ‘always’, but the error remains the same.

That’s good, it means you have the latest versions installed and message encryption was already enabled

If it is failing with delegation maybe it’s a problem with the ansible_password var priority. Delegation should use the password set for the host in question but how it gets the password might be a bit faulty. How are you defining the connection password normally and during the delegation process?

6

The password is set by passing AWX machine credentials to the job template when the job is started within AWX. There are no additional settings of ansible_password in both the normal tasks nor the delegation tasks within the playbook. Nor is the ansible_user/_password set in the inventory for any host.

Also, I see this in the logging, both during normal tasks and during delegation:
creating Kerberos CC at /tmp/tmpkcp5yccl
calling kinit with subprocess for principal user@DOMAIN
kinit succeeded for principal user@DOMAIN

which looks to me that the password is correct, and the kerberos ticket is actually created…

However, I don’t know the actual details of how a kerberos secured connection is set up, but I thought the authorization happened by passing the the valid kerberos ticket to the host where we want to be authenticated?

Thanks
Robin

7

Part of the kerb auth process is for the winrm connection plugin to call kinit to get a temporary Kerberos ticket. This means the Ansible controller contacts a domain controller to verify the password and get a Ticket Granting Ticket TGT that is encrypted using a shared secret (the password). Once it gets the TGT it can use that to generate a shared secret session key known to that user and the remote Windows Server. Finally this session key is used to authenticate from Ansible controller to the WIndows Server which is done by adding a Kerberos token to the HTTP request sent.

Why it is ultimately failing I am honestly not sure. To really rule out any Kerberos side effects I recommend setting ‘ansible_winrm_transport: ntlm’ temporarily for your Windows host and try delegating to that. If it succeeds then something is up with Kerberos like time sync between the host. If it doesn’t work then Kerberos isn’t the problem but either something with the account your are connecting with or some other configuration.

8

Passing ‘ansible_winrm_transport: ntlm’ to the playbook does the trick. Authentication now succeeds on both the targeted machine and the delegated machine using ntlm instead of kerberos… So it seems that kerberos is failing somewhere when using delegate_to…

Could this be a bug in ansible? Targeting the delegation-machine directly makes authentication with kerberos on that same machine with the same credentials work perfectly. It only fails when I delegate to the machine… and it should fail in both cases if time or the account itself would be the problem, not ?

Thanks

Robin