Kerberos Delegation Issues

Hello,

I’m having issues getting the double hop scenario working. To test kerberos delegation I have a simple PowerShell script that does a Get-ChildItem on a UNC path. When running the command manually on the host it works, but when executing as playbook with Ansible I get “Access Denied.” Below is my configuration and the verbose output I receive. Any help or suggestions would be greatly appreciated.

Environment:

user@ansible:~/ansible> pip list 2>/dev/null | grep -i pywinrm
pywinrm (0.2.0)

user@ansible:~/ansible> ansible --version
ansible 2.1.0.0
config file = /home/user/ansible/ansible.cfg
configured module search path = Default w/o overrides

user@ansible:~/ansible> cat /etc/*-release
NAME=“SLES”
VERSION=“11.4”
VERSION_ID=“11.4”
PRETTY_NAME=“SUSE Linux Enterprise Server 11 SP4”
ID=“sles”
ANSI_COLOR=“0;32”
CPE_NAME=“cpe:/o:suse:sles:11:4”
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4

Inventory excerpt:

[all:vars]
ansible_ssh_port=5986
ansible_connection=winrm
ansible_winrm_transport=kerberos
ansible_winrm_kerberos_delegation=yes
ansible_ssh_user=ansible_svc@DOMAIN.COM
ansible_winrm_server_cert_validation=ignore

Playbook output:

user@ansible:~/ansible> ansible-playbook test.yml -i inventories/domain -vvvvv
Using /home/user/ansible/ansible.cfg as config file
Loaded callback default of type stdout, v2.0

PLAYBOOK: test.yml *************************************************************
1 plays in test.yml

PLAY [list unc] ****************************************************************

TASK [list unc] ****************************************************************
task path: /home/user/ansible/test.yml:6
<dc1.domain.com> ESTABLISH WINRM CONNECTION FOR USER: ansible_svc@domain.com on PORT 5986 TO dc1.domain.com
<dc1.domain.com> WINRM CONNECT: transport=kerberos endpoint=https://dc1.domain.com:5986/wsman
<dc1.domain.com> WINRM OPEN SHELL: 33CC652E-0DED-4C66-B898-2860580A29A8
<dc1.domain.com> EXEC Set-StrictMode -Version Latest
(New-Item -Type Directory -Path $env:temp -Name “ansible-tmp-1473809521.62-137672088908702”).FullName | Write-Host -Separator ‘’;
<dc1.domain.com> WINRM EXEC u’PowerShell’ [u’-NoProfile’, u’-NonInteractive’, u’-ExecutionPolicy’, u’Unrestricted’, u’-EncodedCommand’, u’UwBlAHQALQBTAHQAcgBpAGMAdABNAG8AZABlACAALQBWAGUAcgBzAGkAbwBuACAATABhAHQAZQBzAHQACgAoAE4AZQB3AC0ASQB0AGUAbQAgAC0AVAB5AHAAZQAgAEQAaQByAGUAYwB0AG8AcgB5ACAALQBQAGEAdABoACAAJABlAG4AdgA6AHQAZQBtAHAAIAAtAE4AYQBtAGUAIAAiAGEAbgBzAGkAYgBsAGUALQB0AG0AcAAtADEANAA3ADMAOAAwADkANQAyADEALgA2ADIALQAxADMANwA2ADcAMgAwADgAOAA5ADAAOAA3ADAAMgAiACkALgBGAHUAbABsAE4AYQBtAGUAIAB8ACAAVwByAGkAdABlAC0ASABvAHMAdAAgAC0AUwBlAHAAYQByAGEAdABvAHIAIAAnACcAOwA=‘]
<dc1.domain.com> WINRM RESULT u’<Response code 0, out “C:\Users\ansible_svc”, err “”>’
<dc1.domain.com> PUT “/home/user/ansible/test.ps1” TO “C:\Users\ansible_svc\AppData\Local\Temp\ansible-tmp-1473809521.62-137672088908702\test.ps1”
<dc1.domain.com> WINRM PUT “/home/user/ansible/test.ps1” to “C:\Users\ansible_svc\AppData\Local\Temp\ansible-tmp-1473809521.62-137672088908702\test.ps1” (offset=46 size=46)
<dc1.domain.com> EXEC & ‘C:\Users\ansible_svc\AppData\Local\Temp\ansible-tmp-1473809521.62-137672088908702\test.ps1’
<dc1.domain.com> WINRM EXEC ‘PowerShell’ [‘-NoProfile’, ‘-NonInteractive’, ‘-ExecutionPolicy’, ‘Unrestricted’, ‘-EncodedCommand’, ‘JgAgACAAJwBDADoAXABVAHMAZQByAHMAXABhAG4AcwBpAGIAbABlAF8AcwB2AGMAXABBAHAAcABEAGEAdABhAFwATABvAGMAYQBsAFwAVABlAG0AcABcAGEAbgBzAGkAYgBsAGUALQB0AG0AcAAtADEANAA3ADMAOAAwADkANQAyADEALgA2ADIALQAxADMANwA2ADcAMgAwADgAOAA5ADAAOAA3ADAAMgBcAHQAZQBzAHQALgBwAHMAMQAnAA==’]
<dc1.domain.com> WINRM RESULT u’<Response code 0, out “”, err “#< CLIXML\r\n<Objs Ver”>’
<dc1.domain.com> EXEC Set-StrictMode -Version Latest
Remove-Item “C:\Users\ansible_svc\AppData\Local\Temp\ansible-tmp-1473809521.62-137672088908702” -Force -Recurse;
<dc1.domain.com> WINRM EXEC u’PowerShell’ [u’-NoProfile’, u’-NonInteractive’, u’-ExecutionPolicy’, u’Unrestricted’, u’-EncodedCommand’, u’UwBlAHQALQBTAHQAcgBpAGMAdABNAG8AZABlACAALQBWAGUAcgBzAGkAbwBuACAATABhAHQAZQBzAHQACgBSAGUAbQBvAHYAZQAtAEkAdABlAG0AIAAiAEMAOgBcAFUAcwBlAHIAcwBcAGEAbgBzAGkAYgBsAGUAXwBzAHYAYwBcAEEAcABwAEQAYQB0AGEAXABMAG8AYwBhAGwAXABUAGUAbQBwAFwAYQBuAHMAaQBiAGwAZQAtAHQAbQBwAC0AMQA0ADcAMwA4ADAAOQA1ADIAMQAuADYAMgAtADEAMwA3ADYANwAyADAAOAA4ADkAMAA4ADcAMAAyACIAIAAtAEYAbwByAGMAZQAgAC0AUgBlAGMAdQByAHMAZQA7AA==‘]
<dc1.domain.com> WINRM RESULT u’<Response code 0, out “”, err “”>’
<dc1.domain.com> WINRM CLOSE SHELL: 33CC652E-0DED-4C66-B898-2860580A29A8
changed: [dc1.domain.com] => {“changed”: true, “invocation”: {“module_args”: {“_raw_params”: “/home/user/ansible/test.ps1”}, “module_name”: “script”}, “rc”: 0, “stderr”: “Get-ChildItem : Access is denied\r\nAt C:\Users\ansible_svc\AppData\Local\Temp\ansible-tmp-1473809521.62-1376720889\r\n08702\test.ps1:1 char:1\r\n+ Get-ChildItem "\\sccm01\SMS_ABC\Client"\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+ CategoryInfo : PermissionDenied: (\\sccm01\SMS_ABC\Client \r\n:String) [Get-ChildItem], UnauthorizedAccessException\r\n+ FullyQualifiedErrorId : ItemExistsUnauthorizedAccessError,Microsoft.Powe \r\nrShell.Commands.GetChildItemCommand\r\n\r\nGet-ChildItem : Cannot find path ‘\\sccm01\SMS_ABC\Client’ because it \r\ndoes not exist.\r\nAt C:\Users\ansible_svc\AppData\Local\Temp\ansible-tmp-1473809521.62-1376720889\r\n08702\test.ps1:1 char:1\r\n+ Get-ChildItem "\\sccm01\SMS_ABC\Client"\r\n+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+ CategoryInfo : ObjectNotFound: (\\sccm01\SMS_ABC\Client:S \r\ntring) [Get-ChildItem], ItemNotFoundException\r\n+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetCh \r\nildItemCommand\r\n”, “stdout”: “”, “stdout_lines”: }

PLAY RECAP *********************************************************************
dc1.domain.com : ok=1 changed=1 unreachable=0 failed=0

user@ansible:~/ansible>

I just got this working a couple of days ago.
The only differences I can see between your set up and mine are

I set up win connection vars in group vars, rather than host vars (mixed environment - not all my hosts are windows). Might be worth trying to switch to group_vars as at some point I think there was some difference in how host vars and group vars were loaded, although I think that has been resolved now.
I am using 2.0.0.2 / 2.1.1
my ansible controllers are Centos

So I suggest trying to configure with group_vars instead of host vars.

I tested a very similar one line powershell script to do the same as you (access files on a network share), so I’m sure this can be made to work.

Hope this helps,

Jon

Thanks for the response JH. I’ve moved the winrm connection details to group_vars as you suggested, but am still not able to list the files of a network share. You said you are using “2.0.0.2 / 2.1.1” Can you please clarify those version numbers and what they are associated with?

host file:

user@ansible:~/ansible> cat inventories/domain
[test]
dc1.domain.com

group_vars:
user@ansible:~/ansible> cat inventories/group_vars/test.yml

Sorry, I should have been clearer. 2.0.0.2 and 2.1.1 are ansible versions.

JH,

Do you know of any other tests/logging I could try/review to determine why the kerberos delegation is not working in my environment?

Have a look in the event logs. I suspect all you will see is ‘Access is denied’. Worth looking on the network share machine (if it is an actual windows box). If it isn’t a windows box I guess there will be some kind of samba share logging that you could examine too.

Make sure that you are using the same user when logged in via remote desktop as the user that ansible is using.

You could check for logon events in the event viewer and see what privileges are assigned to your ansible… user and see how these differ when you login via RDP.

My understanding is that the auth delegation changes the kerberos ticket in some some way so you could try examining the kerberos ticket using klist - unfortunately I can’t try this myself at the moment.

I wonder if it is possible for the domain controller to disallow granting the necessary kerberos ticket for auth delegation. Perhaps ask Active Directory administrators if they can do anything like this and whether it it is in place.

I still think that you are ‘almost there’ with solving this problem.

Hope the above helps,

Jon

Apologies for the delayed response… I’ve been looking for ways to work around this issue, but I hit a roadblock so I really need to figure this out. Below are the logs from the server hosting the network share. Apparently the login was successful, but it was as an anonymous user using NTLM. I’m still receiving the same Access Denied message in ansible. Any further assistance would be greatly appreciated. Thanks.

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Event ID: 4624
Task Category: Logon
Level: Information
Keywords: Audit Success
User: N/A
Computer: SCCM01.domain.com
Description:
An account was successfully logged on.

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

Impersonation Level: Impersonation

New Logon:
Security ID: ANONYMOUS LOGON
Account Name: ANONYMOUS LOGON
Account Domain: NT AUTHORITY
Logon ID: 0x614767F6
Logon GUID: {00000000-0000-0000-0000-000000000000}

Process Information:
Process ID: 0x0
Process Name: -

Network Information:
Workstation Name: DC1.domain.com
Source Network Address: x.x.x.x
Source Port: 59019

Detailed Authentication Information:
Logon Process: NtLmSsp
Authentication Package: NTLM
Transited Services: -
Package Name (NTLM only): NTLM V1
Key Length: 128

This event is generated when a logon session is created. It is generated on the computer that was accessed.

The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).

The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.

The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The impersonation level field indicates the extent to which a process in the logon session can impersonate.

The authentication information fields provide detailed information about this specific logon request.

  • Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.
  • Transited services indicate which intermediate services have participated in this logon request.
  • Package name indicates which sub-protocol was used among the NTLM protocols.
  • Key length indicates the length of the generated session key. This will be 0 if no session key was requested.
    Event Xml:



    4624
    1
    0
    12544
    0
    0x8020000000000000

    2087408


    Security
    SCCM01.domain.com



    S-1-0-0
    -
    -
    0x0
    S-1-5-7
    ANONYMOUS LOGON
    NT AUTHORITY
    0x614767f6
    3
    NtLmSsp
    NTLM
    DC1.domain.com
    {00000000-0000-0000-0000-000000000000}
    -
    NTLM V1
    128
    0x0
    -
    x.x.x.x
    59019
    %%1833

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Event ID: 5140
Task Category: File Share
Level: Information
Keywords: Audit Success
User: N/A
Computer: SCCM01.domain.com
Description:
A network share object was accessed.

Subject:
Security ID: ANONYMOUS LOGON
Account Name: ANONYMOUS LOGON
Account Domain: NT AUTHORITY
Logon ID: 0x614767F6

Network Information:
Object Type: File
Source Address: x.x.x.x
Source Port: 59019

Share Information:
Share Name: \*\IPC$
Share Path:

Access Request Information:
Access Mask: 0x1
Accesses: ReadData (or ListDirectory)

Event Xml:



5140
1
0
12808
0
0x8020000000000000

2087409


Security
SCCM01.domain.com



S-1-5-7
ANONYMOUS LOGON
NT AUTHORITY
0x614767f6
File
x.x.x.x
59019
\*\IPC$


0x1
%%4416


You mentioned you were using ansible 2.1.0 and that you’d switched to group_vars- that version has an inventory bug where any ansible_winrm_X connection vars are ignored if they live in group_vars. Either upgrade to at least 2.1.1, or move them back. Also, try doing a raw: klist on the Windows host with delegation enabled- you should see a TGT listed.

Thanks for the response Matt! I did verify we are running ansible version 2.1.1.0

user@ansible:~> ansible --version
ansible 2.1.1.0
config file = /etc/ansible/ansible.cfg
configured module search path = Default w/o overrides

I ran the klist command on the windows host (DC1) that ansible directly connects to via winrm and I do not see a cached ticket for the service account ansible is using. Your thoughts?

Don’t know what else to say- works for everyone I know that’s tried it, so I’m suspecting some sort of local configuration or installation issue that hasn’t been covered yet.

Okay. Do you know if it would be possible for paid support from Ansible to assist with troubleshooting?