Kerberos Auth - the specified credentials were rejected by the server

Archives Ansible Project
1

Hi guys,

On my Centos7 ansible control server I’m trying to connect to a windows host using a domain account. I’ve configured kerberos and installed everything (pip install pywinrm[kerberos]) and I have a ticket:

[meaton@MDE-001-IOM ansible-iom]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_g4NS8oC
Default principal: ansible@IOM.DOMAIN.COM

Valid starting Expires Service principal
29/03/17 17:16:35 30/03/17 03:16:35 krbtgt/IOM.DOMAIN.COM@IOM.DOMAIN.COM
renew until 05/04/17 17:16:32

However, trying to use these details it still says invalid credentials, am I missing something?

Group Vars:
ansible_user = ansible@IOM.DOMAIN.COM

ansible_password = password
ansible_port = 5986
ansible_connection = winrm
ansible_winrm_server_cert_validation = ignore
ansible_winrm_realm = IOM.DOMAIN.COM
ansible_winrm_service = HOST (I’ve tried with and without this, following various “solutions” - none of which have worked)

I used the powershell script to configure WinRM. I can run my playbook with no issues if I use a local account rather than a domain account. DNS is configured, realms etc all configured.

cat /etc/krb5.conf

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = IOM.DOMAIN.COM
default_ccache_name = KEYRING:persistent:%{uid}

[realms]
IOM.DOMAIN.COM = {
kdc = dc-01-iom.iom.domain.com

}

[domain_realm]
.iom.domain.com = IOM.DOMAIN.COM

example.com = EXAMPLE.COM

Error from Playbook run:

TASK [Gathering Facts] ******************************************************************************************************************************************
Using module file /root/ansible/lib/ansible/modules/windows/setup.ps1
ESTABLISH WINRM CONNECTION FOR USER: ansible@IOM.DOMAIN.COM on PORT 5986 TO appt-001-iom
WINRM CONNECT: transport=ssl endpoint=https://appt-001-iom:5986/wsman
WINRM CONNECTION ERROR: the specified credentials were rejected by the server
Traceback (most recent call last):
File “/root/ansible/lib/ansible/plugins/connection/winrm.py”, line 211, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File “/usr/lib/python2.7/site-packages/winrm/protocol.py”, line 132, in open_shell
res = self.send_message(xmltodict.unparse(req))
File “/usr/lib/python2.7/site-packages/winrm/protocol.py”, line 207, in send_message
return self.transport.send_message(message)
File “/usr/lib/python2.7/site-packages/winrm/transport.py”, line 190, in send_message
raise InvalidCredentialsError(“the specified credentials were rejected by the server”)
InvalidCredentialsError: the specified credentials were rejected by the server

fatal: [appt-001-iom]: UNREACHABLE! => {
“changed”: false,
“msg”: “ssl: the specified credentials were rejected by the server”,
“unreachable”: true
}

Ansible Version:

ansible 2.4.0 (devel e084e8809e) last updated 2017/03/24 11:58:45 (GMT +100)
config file =
configured module search path = Default w/o overrides
python version = 2.7.5 (default, Nov 6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Thanks,

Michael

2

Are you able to set ansible_winrm_transport to Kerberos and see if that works out. I also believe in 2.4 there was a change made where ansible will get the Kerberos ticket for you removing the need for getting it manually beforehand. Another thing that would be good to know is the output of 'winrm get winrm/config' when running on your windows server.

3

Thanks,

That allowed me to get a bit further:

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************************
Using module file /root/ansible/lib/ansible/modules/windows/setup.ps1
<appt-001-iom.IOM.DOMAIN.COM> ESTABLISH WINRM CONNECTION FOR USER: ansible@IOM.DOMAIN.COM on PORT 5986 TO appt-001-iom.IOM.DOMAIN.COM
creating Kerberos CC at /tmp/tmppm3JWz
calling kinit for principal ansible@IOM.DOMAIN.COM
kinit succeeded for principal ansible@IOM.DOMAIN.COM
<appt-001-iom.IOM.DOMAIN.COM> WINRM CONNECT: transport=kerberos endpoint=https://appt-001-iom.IOM.DOMAIN.COM:5986/wsman
<appt-001-iom.IOM.DOMAIN.COM> WINRM CONNECTION ERROR: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))
Traceback (most recent call last):
  File "/root/ansible/lib/ansible/plugins/connection/winrm.py", line 211, in _winrm_connect
    self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
  File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell
    res = self.send_message(xmltodict.unparse(req))
  File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message
    return self.transport.send_message(message)
  File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 181, in send_message
    prepared_request = self.session.prepare_request(request)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 407, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 306, in prepare
    self.prepare_auth(auth, url)
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 543, in prepare_auth
    r = auth(self)
  File "/usr/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 308, in __call__
    auth_header = self.generate_request_header(None, host, is_preemptive=True)
  File "/usr/lib/python2.7/site-packages/requests_kerberos/kerberos_.py", line 148, in generate_request_header
    raise KerberosExchangeError("%s failed: %s" % (kerb_stage, str(error.args)))
KerberosExchangeError: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))

fatal: [appt-001-iom.IOM.DOMAIN.COM]: UNREACHABLE! => {
    "changed": false,
    "msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))",
    "unreachable": true
}
        to retry, use: --limit @/root/ansible-iom/windows.retry

As you can see the ticket request succeeds but I still get the error about the server not being found. DNS looks good - I can resolve both ways, WinRM config...

Config
    MaxEnvelopeSizekb = 500
    MaxTimeoutms = 60000
    MaxBatchItems = 32000
    MaxProviderRequests = 4294967295
    Client
        NetworkDelayms = 5000
        URLPrefix = wsman
        AllowUnencrypted = false
        Auth
            Basic = true
            Digest = true
            Kerberos = true
            Negotiate = true
            Certificate = true
            CredSSP = false
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        TrustedHosts
    Service
        RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
        MaxConcurrentOperations = 4294967295
        MaxConcurrentOperationsPerUser = 1500
        EnumerationTimeoutms = 240000
        MaxConnections = 300
        MaxPacketRetrievalTimeSeconds = 120
        AllowUnencrypted = true
        Auth
            Basic = true
            Kerberos = true
            Negotiate = true
            Certificate = false
            CredSSP = false
            CbtHardeningLevel = Relaxed
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        IPv4Filter = *
        IPv6Filter = *
        EnableCompatibilityHttpListener = false
        EnableCompatibilityHttpsListener = false
        CertificateThumbprint
        AllowRemoteAccess = true
    Winrs
        AllowRemoteShellAccess = true
        IdleTimeout = 7200000
        MaxConcurrentUsers = 2147483647
        MaxShellRunTime = 2147483647
        MaxProcessesPerShell = 2147483647
        MaxMemoryPerShellMB = 2147483647
        MaxShellsPerUser = 2147483647

Let me know if you need anything further - any ideas?

Best Regards,

Michael Eaton | DevOps Engineer

T: +44 (0) 203 4688271 | M: +44 (0) 7624 267 407
E: meaton@iforium.com
W: www.iforium.com

Twitter | Facebook | Linkedin

4

Is the target host’s HTTP SPN assigned to a user (instead of the computer account) in AD? Pywinrm isn’t currently patching the service override through to the kerb layer (see https://github.com/diyan/pywinrm/pull/144), so if you’re in that situation, you’ll have to wait for the next pywinrm release that includes that bugfix.

5

HI, try to check your search string on CentOS machine. It should be set to your domain name. And try to add short name of target server to hosts file. The 'server not found...' kerberos error is related to target server account name that passed to krb5 libraries. It should be a short name without any domain name.

6

I have struggled with this a lot. I have run into the fact that a new windows host, joined to a domain, sometimes needs to be rebooted again in order to have its domain name reflected correctly in its group policies. I have had GPresult /r say another domain name then what is displayed no the computer properties screen.

7

If I recall I’ve had problems in the past where a machine has moved from one domain to another. You can wind up with an orphaned computer account in active directory in the old domain (IIRC this affected older AD versions).
Fix was to remove the Computer account from the old domain.

That said, I am not sure now if this caused winrm problems or whether it affected something else that stopped us from working - thought I’d mention it anyway.

Jon