Kerberos in Ansible Tower, pulling my hair out.

Archives Ansible Project
,
1

Here’s my setup:

Ansible Tower 3.1.1, Basic License, using the vagrant box, with some post provisioning steps to setup krb5 and join the box to my domain.

Kerberos configuration is good. I can kinit, klist, etc. etc. from command line.

python winrm is good:

`

vagrant@ansible-tower ~]$ sudo su - awx

Last login: Sat Apr 1 23:12:18 JST 2017 on pts/1

Welcome to Ansible Tower!

Log into the web interface here: etc…

-bash-4.2$ cat test.py

import sys

from winrm.protocol import Protocol

HYPERV_SERVER = ‘https://louis.home.cartewright.com:5986/wsman’

class RM():

def init(self):

self.win_connect = Protocol(endpoint=HYPERV_SERVER, transport=‘kerberos’, server_cert_validation=‘ignore’)

def test(self):

shell_id = self.win_connect.open_shell()

cmd = “dir”

command_id = self.win_connect.run_command(shell_id, cmd)

output,error_value,exit_status = self.win_connect.get_command_output(shell_id, command_id)

self.win_connect.cleanup_command(shell_id, command_id)

self.win_connect.close_shell(shell_id)

print output

def main():

rm = RM()

rm.test()

if name == ‘main’:

main()

sys.exit()

-bash-4.2$ python test.py

Volume in drive C is SAMSUNG 512GB SSD

Volume Serial Number is 2C8F-7BFA

Directory of C:\Users\ansible

03/31/2017 11:04 AM .

03/31/2017 11:04 AM …

07/16/2016 06:47 AM Desktop

03/31/2017 11:04 AM Documents

07/16/2016 06:47 AM Downloads

07/16/2016 06:47 AM Favorites

07/16/2016 06:47 AM Links

07/16/2016 06:47 AM Music

07/16/2016 06:47 AM Pictures

07/16/2016 06:47 AM Saved Games

07/16/2016 06:47 AM Videos

0 File(s) 0 bytes

11 Dir(s) 291,787,771,904 bytes free

-bash-4.2$

`

So now, I manually create some inventory in the default directories/files for ansible (not Tower). Works perfectly:

-bash-4.2$ cat /etc/ansible/group_vars/windows.yml

ansible_connection: winrm

ansible_user: ansible@HOME.CARTEWRIGHT.COM

ansible_password: R1pflash

ansible_winrm_server_cert_validation: ignore

-bash-4.2$

-bash-4.2$ ansible windows -m win_ping -v

Using /etc/ansible/ansible.cfg as config file

louis.home.cartewright.com | SUCCESS => {

“changed”: false,

“ping”: “pong”

}

Now, I create the exact same inventory in tower, exact same credentials (ansible@HOME.CARTEWRIGHT.COM) and now matter how I tweak it, always the same thing:

Using /etc/ansible/ansible.cfg as config file SSH password: Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/windows/win_ping.ps1 <louis.home.cartewright.com> ESTABLISH WINRM CONNECTION FOR USER: ansible@HOME.CARTEWRIGHT.COM on PORT 5986 TO louis.home.cartewright.com <louis.home.cartewright.com> WINRM CONNECT: transport=kerberos endpoint=https://louis.home.cartewright.com:5986/wsman <louis.home.cartewright.com> WINRM CONNECTION ERROR: authGSSClientInit() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (“Can’t find client principal ansible@HOME.CARTEWRIGHT.COM in cache collection”, -1765328243)) Traceback (most recent call last): File “/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py”, line 154, in winrm_connect self.shell_id = protocol.open_shell(codepage=65001) # UTF-8 File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 132, in open_shell res = self.send_message(xmltodict.unparse(req)) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 207, in send_message return self.transport.send_message(message) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py”, line 181, in send_message prepared_request = self.session.prepare_request(request) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/sessions.py”, line 394, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/models.py”, line 298, in prepare self.prepare_auth(auth, url) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/models.py”, line 500, in prepare_auth r = auth(self) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests_kerberos/kerberos.py", line 308, in call auth_header = self.generate_request_header(None, host, is_preemptive=True) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests_kerberos/kerberos_.py”, line 148, in generate_request_header raise KerberosExchangeError(“%s failed: %s” % (kerb_stage, str(error.args))) KerberosExchangeError: authGSSClientInit() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (“Can’t find client principal ansible@HOME.CARTEWRIGHT.COM in cache collection”, -1765328243)) <louis.home.cartewright.com> WINRM CONNECT: transport=ssl endpoint=https://louis.home.cartewright.com:5986/wsman <louis.home.cartewright.com> WINRM CONNECTION ERROR: the specified credentials were rejected by the server Traceback (most recent call last): File “/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py”, line 154, in _winrm_connect self.shell_id = protocol.open_shell(codepage=65001) # UTF-8 File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 132, in open_shell res = self.send_message(xmltodict.unparse(req)) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 207, in send_message return self.transport.send_message(message) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py”, line 190, in send_message raise InvalidCredentialsError(“the specified credentials were rejected by the server”) InvalidCredentialsError: the specified credentials were rejected by the server louis.home.cartewright.com | UNREACHABLE! => { “changed”: false, “msg”: “kerberos: authGSSClientInit() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), ("Can’t find client principal ansible@HOME.CARTEWRIGHT.COM in cache collection", -1765328243)), ssl: the specified credentials were rejected by the server”, “unreachable”: true }

Just completely unable to find the credentials in the cache, no matter how I do it. I’ve put the password in the tower credentials, in the inventory, using ASK, no matter what I do, it cannot find credentials.

What am I missing?

~Bill

2

Ansible doesn’t manage the tickets for you until Ansible Core 2.3 (still in release candidate). Anything earlier, you’ll have to do the kinit on the controller yourself (either via a cron job or as part of your playbook with a local action).

3

I think that’s what I’m doing.

I’ve tried doing the kinit from the console, doing the kinit in a cron job, doing the kinit manually in a playbook before running the winrm play book, and doing it as a local_action in the winrm playbook itself.

In all cases (except the last one), the kinit succeeds; I can use klist to see the tickets (logged in as awx user). I can see the credential cache with the correct owner and attributes in the /tmp directory. I’ve also tried using the KEYRING instead of the FILE cache. For whatever reason, the winrm job is unable to see the credentials in the cache.

When I run this playbook, it fails:
`

4

sanity check; is time in sync? windows AD/kerb wont auth if the skew is more than 5 or 10 minutes off. I also saw some weirdness like this recently and a reboot and then kinit again made it work but i think due to another config reason on my part.

5

time is definitely good.

i run ntpdate in my vagrant provisioning script just to be sure.

6

Tower uses an isolation tech called proot that will often break shared ticket caches. If you can’t wait for Ansible 2.3 (should be released within the next couple weeks), I’d suggest disabling proot (IIRC it’s in settings.py, but my Tower-fu is getting rusty).

7

Bingo. I was suspecting some kind of isolation thing, because it was so clear that the environment was different when running a playbook than at the command line, even when sudo’d as awx. Many thanks. I’ll never get that week back, but I love a mystery solved.

~Bill

8

@william:

The latest requests-ntlm in combination of Ansible 2.2.0 you can use NTLM against Windows machines. If this floats your boat as an alternative to kerberos tickets.

ansible_connection: winrm
ansible_winrm_transport: ntlm

9

Thanks, I have tried NTLM and basic, and they both work fine. However, I am evaluating this for a 20000+ node enterprise solution, and Kerberos is a must-have requirement.