# Kerberos in Ansible Tower, pulling my hair out. **URL:** \ **Category:** Ansible Project\ **Tags:** windows, awx\ **Created:** [April 1, 2017, 2:58pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121 "2017-04-01T14:58:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![William\_McKenzie](https://avatars.discourse-cdn.com/v4/letter/w/f14d63/32.png) [@William\_McKenzie](https://forum.ansible.com/u/William_McKenzie)\ **Post date:** [April 1, 2017, 2:58pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/1 "2017-04-01T14:58:51Z")
Here’s my setup: Ansible Tower 3.1.1, Basic License, using the vagrant box, with some post provisioning steps to setup krb5 and join the box to my domain. Kerberos configuration is good. I can kinit, klist, etc. etc. from command line. python winrm is good: ` vagrant@ansible-tower ~]$ sudo su - awx Last login: Sat Apr 1 23:12:18 JST 2017 on pts/1 **Welcome to Ansible Tower!** Log into the web interface here: etc… -bash-4.2$ cat test.py import sys from winrm.protocol import Protocol HYPERV\_SERVER = ‘[https://louis.home.cartewright.com:5986/wsman](https://louis.home.cartewright.com:5986/wsman)’ class RM(): def **init** (self): self.win\_connect = Protocol(endpoint=HYPERV\_SERVER, transport=‘kerberos’, server\_cert\_validation=‘ignore’) def test(self): shell\_id = self.win\_connect.open\_shell() cmd = “dir” command\_id = self.win\_connect.run\_command(shell\_id, cmd) output,error\_value,exit\_status = self.win\_connect.get\_command\_output(shell\_id, command\_id) self.win\_connect.cleanup\_command(shell\_id, command\_id) self.win\_connect.close\_shell(shell\_id) print output def main(): rm = RM() rm.test() if **name** == ‘ **main** ’: main() sys.exit() -bash-4.2$ python test.py Volume in drive C is SAMSUNG 512GB SSD Volume Serial Number is 2C8F-7BFA Directory of C:\Users\ansible 03/31/2017 11:04 AM . 03/31/2017 11:04 AM … 07/16/2016 06:47 AM Desktop 03/31/2017 11:04 AM Documents 07/16/2016 06:47 AM Downloads 07/16/2016 06:47 AM Favorites 07/16/2016 06:47 AM Links 07/16/2016 06:47 AM Music 07/16/2016 06:47 AM Pictures 07/16/2016 06:47 AM Saved Games 07/16/2016 06:47 AM Videos 0 File(s) 0 bytes 11 Dir(s) 291,787,771,904 bytes free -bash-4.2$ ` So now, I manually create some inventory in the default directories/files for ansible (not Tower). Works perfectly: -bash-4.2$ cat /etc/ansible/group\_vars/windows.yml ansible\_connection: winrm ansible\_user: [ansible@HOME.CARTEWRIGHT.COM](mailto:ansible@HOME.CARTEWRIGHT.COM) ansible\_password: R1pflash ansible\_winrm\_server\_cert\_validation: ignore -bash-4.2$ -bash-4.2$ ansible windows -m win\_ping -v Using /etc/ansible/ansible.cfg as config file [louis.home.cartewright.com](http://louis.home.cartewright.com) | SUCCESS =\> { “changed”: false, “ping”: “pong” } Now, I create the exact same inventory in tower, exact same credentials ([ansible@HOME.CARTEWRIGHT.COM](mailto:ansible@HOME.CARTEWRIGHT.COM)) and now matter how I tweak it, always the same thing: Using /etc/ansible/ansible.cfg as config file SSH password: Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/windows/win\_ping.ps1 \<[louis.home.cartewright.com](http://louis.home.cartewright.com)\> ESTABLISH WINRM CONNECTION FOR USER: [ansible@HOME.CARTEWRIGHT.COM](mailto:ansible@HOME.CARTEWRIGHT.COM) on PORT 5986 TO [louis.home.cartewright.com](http://louis.home.cartewright.com) \<[louis.home.cartewright.com](http://louis.home.cartewright.com)\> WINRM CONNECT: transport=kerberos endpoint=[https://louis.home.cartewright.com:5986/wsman](https://louis.home.cartewright.com:5986/wsman) \<[louis.home.cartewright.com](http://louis.home.cartewright.com)\> WINRM CONNECTION ERROR: authGSSClientInit() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (“Can’t find client principal [ansible@HOME.CARTEWRIGHT.COM](mailto:ansible@HOME.CARTEWRIGHT.COM) in cache collection”, -1765328243)) Traceback (most recent call last): File “/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py”, line 154, in _winrm\_connect self.shell\_id = protocol.open\_shell(codepage=65001) # UTF-8 File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 132, in open\_shell res = self.send\_message(xmltodict.unparse(req)) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 207, in send\_message return self.transport.send\_message(message) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py”, line 181, in send\_message prepared\_request = self.session.prepare\_request(request) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/sessions.py”, line 394, in prepare\_request hooks=merge\_hooks(request.hooks, self.hooks), File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/models.py”, line 298, in prepare self.prepare\_auth(auth, url) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/models.py”, line 500, in prepare\_auth r = auth(self) File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests\_kerberos/kerberos_.py", line 308, in **call** auth\_header = self.generate\_request\_header(None, host, is\_preemptive=True) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests\_kerberos/kerberos\_.py”, line 148, in generate\_request\_header raise KerberosExchangeError(“%s failed: %s” % (kerb\_stage, str(error.args))) KerberosExchangeError: authGSSClientInit() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), (“Can’t find client principal [ansible@HOME.CARTEWRIGHT.COM](mailto:ansible@HOME.CARTEWRIGHT.COM) in cache collection”, -1765328243)) \<[louis.home.cartewright.com](http://louis.home.cartewright.com)\> WINRM CONNECT: transport=ssl endpoint=[https://louis.home.cartewright.com:5986/wsman](https://louis.home.cartewright.com:5986/wsman) \<[louis.home.cartewright.com](http://louis.home.cartewright.com)\> WINRM CONNECTION ERROR: the specified credentials were rejected by the server Traceback (most recent call last): File “/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py”, line 154, in \_winrm\_connect self.shell\_id = protocol.open\_shell(codepage=65001) # UTF-8 File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 132, in open\_shell res = self.send\_message(xmltodict.unparse(req)) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py”, line 207, in send\_message return self.transport.send\_message(message) File “/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py”, line 190, in send\_message raise InvalidCredentialsError(“the specified credentials were rejected by the server”) InvalidCredentialsError: the specified credentials were rejected by the server [louis.home.cartewright.com](http://louis.home.cartewright.com) | UNREACHABLE! =\> { “changed”: false, “msg”: “kerberos: authGSSClientInit() failed: ((‘Unspecified GSS failure. Minor code may provide more information’, 851968), ("Can’t find client principal [ansible@HOME.CARTEWRIGHT.COM](mailto:ansible@HOME.CARTEWRIGHT.COM) in cache collection", -1765328243)), ssl: the specified credentials were rejected by the server”, “unreachable”: true } Just completely unable to find the credentials in the cache, no matter how I do it. I’ve put the password in the tower credentials, in the inventory, using ASK, no matter what I do, it cannot find credentials. What am I missing? ~Bill ---
**Author:** ![Matt\_Davis](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@Matt\_Davis](https://forum.ansible.com/u/Matt_Davis)\ **Post date:** [April 3, 2017, 9:37pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/2 "2017-04-03T21:37:11Z")
Ansible doesn’t manage the tickets for you until Ansible Core 2.3 (still in release candidate). Anything earlier, you’ll have to do the kinit on the controller yourself (either via a cron job or as part of your playbook with a local action). ---
**Author:** ![William\_McKenzie](https://avatars.discourse-cdn.com/v4/letter/w/f14d63/32.png) [@William\_McKenzie](https://forum.ansible.com/u/William_McKenzie)\ **Post date:** [April 4, 2017, 1:09pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/3 "2017-04-04T13:09:27Z")
I think that’s what I’m doing. I’ve tried doing the kinit from the console, doing the kinit in a cron job, doing the kinit manually in a playbook before running the winrm play book, and doing it as a local\_action in the winrm playbook itself. In all cases (except the last one), the kinit succeeds; I can use klist to see the tickets (logged in as awx user). I can see the credential cache with the correct owner and attributes in the /tmp directory. I’ve also tried using the KEYRING instead of the FILE cache. For whatever reason, the winrm job is unable to see the credentials in the cache. When I run this playbook, it fails: ` ---
**Author:** ![cupcake](https://avatars.discourse-cdn.com/v4/letter/c/d6d6ee/32.png) [@cupcake](https://forum.ansible.com/u/cupcake)\ **Post date:** [April 4, 2017, 7:20pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/4 "2017-04-04T19:20:29Z")
sanity check; is time in sync? windows AD/kerb wont auth if the skew is more than 5 or 10 minutes off. I also saw some weirdness like this recently and a reboot and then kinit again made it work but i think due to another config reason on my part. ---
**Author:** ![William\_McKenzie](https://avatars.discourse-cdn.com/v4/letter/w/f14d63/32.png) [@William\_McKenzie](https://forum.ansible.com/u/William_McKenzie)\ **Post date:** [April 4, 2017, 7:48pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/5 "2017-04-04T19:48:19Z")
time is definitely good. i run ntpdate in my vagrant provisioning script just to be sure. ---
**Author:** ![Matt\_Davis](https://avatars.discourse-cdn.com/v4/letter/m/ce7236/32.png) [@Matt\_Davis](https://forum.ansible.com/u/Matt_Davis)\ **Post date:** [April 4, 2017, 8:59pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/6 "2017-04-04T20:59:00Z")
Tower uses an isolation tech called proot that will often break shared ticket caches. If you can’t wait for Ansible 2.3 (should be released within the next couple weeks), I’d suggest disabling proot (IIRC it’s in settings.py, but my Tower-fu is getting rusty). ---
**Author:** ![William\_McKenzie](https://avatars.discourse-cdn.com/v4/letter/w/f14d63/32.png) [@William\_McKenzie](https://forum.ansible.com/u/William_McKenzie)\ **Post date:** [April 4, 2017, 11:21pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/7 "2017-04-04T23:21:25Z")
Bingo. I was suspecting some kind of isolation thing, because it was so clear that the environment was different when running a playbook than at the command line, even when sudo’d as awx. Many thanks. I’ll never get that week back, but I love a mystery solved. ~Bill ---
**Author:** ![Jarryd\_Took](https://avatars.discourse-cdn.com/v4/letter/j/ce73a5/32.png) [@Jarryd\_Took](https://forum.ansible.com/u/Jarryd_Took)\ **Post date:** [April 4, 2017, 11:30pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/8 "2017-04-04T23:30:00Z")
@william: The latest requests-ntlm in combination of Ansible 2.2.0 you can use NTLM against Windows machines. If this floats your boat as an alternative to kerberos tickets. ansible\_connection: winrm ansible\_winrm\_transport: ntlm ---
**Author:** ![William\_McKenzie](https://avatars.discourse-cdn.com/v4/letter/w/f14d63/32.png) [@William\_McKenzie](https://forum.ansible.com/u/William_McKenzie)\ **Post date:** [April 5, 2017, 12:30pm UTC](https://forum.ansible.com/t/kerberos-in-ansible-tower-pulling-my-hair-out/24121/9 "2017-04-05T12:30:16Z")
Thanks, I have tried NTLM and basic, and they both work fine. However, I am evaluating this for a 20000+ node enterprise solution, and Kerberos is a must-have requirement.