Ansible.builtin.wait_for issue weird behaviour with kerberos

Get Help
, ,
1

Hi !
I’m used to have many working projects to deal with Windows environment using Active Directory service account, ciffered using Hashicorp Vault. Projects are launched from a Debian 11 server to many Windows Server targets (mostly OS Server 2019-2022)

At the moment, while I wanted to migrate an Ansible project usually launched from my Debian server to AWX, I faced an error that happened on both.

My simple tests do the following :

---
# --------------

- name: test-webrequest-win
  hosts: all

  vars_files:
  - vars.yml

  tasks:
  - name: Test win_ping
    ansible.windows.win_ping:

  - name: Test Sleep 10 sec 
    ansible.builtin.wait_for:
      timeout: 10
    delegate_to: localhost

  - name: Test check webrequest
    ansible.builtin.uri:
      url: http://{{inventory_hostname}}:6666/
      return_content: yes
    delegate_to: localhost

My vars.yml is configured like this :

---
ansible_user: "{{ vault.ansible_account }}"
ansible_password: "{{ vault.ansible_password }}"

#### Configuration Ansible WinRM
ansible_connection: winrm
ansible_port: 5985
ansible_winrm_transport: kerberos

Here’s my issue, as you can see below the win_ping (or every windows tasks actually) is working great, but the big surprise is about the ansible.builtin.wait_for module

PLAY [test-webrequest-win] ************************************************************************************************************************************************************************************
TASK [Gathering Facts] ****************************************************************************************************************************************************************************************jeudi 08 février 2024  15:55:34 +0100 (0:00:00.014)       0:00:00.014 *********
ok: [10.10.10.10]
ok: [my-server.ansible.com]

TASK [test win_ping] ******************************************************************************************************************************************************************************************jeudi 08 février 2024  15:55:38 +0100 (0:00:04.082)       0:00:04.097 *********
ok: [my-server.ansible.com]
ok: [10.10.10.10]

TASK [Test sleep 10 sec] **************************************************************************************************************************************************************************************jeudi 08 février 2024  15:55:40 +0100 (0:00:02.538)       0:00:06.635 *********
fatal: [10.10.10.10 -> localhost]: UNREACHABLE! => {"changed": false, "msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))", "unreachable": true}
fatal: [my-server.ansible.com -> localhost]: UNREACHABLE! => {"changed": false, "msg": "kerberos: authGSSClientStep() failed: (('Unspecified GSS failure.  Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))", "unreachable": true}

I test with a lot of hosts to check if something was wrong from the AD parts but everything is ok about gathering facts and test win_ping, no matter if I use FQDN (which is advised) or IP (in my case it was about testing only)

Any advices out there ? :slight_smile:

Gael

2

Hi, I didn’t make any test for this, but I believe ansible_* variables in vars.yaml overrides the parameters for localhost. This causes making Ansible to connect localhost over ansible_connection: winrm.

See Understanding variable precedence section in the docs: .

When using AWX, it is best to build the inventory properly and use inventory variables or machine credentials for connection parameters.

2 Likes
3

Hi,
Actually don’t know why I never had error on this and now I get one. But i found a solution which “fit” the Windows target.

I have to use ansible.windows.win_wait_for module so there’s no need to delegate_to my localhost.

@kurokobo ,if I can abuse your knowledges, why in AWX you say it’s best to use inventory variables ?

I mean, in my case using vars.yml allow me to get an homogeneous configuration for all of my servers/projects without configuring anything else in AWX (except create credentials once because i’m using service account hosted on Hashicorp Vault)

What should be the inconvenient in my current config ?

4

Hmm, inventory and playbooks should be managed separately to improve the reusability of each. In the first place, they are based on completely defferent life-cycle.

Not only AWX, but any product, in the long run, will be easier to use if it is designed to take advantage of native functionality as much as possible. The AWX inventory is a powerful feature.

This may not be a problem if you have a single project, but if you have multiple projects, it is easier to centralize and manage them with inventory variables rather than storing and managing connection information in vars.yml for all projects. Having connection parameters scattered all over the place in various repositories would make management difficult in the first place.

Inventory variables can be set on inventory level, group level, or host lebel. In your case, if you create an inventory called “Windows Targets” and define ansible_* as inventory-level variables just once, you can remove ansible_* from vars.yml on all project repositories. Even if for some reason you want to configure certain hosts differently, this can also be easily accomplished by defining host-level variables.

2 Likes
5

Alright, it makes sense indeed.
At the moment, our team is at the very beginning of using AWX
(even if we got many projects that runs great on different topics) and we’ll improve some way of working little by little to be honest :slight_smile:

To explain a bit more our organization, from an Ansible perspective, we used to develop a project using a vars.yml on every project.
It means if I got 5 projects for Windows, I got 5x the same vars.yml on every project. Same way of working if i talk about networking project for instance.
=> It’s working great, but not optimized.

In another hand, we have very different inventories depends on the project (and sometimes some hosts may be part of different inventories) so at the moment we create a brand new inventory for
every project we have, even if some hosts are similar. => it’s working but not optimized at all.

From an AWX perspective what do you advise ?
Should I manage all my hosts once and then activate them depends on the project I launch ?
Then create inventory level variables ?

6

One solution that can be tried immediately would be to use the “Groups” feature of the inventory to help. This is more of an Ansible feature than an AWX feature, but it seems more efficient to manage than creating an inventory for each project.

Within a single inventory, any group can be created and any host can be invited to join. A host can join more than one group. You can define any variables at inventory level, group level or host level, and variables are inherited from inventory to group to host.

Example inventory structure:

  • Windows Targets (Inventory)
    • windows_server_2022 (Group)
      • Host A
      • Host B
      • Host C
      • Host D
      • Host E
    • domain_controllers (Group)
      • Host A
      • Host B
    • iis_servers (Group)
      • Host C
      • Host D
      • Host E
    • windows_server_2019 (Group)
      • Host F
      • Host G

In your playbook, you can target any groups in hosts.

# Playbook for all hosts in "Windows Targets" inventory
- hosts: all
  tasks: ...

# Playbook for Windows Server 2022
- hosts: windows_server_2022 
  tasks: ...

# Playbook for IIS Servers
- hosts: iis_servers
  tasks: ...

If you want to further aggregate multiple inventories, you can use the recently released Contructed Inventory feature. This is an AAP blog, but it can also be used on AWX.

2 Likes
7

That sounds pretty useful ! Have to try it very soon !
The way you advise to use groups is like a workaround for AWX hosts thay can’t be set into different inventory, very smart.

(and in a second time, need to give a try to this awesome feature you mentionned)

Once again thanks very very much @kurokobo :slight_smile:

8

To learn how Groups works, I think this introduction is helpful for you: How to build your inventory — Ansible Documentation

This is the docs not for AWX but for Ansible, but the same concepts can be applied to AWX.

And here is the docs for inventory for AWX: 17. Inventories — Ansible AWX community documentation

1 Like
9

The error with the ansible.builtin.wait_for module indicates a Kerberos authentication issue, likely due to missing or misconfigured SPNs in the Kerberos database; for detailed guidance, refer to Ansible wait_for Module Examples.emphasised text

10

The misconfiguration was caused by the vars.yml file overwriting the connection plugin when delegating wait_for to localhost, thereby trying to establish a WinRM connection to localhost with the Windows Credentials intended for the remote hosts.