Win_updates module working ok on one server - fails on another

Get Help
,
1

Hello - I have two Windows 2022 servers that i am executing the same playbook on. One is working ok - the other fails. I tried to use psrp plugin for the ansible connection but it failed for all the Windows hosts so cant use this as a workaround.

Is there a way someone can help me understand where I can further pinpoint the issue with the way the module is working on one server vs the other? I saw @jborean wasnt able to reproduce the issue.

the account from AWX i am using is a domain service account with full local administrator permissions on the Windows OS:

---
- name: Install Windows updates and export results
  hosts: myservergroup
  gather_facts: true
  vars:
    vm_os: "{{ ansible_os_family }}"
    become: true
    become_method: runas

  tasks:

    - name: Search-only
      ansible.windows.win_updates:
        category_names:
        - SecurityUpdates
        state: searched
        log_path: C:\Temp\search_updates_install.txt
      register: win_updates_to_install_result
      vars: 
        ansible_winrm_server_cert_validation: ignore


//////////////////////////////////////////////////////

group_vars for myservergroup

---
ansible_connection: winrm
ansible_port: '5986'
ansible_winrm_server_cert_validation: ignore
ansible_winrm_transport: ntlm
  • win_updates, PermissionDenied, AWX
2

How did psrp fail, was is the task that failed or the connection. Can you share the full error from the task so that we can actually see what the error was. It probably won’t fix it but try removing become as it shouldn’t be needed at all.

1 Like
3 
ansible-playbook [core 2.15.8]
  config file = /runner/project/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.18 (main, Sep  7 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
Using /runner/project/ansible.cfg as config file
SSH password: 
Skipping due to inventory source not ending in "now.yaml" nor "now.yml"
ansible_collections.servicenow.itsm.plugins.inventory.now declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: Install_windows_update.yml *******************************************
1 plays in playbooks/wsus/WUA_SearchDownloadInstall/Install_windows_update.yml
[WARNING]: Found variable using reserved name: become
[WARNING]: Found variable using reserved name: become_user
[WARNING]: Found variable using reserved name: become_method

PLAY [Install Windows updates and export results] ******************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/playbooks/wsus/WUA_SearchDownloadInstall/Install_windows_update.yml:2
redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup
Using module file /runner/requirements_collections/ansible_collections/ansible/windows/plugins/modules/setup.ps1
Pipelining is enabled.
<SERVERNAME.MYDOMAIN.COM> ESTABLISH PSRP CONNECTION FOR USER: serv.awxp01@MYDOMAIN.COM ON PORT 5986 TO SERVERNAME.MYDOMAIN.COM
fatal: [SERVERNAME]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via PSRP: HTTPSConnectionPool(host='SERVERNAME.MYDOMAIN.COM', port=5986): Max retries exceeded with url: /wsman (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1129)')))",
    "unreachable": true
}

PLAY RECAP *********************************************************************
MYSERVERNAME          : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
4

Just like with WinRM if you are using a self signed certificate you need to ignore the validation ansible.builtin.psrp connection – Run tasks over Microsoft PowerShell Remoting Protocol — Ansible Community Documentation. If you don’t want to use HTTPS then you need to change the port or set the scheme/protocol to http.

1 Like
6

I tried again with the correct PSRP connection settings:

{
  "ansible_connection": "psrp",
  "ansible_port": "5986",
  "ansible_psrp_cert_validation": "ignore",
  "ansible_winrm_transport": "ntlm"
}

Still the job fails:

/usr/local/lib/python3.9/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/usr/local/lib/python3.9/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed from this module in 48.0.0.
  "class": algorithms.TripleDES,
ansible-playbook [core 2.15.12]
  config file = /runner/project/ansible.cfg
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.9.19 (main, Jul 18 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.4
  libyaml = True
Using /runner/project/ansible.cfg as config file
SSH password: 
setting up inventory plugins
Loading collection servicenow.itsm from /runner/requirements_collections/ansible_collections/servicenow/itsm
Loading collection ansible.builtin from 
Skipping due to inventory source not ending in "now.yaml" nor "now.yml"
ansible_collections.servicenow.itsm.plugins.inventory.now declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
Loading collection ansible.windows from /runner/requirements_collections/ansible_collections/ansible/windows
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Loading callback plugin awx_display of type stdout, v2.0 from /usr/local/lib/python3.9/site-packages/ansible_runner/display_callback/callback/awx_display.py
Attempting to use 'awx_display' callback.
Skipping callback 'awx_display', as we already have a stdout callback.
Attempting to use 'default' callback.
Skipping callback 'default', as we already have a stdout callback.
Attempting to use 'junit' callback.
Attempting to use 'minimal' callback.
Skipping callback 'minimal', as we already have a stdout callback.
Attempting to use 'oneline' callback.
Skipping callback 'oneline', as we already have a stdout callback.
Attempting to use 'tree' callback.

PLAYBOOK: Install_windows_update.yml *******************************************
Positional arguments: playbooks/wsus/WUA_SearchDownloadInstall/Install_windows_update.yml
verbosity: 5
remote_user: myserviceaccount@MYDOMAIN.COM
connection: smart
timeout: 10
ask_pass: True
become_method: sudo
tags: ('all',)
inventory: ('/runner/inventory/hosts',)
subset: myservername
extra_vars: ('@/runner/env/extravars',)
forks: 5
1 plays in playbooks/wsus/WUA_SearchDownloadInstall/Install_windows_update.yml
[WARNING]: Found variable using reserved name: become
[WARNING]: Found variable using reserved name: become_method

PLAY [Install Windows updates and export results] ******************************

TASK [Gathering Facts] *********************************************************
task path: /runner/project/playbooks/wsus/WUA_SearchDownloadInstall/Install_windows_update.yml:2
redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup
Using module file /runner/requirements_collections/ansible_collections/ansible/windows/plugins/modules/setup.ps1
Pipelining is enabled.
<myserver.domain.com> ESTABLISH PSRP CONNECTION FOR USER: myserviceaccount@MYDOMAIN.COM ON PORT 5986 TO myserver.domain.com
<myserver.domain.com> PSRP OPEN RUNSPACE: auth=negotiate configuration=Microsoft.PowerShell endpoint=https://myserver.domain.com:5986/wsman
PSRP: EXEC (via pipeline wrapper)
<myserver.domain.com> PSRP RC: 0
<myserver.domain.com> PSRP STDOUT: {"changed":false,"invocation":{"module_args":{"_measure_subset":false,"gather_timeout":10,"gather_subset":["all"],"fact_path":null}},"ansible_facts":{"ansible_ip_addresses":["myserveripaddress"],"ansible_windows_domain_role":"Member server","ansible_architecture2":"x86_64","ansible_win_rm_certificate_thumbprint":"8D4B8B7057E4FA8B0BFE5963FC1E9691C2942EAF","ansible_user_gecos":"","ansible_distribution_major_version":"10","ansible_pagefiletotal_mb":1280,"ansible_os_name":"Microsoft Windows Server 2022 Standard","ansible_system_description":"","ansible_machine_id":"S-1-5-21-1981642916-842144808-2934601117","ansible_product_serial":"7DB74CF8-9B9E-4532-8E83-A6D5ADA15F82","ansible_system_vendor":"Nutanix","gather_subset":["all"],"ansible_bios_version":"nutanix-ahv-2.20220304.0.2619.el7","ansible_user_id":"myserviceaccount","ansible_date_time":{"epoch_local":"1725533332.13765","epoch_int":1725554932,"date":"2024-09-05","second":"52","tz":"Mountain Standard Time","iso8601_micro":"2024-09-05T16:48:52.137653Z","iso8601_basic_short":"20240905T104852","minute":"48","day":"05","weekday":"Thursday","iso8601":"2024-09-05T16:48:52Z","tz_offset":"-06:00","iso8601_basic":"20240905T104852137653","epoch":"1725554932.13765","weekday_number":"4","hour":"10","year":"2024","month":"09","time":"10:48:52","weeknumber":"35"},"ansible_user_dir":"C:\\\\Users\\\\myserviceaccount","ansible_win_rm_certificate_expires":"2033-04-27 15:43:59","ansible_processor_vcpus":4,"ansible_user_sid":"S-1-5-21-1752131980-3886936933-2536095260-67910","ansible_owner_contact":"","ansible_distribution":"Microsoft Windows Server 2022 Standard","ansible_processor":["0","GenuineIntel","Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz","1","GenuineIntel","Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz","2","GenuineIntel","Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz","3","GenuineIntel","Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz"],"ansible_powershell_version":5,"ansible_uptime_seconds":113170,"module_setup":true,"ansible_interfaces":[{"dns_domain":null,"connection_name":"Ethernet0","default_gateway":"10.32.199.1","mtu":1500,"interface_name":"Nutanix VirtIO Ethernet Adapter","ipv6":{},"macaddress":"50:6B:8D:8A:97:53","speed":10000,"interface_index":14,"ipv4":{"address":"myserveripaddress","prefix":"24"}}],"ansible_distribution_version":"10.0.20348.0","ansible_windows_domain_member":true,"ansible_swaptotal_mb":0,"ansible_hostname":"myservername","ansible_virtualization_role":"guest","ansible_memfree_mb":4127,"ansible_processor_count":4,"ansible_os_product_type":"server","ansible_netbios_name":"UK1WPSFPXSOL101","ansible_product_name":"AHV","ansible_virtualization_type":"kvm","ansible_os_installation_type":"Server","ansible_lastboot":"2024-09-04 03:22:43Z","ansible_architecture":"64-bit","ansible_bios_date":"04/01/2014","ansible_processor_cores":1,"ansible_fqdn":"myserver.domain.com","ansible_domain":"dcsprod.dcsroot.local","ansible_memtotal_mb":8192,"ansible_kernel":"10.0.20348.0","ansible_pagefilefree_mb":1014,"ansible_nodename":"myserver.domain.com","ansible_os_family":"Windows","ansible_processor_threads_per_core":1,"ansible_owner_name":"PTS","ansible_env":{"JRE_HOME":"D:\\\\Java\\\\jre8","PROCESSOR_IDENTIFIER":"Intel64 Family 15 Model 6 Stepping 1, GenuineIntel","USERDNSDOMAIN":"MYDOMAIN.COM","CURL_HOME":"D:\\\\Utils\\\\cURL","DriverData":"C:\\\\Windows\\\\System32\\\\Drivers\\\\DriverData","LOCALAPPDATA":"C:\\\\Users\\\\myserviceaccount\\\\AppData\\\\Local","CYGWIN_HOME":"D:\\\\cygwin64","CYGWIN_NUTCH_INSTALL":"/cygdrive/d/Solr/nutch-1.0","ComSpec":"C:\\\\Windows\\\\system32\\\\cmd.exe","SystemRoot":"C:\\\\Windows","ALLUSERSPROFILE":"C:\\\\ProgramData","CommonProgramFiles(x86)":"C:\\\\Program Files (x86)\\\\Common Files","ProgramW6432":"C:\\\\Program Files","Path":"C:\\\\Program Files (x86)\\\\Common Files\\\\Oracle\\\\Java\\\\java8path;C:\\\\Program Files (x86)\\\\Common Files\\\\Oracle\\\\Java\\\\javapath;C:\\\\Windows\\\\system32;C:\\\\Windows;C:\\\\Windows\\\\System32\\\\Wbem;C:\\\\Windows\\\\System32\\\\WindowsPowerShell\\\\v1.0\\\\;C:\\\\Windows\\\\System32\\\\OpenSSH\\\\;C:\\\\Program Files\\\\ServiceNow\\\\agent-client-collector\\\\bin;C:\\\\Program Files\\\\ServiceNow\\\\agent-client-collector\\\\embedded\\\\bin;D:\\\\Java\\\\jre8;C:\\\\Users\\\\myserviceaccount\\\\AppData\\\\Local\\\\Microsoft\\\\WindowsApps","USERPROFILE":"C:\\\\Users\\\\myserviceaccount","PROCESSOR_ARCHITECTURE":"AMD64","OS":"Windows_NT","PSModulePath":"C:\\\\Users\\\\myserviceaccount\\\\Documents\\\\WindowsPowerShell\\\\Modules;C:\\\\Program Files\\\\WindowsPowerShell\\\\Modules;C:\\\\Windows\\\\system32\\\\WindowsPowerShell\\\\v1.0\\\\Modules","NUTCH_JAVA_HOME":"/cygdrive/d/Java/jre8","ProgramFiles(x86)":"C:\\\\Program Files (x86)","SystemDrive":"C:","CommonProgramFiles":"C:\\\\Program Files\\\\Common Files","ProgramFiles":"C:\\\\Program Files","USERDOMAIN":"DCSUTIL","USERNAME":"myserviceaccount","PROCESSOR_LEVEL":"15","COMPUTERNAME":"UK1WPSFPXSOL101","NUMBER_OF_PROCESSORS":"4","TEMP":"C:\\\\Users\\\\SERV~1.AWX\\\\AppData\\\\Local\\\\Temp","PUBLIC":"C:\\\\Users\\\\Public","PROCESSOR_REVISION":"0601","APPDATA":"C:\\\\Users\\\\myserviceaccount\\\\AppData\\\\Roaming","ProgramData":"C:\\\\ProgramData","NUTCH_HOME":"D:/Solr/NutchHome","NUTCH_INSTALL":"D:/Solr/nutch-1.0","NUTCH_DATA":"D:/Solr/NutchData","JAVA_HOME":"D:\\\\Java\\\\jre8","CommonProgramW6432":"C:\\\\Program Files\\\\Common Files","windir":"C:\\\\Windows","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL","SN_AGENT_HOME_DIR":"C:\\\\Program Files\\\\ServiceNow\\\\agent-client-collector","TMP":"C:\\\\Users\\\\SERV~1.AWX\\\\AppData\\\\Local\\\\Temp"},"ansible_windows_domain":"dcsprod.dcsroot.local","ansible_reboot_pending":true,"ansible_system":"Win32NT"}}\r
<myserver.domain.com> PSRP STDERR: 
<myserver.domain.com> PSRP CLOSE RUNSPACE: 06F0F391-2F9D-4C0B-BF73-1910B05E287B
ok: [myservername]

TASK [Search-only] *************************************************************
task path: /runner/project/playbooks/wsus/WUA_SearchDownloadInstall/Install_windows_update.yml:12
<myservername> Running win_updates - round 1
<myservername> Starting update task
Using module file /runner/requirements_collections/ansible_collections/ansible/windows/plugins/modules/win_updates.ps1
Pipelining is enabled.
<myserver.domain.com> ESTABLISH PSRP CONNECTION FOR USER: myserviceaccount@MYDOMAIN.COM ON PORT 5986 TO myserver.domain.com
<myserver.domain.com> PSRP OPEN RUNSPACE: auth=negotiate configuration=Microsoft.PowerShell endpoint=https://myserver.domain.com:5986/wsman
PSRP: EXEC (via pipeline wrapper)
<myserver.domain.com> PSRP RC: 1
<myserver.domain.com> PSRP STDOUT: {"changed":false,"invocation":{"module_args":{"reject_list":null,"_operation":"start","category_names":["SecurityUpdates"],"reboot_timeout":1200,"reboot":false,"server_selection":"default","state":"searched","accept_list":null,"_operation_options":{},"skip_optional":false,"log_path":"C:\\\\Temp\\\\search_updates_install.txt"}},"failed":true,"msg":"Failed to start new win_updates task with Task Scheduler: Exception calling \\"NativeCreateProcess\\" with \\"9\\" argument(s): \\"CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)\\"","exception":"Exception calling \\"NativeCreateProcess\\" with \\"9\\" argument(s): \\"CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)\\"\\r\\nAt line:2198 char:24\\r\\n+ ...  $taskPid = Invoke-InProcess @invokeSplat -ParentProcessId $parentPid ...\\r\\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\\r\\n    + CategoryInfo          : NotSpecified: (:) [Invoke-InProcess], MethodInvocationException\\r\\n    + FullyQualifiedErrorId : Win32Exception,Invoke-InProcess\\r\\n\\r\\nScriptStackTrace:\\r\\nat Invoke-InProcess, \\u003cNo file\\u003e: line 503\\r\\nat \\u003cScriptBlock\\u003e, \\u003cNo file\\u003e: line 2198"}\r
<myserver.domain.com> PSRP STDERR: 
<myserver.domain.com> PSRP CLOSE RUNSPACE: C94327C0-6A0B-48F8-89D1-709703E63F1F
The full traceback is:
Exception calling "NativeCreateProcess" with "9" argument(s): "CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)"\r
At line:2198 char:24\r
+ ...  $taskPid = Invoke-InProcess @invokeSplat -ParentProcessId $parentPid ...\r
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r
    + CategoryInfo          : NotSpecified: (:) [Invoke-InProcess], MethodInvocationException\r
    + FullyQualifiedErrorId : Win32Exception,Invoke-InProcess\r
\r
ScriptStackTrace:\r
at Invoke-InProcess, <No file>: line 503\r
at <ScriptBlock>, <No file>: line 2198
fatal: [myservername]: FAILED! => {
    "changed": false,
    "failed_update_count": 0,
    "filtered_updates": {},
    "found_update_count": 0,
    "installed_update_count": 0,
    "invocation": {
        "module_args": {
            "accept_list": null,
            "category_names": [
                "SecurityUpdates"
            ],
            "log_path": "C:\\\\Temp\\\\search_updates_install.txt",
            "reboot": false,
            "reboot_timeout": 1200,
            "reject_list": null,
            "server_selection": "default",
            "skip_optional": false,
            "state": "searched"
        }
    },
    "msg": "Failed to start new win_updates task with Task Scheduler: Exception calling \\"NativeCreateProcess\\" with \\"9\\" argument(s): \\"CreateProcessW() failed (Access is denied, Win32ErrorCode 5 - 0x00000005)\\"",
    "updates": {}
}

PLAY RECAP *********************************************************************
myserverthatfailed            : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
7

You’re now getting an access denied error from Windows’ Task Scheduler.

This is a bit of a known recurring issue that hasn’t been pinned down to a root cause.

The issue is inconsistent, and occurs both with WinRM and PSRP plugins, but it’s hard to say that anything is wrong with the win_updates module because the issue will just “go away” after some time.

8

A couple of things that I can think of to check:

  • Are any Task Scheduler related GPO’s set?
    • There are both Computer/User scoped Windows Components/Task Scheduler/* policies
    • There are also Security Policies that might impact this such as Local Policies/User Rights Assignment/(Deny )Log on as Batch Job

These policies could be set locally though gpedit.msc/secpol.msc, or applied by domain level GPO’s.

Are these servers all in the same OU? (or at least getting the same linked GPOs applied?)

Are there any notable error logs in Windows’ Event Viewer around the time you try to run updates via Ansible?

Something else that could be happening is some kind of weird race condition relating to auth. Is there any significant uptime on the failing server vs the successful one (when was the computer object’s lastLogon/lastLogonTimestamp in AD)? Is it geographically distant from the DC? (Higher latency than your other succeeding host?) What are the CPU/RAM specs?

9

@jborean I dove a little deeper, and realized that this particular error message is not coming from Task Scheduler.

I noticed in win_updates.ps1#L2198 that the module is apparently spawning the scheduled task via cmd and hooking into cmd as a parent process. Going upstream a bit through the trace at win_updates.ps1#L503, the parent process is being handed off in the $si to Process.cs#L484 as part of the startupInfo. Then down at Process.cs#L616, we see a throw from NativeCreateProcess() that matches the exception that we see.

So the issue isn’t necessarily that Task Scheduler is denying the task creation, but that Windows doesn’t like the way the process itself for creating the task is being spawned?

10

Yep there’s a problem attempting to spawn a process with an explicit parent process here. I’ve never been able to reproduce it and this is the first time I’ve seen someone show it fails on psrp.