Windows: CS1504 Failed to compile C# code

Archives Ansible Project
1

Dear Ansible community,

Initially, I wanted to reply to this old conversation: https://groups.google.com/g/ansible-project/c/UVyj6hxgDAA/m/W7VjjynPGwAJ

As it has been closed for a few years, I can’t directly answer to it. Back then, it was noted that one hadn’t come across this issue before, and I couldn’t find any other information about this online as well.

We’re just in the middle of setup and migration of about 70 Windows 11 PCs (23H2, x64, German) with Ansible Community Edition, Active Directory and Group Policies. The Ansible connection is made with WSMan & CredSSP. We managed to set up and migrate about 10 PCs in the first week and everything went smooth until Thursday last week (02/08/2024). Suddenly, all the PCs we set up with Ansible couldn’t be managed by Ansible anymore — every task/playbook/ad-hoc connection with Ansible to the hosts failed with the mentioned error.

Here is a “-vvv” output of the error (partly German) for one of the hosts:

The full traceback is:
Failed to compile C# code:
error CS1504: Die Quelldatei ‘c:\Users\admin\AppData\Local\Temp\s3jrbyhw.0.cs’ konnte nicht geöffnet werden ('Unbekannter Fehler ').
In Zeile:369 Zeichen:13
+ throw [InvalidOperationException]$msg
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:slight_smile: [], InvalidOperationException
+ FullyQualifiedErrorId : Failed to compile C# code:
error CS1504: Die Quelldatei ‘c:\Users\admin\AppData\Local\Temp\s3jrbyhw.0.cs’ konnte nicht geöffnet werden ('Unbekannter Fehler ').

ScriptStackTrace:
bei Add-CSharpType, : Zeile 369
bei , : Zeile 19
bei , : Zeile 143
bei , : Zeile 11
fatal: [WINDOWSPC]: FAILED! => {
“changed”: false,
“msg”: “internal error: failed to run exec_wrapper action module_powershell_wrapper: Failed to compile C# code: \r\nerror CS1504: Die Quelldatei ‘c:\Users\admin\AppData\Local\Temp\s3jrbyhw.0.cs’ konnte nicht geöffnet werden ('Unbekannter Fehler ').”
}

Interestingly, we found that all PCs we set up since Monday this week (02/12/2024) are not affected by the error and continue working. All the PCs were set up and configured identically to the previous affected PCs and are of the same hardware batch, make and model. After a lot of digging, we found that PCs that were set up before that date and shutdown since then (for later installation) are not impacted by the issue for the first minutes after starting those PCs. Just after installation of the latest 3 Windows Updates on these PCs — even without rebooting — the issue starts. We also noticed that Windows Defender’s (no other AV/EDR/… software is present on the PCs) CPU usage spikes for a few seconds if any Ansible connection attempt is made to those PCs (causing the error message). Unfortunately, no detections or blocks are reported by Windows Defender. Disabling all protections of Windows Defender or Windows Firewall doesn’t help as well. AppLocker is not active / in use. The permissions to the TEMP directory are correct — even deleting the Windows user profile (and folder) and recreating it doesn’t help.

As a result, we are very positive that there has to be an issue with one of those three Windows Updates:

  • 2024-02 Cumulative Update for Windows 11 Version 23H2 for x64-based systems (KB5034765)
  • 2024-02 Cumulative Update for .NET Framework 3.5 and 4.8.1 for Windows 11, version 23H2 for x64 (KB5034467)
  • Windows Malicious Software Removal Tool x64 - v5.121 (KB890830)
    Unfortunately, removing all of these updates (the last one with “wusa /uninstall /kb:890830 /quiet /norestart”) didn’t help. As a correlation with Windows Defender is to be expected, it’s worth noting that all the older/affected Windows PCs had the previous version of Windows Malicious Software Removal Tool x64 v5.120 installed before — while the not affected / “newer” (technically identical) PCs never had v5.120 installed (fresh installation of Windows 11) and installed v5.121 directly.

After lots of debugging and tests, we were unable to find a solution to the problem. Our current solution is to format the affected PCs and start again — which works and the issue doesn’t appear anymore on those devices. Still, this isn’t very satisfying, since we expect the issue to reappear with a later Windows Update again.

Does anybody have any idea on how to further debug the issue or even fix it?

Thanks in advance!

Best

Alexander Schomburg

2

Another week passed and unfortunately all other Windows 11 PCs we set up during the last week are now affected by the issue too. They’ve been working on Thursday evening and the issue appeared on Monday morning (for this client, Friday to Sunday are limited working days). Interestingly, we have around 80 Windows 10 PCs setup previously with Ansible that are in the same Windows Domain and that are applied the same Group Policies, none of those Windows 10 PCs are impacted by the issue. But all Windows 11 PCs are impacted. Pretty sure this has to be some recent Windows Update or strange time-based internal Windows 11 policy that causes the problem.

We also did some tests and tested connections with OpenSSH and NTLM (instead of CredSSP) for the impacted PCs. The issue happens with all those connection types — for any Ansible remote command (Gathering facts, win_ping, chocolatey, …) and any local/domain Windows user/admin (obviously the path to the CS1504 reported source file changes for different users).

Can anybody help?

3

FYI, We managed to find the reason and a possible solution for this issue which is described in detail in this GitHub Issue: https://github.com/ansible-collections/ansible.windows/issues/588

TLDR: Make sure to never enable Dynamic Code Security with AppLocker, even if AppLocker is just run in Audit only mode (which means it shouldn’t block anything — it will block Ansible with DCS anyway…)