Debugging Ansible 2 on Windows Server?

Archives Ansible Project
,
1

I’m using the Ansible “raw” module to compile a Java app on Windows Server 2008 and 2012. Both have the Oracle 64-bit Java 8 JDK. Windows Server 2008 has Powershell 3 and
Windows Server 2012 has Powershell 4. These servers are the stock Windows AMIs running on AWS.

On Windows Server 2012, it almost completes but always eventually bombs out with:

error occurred during error reporting (null), id 0xc0000005]", "", "#", "# There is insufficient memory for the Java Runtime Environment to continue.", "# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate",

This is on a machine with 15 gig of RAM with max RAM for the compilation at 8gig.
Notably, the compilation always completes if I go to the machine’s command line and run it by hand (it only fails when Ansible runs it).

On Windows Server 2008, it mysteriously fails every time:

`

TASK [compile the app] **************************************************
task path: /mycorp_CODE/ansible/building_scripts/build_win.yml:126
<52.24.206.112> ESTABLISH WINRM CONNECTION FOR USER: Administrator on PORT 5986 TO 52.24.28.18
<52.24.206.112> WINRM CONNECT: transport=ssl endpoint=https://52.24.28.18:5986/wsman
<52.24.206.112> EXEC .\activator dist
<52.24.206.112> WINRM OPEN SHELL: E6D0B5A6-99CD-47C7-845E-BF9B7DB72FB2
<52.24.206.112> WINRM EXEC ‘PowerShell’ [‘-NoProfile’, ‘-NonInteractive’, ‘-ExecutionPolicy’, ‘Unrestricted’, ‘-EncodedCommand’, ‘YwBoAGQAaQByACAAQwA6AFwAYwBhAHAAcwBlAAdQB3AG0AYQBwAHAAZQByAF8AcgBlAHAAbwBcAFUATABUAFIAQQBXAFAIAAuAFwAYQBjAHQAAHIALgBiAGEAdAaQBzAHQA’]
<52.24.206.112> WINRM RESULT u’<Response code 0, out “”, err “”>’
<52.24.206.112> WINRM STDOUT
<52.24.206.112> WINRM STDERR
<52.24.206.112> WINRM CLOSE SHELL: E6D0B5A6-99CD-47C7-845E-BF9B7DB72FB2
ok: [52.24.28.18] => {“changed”: false, “invocation”: {“module_args”: {“_raw_params”: “.\activator dist”}, “module_name”: “raw”}, “rc”: 0, “stderr”: “”, “stdout”: “”, “stdout_lines”: }
`

But again, if I go to the Windows command line, I can run it by hand successfully every time.

Any ideas?

J

2

Bit of a guess but I wonder if you are hitting a quota limit in the winrm configuration on your windows machine?

MaxMemoryPerShellMB looks the most likely - see https://msdn.microsoft.com/en-us/library/windows/desktop/ee309367(v=vs.85).aspx

Unpatched Server 2008 R2 had a bug that mis-set the quotas which makes me think that might explain the difference - bug detailed here: https://social.technet.microsoft.com/Forums/windows/en-US/51810d3b-6236-44f8-99fd-10f004ad8002/winrm-quotas-setting-ignored-since-kb2506143?forum=w7itproinstall

To debug… you could try running with -vvvvvv to get full stdout/stderr from when the compile runs.
Also check windows event logging for winrm / winrs to see if winrm is reporting any errors.

If the quota size is the problem, I have read somewhere, -sorry can’t find the link now - that you can’t reconfigure winrm remotely so you might have to workaround changing the quota settings if that is in fact the problem. If it can’t be done in the configure ansible for remoting script then might be possible by creating a scheduled task.

Hope this helps,

Jon

3

Hi Jon,

Thanks so much for your reply. I carefully followed the instructions at this link to double the amount of winrm ram from 1 gig to 2gig:

https://blogs.technet.microsoft.com/heyscriptingguy/2013/07/30/learn-how-to-configure-powershell-memory/

That solved the out-of-memory problem on Windows Server 2012, but for Windows Server 2008R2, I’m getting the same result as before.
I did a full Windows Update and restart. The MaxMemoryPerShellMB shows as 2048m. But still the same results. The
output (shown above in my original posting) is done with -vvvvvv, btw. I looked through the event logs and couldn’t find
anything for winrm that would explain things.

Do you think I need to manually apply that patch somehow? Is there an easy way to tell if I need to apply it or not? (sorry, I’m not a Windows expert).

Thanks,

J

4

Glad increasing the quota got you going on S 2012.

You can manually apply the hotfix, although my prefered way of getting round it was to upgrade to Windows Management Framework 4.0 (which includes powershell 4) if that’s an option for you. I think there’s an msi or msu to run to install it - I’m not actually using 2008 any more.

You can see if you have the hotfix installed by running the powershell command

Get-Hotfix

Hope this helps,

Jon

5

Windows 2008R2 is still a mystery. I’m using the stock AWS AMI with all updates applied and upped the
PS RAM as I did for Windows 2012R2. Still, I get the same thing as before. How can it just fail with no
output? I don;t see anything in the Windows event logs, but I could be missing something.
I’m hoping that Ansible 2.01 might work a bit better.

J

6

The only time I’ve seen a raw really not produce any output was on ansible 1.9.4 where one of the command args contained a > - however the same problem doesn’t occurr in 2.0.

Does it fail instantly or after a second or two?

Only thing other I can think of is to try the update to Windows Management Framework 4.0 then. I heard Dave Wyatt (powershell MVP) say at WinOps last year that 4.0 includes a re-write of WMI, which in his opinion made it much more stable and a better choice for automating against.
Unless it would invalidate the purpose of compiling on Server 2008R2 I reckon it would be worth a try.

Jon

7

It sits there for a few moments and then moves on.

J

8

I would set export ANSIBLE_KEEP_REMOTE_FILES=1 to have ansible not delete the generated powershell files, and then debug from there. They will end up in the temp directory of the user ansible logs onto the windows box with. Executing the generated scripts from within a regular PowerShell remoting session wold probably be interesting.