Unexpected failure during module execution during Windows build

72clg · October 25, 2023, 11:22am

I’m trying to build a Windows Server (2016, 2019 and 2022) using Packer and Ansible within AWS and reusing code where I can. As part of the AMI creation it builds the image and sysprep it all OK, but then it creates an instance to perform some tests on the newly created AMI, but this is failing, it use to work until around July. I’ve even removed all the hardening to ensure its nothing from that.

- Error
test.amazon-ebs.test: Prevalidating any provided VPC information
test.amazon-ebs.test: Prevalidating AMI Name: windows-2016-2023-10-24t09-39-10z
test.amazon-ebs.test: Found Image ID: ami-0f191d7afe71ef99c
test.amazon-ebs.test: Found VPC ID: vpc-0688c234550620b82
test.amazon-ebs.test: Found Subnet ID: subnet-03f031d8fb5ec4473
test.amazon-ebs.test: Creating temporary keypair: packer_653790be-8348-6dea-8bb9-4229ad9e4511
test.amazon-ebs.test: Saving key for debug purposes: ec2_test.pem
test.amazon-ebs.test: Found Security Group(s): sg-0708434681fcea5ac
test.amazon-ebs.test: Launching a source AWS instance...
test.amazon-ebs.test: Adding tag: "Name": "windows-2016-185-test"
test.amazon-ebs.test: Adding tag: "Project": "image-hardening"
test.amazon-ebs.test: Adding tag: "trend-plan": "windows-min"
test.amazon-ebs.test: Adding tag: "BuildNumber": "185"
test.amazon-ebs.test: Adding tag: "ImageFamily": "windows-2016"
test.amazon-ebs.test: Adding tag: "CommitId": "8f6ceee8601f1510f9946124ba3db0892454ac27"
test.amazon-ebs.test: Adding tag: "InspectorID": "windows-2016-185"
test.amazon-ebs.test: Adding tag: "Project": "image-hardening"
test.amazon-ebs.test: Adding tag: "BuildNumber": "185"
test.amazon-ebs.test: Adding tag: "trend-plan": "windows-min"
test.amazon-ebs.test: Adding tag: "InspectorID": "windows-2016-185"
test.amazon-ebs.test: Adding tag: "CommitId": "8f6ceee8601f1510f9946124ba3db0892454ac27"
test.amazon-ebs.test: Adding tag: "ImageFamily": "windows-2016"
test.amazon-ebs.test: Adding tag: "Name": "windows-2016-185-test"
test.amazon-ebs.test: Adding tag: "Environment": "DEV"
test.amazon-ebs.test: Instance ID: i-03ed24d8c203b8c9a
test.amazon-ebs.test: Waiting for instance (i-03ed24d8c203b8c9a) to become ready...
test.amazon-ebs.test: Public DNS: ec2-79-125-76-178.eu-west-1.compute.amazonaws.com
test.amazon-ebs.test: Public IP: 79.125.76.178
test.amazon-ebs.test: Private IP: 10.0.0.254
test.amazon-ebs.test: Waiting for auto-generated password for instance...
test.amazon-ebs.test: It is normal for this process to take up to 15 minutes,
test.amazon-ebs.test: but it usually takes around 5. Please wait.
test.amazon-ebs.test:
test.amazon-ebs.test: Password retrieved!
test.amazon-ebs.test: Password (since debug is enabled): 6oMM4ekTakKcZKD=yu(R6CFfevmRZYHP
test.amazon-ebs.test: Using WinRM communicator to connect: 79.125.76.178
test.amazon-ebs.test: Waiting for WinRM to become available...
test.amazon-ebs.test: WinRM connected.
test.amazon-ebs.test: Connected to WinRM!
test.amazon-ebs.test: Setting a 1h30m0s timeout for the next provisioner...
test.amazon-ebs.test: Provisioning with Ansible...
test.amazon-ebs.test: Not using Proxy adapter for Ansible run:
test.amazon-ebs.test: Using WinRM Password from Packer communicator...
test.amazon-ebs.test: Executing Ansible: ansible-playbook -e packer_build_name="test" -e packer_builder_type=amazon-ebs -vvvvvv --diff -eansible_connection=winrm -e ansible_winrm_server_cert_validation=ignore -e ansible_winrm_operation_timeout_sec=3600 -e ansible_winrm_read_timeout_sec=7200 -e cis_min_score=90 -e cis_benchmark=CIS_Microsoft_Windows_Server_2016_Benchmark_v2.0.0-xccdf.xml -e profile=xccdf_org.cisecurity.benchmarks_profile_Level_1_-_Member_Server -e build_number=185 -e commit_id=8f6ceee8601f1510f9946124ba3db0892454ac27 -e image_family=windows-2016 -e ansible_password=***** -i /tmp/packer-provisioner-ansible4047541749 /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/playbook.yml
test.amazon-ebs.test: ansible-playbook 2.10.17
test.amazon-ebs.test:   config file = None
test.amazon-ebs.test:   configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
test.amazon-ebs.test:   ansible python module location = /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible
test.amazon-ebs.test:   executable location = /root/.pyenv/versions/3.8.16/bin/ansible-playbook
test.amazon-ebs.test:   python version = 3.8.16 (default, Feb  6 2023, 02:19:23) [GCC 7.5.0]
test.amazon-ebs.test: No config file found; using defaults
test.amazon-ebs.test: setting up inventory plugins
test.amazon-ebs.test: host_list declined parsing /tmp/packer-provisioner-ansible4047541749 as it did not pass its verify_file() method
test.amazon-ebs.test: script declined parsing /tmp/packer-provisioner-ansible4047541749 as it did not pass its verify_file() method
test.amazon-ebs.test: auto declined parsing /tmp/packer-provisioner-ansible4047541749 as it did not pass its verify_file() method
test.amazon-ebs.test: Parsed /tmp/packer-provisioner-ansible4047541749 inventory source with ini plugin
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/validity/tasks/validity.yml
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/cleanup/tasks/cleanup.yml
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/checks/tasks/checks.yml
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/ssm/tasks/win-ssm.yml
test.amazon-ebs.test: redirecting (type: modules) ansible.builtin.win_service to ansible.windows.win_service
test.amazon-ebs.test: Loading collection ansible.windows from /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible_collections/ansible/windows
test.amazon-ebs.test: redirecting (type: modules) ansible.builtin.win_scheduled_task_stat to community.windows.win_scheduled_task_stat
test.amazon-ebs.test: Loading collection community.windows from /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible_collectis_pam_valid.yml
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/mcs/tasks/mc
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/cis/tasks/cis.yml
test.amazon-ebs.test: statically imported: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/roles/cis/tasks/win-cis.yml
test.amazon-ebs.test: redirecting (type: modules) ansible.builtin.win_stat to ansible.windows.win_stat
test.amazon-ebs.test: redirecting (type: modules) ansible.builtin.win_shell to ansible.windows.win_shell
test.amazon-ebs.test: redirecting (type: modules) ansible.builtin.win_shell to ansible.windows.win_shell
test.amazon-ebs.test: Loading callback plugin default of type stdout, v2.0 from /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/plugins/callback/default.py
test.amazon-ebs.test: Attempting to use 'default' callback.
test.amazon-ebs.test: Skipping callback 'default', as we already have a stdout callback.
test.amazon-ebs.test: Attempting to use 'junit' callback.
test.amazon-ebs.test: Attempting to use 'minimal' callback.
test.amazon-ebs.test: Skipping callback 'minimal', as we already have a stdout callback.
test.amazon-ebs.test: Attempting to use 'oneline' callback.
test.amazon-ebs.test: Skipping callback 'oneline', as we already have a stdout callback.
test.amazon-ebs.test: Attempting to use 'tree' callback.
test.amazon-ebs.test:
test.amazon-ebs.test: PLAYBOOK: playbook.yml *********************************************************
test.amazon-ebs.test: Positional arguments: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/playbook.yml
test.amazon-ebs.test: verbosity: 6
test.amazon-ebs.test: connection: smart
test.amazon-ebs.test: timeout: 10
test.amazon-ebs.test: become_method: sudo
test.amazon-ebs.test: tags: ('all',)
test.amazon-ebs.test: diff: True
test.amazon-ebs.test: inventory: ('/tmp/packer-provisioner-ansible4047541749',)
test.amazon-ebs.test: extra_vars: ('packer_build_name="test"', 'packer_builder_type=amazon-ebs', 'ansible_connection=winrm', 'ansible_winrm_server_cert_validation=ignore', 'ansible_winrm_operation_timeout_sec=3600', 'ansible_winrm_read_timeout_sec=7200', 'cis_min_score=90', 'cis_benchmark=CIS_Microsoft_Windows_Server_2016_Benchmark_v2.0.0-xccdf.xml', 'profile=xccdf_org.cisecurity.benchmarks_profile_Level_1_-_Member_Server', 'build_number=185', 'commit_id=8f6ceee8601f1510f9946124ba3db0892454ac27', 'image_family=windows-2016', 'ansible_password=6oMM4ekTakKcZKD=yu(R6CFfevmRZYHP')
test.amazon-ebs.test: forks: 5
test.amazon-ebs.test: 1 plays in /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/playbook.yml
test.amazon-ebs.test:
test.amazon-ebs.test: PLAY [Test] ********************************************************************
test.amazon-ebs.test:
test.amazon-ebs.test: TASK [Gathering Facts] *********************************************************
test.amazon-ebs.test: task path: /codebuild/output/src4191/src/api.github.myidtest.com/image-hardening/tests/playbook.yml:2
test.amazon-ebs.test: redirecting (type: modules) ansible.builtin.setup to ansible.windows.setup
test.amazon-ebs.test: Using module file /root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible_collections/ansible/windows/plugins/modules/setup.ps1
test.amazon-ebs.test: Pipelining is enabled.
test.amazon-ebs.test: <79.125.76.178> ESTABLISH WINRM CONNECTION FOR USER: Administrator on PORT 5985 TO 79.125.76.178
test.amazon-ebs.test: <79.125.76.178> WINRM CONNECT: transport=basic endpoint=http://79.125.76.178:5985/wsman
test.amazon-ebs.test: <79.125.76.178> WINRM OPEN SHELL: DCBA94C9-90C0-47C3-9487-D7A39D4179D2
test.amazon-ebs.test: EXEC (via pipeline wrapper)
test.amazon-ebs.test: <79.125.76.178> WINRM EXEC 'PowerShell' ['-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Unrestricted', '-EncodedCommand', 'UABvAHcAZQByAFMAaABlAGwAbAAgAC0ATgBvAFAAcgBvAGYAaQBsAGUAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0ARQB4AGUAYwB1AHQAaQBvAG4AUABvAGwAaQBjAHkAIABVAG4AcgBlAHMAdAByAGkAYwB0AGUAZAAgAC0ARQBuAGMAbwBkAGUAZABDAG8AbQBtAGEAbgBkACAASgBnAEIAagBBAEcAZwBBAFkAdwBCAHcAQQBDADQAQQBZAHcAQgB2AEEARwAwAEEASQBBAEEAMgBBAEQAVQBBAE0AQQBBAHcAQQBEAEUAQQBJAEEAQQArAEEAQwBBAEEASgBBAEIAdQBBAEgAVQBBAGIAQQBCAHMAQQBBAG8AQQBhAFEAQgBtAEEAQwBBAEEASwBBAEEAawBBAEYAQQBBAFUAdwBCAFcAQQBHAFUAQQBjAGcAQgB6AEEARwBrAEEAYgB3AEIAdQBBAEYAUQBBAFkAUQBCAGkAQQBHAHcAQQBaAFEAQQB1AEEARgBBAEEAVQB3AEIAVwBBAEcAVQBBAGMAZwBCAHoAQQBHAGsAQQBiAHcAQgB1AEEAQwBBAEEATABRAEIAcwBBAEgAUQBBAEkAQQBCAGIAQQBGAFkAQQBaAFEAQgB5AEEASABNAEEAYQBRAEIAdgBBAEcANABBAFgAUQBBAGkAQQBEAE0AQQBMAGcAQQB3AEEAQwBJAEEASwBRAEEAZwBBAEgAcwBBAEMAZwBBAG4AQQBIAHMAQQBJAGcAQgBtAEEARwBFAEEAYQBRAEIAcwBBAEcAVQBBAFoAQQBBAGkAQQBEAG8AQQBkAEEAQgB5AEEASABVAEEAWgBRAEEAcwBBAEMASQBBAGIAUQBCAHoAQQBHAGMAQQBJAGcAQQA2AEEAQwBJAEEAUQBRAEIAdQBBAEgATQBBAGEAUQBCAGkAQQBHAHcAQQBaAFEAQQBnAEEASABJAEEAWgBRAEIAeABBAEgAVQBBAGEAUQBCAHkAQQBHAFUAQQBjAHcAQQBnAEEARgBBAEEAYgB3AEIAMwBBAEcAVQBBAGMAZwBCAFQAQQBHAGcAQQBaAFEAQgBzAEEARwB3AEEASQBBAEIAMgBBAEQATQBBAEwAZwBBAHcAQQBDAEEAQQBiAHcAQgB5AEEAQwBBAEEAYgBnAEIAbABBAEgAYwBBAFoAUQBCAHkAQQBDAEkAQQBmAFEAQQBuAEEAQQBvAEEAWgBRAEIANABBAEcAawBBAGQAQQBBAGcAQQBEAEUAQQBDAGcAQgA5AEEAQQBvAEEASgBBAEIAbABBAEgAZwBBAFoAUQBCAGoAQQBGADgAQQBkAHcAQgB5AEEARwBFAEEAYwBBAEIAdwBBAEcAVQBBAGMAZwBCAGYAQQBIAE0AQQBkAEEAQgB5AEEAQwBBAEEAUABRAEEAZwBBAEMAUQBBAGEAUQBCAHUAQQBIAEEAQQBkAFEAQgAwAEEAQwBBAEEAZgBBAEEAZwBBAEUAOABBAGQAUQBCADAAQQBDADAAQQBVAHcAQgAwAEEASABJAEEAYQBRAEIAdQBBAEcAYwBBAEMAZwBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQQBnAEEARAAwAEEASQBBAEEAawBBAEcAVQBBAGUAQQBCAGwAQQBHAE0AQQBYAHcAQgAzAEEASABJAEEAWQBRAEIAdwBBAEgAQQBBAFoAUQBCAHkAQQBGADgAQQBjAHcAQgAwAEEASABJAEEATABnAEIAVABBAEgAQQBBAGIAQQBCAHAAQQBIAFEAQQBLAEEAQgBBAEEAQwBnAEEASQBnAEIAZwBBAEQAQQBBAFkAQQBBAHcAQQBHAEEAQQBNAEEAQgBnAEEARABBAEEASQBnAEEAcABBAEMAdwBBAEkAQQBBAHkAQQBDAHcAQQBJAEEAQgBiAEEARgBNAEEAZABBAEIAeQBBAEcAawBBAGIAZwBCAG4AQQBGAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAUABBAEgAQQBBAGQAQQBCAHAAQQBHADgAQQBiAGcAQgB6AEEARgAwAEEATwBnAEEANgBBAEYASQBBAFoAUQBCAHQAQQBHADgAQQBkAGcAQgBsAEEARQBVAEEAYgBRAEIAdwBBAEgAUQBBAGUAUQBCAEYAQQBHADQAQQBkAEEAQgB5AEEARwBrAEEAWgBRAEIAegBBAEMAawBBAEMAZwBCAEoAQQBHAFkAQQBJAEEAQQBvAEEAQwAwAEEAYgBnAEIAdgBBAEgAUQBBAEkAQQBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQQB1AEEARQB3AEEAWgBRAEIAdQBBAEcAYwBBAGQAQQBCAG8AQQBDAEEAQQBMAFEAQgBsAEEASABFAEEASQBBAEEAeQBBAEMAawBBAEkAQQBCADcAQQBDAEEAQQBkAEEAQgBvAEEASABJAEEAYgB3AEIAMwBBAEMAQQBBAEkAZwBCAHAAQQBHADQAQQBkAGcAQgBoAEEARwB3AEEAYQBRAEIAawBBAEMAQQBBAGMAQQBCAGgAQQBIAGsAQQBiAEEAQgB2AEEARwBFAEEAWgBBAEEAaQBBAEMAQQBBAGYAUQBBAEsAQQBGAE0AQQBaAFEAQgAwAEEAQwAwAEEAVgBnAEIAaABBAEgASQBBAGEAUQBCAGgAQQBHAEkAQQBiAEEAQgBsAEEAQwBBAEEATABRAEIATwBBAEcARQBBAGIAUQBCAGwAQQBDAEEAQQBhAGcAQgB6AEEARwA4AEEAYgBnAEIAZgBBAEgASQBBAFkAUQBCADMAQQBDAEEAQQBMAFEAQgBXAEEARwBFAEEAYgBBAEIAMQBBAEcAVQBBAEkAQQBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQgBiAEEARABFAEEAWABRAEEASwBBAEMAUQBBAFoAUQBCADQAQQBHAFUAQQBZAHcAQgBmAEEASABjAEEAYwBnAEIAaABBAEgAQQBBAGMAQQBCAGwAQQBIAEkAQQBJAEEAQQA5AEEAQwBBAEEAVwB3AEIAVABBAEcATQBBAGMAZwBCAHAAQQBIAEEAQQBkAEEAQgBDAEEARwB3AEEAYgB3AEIAagBBAEcAcwBBAFgAUQBBADYAQQBEAG8AQQBRAHcAQgB5AEEARwBVAEEAWQBRAEIAMABBAEcAVQBBAEsAQQBBAGsAQQBIAE0AQQBjAEEAQgBzAEEARwBrAEEAZABBAEIAZgBBAEgAQQBBAFkAUQBCAHkAQQBIAFEAQQBjAHcAQgBiAEEARABBAEEAWABRAEEAcABBAEEAbwBBAEoAZwBBAGsAQQBHAFUAQQBlAEEAQgBsAEEARwBNAEEAWAB3AEIAMwBBAEgASQBBAFkAUQBCAHcAQQBIAEEAQQBaAFEAQgB5AEEAQQA9AD0A']
test.amazon-ebs.test: <79.125.76.178> WINRM CLOSE SHELL: DCBA94C9-90C0-47C3-9487-D7A39D4179D2
test.amazon-ebs.test: The full traceback is:
test.amazon-ebs.test: Traceback (most recent call last):
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
test.amazon-ebs.test:     httplib_response = self._make_request(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
test.amazon-ebs.test:     six.raise_from(e, None)
test.amazon-ebs.test:   File "<string>", line 3, in raise_from
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
test.amazon-ebs.test:     httplib_response = conn.getresponse()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 1348, in getresponse
test.amazon-ebs.test:     response.begin()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 316, in begin
test.amazon-ebs.test:     version, status, reason = self._read_status()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 277, in _read_status
test.amazon-ebs.test:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/socket.py", line 669, in readinto
test.amazon-ebs.test:     return self._sock.recv_into(b)
test.amazon-ebs.test: ConnectionResetError: [Errno 104] Connection reset by peer
test.amazon-ebs.test:
test.amazon-ebs.test: During handling of the above exception, another exception occurred:
test.amazon-ebs.test:
test.amazon-ebs.test: Traceback (most recent call last):
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
test.amazon-ebs.test:     resp = conn.urlopen(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
test.amazon-ebs.test:     retries = retries.increment(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
test.amazon-ebs.test:     raise six.reraise(type(error), error, _stacktrace)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
test.amazon-ebs.test:     raise value.with_traceback(tb)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
test.amazon-ebs.test:     httplib_response = self._make_request(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
test.amazon-ebs.test:     six.raise_from(e, None)
test.amazon-ebs.test:   File "<string>", line 3, in raise_from
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
test.amazon-ebs.test:     httplib_response = conn.getresponse()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 1348, in getresponse
test.amazon-ebs.test:     response.begin()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 316, in begin
test.amazon-ebs.test:     version, status, reason = self._read_status()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 277, in _read_status
test.amazon-ebs.test:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/socket.py", line 669, in readinto
test.amazon-ebs.test:     return self._sock.recv_into(b)
test.amazon-ebs.test: urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
test.amazon-ebs.test:
test.amazon-ebs.test: During handling of the above exception, another exception occurred:
test.amazon-ebs.test:
test.amazon-ebs.test: Traceback (most recent call last):
_winrm_exec
test.amazon-ebs.test:     resptuple = self.protocol.get_command_output(self.shell_id, command_id)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/protocol.py", line 449, in get_command_output
test.amazon-ebs.test:     self._raw_get_command_output(shell_id, command_id)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/protocol.py", line 468, in _raw_get_command_output
test.amazon-ebs.test:     res = self.send_message(xmltodict.unparse(req))
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/protocol.py", line 243, in send_message
test.amazon-ebs.test:     resp = self.transport.send_message(message)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/transport.py", line 322, in send_message
test.amazon-ebs.test:     response = self._send_message_request(prepared_request, message)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/transport.py", line 327, in _send_message_request
test.amazon-ebs.test:     response = self.session.send(prepared_request, timeout=self.read_timeout_sec)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
test.amazon-ebs.test:     r = adapter.send(request, **kwargs)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
test.amazon-ebs.test:     raise ConnectionError(err, request=request)
test.amazon-ebs.test: requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
test.amazon-ebs.test:
test.amazon-ebs.test: During handling of the above exception, another exception occurred:
test.amazon-ebs.test:
test.amazon-ebs.test: Traceback (most recent call last):
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
test.amazon-ebs.test:     conn = connection.create_connection(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
test.amazon-ebs.test:     raise err
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
test.amazon-ebs.test:     sock.connect(sa)
test.amazon-ebs.test: ConnectionRefusedError: [Errno 111] Connection refused
test.amazon-ebs.test:
test.amazon-ebs.test: During handling of the above exception, another exception occurred:
test.amazon-ebs.test:
test.amazon-ebs.test: Traceback (most recent call last):
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
test.amazon-ebs.test:     httplib_response = self._make_request(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
test.amazon-ebs.test:     conn.request(method, url, **httplib_request_kw)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
test.amazon-ebs.test:     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 1256, in request
test.amazon-ebs.test:     self._send_request(method, url, body, headers, encode_chunked)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 1302, in _send_request
test.amazon-ebs.test:     self.endheaders(body, encode_chunked=encode_chunked)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 1251, in endheaders
test.amazon-ebs.test:     self._send_output(message_body, encode_chunked=encode_chunked)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 1011, in _send_output
test.amazon-ebs.test:     self.send(msg)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/http/client.py", line 951, in send
test.amazon-ebs.test:     self.connect()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
test.amazon-ebs.test:     conn = self._new_conn()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
test.amazon-ebs.test:     raise NewConnectionError(
test.amazon-ebs.test: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f632ecf9670>: Failed to establish a new connection: [Errno 111] Connection refused
test.amazon-ebs.test:
test.amazon-ebs.test: During handling of the above exception, another exception occurred:
test.amazon-ebs.test:
test.amazon-ebs.test: Traceback (most recent call last):
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
test.amazon-ebs.test:     resp = conn.urlopen(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
test.amazon-ebs.test:     retries = retries.increment(
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
test.amazon-ebs.test:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
test.amazon-ebs.test: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='79.125.76.178', port=5985): Max retries exceeded with url: /wsman (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f632ecf9670>: Failed to establish a new connection: [Errno 111] Connection refused'))
test.amazon-ebs.test:
test.amazon-ebs.test: During handling of the above exception, another exception occurred:
test.amazon-ebs.test:
test.amazon-ebs.test: Traceback (most recent call last):
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 158, in run
test.amazon-ebs.test:     res = self._execute()
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/executor/task_executor.py", line 663, in _execute
test.amazon-ebs.test:     result = self._handler.run(task_vars=variables)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/plugins/action/gather_facts.py", line 94,
test.amazon-ebs.test:     res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/plugins/action/__init__.py", line 969, in_execute_module
test.amazon-ebs.test:     res = self._low_level_execute_command(cmd, sudoable=sudoable, in_data=in_data)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/plugins/action/__init__.py", line 1121, in _low_level_execute_command
test.amazon-ebs.test:     rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/plugins/connection/winrm.py", line 549, in exec_command
test.amazon-ebs.test:     result = self._winrm_exec(cmd_parts[0], cmd_parts[1:], from_exec=True, stdin_iterator=stdin_iterator)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/ansible/plugins/connection/winrm.py", line 508, in_winrm_exec
test.amazon-ebs.test:     self.protocol.cleanup_command(self.shell_id, command_id)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/protocol.py", line 390, in cleanup_command
test.amazon-ebs.test:     res = self.send_message(xmltodict.unparse(req))
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/protocol.py", line 243, in send_message
test.amazon-ebs.test:     resp = self.transport.send_message(message)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/transport.py", line 322, in send_message
test.amazon-ebs.test:     response = self._send_message_request(prepared_request, message)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/winrm/transport.py", line 327, in _send_message_request
test.amazon-ebs.test:     response = self.session.send(prepared_request, timeout=self.read_timeout_sec)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
test.amazon-ebs.test:     r = adapter.send(request, **kwargs)
test.amazon-ebs.test:   File "/root/.pyenv/versions/3.8.16/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
test.amazon-ebs.test:     raise ConnectionError(e, request=request)
test.amazon-ebs.test: requests.exceptions.ConnectionError: HTTPConnectionPool(host='79.125.76.178', port=5985): Max retries exceeded with url:/wsman (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f632ecf9670>: Failed to establish a new connection: [Errno 111] Connection refused'))
test.amazon-ebs.test: fatal: [default]: FAILED! => {
test.amazon-ebs.test:     "msg": "Unexpected failure during module execution.",
test.amazon-ebs.test:     "stdout": ""
test.amazon-ebs.test: }
test.amazon-ebs.test:
test.amazon-ebs.test: PLAY RECAP *********************************************************************
test.amazon-ebs.test: default                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

72clg · October 25, 2023, 11:33am

test.pkr.hcl

source "amazon-ebs" "test" {
  ami_name          = local.image_name
  region            = var.region
  availability_zone = var.az
  skip_create_ami   = true
  assume_role {
    role_arn     = "arn:aws:iam::${var.account_id_dev}:role/${var.pipeline_role}"
    session_name = "${var.image_family}-${var.build_number}"
  }
  source_ami                  = var.image_id
  instance_type               = var.instance_type
  ena_support                 = var.ena_support
  ami_virtualization_type     = var.virtualization_type
  sriov_support               = var.sriov_support
  ebs_optimized               = var.ebs_optimized
  shutdown_behavior           = "terminate"
  user_data_file              = "../../build/windows/scripts/bootstrap_win.txt"  
  communicator                = "winrm"
  winrm_username              = var.username
  winrm_insecure              = true
  associate_public_ip_address = true

  security_group_filter {
    filters = {
      "owner-id"                  = var.account_id_dev
      "ip-permission.protocol"    = "tcp"
      "ip-permission.to-port"     = "5985"
      "ip-permission.from-port"   = "5985"
      "ip-permission.cidr"        = "34.246.47.146/32"
      "egress.ip-permission.cidr" = "0.0.0.0/0"
      "tag:Project"               = local.tags.Project
      "tag:SecurityZone"          = "S1"
    }
  }

  vpc_filter {
    filters = {
      "owner-id"    = var.account_id_dev
      "state"       = "available"
      "tag:Project" = local.tags.Project
    }
  }
  subnet_filter {
    filters = {
      "state"             = "available"
      "owner-id"          = var.account_id_dev
      "availability-zone" = var.az
      "tag:Project"       = local.tags.Project
      "tag:SecurityZone"  = "P"
      "tag:Purpose"       = "build"
    }
  }
  iam_instance_profile    = var.instance_profile
  launch_block_device_mappings {
    device_name           = var.volume_device_name
    volume_type           = var.volume_type
    volume_size           = var.volume_size
    delete_on_termination = true
    iops                  = var.volume_type == "gp3" ? 3000 : null 
    throughput            = var.volume_type == "gp3" ? 125 : null
  }
  run_volume_tags = local.tags
  run_tags        = local.tags
}

build {
  name    = "test"
  sources = ["source.amazon-ebs.test"]
  
  provisioner "ansible" {
    playbook_file     = abspath("${var.cloud_repo}/tests/playbook.yml")
    user              = var.ssh_username
    ansible_env_vars  = ["WINRM_PASSWORD=${build.WinRMPassword}"]
    use_proxy         = false
    extra_arguments   = local.ansible_args
    timeout           = "90m"
  }

  provisioner "ansible" {
    playbook_file     = abspath("${var.aws_repo}/tests/playbook.yml")
    user              = var.ssh_username
    ansible_env_vars  = ["WINRM_PASSWORD=${build.WinRMPassword}"]
    use_proxy         = false
    extra_arguments   = local.ansible_args    
    timeout           = "90m"
  }
}

locals.pkr.hcl

locals {
  timestamp         = join("", [formatdate("YYYY-MM-DD", timestamp()), "t", formatdate("hh-mm-ss", timestamp()), "z"])
  image_name        = "${var.image_family}-${join("", [formatdate("YYYY-MM-DD", timestamp()), "t", formatdate("hh-mm-ss", timestamp()), "z"])}"
  tags = {
    "Name"            = "${var.image_family}-${var.build_number}-test"
    "trend-plan"      = "windows-min"
    "CommitId"        = var.commit_id
    "ImageFamily"     = var.image_family
    "BuildNumber"     = var.build_number
    "Project"         = "image-hardening"
    "InspectorID"     = "${var.image_family}-${var.build_number}"
  }
  ansible_args = [
    "-vvvvvv",
    "--diff",
    "-e",
    "ansible_connection=winrm",
    "-e", 
    "ansible_winrm_server_cert_validation=ignore",
    "-e",
    "ansible_winrm_operation_timeout_sec=3600",
    "-e",
    "ansible_winrm_read_timeout_sec=7200",
    "-e",
    "cis_min_score=${var.cis_min_score}",
    "-e",
    "cis_benchmark=${var.cis_benchmark}",
    "-e",
    "profile=${var.cis_profile}",
    "-e",
    "build_number=${var.build_number}",
    "-e",
    "commit_id=${var.commit_id}",
    "-e",
    "image_family=${var.image_family}"
  ]
}

bootstrap_win.txt

<powershell>
net user Administrator
wmic useraccount where "name='Administrator'" set PasswordExpires=FALSE
netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new enable=yes action=block
winrm delete winrm/config/listener?Address=*+Transport=HTTP  2>$Null
winrm delete winrm/config/listener?Address=*+Transport=HTTPS 2>$Null
Set-ItemProperty -Path HKLM:\Software\Policies\Microsoft\Windows\WinRM\Client -Name AllowBasic -Value 1
Set-ItemProperty -Path HKLM:\Software\Policies\Microsoft\Windows\WinRM\Client -Name AllowUnencryptedTraffic -Value 1
Set-ItemProperty -Path HKLM:\Software\Policies\Microsoft\Windows\WinRM\Service -Name AllowBasic -Value 1
Set-ItemProperty -Path HKLM:\Software\Policies\Microsoft\Windows\WinRM\Service -Name AllowUnencryptedTraffic -Value 1
winrm create winrm/config/listener?Address=*+Transport=HTTP
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="0"}'
winrm set winrm/config '@{MaxTimeoutms="7200000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service '@{MaxConcurrentOperationsPerUser="12000"}'
winrm set winrm/config/service/auth '@{Basic="true"}'
winrm set winrm/config/client/auth '@{Basic="true"}'
$Key = 'HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System'
$Setting = 'LocalAccountTokenFilterPolicy'
Set-ItemProperty -Path $Key -Name $Setting -Value 1 -Force
Stop-Service -Name WinRM
Set-Service -Name WinRM -StartupType Automatic
netsh advfirewall firewall set rule name="Windows Remote Management (HTTP-In)" new action=allow localip=any remoteip=any
Start-Service -Name WinRM
Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"
</powershell>

ptn · October 25, 2023, 5:21pm

Hi,

For starter, you might want to hide your instance public IP address from the output.

I don’t use Packer much, but your config seems right according to builder documentation.

A few questions / observations, top of my head:

Are you sure your script (user_data_file) got executed on your instance ?
Have you tried accessing your instance through winrm manually, using the same credentials ?
Have you check security groups logs, for instance using this ? On that matter, instance public IP addr (security_group_filter) doesn’t seem to be the one you’re targeting with Ansible later on. Either I misread that or you should variabilize it.

72clg · October 26, 2023, 8:36am

Hi thanks for your reply.

I’m already one step ahead regarding the IP address as I’ve ensured I’ve changed things like the public IP, account number, GitHub etc info so the real info is not on display. Also the instances get terminated after they fail so in theory the IP address doesn’t matter as next time a new one will be generated anyway.

The user data file is the same file which is run during the build phase which runs successfully the same with the security group is using the same code/settings which allows access when building (same with VPC etc)

I haven’t tried to connect due to the fact that the instance is basically built and terminated as soon as the error happens with is within a few minutes.

As mentioned it use to work until around July but saying that the Windows 2016 build did work successfully once a couple of weeks ago?

ptn · October 26, 2023, 7:05pm

Hey,

I’m already one step ahead regarding the IP address

with the security group is using the same code/settings which allows access when building

Sure, though I was saying this filter: "ip-permission.cidr" = "34.246.47.146/32" in the security_group_filter block is not the same as the one in previous output (might just be a paste from another build), and seems hardcoded to me.

I haven’t tried to connect due to the fact that the instance is basically built and terminated as soon as the error happens with is within a few minutes.

I get that, you are building an AMI, not provisioning a live server. Again, I don’t use Packer much, but could you perhaps try running builder in debug mode ?
Reading doc, I’m not sure instance stays up for debug, but if not, you could add an infinite sleep loop or tail -f /dev/null equivalent to give you time doing so.

Also perhaps consider updating Ansible; ansible-core 2.10 is EOL for a while and as much as I don’t think your issue lies here, I’ve seen dumb issues getting fixed just by upgrading packages.