Amazon.aws collection - using sts credential for ec2_instance module authentication

Get Help
1

Hey all! Running into what appears to me to be a bug, but may in fact be user error with the amazon.aws.ec2_instance module.

Task: Provision an EC2 instance using the ec2_instance module
Result: An EC2 instance in my AWS Account
Context:

  • Python Version: 3.12.8
  • Boto3/botocore Version: 1.35.80
  • Ansible Version: 9.13.0
  • amazon.aws collection Version: 9.3.0

Steps To Reproduce:

# Playbook Snippets
- name: Obtain STS Credential
  hosts: localhost
  tasks:
    - name: Create STS Token
      amazon.aws.sts_assume_role:
        role_arn: "{{ aws_role_arn }}" # Set in full playbook
        role_session_name: "{{ aws_session_name }}" # Set in full playbook
        profile: "{{ aws_profile }}" # Set in full playbook
      register: _sts_credential_response

- Name: Create EC2 Instance
  hosts: localhost
  environment:
    AWS_ACCESS_KEY_ID: "{{ _sts_credential_response.sts_creds.access_key }}"
    AWS_SECRET_ACCESS_KEY: "{{ _sts_credential_response.sts_creds.secret_key }}"
    AWS_SESSION_TOKEN: "{{ _sts_credential_response.sts_creds.session_token }}"
    AWS_DEFAULT_REGION: "{{ aws_region }}" # Set in full playbook
  tasks:
    - name: Create EC2 Instance
      amazon.aws.ec2_instance:
        name: "{{ ec2_instance_name }}" # Set in full playbook
        # ... full list of parameters
      register: _ec2_instance_response

Expected Behavior: The EC2 instance is created and the expected return of the module is registered in _ec2_instance_response

Actual Behavior: receive error: “Failed to describe instances: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials”

Verbose Logs:

Using module file /root/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/ec2_instance.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-23778btuz5n38/tmpyvb3mvhy TO /root/.ansible/tmp/ansible-tmp-1741969693.43713-24081-88772827775964/AnsiballZ_ec2_instance.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1741969693.43713-24081-88772827775964/ /root/.ansible/tmp/ansible-tmp-1741969693.43713-24081-88772827775964/AnsiballZ_ec2_instance.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'AWS_ACCESS_KEY_ID=<redacted_from_log> AWS_SECRET_ACCESS_KEY=<reacted_from_log> AWS_SESSION_TOKEN=<redacted_from_log> AWS_DEFAULT_REGION=<redacted_from_log> /work/venv/bin/python /root/.ansible/tmp/ansible-tmp-1741969693.43713-24081-88772827775964/AnsiballZ_ec2_instance.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1741969693.43713-24081-88772827775964/ > /dev/null 2>&1 && sleep 0'

Stack Trace From Log:

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/errors.py", line 41, in handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/errors.py", line 71, in handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
           ^^^^^^^^^^^^
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
           ^^^^^^
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/ec2.py", line 768, in describe_instances
    return paginator.paginate(**params).build_full_result()["Reservations"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/venv/lib64/python3.12/site-packages/botocore/paginate.py", line 479, in build_full_result
    for response in self:
                    ^^^^
  File "/work/venv/lib64/python3.12/site-packages/botocore/paginate.py", line 269, in __iter__
    response = self._make_request(current_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/venv/lib64/python3.12/site-packages/botocore/paginate.py", line 357, in _make_request
    return self._method(**current_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/venv/lib64/python3.12/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/work/venv/lib64/python3.12/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_instance.py", line 2869, in main
    existing_matches = find_instances(client, module, filters=filters)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_instance.py", line 2071, in find_instances
    for reservation in describe_instances(client, **params):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_afd5418g/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/errors.py", line 45, in handler
    raise cls._CUSTOM_EXCEPTION(message=f"Failed to {description}", exception=e) from e
ansible_collections.amazon.aws.plugins.module_utils.ec2.AnsibleEC2Error: Failed to describe instances: An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS was not able to validate the provided access credentials

As seen in the logs, the environment variables set for Ansible are being properly captured to pass to the module. The stack trace points to the ec2 utility module, but I’d be really surprised if this is an actual bug in what I would imagine is a common positive path for using session tokens as an AWS authentication method. Further validation of the STS methodolgy in my setup is evidenced by the successful use of other amazon.aws modules in the same playbook (eg. amazon.aws.ec2_instance_info, amazon.aws.ec2_vpc_subnet_info, and numerous others).

Has anyone impleented this STS pattern or anything similar and run into this isuse with this or any other Amazon collection modules? Thanks in advance for the engagement and help! :slight_smile:

2

Hopefully you solved it already, but in case not, this error can occur if the credentials provided conflict. It could be caused by passing credential options to the task, which have precedence over the env vars. I’m not able to reproduce as described though, so there’s some missing context.

Check no credential options are directly passed to that task.

If that doesn’t fix it, try setting module_defaults on the task to ensure any module_defaults set on a play/block aren’t causing the credential conflict:

    - name: Create EC2 Instance
      amazon.aws.ec2_instance:
        name: "{{ ec2_instance_name }}" # Set in full playbook
        # ... full list of parameters
      register: _ec2_instance_response
      module_defaults:
        group/aws: {}
        amazon.aws.ec2_instance: {}

For security, I use module_defaults on a play/block instead of using environment vars, like amazon.aws/tests/integration/targets/ec2_instance_type/tasks/main.yml at 9.3.0 · ansible-collections/amazon.aws · GitHub.

It’s also can be convenient to use a ~/.aws/config profile instead of manually assuming roles:

# ~/.aws/credentials
[shertel]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx

# ~/.aws/config
[profile ec2]
role_arn = arn:aws:iam::xxxx:role/rw_ec2
source_profile = shertel

    - name: Create EC2 Instance
      amazon.aws.ec2_instance:
        profile: ec2  # assumes role for me
        name: "{{ ec2_instance_name }}" # Set in full playbook
        # ... full list of parameters
      register: _ec2_instance_response