Struggling with AWS Dynamic Inventory: Created hosts unresolvable

Get Help
, ,
1

I’m a beginner struggling with making AWS dynamic Inventory work for me. This is on a newly created Ubuntu 22.04 VM, Ansible 2.16.11 from PPA, amazon.aws 8.2.1 and 7.6.1 installed (not sure why two versions exist here, I installed the collection a single time). The problem is that the hosts I’m creating are not reachable and not matched in the playbook. What am I doing wrong?

Listed below are my exercise playbook (with only one task besides instance creation), ansible.cfg, and the lsn13.aws_ec2.yml dynamic inventory config file. After that, I present the output of ansible-playbook, ansible-inventory --list, and ansible all -m ping.

lsn13.yml:

---
- name: Create AWS instances
  hosts: localhost
  gather_facts: no
  tasks:

  - name: Create instances
    amazon.aws.ec2_instance:
      name: "{{ item }}"
      key_name: user18kp
      region: us-east-1
      instance_type: t2.micro
      security_group: user18sg
      network_interfaces:
       - assign_public_ip: true
      image_id: ami-0a0e5d9c7acc336f1
      wait: yes
    with_items:
      - build
      - deploy

- name: Provision build host and build app
  hosts: build
  become: yes
  tasks:

  - name: Ensure git and maven are installed
    apt:
      name:
        - git
        - maven
      state: present
      update_cache: true

ansible.cfg:

[defaults]
inventory = lsn13.aws_ec2.yml
host_key_checking = false
remote_user = ubuntu
private_key_file = /root/.ssh/user18kp.pem

lsn13.aws_ec2.yml:

plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
hostnames:
  - 'tag:Name'

Output:

. ~/secrets && ansible-playbook --flush-cache lsn13.yml

PLAY [Create AWS instances] *************************************************************************************************************************************************************************

TASK [Create instances] *****************************************************************************************************************************************************************************
changed: [localhost] => (item=build)
changed: [localhost] => (item=deploy)
[WARNING]: Could not match supplied host pattern, ignoring: build

PLAY [Provision build host and build app] ***********************************************************************************************************************************************************
skipping: no hosts matched

PLAY RECAP ******************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


ansible-inventory -i lsn13.aws_ec2.yml --list
{
    "_meta": {
        "hostvars": {
            "Elastic01": {
                "ami_launch_index": 0,
                "architecture": "x86_64",
                "block_device_mappings": [
                    {
                        "device_name": "/dev/sda1",
                        "ebs": {
                            "attach_time": "2024-08-01T16:07:17+00:00",
                            "delete_on_termination": true,
                            "status": "attached",
                            "volume_id": "vol-027caeadc08aa38dc"
                        }
                    }
                ],
                "boot_mode": "uefi-preferred",
                "capacity_reservation_specification": {
                    "capacity_reservation_preference": "open"
                },
                "client_token": "c9af4461-9a33-48bb-991e-895ededbf2e3",
                "cpu_options": {
                    "core_count": 2,
                    "threads_per_core": 1
                },
                "current_instance_boot_mode": "legacy-bios",
                "ebs_optimized": false,
                "ena_support": true,
                "enclave_options": {
                    "enabled": false
                },
                "hibernation_options": {
                    "configured": false
                },
                "hypervisor": "xen",
                "image_id": "ami-0a0e5d9c7acc336f1",
                "instance_id": "i-0b188573888c4b973",
                "instance_type": "t2.large",
                "key_name": "root-01",
                "launch_time": "2024-08-27T15:14:59+00:00",
                "maintenance_options": {
                    "auto_recovery": "default"
                },
                "metadata_options": {
                    "http_endpoint": "enabled",
                    "http_protocol_ipv6": "disabled",
                    "http_put_response_hop_limit": 2,
                    "http_tokens": "required",
                    "instance_metadata_tags": "disabled",
                    "state": "applied"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "network_interfaces": [
                    {
                        "attachment": {
                            "attach_time": "2024-08-01T16:07:16+00:00",
                            "attachment_id": "eni-attach-0431e5d04c7efb1c1",
                            "delete_on_termination": true,
                            "device_index": 0,
                            "network_card_index": 0,
                            "status": "attached"
                        },
                        "description": "",
                        "groups": [
                            {
                                "group_id": "sg-09d3e51b44970b195",
                                "group_name": "launch-wizard-6"
                            }
                        ],
                        "interface_type": "interface",
                        "ipv6_addresses": [],
                        "mac_address": "0a:ff:cc:b3:da:b7",
                        "network_interface_id": "eni-0fe807284a4a0563d",
                        "owner_id": "107744273479",
                        "private_dns_name": "ip-172-31-21-181.ec2.internal",
                        "private_ip_address": "172.31.21.181",
                        "private_ip_addresses": [
                            {
                                "primary": true,
                                "private_dns_name": "ip-172-31-21-181.ec2.internal",
                                "private_ip_address": "172.31.21.181"
                            }
                        ],
                        "source_dest_check": true,
                        "status": "in-use",
                        "subnet_id": "subnet-0481c8fb57adc815c",
                        "vpc_id": "vpc-0a48b0ba83bd3a816"
                    }
                ],
                "owner_id": "107744273479",
                "placement": {
                    "availability_zone": "us-east-1d",
                    "group_name": "",
                    "region": "us-east-1",
                    "tenancy": "default"
                },
                "platform_details": "Linux/UNIX",
                "private_dns_name": "ip-172-31-21-181.ec2.internal",
                "private_dns_name_options": {
                    "enable_resource_name_dns_a_record": true,
                    "enable_resource_name_dns_aaaa_record": false,
                    "hostname_type": "ip-name"
                },
                "private_ip_address": "172.31.21.181",
                "product_codes": [],
                "public_dns_name": "",
                "requester_id": "",
                "reservation_id": "r-0a517ab1db76e35d3",
                "root_device_name": "/dev/sda1",
                "root_device_type": "ebs",
                "security_groups": [
                    {
                        "group_id": "sg-09d3e51b44970b195",
                        "group_name": "launch-wizard-6"
                    }
                ],
                "source_dest_check": true,
                "state": {
                    "code": 80,
                    "name": "stopped"
                },
                "state_reason": {
                    "code": "Client.UserInitiatedShutdown",
                    "message": "Client.UserInitiatedShutdown: User initiated shutdown"
                },
                "state_transition_reason": "User initiated (2024-08-27 21:31:18 GMT)",
                "subnet_id": "subnet-0481c8fb57adc815c",
                "tags": {
                    "Name": "Elastic01"
                },
                "usage_operation": "RunInstances",
                "usage_operation_update_time": "2024-08-01T16:07:16+00:00",
                "virtualization_type": "hvm",
                "vpc_id": "vpc-0a48b0ba83bd3a816"
            },
            "build": {
                "ami_launch_index": 0,
                "architecture": "x86_64",
                "block_device_mappings": [
                    {
                        "device_name": "/dev/sda1",
                        "ebs": {
                            "attach_time": "2024-09-25T13:40:02+00:00",
                            "delete_on_termination": true,
                            "status": "attached",
                            "volume_id": "vol-03393f23aff981241"
                        }
                    }
                ],
                "boot_mode": "uefi-preferred",
                "capacity_reservation_specification": {
                    "capacity_reservation_preference": "open"
                },
                "client_token": "aac6f4cd64bb4925876d4e5c67139aca",
                "cpu_options": {
                    "core_count": 1,
                    "threads_per_core": 1
                },
                "current_instance_boot_mode": "legacy-bios",
                "ebs_optimized": false,
                "ena_support": true,
                "enclave_options": {
                    "enabled": false
                },
                "hibernation_options": {
                    "configured": false
                },
                "hypervisor": "xen",
                "image_id": "ami-0a0e5d9c7acc336f1",
                "instance_id": "i-0f3a6f50d503ff3ac",
                "instance_type": "t2.micro",
                "key_name": "user18kp",
                "launch_time": "2024-09-25T13:40:01+00:00",
                "maintenance_options": {
                    "auto_recovery": "default"
                },
                "metadata_options": {
                    "http_endpoint": "enabled",
                    "http_protocol_ipv6": "disabled",
                    "http_put_response_hop_limit": 1,
                    "http_tokens": "optional",
                    "instance_metadata_tags": "disabled",
                    "state": "applied"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "network_interfaces": [
                    {
                        "association": {
                            "ip_owner_id": "amazon",
                            "public_dns_name": "ec2-54-89-132-103.compute-1.amazonaws.com",
                            "public_ip": "54.89.132.103"
                        },
                        "attachment": {
                            "attach_time": "2024-09-25T13:40:01+00:00",
                            "attachment_id": "eni-attach-0ac92cbde2d703623",
                            "delete_on_termination": true,
                            "device_index": 0,
                            "network_card_index": 0,
                            "status": "attached"
                        },
                        "description": "",
                        "groups": [
                            {
                                "group_id": "sg-0956ef2aa27fb24df",
                                "group_name": "user18sg"
                            }
                        ],
                        "interface_type": "interface",
                        "ipv6_addresses": [],
                        "mac_address": "0e:81:1c:9b:14:e7",
                        "network_interface_id": "eni-018a1018fa2afff9a",
                        "owner_id": "107744273479",
                        "private_dns_name": "ip-172-31-37-45.ec2.internal",
                        "private_ip_address": "172.31.37.45",
                        "private_ip_addresses": [
                            {
                                "association": {
                                    "ip_owner_id": "amazon",
                                    "public_dns_name": "ec2-54-89-132-103.compute-1.amazonaws.com",
                                    "public_ip": "54.89.132.103"
                                },
                                "primary": true,
                                "private_dns_name": "ip-172-31-37-45.ec2.internal",
                                "private_ip_address": "172.31.37.45"
                            }
                        ],
                        "source_dest_check": true,
                        "status": "in-use",
                        "subnet_id": "subnet-068bdc5c304901a8b",
                        "vpc_id": "vpc-0a48b0ba83bd3a816"
                    }
                ],
                "owner_id": "107744273479",
                "placement": {
                    "availability_zone": "us-east-1a",
                    "group_name": "",
                    "region": "us-east-1",
                    "tenancy": "default"
                },
                "platform_details": "Linux/UNIX",
                "private_dns_name": "ip-172-31-37-45.ec2.internal",
                "private_dns_name_options": {
                    "enable_resource_name_dns_a_record": false,
                    "enable_resource_name_dns_aaaa_record": false,
                    "hostname_type": "ip-name"
                },
                "private_ip_address": "172.31.37.45",
                "product_codes": [],
                "public_dns_name": "ec2-54-89-132-103.compute-1.amazonaws.com",
                "public_ip_address": "54.89.132.103",
                "requester_id": "",
                "reservation_id": "r-0aa71f89062339c78",
                "root_device_name": "/dev/sda1",
                "root_device_type": "ebs",
                "security_groups": [
                    {
                        "group_id": "sg-0956ef2aa27fb24df",
                        "group_name": "user18sg"
                    }
                ],
                "source_dest_check": true,
                "state": {
                    "code": 16,
                    "name": "running"
                },
                "state_transition_reason": "",
                "subnet_id": "subnet-068bdc5c304901a8b",
                "tags": {
                    "Name": "build"
                },
                "usage_operation": "RunInstances",
                "usage_operation_update_time": "2024-09-25T13:40:01+00:00",
                "virtualization_type": "hvm",
                "vpc_id": "vpc-0a48b0ba83bd3a816"
            },
            "deploy": {
                "ami_launch_index": 0,
                "architecture": "x86_64",
                "block_device_mappings": [
                    {
                        "device_name": "/dev/sda1",
                        "ebs": {
                            "attach_time": "2024-09-25T13:40:08+00:00",
                            "delete_on_termination": true,
                            "status": "attached",
                            "volume_id": "vol-079863d419e5d4979"
                        }
                    }
                ],
                "boot_mode": "uefi-preferred",
                "capacity_reservation_specification": {
                    "capacity_reservation_preference": "open"
                },
                "client_token": "2da9e6786ea545099249acf4cbca44a6",
                "cpu_options": {
                    "core_count": 1,
                    "threads_per_core": 1
                },
                "current_instance_boot_mode": "legacy-bios",
                "ebs_optimized": false,
                "ena_support": true,
                "enclave_options": {
                    "enabled": false
                },
                "hibernation_options": {
                    "configured": false
                },
                "hypervisor": "xen",
                "image_id": "ami-0a0e5d9c7acc336f1",
                "instance_id": "i-0c86cc09d6c7186c4",
                "instance_type": "t2.micro",
                "key_name": "user18kp",
                "launch_time": "2024-09-25T13:40:07+00:00",
                "maintenance_options": {
                    "auto_recovery": "default"
                },
                "metadata_options": {
                    "http_endpoint": "enabled",
                    "http_protocol_ipv6": "disabled",
                    "http_put_response_hop_limit": 1,
                    "http_tokens": "optional",
                    "instance_metadata_tags": "disabled",
                    "state": "applied"
                },
                "monitoring": {
                    "state": "disabled"
                },
                "network_interfaces": [
                    {
                        "association": {
                            "ip_owner_id": "amazon",
                            "public_dns_name": "ec2-3-95-232-92.compute-1.amazonaws.com",
                            "public_ip": "3.95.232.92"
                        },
                        "attachment": {
                            "attach_time": "2024-09-25T13:40:07+00:00",
                            "attachment_id": "eni-attach-0a263144210113ba7",
                            "delete_on_termination": true,
                            "device_index": 0,
                            "network_card_index": 0,
                            "status": "attached"
                        },
                        "description": "",
                        "groups": [
                            {
                                "group_id": "sg-0956ef2aa27fb24df",
                                "group_name": "user18sg"
                            }
                        ],
                        "interface_type": "interface",
                        "ipv6_addresses": [],
                        "mac_address": "0e:48:13:ac:98:b3",
                        "network_interface_id": "eni-0357012ad53249acf",
                        "owner_id": "107744273479",
                        "private_dns_name": "ip-172-31-44-98.ec2.internal",
                        "private_ip_address": "172.31.44.98",
                        "private_ip_addresses": [
                            {
                                "association": {
                                    "ip_owner_id": "amazon",
                                    "public_dns_name": "ec2-3-95-232-92.compute-1.amazonaws.com",
                                    "public_ip": "3.95.232.92"
                                },
                                "primary": true,
                                "private_dns_name": "ip-172-31-44-98.ec2.internal",
                                "private_ip_address": "172.31.44.98"
                            }
                        ],
                        "source_dest_check": true,
                        "status": "in-use",
                        "subnet_id": "subnet-068bdc5c304901a8b",
                        "vpc_id": "vpc-0a48b0ba83bd3a816"
                    }
                ],
                "owner_id": "107744273479",
                "placement": {
                    "availability_zone": "us-east-1a",
                    "group_name": "",
                    "region": "us-east-1",
                    "tenancy": "default"
                },
                "platform_details": "Linux/UNIX",
                "private_dns_name": "ip-172-31-44-98.ec2.internal",
                "private_dns_name_options": {
                    "enable_resource_name_dns_a_record": false,
                    "enable_resource_name_dns_aaaa_record": false,
                    "hostname_type": "ip-name"
                },
                "private_ip_address": "172.31.44.98",
                "product_codes": [],
                "public_dns_name": "ec2-3-95-232-92.compute-1.amazonaws.com",
                "public_ip_address": "3.95.232.92",
                "requester_id": "",
                "reservation_id": "r-0509c0e2c81cebf27",
                "root_device_name": "/dev/sda1",
                "root_device_type": "ebs",
                "security_groups": [
                    {
                        "group_id": "sg-0956ef2aa27fb24df",
                        "group_name": "user18sg"
                    }
                ],
                "source_dest_check": true,
                "state": {
                    "code": 16,
                    "name": "running"
                },
                "state_transition_reason": "",
                "subnet_id": "subnet-068bdc5c304901a8b",
                "tags": {
                    "Name": "deploy"
                },
                "usage_operation": "RunInstances",
                "usage_operation_update_time": "2024-09-25T13:40:07+00:00",
                "virtualization_type": "hvm",
                "vpc_id": "vpc-0a48b0ba83bd3a816"
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "aws_ec2"
        ]
    },
    "aws_ec2": {
        "hosts": [
            "Elastic01",
            "deploy",
            "build"
        ]
    }
}

ansible all -m ping
Elastic01 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname elastic01: Temporary failure in name resolution",
    "unreachable": true
}
deploy | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname deploy: Temporary failure in name resolution",
    "unreachable": true
}
build | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname build: Temporary failure in name resolution",
    "unreachable": true
}

(Nevermind the “Elastic01” host, it’s an unrelated stopped instance, should have filtered it out in the inventory.)

2

Have you tried not using a list, and just trying a single build first?

For me, whenever I run an AWS call, I have to set the environment variable for AWS_PROFILE. I had issues early on and that resolved it - but I know others do not need to do it.

So for example:

gather_facts: false
environment:
  AWS_PROFILE: "{{ aws_profile_used }}"
tasks:
3

Thanks, I’ll look into setting the profile and report back.

4

UPDATE: After much experimenting (e.g. involving using keyed groups), I’ve found that the second play in the playbook does work when I re-run the playbook, but never at the first run. First I thought this was related to instances not having properly come up yet (as in, “2/2 checks passed” not yet displayed in the AWS web UI) but found that re-running works even if they are in the “Initializing” status. Inserting a wait makes no difference.

Any idea why this might happen?

5

Are these 2 tasks in the same playbook? I personally have always kept the “Create EC2” in a separate playbook, and any “Configure EC2” actions in a different playbook.

That being said, if you are running tasks in the same playbook after creating EC2s, you likely have to refresh the inventory:

- name: Refresh inventory to ensure new instance data exist in inventory
  meta: refresh_inventory

And I would add a wait, something like:

- name: Wait up to 600 seconds for target connection to become reachable/usable
  ansible.builtin.wait_for_connection:
6

Wow, this is golden. This fixed it, the playbook now runs on the first attempt. Thank you so much! How come this isn’t mentioned in the docs for the dynamic inventory plugin?

7

I should have also mentioned that the working version of this project uses keyed groups for host matching and public IP addresses instead of tag:Name for hostnames in the inventory, so the inventory config file now looks like this:

plugin: amazon.aws.aws_ec2
regions:
  - us-east-1
hostnames:
  - ip-address
keyed_groups:
  - key: tags.Name
    separator: ''
filters:
  instance-state-name: running

I couldn’t make it work using tag:Name in hostnames. I guess it’s impossible at least when accessing the hosts via the Internet, which is what I do here.