Slow Dynamic Inventory Performance with Execution Environments

I am in the process of testing the use of execution environments to eventually transition my environment into using ansible-navigator. I’ve noticed an unexpected behavior regarding my execution environment and the vmware.vmware.vms dynamic inventory plugin. I notice that it takes my execution environment about 9x more time to generate the dynamic inventory compared to using ansible-inventory without the execution environment.

Has anyone else experienced this?

I’ve tried to test as close to apples-to-apples as I can. Below are the parameters of my tests. My initial guess is that there may be something with how networking works with the execution environment container that’s adding some kind of non-trivial latency.

I have two VMs. Both are RHEL 9.5. VM1 has Ansible core installed from PyPi. VM2 has Ansible Navigator installed from PyPi with an execution environment built from Fedora 41. I intended to use RHEL UBI 9 as the base for the execution environment, but the current Ansible Navigator doesn’t like that, but that’s an issue for another day :slight_smile: .

Results according to time
Using an execution environment.

time ansible-navigator inventory -i vmware_vms.yml --graph -m stdout
real    18m24.069s
user    0m1.259s
sys     0m0.758s

Without an execution environment.

time ansible-inventory -i vmware_vms.yml --graph
real    2m32.579s
user    0m42.744s
sys     0m4.594s

The dynamic inventory configuration:

[me@both-vms ansible]$ cat vmware_vms.yml
---
plugin: vmware.vmware.vms
strict: true
hostname: fqdn.of.vcenter
username: the_vcenter_user
password: vault-encrypted_vcenter_password
validate_certs: False

filters:
  - "tags['urn:vmomi:InventoryServiceTag:THEUUID:GLOBAL'] is not defined"
hostnames:
  - "name"
properties:
  - "name"
  - "guest.ipAddress"
keyed_groups: []
gather_tags: True
groups:
  datacenter1: "'/datacenter1' in path"
  datacenter2: "'/datacenter2' in path"
  datacenter3: "'/datacenter3' in path"
...

Ansible information from the execution environment:

[me@VM2 ansible]$ podman run -it fqdn.to.ee:v0.1 ansible --version
ansible [core 2.18.4]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.13/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.13.2 (main, Feb  4 2025, 00:00:00) [GCC 14.2.1 20250110 (Red Hat 14.2.1-7)] (/usr/bin/python3.13)
  jinja version = 3.1.6
  libyaml = True

Ansible information for VM1 that’s not using the execution environment:

[me@VM1 ansible]$ ansible --version
ansible [core 2.18.5]
  config file = /home/me/ansible/ansible.cfg
  configured module search path = ['/home/me/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/venv-ansible/lib64/python3.12/site-packages/ansible
  ansible collection location = /home/me/ansible/collections
  executable location = /opt/venv-ansible/bin/ansible
  python version = 3.12.5 (main, Apr  2 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] (/opt/venv-ansible/bin/python3.12)
  jinja version = 3.1.6
  libyaml = True

The execution-environment.yml file used to build the execution environment:

---
version: 3
images:
  base_image:
    name: quay.io/fedora/fedora-minimal:41-x86_64
dependencies:
  python_interpreter:
    package_system: "python3"
    python_path: "/usr/bin/python3.13"
  ansible_core:
    package_pip: ansible-core
  ansible_runner:
    package_pip: ansible-runner
  galaxy:
    collections:
      - name: ansible.utils
      - name: ansible.posix
      - name: ansible.windows
      - name: community.crypto
      - name: community.general
      - name: community.vmware
      - name: community.windows
      - name: microsoft.ad
      - name: vmware.vmware
  system:
    - git-core
    - sshpass
additional_build_steps:
  append_final:
    - LABEL org.opencontainers.image.source=https://fqdn.to.my.ansible.repo
    - LABEL org.opencontainers.image.description="My descripition"   
...

Is it possible for you to enter the execution environment, container, and time ansible-inventory from within it? My hunch is you’re observing the initialization of the container runtime.

Is it possible for you to enter the execution environment, container, and time ansible-inventory from within it? My hunch is you’re observing the initialization of the container runtime.

That’s a good idea and it turns out I can. I launched a container using information from the ansible-navigator log file and ran the inventory command. The result was basically the same unfortunately.

[me@VM02 ansible]$ podman run --rm --tty --interactive -v /home/me/ansible/:/home/me/ansible/ --workdir /home/me/ansible -v /home/me/ansible/playbooks/:/home/me/ansible/playbooks/ -v /home/me/ansible/inventory/:/home/me/ansible/inventory/ -v /home/me/.ssh/:/home/runner/.ssh/ -v /home/me/.ssh/:/root/.ssh/ --group-add=root --ipc=host --user=root fqdn.to.my.execution-environment:v0.1 /bin/bash
bash-5.2# ls
CODEOWNERS  ansible-navigator.log  ansible.cfg  inventory  profile.pstats  tasks  vmware_vms.yml
README.md   ansible-navigator.yml  files        playbooks  roles           vault
bash-5.2# time ansible-inventory -i vmware_vms.yml --graph

And the result

real    16m59.169s
user    0m54.647s
sys     0m4.863s

Can you check the ansible-navigator.yml config file for the image pull policy? It could be pulling the image on always instead of only on missing.

My pull policy is set to missing. I submitted a bug to the vmware.vmware project. Hopefully they’ll have some troubleshooting suggestions or can replicate the behavior.

Are the two RHEL VMs on the same hypervisor? Do they have the same virtual hardware? Is the system running ansible-navigator talking to a container runtime like podman locally or is it using a separate machine, podman machine?

Is there a difference in Python version you are using in the execution environment to the RHEL 9 VM running ansible-core?