# Slow Dynamic Inventory Performance with Execution Environments

**URL:** <https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018>\
**Category:** Get Help\
**Tags:** rhel, vmware, ansible-navigator, fedora, inventory, ee\
**Created:** [April 24, 2025, 9:42pm UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018 "2025-04-24T21:42:04Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![eddiejennings](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/eddiejennings/32/7960_2.png) [@eddiejennings](https://forum.ansible.com/u/eddiejennings)\
**Post date:** [April 24, 2025, 9:42pm UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/1 "2025-04-24T21:42:04Z")

</div>

I am in the process of testing the use of execution environments to eventually transition my environment into using `ansible-navigator`. I’ve noticed an unexpected behavior regarding my execution environment and the `vmware.vmware.vms` dynamic inventory plugin. I notice that it takes my execution environment about 9x more time to generate the dynamic inventory compared to using `ansible-inventory` without the execution environment.

Has anyone else experienced this?

I’ve tried to test as close to apples-to-apples as I can. Below are the parameters of my tests. My initial guess is that there may be something with how networking works with the execution environment container that’s adding some kind of non-trivial latency.

I have two VMs. Both are RHEL 9.5. VM1 has Ansible core installed from PyPi. VM2 has Ansible Navigator installed from PyPi with an execution environment built from Fedora 41. I intended to use RHEL UBI 9 as the base for the execution environment, but the current Ansible Navigator doesn’t like that, but that’s an issue for another day 🙂 .

**Results according to time**  
Using an execution environment.

```auto
time ansible-navigator inventory -i vmware_vms.yml --graph -m stdout
real 18m24.069s
user 0m1.259s
sys 0m0.758s

```

Without an execution environment.

```auto
time ansible-inventory -i vmware_vms.yml --graph
real 2m32.579s
user 0m42.744s
sys 0m4.594s

```

**The dynamic inventory configuration:**

```auto
[me@both-vms ansible]$ cat vmware_vms.yml
---
plugin: vmware.vmware.vms
strict: true
hostname: fqdn.of.vcenter
username: the_vcenter_user
password: vault-encrypted_vcenter_password
validate_certs: False

filters:
  - "tags['urn:vmomi:InventoryServiceTag:THEUUID:GLOBAL'] is not defined"
hostnames:
  - "name"
properties:
  - "name"
  - "guest.ipAddress"
keyed_groups: []
gather_tags: True
groups:
  datacenter1: "'/datacenter1' in path"
  datacenter2: "'/datacenter2' in path"
  datacenter3: "'/datacenter3' in path"
...

```

**Ansible information from the execution environment:**

```auto
[me@VM2 ansible]$ podman run -it fqdn.to.ee:v0.1 ansible --version
ansible [core 2.18.4]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.13/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.13.2 (main, Feb 4 2025, 00:00:00) [GCC 14.2.1 20250110 (Red Hat 14.2.1-7)] (/usr/bin/python3.13)
  jinja version = 3.1.6
  libyaml = True

```

**Ansible information for VM1 that’s not using the execution environment:**

```auto
[me@VM1 ansible]$ ansible --version
ansible [core 2.18.5]
  config file = /home/me/ansible/ansible.cfg
  configured module search path = ['/home/me/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/venv-ansible/lib64/python3.12/site-packages/ansible
  ansible collection location = /home/me/ansible/collections
  executable location = /opt/venv-ansible/bin/ansible
  python version = 3.12.5 (main, Apr 2 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] (/opt/venv-ansible/bin/python3.12)
  jinja version = 3.1.6
  libyaml = True

```

**The execution-environment.yml file used to build the execution environment:**

```auto
---
version: 3
images:
  base_image:
    name: quay.io/fedora/fedora-minimal:41-x86_64
dependencies:
  python_interpreter:
    package_system: "python3"
    python_path: "/usr/bin/python3.13"
  ansible_core:
    package_pip: ansible-core
  ansible_runner:
    package_pip: ansible-runner
  galaxy:
    collections:
      - name: ansible.utils
      - name: ansible.posix
      - name: ansible.windows
      - name: community.crypto
      - name: community.general
      - name: community.vmware
      - name: community.windows
      - name: microsoft.ad
      - name: vmware.vmware
  system:
    - git-core
    - sshpass
additional_build_steps:
  append_final:
    - LABEL org.opencontainers.image.source=https://fqdn.to.my.ansible.repo
    - LABEL org.opencontainers.image.description="My descripition"   
...

```

---

<div class="post-metadata">

**Author:** ![wayt](https://avatars.discourse-cdn.com/v4/letter/w/3be4f8/32.png) [@wayt](https://forum.ansible.com/u/wayt)\
**Post date:** [April 25, 2025, 4:22am UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/2 "2025-04-25T04:22:25Z")

</div>

Is it possible for you to enter the execution environment, container, and time `ansible-inventory` from within it? My hunch is you’re observing the initialization of the container runtime.

---

<div class="post-metadata">

**Author:** ![eddiejennings](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/eddiejennings/32/7960_2.png) [@eddiejennings](https://forum.ansible.com/u/eddiejennings)\
**Post date:** [April 25, 2025, 2:07pm UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/3 "2025-04-25T14:07:42Z")

</div>

> Is it possible for you to enter the execution environment, container, and time `ansible-inventory` from within it? My hunch is you’re observing the initialization of the container runtime.

That’s a good idea and it turns out I can. I launched a container using information from the ansible-navigator log file and ran the inventory command. The result was basically the same unfortunately.

```auto
[me@VM02 ansible]$ podman run --rm --tty --interactive -v /home/me/ansible/:/home/me/ansible/ --workdir /home/me/ansible -v /home/me/ansible/playbooks/:/home/me/ansible/playbooks/ -v /home/me/ansible/inventory/:/home/me/ansible/inventory/ -v /home/me/.ssh/:/home/runner/.ssh/ -v /home/me/.ssh/:/root/.ssh/ --group-add=root --ipc=host --user=root fqdn.to.my.execution-environment:v0.1 /bin/bash
bash-5.2# ls
CODEOWNERS ansible-navigator.log ansible.cfg inventory profile.pstats tasks vmware_vms.yml
README.md ansible-navigator.yml files playbooks roles vault
bash-5.2# time ansible-inventory -i vmware_vms.yml --graph

```

And the result

```auto
real 16m59.169s
user 0m54.647s
sys 0m4.863s

```

---

<div class="post-metadata">

**Author:** ![r3dact3d](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/r3dact3d/32/7986_2.png) [@r3dact3d](https://forum.ansible.com/u/r3dact3d)\
**Post date:** [April 26, 2025, 3:23am UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/4 "2025-04-26T03:23:57Z")

</div>

Can you check the ansible-navigator.yml config file for the image pull policy? It could be pulling the image on `always` instead of only on `missing`.

---

<div class="post-metadata">

**Author:** ![eddiejennings](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/eddiejennings/32/7960_2.png) [@eddiejennings](https://forum.ansible.com/u/eddiejennings)\
**Post date:** [April 26, 2025, 3:49am UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/5 "2025-04-26T03:49:18Z")

</div>

My pull policy is set to `missing`. I submitted a bug to the [vmware.vmware project.](https://github.com/ansible-collections/vmware.vmware/issues/172) Hopefully they’ll have some troubleshooting suggestions or can replicate the behavior.

---

<div class="post-metadata">

**Author:** ![wayt](https://avatars.discourse-cdn.com/v4/letter/w/3be4f8/32.png) [@wayt](https://forum.ansible.com/u/wayt)\
**Post date:** [April 26, 2025, 4:42pm UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/6 "2025-04-26T16:42:28Z")

</div>

Are the two RHEL VMs on the same hypervisor? Do they have the same virtual hardware? Is the system running ansible-navigator talking to a container runtime like podman locally or is it using a separate machine, podman machine?

Is there a difference in Python version you are using in the execution environment to the RHEL 9 VM running ansible-core?

---

<div class="post-metadata">

**Author:** ![eddiejennings](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/eddiejennings/32/7960_2.png) [@eddiejennings](https://forum.ansible.com/u/eddiejennings)\
**Post date:** [April 28, 2025, 3:46pm UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/7 "2025-04-28T15:46:03Z")

</div>

Yes, both VMs are on the same hypervisor, have the same VM hardware configuration, and on the same subnet. The one running `ansible-navigator` is using a container that’s downloaded locally to the VM (by Podman). Python configuration between the execution environment and the VM does differ. The VM has 3.12.5 and the execution environment uses 3.13.2, since the execution environment is built from Fedora 41 and the VM is RHEL 9.5

One thing that’s curious is I’ve also tested this with an execution environment built from `ghcr.io/ansible/community-ansible-dev-tools` and the behavior is the same.

What I’m most curious to learn is if anyone else who uses the `vmware.vmware.vms` inventory plugin has noticed such a performance difference with an execution environment.

---

<div class="post-metadata">

**Author:** ![eddiejennings](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/eddiejennings/32/7960_2.png) [@eddiejennings](https://forum.ansible.com/u/eddiejennings)\
**Post date:** [April 28, 2025, 9:57pm UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/8 "2025-04-28T21:57:48Z")

</div>

I had a suspicion, which resulted in what seems like a breakthrough.

My suspicion is that the abstracted network bits used for a container’s network stack might be killing the performance, since all other things seem equal (python version, etc.).

I did a test with adding this container option to the execution environment block in the `ansible-navigator.yml` configuration file.

```auto
container-options:
  - "--network=host"

```

With that change, the performance now matches `ansible-navigator inventory` without the execution environment. Of course keeping this configuration presents a question about defeating an inherent security feature of the execution environment container, but at least now I know the general reason behind the performance issue.

Thanks to all of your suggestions!

---

<div class="post-metadata">

**Author:** ![wayt](https://avatars.discourse-cdn.com/v4/letter/w/3be4f8/32.png) [@wayt](https://forum.ansible.com/u/wayt)\
**Post date:** [April 29, 2025, 12:30am UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/9 "2025-04-29T00:30:17Z")

</div>

Awesome find! Good work. I’d be curious what the runtime folks say in regards to performance. I assume this is podman and user mode networking, probably slirp4netns?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/ansible/original/1X/1882d9a01da5d8996a862359b0913cc568a39b84.svg) [@system](https://forum.ansible.com/u/system)\
**Post date:** [May 29, 2025, 12:31am UTC](https://forum.ansible.com/t/slow-dynamic-inventory-performance-with-execution-environments/42018/10 "2025-05-29T00:31:04Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.