Ansible Builtin Modules not found using Python PlaybookExecutor

hi, folks

On a RHEL 9 system using Python 3.12, a private virtual environment and latest ansible-core (1.18)
and the PlaybookExecutor API, I get errors if I call ansible.builtin modules full qualified in my playbook.

Error message is:
couldn't resolve module/action 'ansible.builtin.command'. This often indicates a misspelling, missing collection, or incorrect module path.

the playbook is as follows:

- name: Test Playbook
  hosts: localhost
  tasks:
    - name: Echo a message
      ansible.builtin.command: echo "Hello from Ansible"

Python code is as follows:

from ansible.executor.playbook_executor import PlaybookExecutor
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager
from ansible.vars.manager import VariableManager
import sys
import ansible_runner
import os
from ansible.playbook.play import Play
from ansible import context
from ansible.module_utils.common.collections import ImmutableDict
from ansible.plugins.loader import module_loader
from ansible.executor.task_queue_manager import TaskQueueManager
from ansible.plugins.callback import CallbackBase

mypath="~/automation/.venv/lib/python3.12/site-packages/ansible/modules/:."


def run_playbook(playbook_path):

    context.CLIARGS = ImmutableDict({
        'connection': 'local',
        'module_path': [mypath],
        'forks': 10,
        'become': None,
        'become_method': None,
        'become_user': None,
        'check': False,
        'diff': False,
        'syntax': None,
        'start_at_task': None
    })
    loader = DataLoader()
    os.environ['ANSIBLE_COLLECTIONS_PATHS'] = mypath
    os.environ['ANSIBLE_LIBRARY'] = mypath
    inventory = InventoryManager(loader=loader, sources=["localhost,"])
    variable_manager = VariableManager(loader=loader, inventory=inventory)
        
    executor = PlaybookExecutor(
        playbooks=[playbook_path],
        inventory=inventory,
        variable_manager=variable_manager,
        loader=loader,
        passwords={}
    )

    executor.run()

if __name__ == "__main__":
    run_playbook("ansible_test/test_playbook.yml")

when I run the playbook natively, its working:
ansible-playbook test_playbook.yml -i “localhost,”

the crazy thing: When I use non-qualified module, it also works from Python:

- name: Test Playbook
  hosts: localhost
  tasks:
    - name: Echo 
      command: echo "This works fine !!!!!"

this tells me that the builtin modules are installed correctly in my environment, but not accessible within Python runtime - although I set (and verified) the module path.

any help is apprechiated.

Kind regards
Stephan

Hi
What say ansible --version ?
Probably your modules path isn’t properly configured.

Also your code seems to be based on venv, so provably your modules path isn’t updated into the venv.

Interfacing with ansible module directly can be quite flaky in the future as there are no promises regarding the api and how everything is put together.

I’d rather suggest to take a look at Ansible runner which provides stable interface to run ansible programmatically.

1 Like

result is as follows:

ansible --version
ansible [core 2.18.3]
config file = /home/d7yc7x3@xxx/.ansible.cfg
configured module search path = [‘/home/d7yc7x3@xxx/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]
ansible python module location = /home/d7yc7x3@xxx/automation/.venv/lib64/python3.12/site-packages/ansible
ansible collection location = /home/d7yc7x3@xxx/.ansible/collections:/usr/share/ansible/collections
executable location = /home/d7yc7x3@xxx/automation/.venv/bin/ansible
python version = 3.12.5 (main, Dec 3 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/home/d7yc7x3@xxx/automation/.venv/bin/python)
jinja version = 3.1.5
libyaml = True

noticed, but ansible-runner is too slow for our usecase.

some more details:

[d7yc7x3@Kallie02i modules]$ ansible-config dump | grep COLLECTIONS_PATHS
COLLECTIONS_PATHS(/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/.ansible.cfg) = ['/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib64/python3.12/site-packages/ansible/modules']
[d7yc7x3@Kallie02i modules]$ ansible-test integration ping
FATAL: The current working directory must be within the source tree being tested.

Testing an Ansible collection: {...}/ansible_collections/{namespace}/{collection}/
Example #1: community.general -> ~/code/ansible_collections/community/general/
Example #2: ansible.util -> ~/.ansible/collections/ansible_collections/ansible/util/

Current working directory: /home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib/python3.12/site-packages/ansible/modules/
No "ansible_collections" parent directory was found.

in my modules dir, all moduls are present:

[d7yc7x3@Kallie02i modules]$ ls /home/d7yc7x3@xxx/automation/.venv/lib64/python3.12/site-packages/ansible/modules
add_host.py           fail.py             known_hosts.py    setup.py
apt_key.py            fetch.py            lineinfile.py     shell.py
apt.py                file.py             meta.py           slurp.py
apt_repository.py     find.py             mount_facts.py    stat.py
assemble.py           gather_facts.py     package_facts.py  subversion.py
assert.py             getent.py           package.py        systemd.py
async_status.py       get_url.py          pause.py          systemd_service.py
async_wrapper.py      git.py              ping.py           sysvinit.py
blockinfile.py        group_by.py         pip.py            tempfile.py
command.py            group.py            __pycache__       template.py
copy.py               hostname.py         raw.py            unarchive.py
cron.py               import_playbook.py  reboot.py         uri.py
deb822_repository.py  import_role.py      replace.py        user.py
debconf.py            import_tasks.py     rpm_key.py        validate_argument_spec.py
debug.py              include_role.py     script.py         wait_for_connection.py
dnf5.py               include_tasks.py    service_facts.py  wait_for.py
dnf.py                include_vars.py     service.py        yum_repository.py
dpkg_selections.py    __init__.py         set_fact.py
expect.py             iptables.py         set_stats.py

so, it’s pretty native what I am doing here :frowning: :cold_sweat:

It seems that ansibleconfigured module search path (seen in ansible --version) doesn’t include the path you’re using.

Ansible configured module search path is : [‘/home/d7yc7x3@xxx/.ansible/plugins/modules’, ‘/usr/share/ansible/plugins/modules’]

versus
what you show/use is : /home/d7yc7x3@xxx/automation/.venv/lib64/python3.12/site-packages/ansible/modules

I’m using venv too, however my configuration for module search path is the same as yours in ansible --version output, and everything works well.

I think you got a misconfiguration with your venv.

thanks, Gael.

but that was obvious not the solution. I have change the module path in ansible.cfg but without result to my issue:

ansible --version
ansible [core 2.18.3]
  config file = /home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/.ansible.cfg
  configured module search path = ['/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib64/python3.12/site-packages/ansible/modules', '/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib/python3.12/site-packages/ansible/modules', '/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib/python3.12/site-packages/ansible/plugins', '/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/.local/lib/python3.12/site-packages/ansible/plugins', '/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib/python3.12/site-packages/ansible', '/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib/python3.12/site-packages']
  ansible python module location = /home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib64/python3.12/site-packages/ansible
  ansible collection location = /home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/lib64/python3.12/site-packages/ansible/modules
  executable location = /home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/bin/ansible
  python version = 3.12.5 (main, Dec  3 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/home/d7yc7x3@shsr.ing.private.cloud.vwgroup.com/automation/.venv/bin/python)
  jinja version = 3.1.5
  libyaml = True

in addtion to my first mail: The error occurs on ansible-core (2.18). And it works with/until ansible-core 2.14.17.

coughinit_plugin_loadercough

Excuse me, might be catching a cold :face_with_thermometer: :wink:

hello Sivel - you anwer was pretty unclear. My issue is still open and any help is appreciated.

@Stephan4711 i had a look on various issues regarding yours, however the official doc says Python API isn’t for external use unfortunately.

You probably better move to ansible-runner as @kristianheljas said.

By the way, why do you say it’s too slow for your usecase ? :slight_smile:

1 Like

Thanks, Gael. I have overseen this usage hint.

Runner is fine, it just takes some more seconds during initialzation , but that’s ok .

1 Like

What sivel is truing to say thet you are missing init_plugin_loader function call.

from ansible.plugins.loader import init_plugin_loader
init_plugin_loader([])

See cli run method

And regarding ansible-runner:

Are you intefacing ansible-runner using python?
The example code below only costs few hundred milliseconds to start:

import ansible_runner

r = ansible_runner.run(
    project_dir='/my/project/dir',
    playbook='playbooks/site.yml'
)

print("{}: {}".format(r.status, r.rc))
# successful: 0
for each_host_event in r.events:
    print(each_host_event['event'])
print("Final status:")
print(r.stats)
1 Like