How to use _ANSIBALLZ_DEBUGPY_CONFIG / debugpy to debug & develop ansible.netcommon network_cli module

While developing an Ansible module that uses ansible.netcommon / network_cli connection plugin, I need to debug what is going on with the connection flow and internal module, actions & plugin code. The particular combination of AnsiballZ zip file and local execution with Ansible network_cli connection type seems to add some extra complexity to navigate when trying to debug this.

This guide gives some helpful instructions. Yet, due to how AnsiballZ deals with the internal zipped module code paths seems to cause problems for the debugger when it tries to find the code files being executed.

I have instrumented my network_cli plugin code using debugpy like so:

def run_commands(module, commands, check_rc=True):
    # BEGIN DEBUG INSTRUMENTATION
    import debugpy

    debugpy.listen(("0.0.0.0", 5678))
    debugpy.wait_for_client()
    debugpy.breakpoint()
    # END DEBUG INSTRUMENTATION

    connection = get_connection(module)
    try:
        return connection.run_commands(commands=commands, check_rc=check_rc)
    except ConnectionError as exc:
        module.fail_json(msg=to_text(exc))

.vscode/launch.json:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python Debugger: Remote Attach",
            "type": "debugpy",
            "request": "attach",
            "connect": {
                "host": "127.0.0.1",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "${workspaceFolder}" // does not work
                    // "remoteRoot": "." // also did not work
                    // Maybe a hack like this could work... but the tmp dir is random at runtime...
                    // "remoteRoot": "/tmp/ansible_lyraphase.opnsense.facts_payload_XXXXXXXX/ansible_lyraphase.opnsense.facts_payload.zip/.../path/to/local/python/code"
                }
            ]
        }
    ]
}

When I try each of the following, I run into a different roadblock:

  1. Running the debugpy-instrumented module via a playbook seems to work, but paths are given within a random temporary directory path, and within the Ansiballz payload.zip. Thus, VSCode debugger fails to find the local python file path to debug. For example:

    Could not load source '/tmp/ansible_lyraphase.opnsense.facts_payload_sc4ozorn/ansible_lyraphase.opnsense.facts_payload.zip/ansible/module_utils/connection.py': Unable to retrieve source for /tmp/ansible_lyraphase.opnsense.facts_payload_sc4ozorn/ansible_lyraphase.opnsense.facts_payload.zip/ansible/module_utils/connection.py.
    
  2. Unpacking the module and running it manually (as in the previously mentioned guide) results in an error when the temporary Ansible connection socket is not found:

    $ python3 AnsiballZ_opnsense_facts.py explode
    Module expanded into:
    /home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir
    
    $ python3 AnsiballZ_opnsense_facts.py execute
    
    {"failed": true, "msg": "socket path /home/exampleuser/.ansible/pc/a6f4bd23f0 does not exist or cannot be found. See Troubleshooting socket path issues in the Network Debug and Troubleshooting Guide", "exception": "  File \"/home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/network.py\", line 217, in get_capabilities\n    capabilities = Connection(module._socket_path).get_capabilities()\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir/ansible/module_utils/connection.py\", line 177, in __rpc__\n    response = self._exec_jsonrpc(name, *args, **kwargs)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir/ansible/module_utils/connection.py\", line 124, in _exec_jsonrpc\n    raise ConnectionError(\n", "invocation": {"module_args": {"gather_subset": ["default"], "gather_network_resources": null, "opnsense_shell_option": null, "passwords": null}}}
    
Expand for stacktrace with newlines rendered
File "/home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir/ansible_collections/ansible/netcommon/plugins/module_utils/network/common/network.py", line 217, in get_capabilities
    capabilities = Connection(module._socket_path).get_capabilities()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir/ansible/module_utils/connection.py", line 177, in __rpc__
    response = self._exec_jsonrpc(name, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exampleuser/src/pub/ansible/lyraphase-ansible-playbooks/tmp/debug_dir/ansible/module_utils/connection.py", line 124, in _exec_jsonrpc
    raise ConnectionError(
", "invocation": {"module_args": {"gather_subset": ["default"], "gather_network_resources": null, "opnsense_shell_option": null, "passwords": null}}}

The first case seems to work, as the VSCode debugger can connect to debugpy server port 5678, and the breakpoint, variables, and call stack are visible in the VSCode debug UI. Yet, the python code file cannot be found to show the currently executing line to step through.

The second case (explode + execute) fails because it expects Ansible to have setup a socket file (via task_executor.py + ansible_connection_cli_stub.py perhaps?)

Searching through Ansible’s source code, I found mention of an environment variable that seems related to this: _ANSIBALLZ_DEBUGPY_CONFIG

_ANSIBALLZ_DEBUGPY_CONFIG:
  name: Configure the AnsiballZ remote debugging extension for debugpy
  description:
    - Enables and configures the AnsiballZ remote debugging extension for debugpy.
    - This is for internal use only.

How can this be used to configure debugpy for debugging of AnsiballZ-packed code?

1 Like

_ANSIBALLZ_DEBUGPY_CONFIG

This is a newly released feature that is mostly kept as an internal API but have currently exposed a more public interface through ansible-test and the --dev-debug-on-demand argument. To use this in VSCode try to run this launch configuration

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "ansible-test shell module debugging",
            "type": "debugpy",
            "request": "launch",
            "module": "ansible",
            "args": ["test", "shell", "--dev-debug-on-demand"]
        },
    ]
}

This will launch an interactive shell in VSCode that has set the env var _ANSIBALLZ_DEBUGPY_CONFIG with the required values needed for Ansible to attach back to the running debug session like the below.

I ran the above launch configuration then in the terminal below I just ran ansible localhost -m ping and the breakpoint I have set in the ping.py module is hit when run. It deals with all the required path mappings for you and you can step through the code, including action plugins, the main Ansible process, any worker code, plus the module execution.

What I have no tested is this with networking modules. There’s a chance that this doesn’t work as I’ve not got an environment to test them against and there’s a chance that how the modules are run won’t work with how this is all setup. But AFAIK for networking, the module code is run on localhost so the existing ansible-test shell --dev-debug-on-demand should just have it all hooked up to work normally. If it doesn’t we can certainly try and look into what’s happening to make it work.

I forgot to mention, you can manually set this config and setup the required debug server. The documentation for this config option is found in ansible/lib/ansible/module_utils/_internal/_ansiballz/_extensions/_debugpy.py at 89ba882b085a47a8cd064972dfe3348249652686 · ansible/ansible · GitHub and shows you a launch configuration that sets up a debugpy listener and a configuration that launches your playbook (or adhoc command) to connect back to that listener. Please keep in mind this is not a public API and is subject to change.