Background
The Python packaging ecosystem has recently adopted PEP668 as a solution to the long-standing problem of packaging tools (e.g., pip) clobbering dependencies in shared Python environments managed by an OS package manager (e.g., dnf). PEP668 defines a standard way for a Python environment to be marked as externally-managed, so that Python packaging tools can refuse to manipulate packages in those environments.
As distributions and tools begin to implement PEP668 support, long-standing methods of Python environment setup that use pip install in externally-managed environments will fail by default, requiring either explicit overrides to disable PEP668 enforcement (restoring the old behavior) or restructuring of the Python environment in question. There are several areas where Ansible users may encounter problems with PEP668; this discussion will describe some of the known common cases and workarounds.
Potential user problem areas / solutions
Ansible’s pip module
To use the Ansible pip module in an externally managed environment, refer to the example for how to run tasks in a virtual environment.
If use of a virtual environment is not acceptable, pip can be instructed to override the restriction on installation in externally managed environments (see below for details). Use caution when taking this approach, as this could potentially break system managed packages.
Starting with ansible-core 2.17 (currently under development), the Ansible pip module provides a break_system_packages option. This option can be used to force installation of packages in externally managed environments. For ansible-core 2.16 and earlier, set the PIP_BREAK_SYSTEM_PACKAGES=1 environment variable on the task when invoking the Ansible pip module. The environment variable will be ignored by versions of pip which do not support the feature. [1]
[1] pip version 23.0 enforces externally managed environments, but does not support the override. Upgrade to 23.0.1 or later, or use an earlier version of pip.
ansible-builder
As ansible-builder’s current dependency management design relies heavily on pip manipulation of an OS-managed Python environment, PEP668 behavior must be unconditionally disabled during execution environment builds. An upcoming bugfix release of ansible-builder will include this change, which should be largely transparent to users. [2] Longer-term, we’ll be looking at enhancements to builder’s dependency mechanism that could support the use of a Python virtual environment that inherits system packages, as well as other ways of merging OS and Python dependencies.
[2] At this time, it appears that no distributions directly supported by ansible-builder have implemented PEP668, but should you encounter one, ensuring that a usable pip version is already present in the target base image’s Python environment, and setting
ENV PIP_BREAK_SYSTEM_PACKAGES=1in anadditional_build_steps->prepend_baseblock in your execution environment definition should allow your build to proceed as expected.
Running ansible-test
When running in an externally managed environment, the --requirements option for ansible-test cannot be used outside of a user-provided virtual environment. Alternatives are to use the --venv option for ansible-test or the --docker option to run tests from a container.
Sanity tests are unaffected, as they always run in an ansible-test managed virtual environment.
Integration tests using pip
Integration test targets that require use of pip to install modules will not function in an externally managed environment without the use of a virtual environment or enabling of the pip override.
When possible, integration tests should avoid using the override to break system packages, instead preferring use of a virtual environment.
For runme.sh based integration tests, use of source virtualenv.sh (a script provided by ansible-test) to activate an ephemeral virtual environment is recommended. For role based integration tests, see details above on using the Ansible pip module to create a virtual environment for running subsequent tasks.
Special thanks to @mattclay for helping work through the various implications of these changes and documenting workarounds.