Breaking changes with pip and system packages


The Python packaging ecosystem has recently adopted PEP668 as a solution to the long-standing problem of packaging tools (e.g., pip) clobbering dependencies in shared Python environments managed by an OS package manager (e.g., dnf). PEP668 defines a standard way for a Python environment to be marked as externally-managed, so that Python packaging tools can refuse to manipulate packages in those environments.

As distributions and tools begin to implement PEP668 support, long-standing methods of Python environment setup that use pip install in externally-managed environments will fail by default, requiring either explicit overrides to disable PEP668 enforcement (restoring the old behavior) or restructuring of the Python environment in question. There are several areas where Ansible users may encounter problems with PEP668; this discussion will describe some of the known common cases and workarounds.

Potential user problem areas / solutions

Ansible’s pip module

To use the Ansible pip module in an externally managed environment, refer to the example for how to run tasks in a virtual environment.

If use of a virtual environment is not acceptable, pip can be instructed to override the restriction on installation in externally managed environments (see below for details). Use caution when taking this approach, as this could potentially break system managed packages.

Starting with ansible-core 2.17 (currently under development), the Ansible pip module provides a break_system_packages option. This option can be used to force installation of packages in externally managed environments. For ansible-core 2.16 and earlier, set the PIP_BREAK_SYSTEM_PACKAGES=1 environment variable on the task when invoking the Ansible pip module. The environment variable will be ignored by versions of pip which do not support the feature. [1]

[1] pip version 23.0 enforces externally managed environments, but does not support the override. Upgrade to 23.0.1 or later, or use an earlier version of pip.


As ansible-builder’s current dependency management design relies heavily on pip manipulation of an OS-managed Python environment, PEP668 behavior must be unconditionally disabled during execution environment builds. An upcoming bugfix release of ansible-builder will include this change, which should be largely transparent to users. [2] Longer-term, we’ll be looking at enhancements to builder’s dependency mechanism that could support the use of a Python virtual environment that inherits system packages, as well as other ways of merging OS and Python dependencies.

[2] At this time, it appears that no distributions directly supported by ansible-builder have implemented PEP668, but should you encounter one, ensuring that a usable pip version is already present in the target base image’s Python environment, and setting ENV PIP_BREAK_SYSTEM_PACKAGES=1 in an additional_build_steps->prepend_base block in your execution environment definition should allow your build to proceed as expected.

Running ansible-test

When running in an externally managed environment, the --requirements option for ansible-test cannot be used outside of a user-provided virtual environment. Alternatives are to use the --venv option for ansible-test or the --docker option to run tests from a container.

Sanity tests are unaffected, as they always run in an ansible-test managed virtual environment.

Integration tests using pip

Integration test targets that require use of pip to install modules will not function in an externally managed environment without the use of a virtual environment or enabling of the pip override.

When possible, integration tests should avoid using the override to break system packages, instead preferring use of a virtual environment.

For based integration tests, use of source (a script provided by ansible-test) to activate an ephemeral virtual environment is recommended. For role based integration tests, see details above on using the Ansible pip module to create a virtual environment for running subsequent tasks.

Special thanks to @mattclay for helping work through the various implications of these changes and documenting workarounds.


Thanks for the detailed writeup @nitzmahone! This seems pretty crucial information, so I’m going to pin this to the top of Project Discussions for at least a few weeks as an absolute minimum. Two questions for you:

  1. Is there any particular call-to-action for things the community needs to work on here?
  2. Do we need to get this into wider distribution? I’m happy to help do a News & Announcements post which will go out to all users, and I’m sure @cybette can help make sure it goes into the Bullhorn for a few weeks too.