Ansible supported Python version on _target_

[Thulium-Drake (Jeffrey van Pelt)]

Hi there

When looking at the table of supported versions of Python on the target nodes in Ansible over at Ansible-core | endoflife.date I see a pattern that is slightly concerning to me.

While I really like the fact that Ansible is very much alive and under heavy development, as a sysadmin I see a concerning pattern when looking at the versions of Python required by the clients/targets for the automation.

A while ago, when Ansible 2.17 was released, we also ran into this issue, where the Core team decided to drop Python 3.6. Some of us voiced concerns about dropping a python version for a (big) OS that is still around for another 3,5 years. Also see the discussions here (ansible-core 2.17 removing target support for Python 2.7 and 3.6 · Issue #54 · ansible-collections/news-for-maintainers · GitHub) and here (Discussion: ansible-core 2.17 removing target support for Python 2.7 and 3.6 · ansible-collections/news-for-maintainers · Discussion #55 · GitHub)

At the time, this was inconvenient at best, because there was still a broad support of community based collections and we (in our RHEL8 environment) have been running with Ansible 2.16 quite happily ever since, getting patches via Red Hat etc.

Which brings us to the current state, for some usecases we use community code for which there is no certified alternative, but we also have environments that use non-Red Hat EL8). And the current state of things is much less peachy, as we currently see breakage because the collections follow (mostly) whatever requirements ansible-core ‘demands’ on the controller/client. Which is how it’s supposed to be, no argument there.

But, to get back to the point, the pattern I see is that every new release (which comes fast) seems to drop the lowest version of Python on the target compared to the previous version. If this pattern continues it’s not too farfetched to think that by the time Python 3.15 hits the streets, the next release of Ansible will drop EL9. Six (6) years before EL9 is slated for end of maintenance (not even mentioning ELS).

So I have some questions:

• What is the current plan of the Core team on the supported requirements on the targets?
• What recourse do we, the sysadmins that have to keep systems running, have in case this does change on the Ansible side? What reasonable expectations can we have that the ‘magic glue’ we use to connect “everything and the kitchen sink™” will keep supporting the systems we’ll need to run for years to come without resorting to all kinds of hacks that fix compatibility
• Same goes for the collection maintainers, some of the OS-es (Proxmox, based on Debian for example) don’t move as fast as Python does. What do we do when we have a collection that cannot support the new ansible-core because the target $thing does not work with that version of Python?

I get that using all of the new fancy Python features is cool and it’s nice to keep the spectrum of supported versions to a minimum, but again, my primary job is a sysadmin and I have to use Ansible on whatever systems I encounter. And I am not always in the position to ‘demand’ certain OS versions. Or the other way around, Red Hat releases their OS-es with a specific Python version as /usr/libexec/platform-python with the intention for it to be the stable interface for other system modules (be it Ansible or otherwise) that will not change during the lifecycle of the OS.

TL;DR, I fear that Ansible is chasing new Python versions so fast that it might become irrelevant because it cannot target nothing because the target does not run a supported version of Python.

[felixfontein]

I fully agree with you (and it’s one of the reason why I continue to hope for “execution plugins” - or however they should be named - which allow to support older Python versions without having to drop support for module_utils etc.)

I think dropping support for older Python versions on the controller is perfectly fine. It’s also perfectly fine for many collections that (mostly, or even exclusively) contain modules that are almost always run on the controller. But some things definitely should better keep compatible with old Pythons, especially all the low-level things.

(I’m thinking of whether it would make sense to create a new collection community.core for that, and migrate modules from other collections such as community.general to there. I really don’t want to support old Pythons and ansible-cores for community.general, but for a smaller, more specialized collection that’s perfectly fine IMO. But that still doesn’t solve the ansible-core problem, where a new enough ansible-core version effectively prevents execution of modules and module utils - even if they still support Python 2.6 - on older Python versions like Python 3.6.)

RH’s official answer to all this are execution environments, which are better than nothing, but IMO they are more like a band-aid than a proper solution. (And you really don’t want to split up playbooks into different playbooks that each needs a different EE… )

---

There’s one thing in your post I’d like to clarify:

@Thulium-Drake

But, to get back to the point, the pattern I see is that every new release (which comes fast) seems to drop the lowest version of Python on the target compared to the previous version.

I’m assuming you mean a new ansible-core 2.x.0 release. (And not a new Python 3.y.0 release.)

It’s only every second release: basically once a new Python version is added (which happens once per year), an old Python is dropped. The ansible-core versions 2.15 - 2.18 were kind of outliers: there it looks like one version is dropped every release, i.e. every half a year. But that was mainly to reduce the number of supported Python versions to the current count (3 on controller, 6 on target) without dropping too much at once. (I’m not sure whether that was the official strategy, but it felt very much like that.)

[mariolenz]

Talking about RHEL8, I think that Python 3.6 is the default and supported for the full lifecycle of this major RHEL release. But it looks that there are also newer versions available.

Can’t you just install a newer version the target and it just works? Or wouldn’t this work because the default python version (3.6) is used?

I’m sorry if this is a stupid question, but I’m mostly involved with collections that run on the controller node and talk to some API. Not with ones that run on the target.

[felixfontein]

1. You cannot use a modern ansible-core to install these older Python versions in a nice way (i.e. without using the raw module).
2. Certain modules, including the package managing modules, require to be run with the system Python. So you can run a lot of things, but not basic tasks such as installing/removing/upgrading packages.

Especially 2. is the main reason why the situation sucks so much… If the only disadvantage would be that you need to install a newer Python first (somehow), it would be a lot easier.

[rfc2549]

I have worn many hats and can relate to the challenges of using, packaging and maintaining ansible things

I don’t really have any good solution to suggest right now but some thoughts and comments.

@felixfontein

RH’s official answer to all this are execution environments, which are better than nothing, but IMO they are more like a band-aid than a proper solution.

Execution environments (as built/meant for awx/aap) lets you pick your version of python, ansible-core, ansible, dependencies and other things but at the end of the day it’s not fundamentally different than using a regular container image or a virtualenv of your preferred python interpreter..

I think dropping support for older Python versions on the controller is perfectly fine.

That’s the controller side and I agree, it’s probably fine. You can install it where you want and you run it however you want. Run it in k8s if you want

For the targets, I understand and can totally relate to wanting to reduce the maintenance burden but the reality is that there are a lot of LTS distros (RHEL, Debian, Ubuntu LTS to name a few) that have lifecycles that are supported far beyond the EOL of any python version.

Dropping support for the python version of these distros makes it a bit more difficult to manage a living datacenter. We can have thousands of these, it makes upgrading ansible a bit harder.

For example, CentOS 7, RHEL 7, Debian 10 (Buster) and Ubuntu 16.04 shipped python 2.7 by default.

RHEL 7 supported ended in 2024, Ubuntu 16.04 until (at least) 2026 and Debian 10 is no longer supported since 2024.

python2.7 went EOL on 2020-01-01 but it’s actually kind of supported until 2026, actually ?

I don’t know what to do about this.

It’s ok to expect users to update to modern distributions but reality is we are busy and we do not always have the luxury of keeping thousands of machines upgraded easily

I just mean to point out that dropping python versions for distros that are still supported can be a big papercut.

You cannot use a modern ansible-core to install these older Python versions in a nice way (i.e. without using the raw module).

Yes, we can probably jump through hoops to get the right version of a python interpreter installed somehow.

But I think Ansible is supposed to be agentless, simple to use, because you don’t (usually) need to install anything, it’s just ssh, but actually python too, but at whatever the default version of python each distro happened to pick years ago.

Edit: I mixed up RHEL8 and RHEL7

[Thulium-Drake (Jeffrey van Pelt)]

Thanks for all the replies!

@felixfontein

I think dropping support for older Python versions on the controller is perfectly fine.

Agreed, because if for whatever reason the controller is not able to run the modules, this is exactly where EE’s come in (and you can also run those with ansible-navigator if you don’t have AWX/AAP

@felixfontein

But that still doesn’t solve the ansible-core problem, where a new enough ansible-core version effectively prevents execution of modules and module utils

Yes, exactly, and the EL distro’s (RHEL, Rocky, Alma etc.) are impacted most by these due to the /usr/libexec/platform-python begin static

@felixfontein

I’m assuming you mean a new ansible-core 2.x.0 release. (And not a new Python 3.y.0 release.)

Yes, I’m guessing that once Ansible 2.22 rolls around they drop Pyton 3.9 (given that Python 3.15 is around by then)

@mariolenz

Can’t you just install a newer version the target and it just works? Or wouldn’t this work because the default python version (3.6) is used?

Well, not on all distros, I’m not too sure how this works on Debian (it’s been a while), but for EL the modules that interact with for example Yum/DNF have to use /usr/libexec/platform-python, which means for EL8 3.6.8, EL9 3.9.18. And Ansible’s interpreter discovery plugin will always attempt to run modules with that interpreter as well.

And when trying to run a newer version of Ansible on those targets will either result in an error finding a suitable interpreter, or Python errors because the interpreter is not supported.

@felixfontein

Especially 2. is the main reason why the situation sucks so much… If the only disadvantage would be that you need to install a newer Python first (somehow), it would be a lot easier.

Indeed, but that’s the point with EL at least, that’s something they explicitly changed when designing RHEL8 (https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8-3 and Chapter 16. Installing and using dynamic programming languages | Configuring basic system settings | Red Hat Enterprise Linux | 8 | Red Hat Documentation)

@rfc2549

Execution environments (as built/meant for awx/aap) lets you pick your version of python, ansible-core, ansible, dependencies and other things but at the end of the day it’s not fundamentally different than using a regular container image or a virtualenv of your preferred python interpreter..

Agreed and I’m fully in favor of EE’s, because it allows one to have a static, immutable, personalize Ansible executioner. And that on it’s own is brilliant, no more juggling dependencies, installing obscure Python packages from outside of PyPI (looking at you VMware) and hoping nothing breaks this time after upgrading your system. Heck, you can even stick it inside a CI job that runs a test playbook to ensure you can still talk to <insert thing here>!

@rfc2549

It’s ok to expect users to update to modern distributions but reality is we are busy and we do not always have the luxury of keeping thousands of machines upgraded easily

I just mean to point out that dropping python versions for distros that are still supported can be a big papercut.

Well this exactly, I already have to keep everything updated, secured and functioning. And we’re trying real hard to keep the pace (we’re already planning RHEL9 upgrades across the board, because RHEL10 has been released, which mean N-1 is now 9 ), but we can’t keep up if the tools we need to use drop support for the OS-es we need (again, sysadmins don’t always have a choice here).

@rfc2549

Yes, we can probably jump through hoops to get the right version of a python interpreter installed somehow.

Well these are the ‘all kinds of hacks that fix compatibility’ that I meant in my post

And frankly, my work is impossible to do in a reasonable amount of time without tools like Ansible, there’s just too many systems, hardening guidelines to apply, patches to install, you name it.

When doing some more searching I ran across this post from @sivel (Python 3.7+ Impact on EL8 (future for EL9) - #13 by sivel) in which he explained the viewpoint from the Core team’s side of things. Which I can fully understand, but I think we’re going in the wrong direction as of now.

And with regards to what we could possibly do to fix this, I think we have a few options/directions in which we can look:

• Investigate ‘LTS’ releases of Ansible, because most of the major distro’s seem to hover around the same versions, this might just work
• Expand the range of supported Python on the targets only. Controller is fine, because that’s relatively easy to fix with EE’s, vens or some other means (it’s just 1 system/container image after all)

Possible AI lies incoming: I had Chat whip up a table of current supported versions of Debian, Suse, Ubuntu and RHEL and their supposed default version of Python:

Distro family	Specific distro / version	Default Python version (system)
EL	RHEL 8 (platform-python)	3.6 (Red Hat Developer)
EL	RHEL 9 (platform-python)	3.9 (Red Hat Docs)
EL	RHEL 10 (platform-python)	3.12 (Red Hat Docs)
Debian	Debian 11 (“Bullseye”)	3.9 (Debian Wiki)
Debian	Debian 12 (“Bookworm”)	3.11 (Debian Wiki)
Debian	Debian 13 (“Trixie”)	3.13 (Debian Wiki)
Ubuntu	Ubuntu 20.04 LTS (“Focal”)	3.8 (documentation.ubuntu.com)
Ubuntu	Ubuntu 22.04 LTS (“Jammy”)	3.10 (documentation.ubuntu.com)
Ubuntu	Ubuntu 24.04 LTS (“Noble”)	3.12 (documentation.ubuntu.com)
Ubuntu	Ubuntu 25.04 (“Plucky”)	3.13 (documentation.ubuntu.com)
SUSE (SLE)	SLES 15 (Base system)	3.6 (SUSE Documentation)
SLES 15 SP4+ (Python 3 module)	SLES 15 SP4 / SP5	3.11 (SUSE Documentation)

[bvitnik (Bojan Vitnik)]

Sorry for asking but what is the exact criteria for dropping support for certain Python version? I kinda guess that Python versions EOLed upstream are also dropped from ansible-core.

This does not make much sense to me. A Python version being EOL does not mean it is not massively used by users. Yes, dropping some legacy code can make the code cleaner but your tool becomes less useful. You want your tool to be useful to as many people as possible right? The decision to deprecate some Python version should be made on some statistics of Python versions used in the wild, not by following upstream EOL plan blindly. This would allow for a natural deprecation of Python versions. Python 2.6 and 2.7 more or less died off naturally, why should 3.x be any different?

In any case, every Python version will eventually be deprecated and the code base will become cleaner and new language features and standards will be employed. It just doesn’t have to be as soon as possible. Dropping support for certain Python version as soon as possible looks more like a coding elitism (look ma, my code is super modern and leet) than considering usefulness of the tool and richness of the environments it will likely be used in.

[sivel]

@bvitnik

Sorry for asking but what is the exact criteria for dropping support for certain Python version?

Largely, this is down to 2 or 3 things, depending on how you want to separate 1 of them, I’ll just do it for clarity.

1. Engineering overhead and limitations to supporting many python versions
2. Cost limitations and CI complexity for testing these python versions
3. The specific need for features in a minimum python for implementation of functionality

Python 3.6 was dropped due to a specific need of the contextvars module added in Python 3.7.

And with Python now being on a yearly release cadence, the new python versions are coming at us faster than ever, and these new Python versions need to be supported, to support OSes that release and include these versions.

We’ve experienced, at least in the importlib library which the collection loading is strongly dependent on changing APIs and deprecating features at a very fast rate, to the point where a new feature was added, deprecated, replaced, and the replacement deprecated and itself replaced.

Ultimately, while it sounds easy to support any arbitrary number of python versions, it comes down to what we can actually commit to supporting, and delivering a usable project on.

It’s all a balancing act, and in the end, instead of it feeling more arbitrary than it is from release to release, we decided to define the limits of the number of versions we would support. These definitions can be found at:

Releases and maintenance — Ansible Community Documentation

Releases and maintenance — Ansible Community Documentation

[Thulium-Drake (Jeffrey van Pelt)]

@sivel

Ultimately, while it sounds easy to support any arbitrary number of python versions, it comes down to what we can actually commit to supporting, and delivering a usable project on.

Thanks for the reply! And it does sound easy, but I know it isn’t, I’ve seen similar issues that someone needs to have a piece of software which does not match the OS library and that library cannot be replaced, so we have to compile and install it, which leads to a different library and so forth.

I can imagine the maintenance burden for Ansible is similar, so you have to limit it somewhere.

@sivel

Releases and maintenance — Ansible Core Documentation

I haven’t read that before, thanks for this! But I will admit that it stings a bit that Windows enjoys a massive spectrum of supported Ansible version because Powershell isn’t releasing that often (and no, this is not Ansible’s fault, if anything, it’s Python’s )

However, I still think my point holds, any user of EL-based or a different LTS distro might run into a situation at some point where they can no longer update Ansible, or their collections because of the interpreter restrictions on the target. Which would basically cut them off and leave them with either outdated tools or a really good excuse to look for different tools.

Maybe @gundalow knows, does PyPI keep track of downloads per version of the package published? This might give some insight which versions are more in demand (and hopefully if there’s some outliers that wouldn’t make sense if it’s EOL)

[rfc2549]

@Thulium-Drake

Maybe @gundalow knows, does PyPI keep track of downloads per version of the package published? This might give some insight which versions are more in demand (and hopefully if there’s some outliers that wouldn’t make sense if it’s EOL)

Yes: PyPI Download Stats

What is the source of the download data?

PyPI provides download records as a publicly available dataset on Google’s BigQuery. You can access the data with a Google Cloud account here.