Ansible.module_utils.six is deprecated - what to do for collections that support both ansible-core 2.16 (or even before) and 2.20?

Project Discussions Collection Development
, , , ,
1

Recently ansible-core’s ansible-test’s pylint sanity test started to flag ansible.module_utils.six imports as ansible-bad-import-from. This was easy to ignore with some tests/sanity/ignore-2.20.txt entries, but raised the question whether ansible-core is planning to remove their vendored six or not (there was no changelog entry, and there’s no porting guide yet). I created an issue to get clarification on the status of ansible.module_utils.six, and the result was that it should be deprecated (apparently the deprecation will go to ansible-core 2.20, since the removal version is 2.24).

Unfortunately this is implemented in a way that every single plugin and module that imports ansible.module_utils.six will result in a deprecation message shown to the user. This means that every collection that uses ansible.module_utils.six right now will annoy users of ansible-core 2.20, unless they stop importing ansible.module_utils.six. Now getting rid of ansible.module_utils.six is pretty easy if the affected code is Python 3 only, like in plugin code if you support ansible-core 2.12+, or in all code if you support ansible-core 2.17+ (ansible-core Python support matrix). But if you have a collection that still supports Python 2.7 for modules (by supporting ansible-core 2.16 or older), like for example community.general (up to version 11, the currently released major version), you can’t simply get rid of ansible.module_utils.six.

One solution would of course be to vendor six yourself in your own collection. Unfortunately, that doesn’t work well, since six contains a large list of “virtual” submodules (of the form six.moves.xxx) that do not correspond to files, and which lead to errors when using them in ansible-core’s ansiballz module_utils file collector. (It will look for explicit files, which don’t exist, and then it will fail.)

All collections that still need to support Python 2 (for whatever reasons) and also ansible-core 2.20 (for example because they don’t want to do a new major release for no other reason than to stop supporting older ansible-core versions) seem to have basically two choices:

  1. Continue to use ansible.module_utils.six, which means all users using ansible-core 2.20+ will see ugly deprecation message for every plugin/module from that collection they use that uses six;
  2. Somehow work around this by conditional imports, lots of if sys.version_info[0] == 2:, vendoring some own versions of six that don’t make use of “virtual” submodules, etc.

For most collections I maintain, I implemented 2. This is ugly, and in cases like community.docker resulted in a lot of changes and boilerplate that I can remove again for the next major release (PRs 1, 2, 3, 4). I’m not happy about this, but I think it was worth the effort, as I really don’t want to annoy users.

But now there’s community.general. I’ve created a PR to remove all six usage from plugins (community.general hasn’t supported Python 2 for non-modules for some time now), which was easy. But there are also a lot of module utils and modules that use six; in total, there are 105 Python files:

  • 20 module utils (which means a lot more modules that use them);
  • 63 modules;
  • 22 unit tests for modules or module utils.
$ grep -Rh ansible.module_utils.six $(find plugins/ tests/unit/ -name '*.py') | sort | uniq -c
      1 from ansible.module_utils.six import binary_type, string_types, text_type
      1 from ansible.module_utils.six import binary_type, text_type
      1 from ansible.module_utils.six import integer_types
      1 from ansible.module_utils.six import integer_types, string_types
      9 from ansible.module_utils.six import iteritems
      3 from ansible.module_utils.six import iteritems, string_types
      1 from ansible.module_utils.six import PY2
      5 from ansible.module_utils.six import PY3
      1 from ansible.module_utils.six import python_2_unicode_compatible
      1 from ansible.module_utils.six import raise_from
      1 from ansible.module_utils.six import raise_from, PY2
     16 from ansible.module_utils.six import StringIO
      6 from ansible.module_utils.six import string_types
      2 from ansible.module_utils.six import text_type
      1 from ansible.module_utils.six import text_type, binary_type
      2 from ansible.module_utils.six.moves.collections_abc import Mapping
      2 from ansible.module_utils.six.moves.collections_abc import MutableMapping
      2 from ansible.module_utils.six.moves import configparser
      1 from ansible.module_utils.six.moves import configparser, StringIO
      2 from ansible.module_utils.six.moves import http_client
      2 from ansible.module_utils.six.moves import http_cookiejar as cookiejar
      4 from ansible.module_utils.six.moves import shlex_quote
      2 from ansible.module_utils.six.moves import xmlrpc_client
      1 from ansible.module_utils.six.moves import xrange
      1 from ansible.module_utils.six.moves import zip_longest
      3 from ansible.module_utils.six.moves.urllib.error import HTTPError
      2 from ansible.module_utils.six.moves.urllib.error import URLError, HTTPError
      1 from ansible.module_utils.six.moves.urllib import error as urllib_error
      7 from ansible.module_utils.six.moves.urllib.parse import quote
      1 from ansible.module_utils.six.moves.urllib.parse import quote_plus
     26 from ansible.module_utils.six.moves.urllib.parse import urlencode
      1 from ansible.module_utils.six.moves.urllib.parse import urlencode, quote
      3 from ansible.module_utils.six.moves.urllib.parse import urljoin
      7 from ansible.module_utils.six.moves.urllib.parse import urlparse
      1 from ansible.module_utils.six.moves.urllib.parse import urlparse, urlencode, urlunparse
      1     from ansible.module_utils.six.moves.urllib.parse import urlparse, urlunparse
      1 from ansible.module_utils.six.moves.urllib.parse import urlparse, urlunparse
      1 from ansible.module_utils.six.moves.urllib.request import getproxies
      1 from ansible.module_utils.six.moves.urllib.request import pathname2url
      1 import ansible.module_utils.six.moves.urllib.error as urllib_error

Fixing them all would be possible, but require some more effort. Especially for the ansible.module_utils.six.moves.xxx imports this at least doubles the work necessary to eventually move everything to Python 3 only (which will happen for community.general 12.0.0, to be released later this year).

What do you think? Should we fix this, so that community.general 10 and 11 work fine with ansible-core 2.20.0 (as promised)? Or should we add known_issues changelog/porting guide entries that basically tell the user “you have to live with the deprecation messages unless you update community.general to 12.0.0”?