Announcing antsibull-nox: test runner for extra sanity tests, various Ansible test tools, and custom tests for your collection!

Project Discussions Collection Development
, , , , , , , , ,
1

I’d like to announce the first ‘stable’ version of antsibull-nox (code), antsibull-nox 0.3.0. Antsibull-nox is a Nox helper library for Ansible collections.

Nox…?

Nox is a test runner, similar to tox, hatch, and pre-commit, that can be programmed in Python (instead of using some DSL). Antsibull-nox allows to run various tests using a TOML based config (with a minimal Python noxfile.py), and provides powerful tools to add own sessions to noxfile.py that need collection dependencies present and/or the collection itself in an importable manner. Antsibull-nox will download/install required collections if necessary, and create temporary tree structures to allow importing and running tools like ansible-test no matter where you clone the collection’s repository.

Examples

Antsibull-nox is somewhat opinionated in the selection of sessions it offers, but tries to be highly customizable. For an example antsibull-nox.toml config file, see community.dns/antsibull-nox.toml at 067401de97e19871cc782a88f0abf72b41d5c361 · ansible-collections/community.dns · GitHub. The corresponding noxfile.py is community.dns/noxfile.py at 067401de97e19871cc782a88f0abf72b41d5c361 · ansible-collections/community.dns · GitHub, which loads the antsibull-nox.toml config file and adds two special sessions that are community.dns specific.

The config file generates sessions for:

  • linting (including code formatting, Python code linting, YAML linting, type checking),
  • docs checking,
  • license checking,
  • extra tests like no unwanted files and action group checking (never forget to add modules to an action group again, or forget to document that a module is part of an action group),
  • Galaxy import checking,
  • running Ansible-lint (not present in that example),
  • and running ansible-test sanity, unit, and integration tests.

Antsibull-nox comes with a GitHub Action and a reusable workflow that simplifies calling nox and creating CI matrices for ansible-test:

Design decisions

  1. Why use nox instead of tox, hatch, pre-commit, …? The main reason is that I wasn’t able to do what I wanted (setting up collections and composing commands to run) with tox and hatch; I only managed to do it with nox in a sane way (i.e. without monkeypatching). The main reason not to use pre-commit is that the main idea there is to run everything in a pre-commit hook. Some of the sessions antsibull-nox offers are fine for that, while others (like running ansible-test xxx --docker) are not. (You should be able to combine antsibull-nox with pre-commit though.)

    Not having this flexibility would result in users having to clone the repository into a special directory structure (so that ansible-test can run, for example, or that importing things works when setting the Python path programmatically), take care of installing dependent collections in the same tree structure (for example ansible-test won’t find them otherwise), etc. Being able to simply run nox -R or pipx run noxfile.py -R or uv run noxfile.py -R in a collection directory to run all tests (the -R reuses venvs that nox creates, which will make this faster if you run it more than once).

  2. Why use a TOML configuration file instead of, say, YAML? TOML has the big advantage of being simpler and having better tooling for programmatically modifying it while keeping the original formatting. There is partial support for YAML for doing that as well (using ruamel.yaml), but it’s not perfect.

    Automatically updating is very useful if versions of dependencies are pinned in the config file, and you want these to be automatically bumped (similar to Dependabot), for example.

    (Also if there’s a huge demand, we can still add support for a YAML config file as well.)

How does the roadmap look like?

The current 0.3.0 release is the first “stable” release in the sense that I do not want any breaking changes until 1.0.0 is there (and since antsibull-nox should adhere to semantic versioning, there won’t be breaking changes until 2.0.0). I’m using quotes for “stable” since there is still a certain risk for breaking changes since the tool is rather new, and we might notice once more folks start using it that some decisions I made weren’t so great, and not fixing them in a breaking way would be really bad. I hope that won’t happen, but I cannot guarantee that until 1.0.0 is out.

I still think it’s ready to use, and I’ve started using it in all collections I’m maintaining (community.general, community.crypto, community.dns, community.docker, community.hrobot, community.internal_test_tools, community.library_inventory_filtering_v1, community.routeros, community.sops, felixfontein.acme). Until a few days ago it was just three of them, to avoid too much work when introducing breaking changes (there were a few). Me using it in all these collections should also tell you that I’m really serious about trying to avoid breaking changes :slight_smile:

About adding support for more sessions: I’m absolutely open for that! I mainly added the sessions which I use/need so far (that apply to more than a single collection) (only exception: running basic EE tests)

Ideas? Questions? Feedback?

I’m happy to hear about that! If you want to chat, you can also take a look at the #antsibull:ansible.com Matrix room.

3 Likes
2

Thanks for working on this and for the informative announcement! One thing I’ve wanted to read through but haven’t gotten to is the updated collection downloading/installation/caching code. Can you explain a bit how it works? Where is the cache stored? How are dependencies resolved? What are the collection_sources in the config used for?

3

We could also considering adding ruff as an alternative for flake8 and optionally black and isort, although people could always configure those in nox themselves. I’m interested to hear what potential adopters think.

I guess the pylint set up built-in to antsibull-nox is more involved as it needs all of the dependencies available for its type inference features.

We should probably migrate antsibull* to ruff at some point as well. I’ve been using ruff as a replacement for flake8 in my personal projects, but I’m still using black and isort instead of ruff’s native support for formatting, as it still has some shortcomings for my usecases.

4

The cache is stored in .nox/.cache/, which is Nox’s session-shared cache directory. It will store both the collection artifacts downloaded/built by ansible-galaxy collection download there (in .nox/.cache/downloaded), as well as the extracted collections (in .nox/.cache/extracted). It will download collections on-demand (with a lock in case Nox adds parallelism at some point), and it will simply download the latest released version of a collection (ignoring any version specifiers) without dependencies. (The dependencies will then be downloaded in the next round, if not already in the cache, until it has the full dependency tree.) To not download a specific version, or clone the collection from git instead, you can use collection_sources to tell Nox what to pass to ansible-galaxy collection download to download a specific collection. I usually clone the latest release from git for all dependencies.

The main reason I’m not using ansible-galaxy collection download’s version resolver (or care about versions in general) is that it won’t use collection_sources for transitive dependencies, and that it’s usually better to test against the latest (development) versions. This might not work well for collections with transitive dependencies (if you’re not keeping track of these), but for my collections this works pretty well (I’m mostly thinking of community.routeros, which depends on ansible.netcommon, which in turn depends on ansible.utils). It’s also something that can be changed at a later point if really needed.

There’s also no mechanism to update dependencies yet. Once they are in the cache, they will be used. To get newer versions, you have to delete the cache (or let Nox do it, I’m not sure right now in which cases it does that, if ever). That’s not a problem in CI, but could be a problem locally.

5

So far I didn’t add support for ruff since I’m not really familiar with it (for example I didn’t knew that it can also format code). I definitely like the idea of supporting it!

Yes, and same for mypy. One of the reasons for antsibull-nox is that it’s next to impossible to run linters like pylint and mypy in a sane way without requiring a very specific directory structure where the collection is part of, and requiring that all (transitive) dependencies are also in the same directory structure.

Replacing flake8 with it sounds good to me. I’m not sure yet about replacing black and isort. I’m not sure what the advantages of ruff over these tools are. (Also of interest: Known Deviations from Black | Ruff)