How to avoid regressions in ansible-modules-core ?

Archives Ansible Developer
1

Hi,

I am one of the small-time contributors on the repo. Adding some features where I need them.

Currently my testing methodology is very crude. I have a test playbook that exercised the module in various ways and hope that I test all the possible combinations. Once happy with the changes they get committed and the test playbook is deleted. How do you generally exercise your modules yourself ?

The more I am doing this the more it dawns on me that regression is bound to happen. Given the high rate of change ( around 20 commits per day ) and that testing is probably also done manually by the reviewers I am afraid of the result. We’re talking about changes that are potentially critical to an infrastructure.

Has there been any talk to introduce runnable tests for ansible modules ?

Cheers,
z

2

Just adding to this that some modules do have unit tests, etc but they sit in the core repo, making them harder to run and merge.

3

We setup travis to run the unit tests automatically when you merge, it
is still not at 100% as we sometimes break it when adding new tests,
also the v2 switch also broke some tests, some are invalid, others are
because we have not finished implementing issues.

Hopefully this will be more stable soon and you'll be able to rely on
those results going forward, still not all modules have tests nor full
coverage for those which do. We also have integration tests we run
internally, most dealing with clouds which require a 'pay account'.

4

I think I’m missing something. Can you point me toward the unit tests that test the core modules ?

The .travis.yml in the repo seems to only validate that the code compiles properly: https://github.com/ansible/ansible-modules-core/blob/devel/.travis.yml

5

The main repo is where the integration tests live for modules. Ansible does not use unit tests for modules.

Also, the integration tests do not run automatically in a publicly viewable location. Ansible has a Jenkins server that runs them, but it is private.

It would be the responsibility of others to run them manually.

There is a Makefile that is used to run the tests in the test/integration directory.

6

The main repo is where the integration tests live for modules. Ansible does not use unit tests for modules.

Many modules have doctests, as well- are those run as a part of the full test suite? I’ve never actually checked.

7

I’m unsure what you are referring to. What are “doctests”? Are you referring to the examples provided with a module? If so then not really, that would be part of an integration test.

At this time we do limited auto testing via travis (To basically do syntax and python version checking), and there are integration tests found at https://github.com/ansible/ansible/tree/devel/test/integration

Outside of that I am not aware of any other tests. Eventually there will probably be some additional tests done via travis, but not for the sake of validating functionality, but validating that it conforms to requirements.

8

I'm unsure what you are referring to. What are "doctests"? Are you
referring to the examples provided with a module? If so then not really,
that would be part of an integration test.

Like this: https://github.com/ansible/ansible-modules-core/blob/devel/database/mysql/mysql_variables.py#L138

Although grepping around shows they aren't as common as I remembered.

9

2 all. If unit-tests for modules would have exited, how would they are look like?