Automated testing of modules

Archives Ansible Project
1

I’ve found some testing of modules in test/TestRunner.py https://github.com/ansible/ansible/blob/devel/test/TestRunner.py

Is this the place where module testing occurs? Are there any other automated tests which I haven’t found that exercise modules?

Is there any way that we can measure coverage of modules?

Is this the way automated tests will need to be written for modules? That is, firing up a runner and actually kinda’ “doing it live” against the local machine? Do we envision a different way to write automated tests against modules? It seems like moving more towards unit testing with the modules (instead of the current more integration-like approach) is complicated by the way that the module_common.py hack works.

I’d like to work on a somewhat comprehensive set of tests for the source_control modules.

The motivation is this little pull request: https://github.com/ansible/ansible/pull/4267 Maybe this is good to go. But, the bzr, subversion and hg modules seem to be without automated tests; so, I’m not really sure if I have made any errors in my refactoring of those modules. I’m pretty sure the changes to git are ok, b/c those changes are more simple and git has a unit test in TestRunner.

Having all module tests in a single TestCase class doesn’t seem like the best thing. I’d like to move some tests into their own classes and do a little bit of a reorganization when I add tests for bzr, hg, subversion and git. Here is that idea:

https://github.com/skyl/ansible/commit/11aabb6edb88a89f0a7399d9a70bb7c8ec42eda6

Anyone feel free to comment there.

Thanks,
Skylar

2

The quick answer is that unit tests aren’t integration tests, and what I really want to see if fully deployed playbooks verifying that apps get fully deployed.

Unit tests can’t make network activity, they can’t mess with the system in dangerous ways, etc. They are meant to be lightweight. And I’m not really happy with our framework for testing playbooks right now.

James Laska is leading up our QA team – I’ll admit we’re not quite ready to organize something, but our intent is to have a integration test framework that does allow some degree of outside contribution.

You’ll have to be a little patient as we are not quite ready for that at this point.

In the future, also, this type of discussion really belongs on ansible-devel, so please join that list if you are interested in this sort of thing.