Unit Testing modules like rax and ec2

Archives Ansible Developer
1

James Tanner approached me about thinking of ways to unit test our cloud modules a day or so ago, and since then he and I have been chatting about ideas to make unit testing modules like rax and ec2 possible. Currently they are not easily unit testable, and require integration testing.

Not everyone is going to have access to cloud environments that they will want to use for testing, and this could still help us catch issues with updates and such. James pointed out that #5285 as a prime example of needing this functionality.

Thinking over the process, I couldn’t really see a way in, other than taking the approach of how test-module works. For those that don’t know, test-module basically uses ModuleReplacer to build out the module as it would appear during ansible execution, writes it to a file, and then subprocess.Popen()s it, and evaluates stdout.

I took this a little further, and after ModuleReplacer does it’s thing, I strip out the main() call, and replace it with Mock, monkey patching, and test cases, from a file that would live in test/inject_module_unit_test. The files in there would be named something like MODULE_NAME_tests.py, such as test/inject_module_unit_test/rax_facts_tests.py.

Then we run subprocess.Popen on the file, optionally running it via coverage.

I’ve only created tests for rax_facts right now, as it was simple and required minimal test cases an Mock/Monkey patching for testing.

A sample run looks like:

./unit-test-module -c -v -m …/library/cloud/rax_facts

  • including generated source, if any, saving to: /Users/matt/.ansible_unit_test_rax_facts
  • this may offset any line numbers in tracebacks/debuggers!

Name Stmts Miss Cover Missing

2

This looks great. Hopefully I’ll find some time next week to work on some similar tests for ec2 using the same framework.

I’ll also need to do some reading around fixtures and mocks and the like, I must confess that testing is not my strong point.

I’d say this should be a full fledged pull request.

Will

3

So I strongly want to see integration tests here, not units.

This is a time/value investment calculation, more than anything, and as I’ve mentioned we’re meeting a bit next week (internally) to discuss a structure in the Ansible core project that will allow for this, and we’re going to wire this up and make it possible for everybody to extend it too.

If someone doesn’t have access to Rackspace infrastructure, most likely they won’t be working on the modules, and we also plan to run the integration battery for everyone on a regular basis.

Let’s hold off until you see what we have going (very soon) and I think you’ll like what it’s going to be capable of.

It will take some while above, but I don’t want to invest in a lot of mock infrastructure when the real world testing against those APIs is more lacking, and easily fixable.