One of the most important aspects of software development is a commitment to quality testing.
It’s for this reason we (as a company) invest a lot in automated QA.
However, it’s a dirty secret that systems management software is one of the things that is usually some of the worst stuff to write Test Driven tests for. When talking to 500 different remote APIs, a mock interface for each one will never tell you that something upstream has drastically changed. The most important thing you can do is always to test for real. And there are tons of error scenario paths!
Also, as Ansible has grown, what started out with some basic unit tests quickly blended into a hodgepodge of unit tests and not, and for many modules there was no clear way to test them safely in a unit test environment. Many tests revolved around a playbook returning 37 events, and that’s clearly not what we want. What we want to know is, did something work?
A further problem is actually a good one to have – Ansible makes itself easy to contribute by all kinds of people, and many of our contributors are not well versed in writing, say, RSpec style mock tests.
Though as we said, while contributions to Ansible were sky-high, the tests were hard to contribute to, and really disorganized, with a lot of random files in different directories.
Also while we appreciated acquired test code, it was frequently hard to maintain because it would rely on executing playbooks and counting events via callbacks. Clearly not the way to go.
Of course, one of the easiest ways to test ansible modules and features is use the interplay inside of Ansible to do this, and in just a few short days several of us have done this to (A) make tests drastically easier to write and (B) extend our coverage a good bit.
There are still a few gaps, and a few of us have homework assignments to extend them.
There are also future places we are going to upgrade this, including AWS and Rackspace tests that take optional credentials files, to use all these modules for real, and delete the test infrastructure when done.
The new test infrastructure lives where the old did:
Unit tests are a subset of the old tests, which still need cleanup to a degree, but are generally things that can be cleanly mocked out without side effects. With the exception of inventory, these tests should not require input, and in any case, should not generate filesystem changes as output.
https://github.com/ansible/ansible/tree/devel/test/units
More interesting are the new integration tests, which are being broken up into a few categories.
“non destructive” tests are things that don’t ever do anything like install a package. These should be always safe to run as if they were unit tests.
“destructive” tests aren’t that destructive, but might install and uninstall a package. You’d want to run these on a VM.
There are also going to be tests pending for cloud provider utilization, which will also allow multi-tier operations. This is pending – and once we get a pattern going for it, should be equally easy to contribute to. Tower already has some of this for the cloud integrations we need, but we really want this to be free and easy for people to contribute to.
You can find the new test code here:
https://github.com/ansible/ansible/tree/devel/test/integration
The brunt of everything lives in roles, which will be a standard for these new tests:
https://github.com/ansible/ansible/tree/devel/test/integration/roles
What is there today is basic in some regards, but it’s designed to be really easy to take pull request contributions to.
Here’s an example of some code that tests the git module.
https://github.com/ansible/ansible/blob/devel/test/integration/roles/test_git/tasks/main.yml
If you compare that with the code we originally had to test the git module, this is radically simpler to add, and also understand.
And it’s written in Ansible, so that makes it even easier – there isn’t another framework to learn.
I hope everybody finds this to be a huge improvement – we are definitely very very excited, and now have a place where we can assemble the test battery that we always wanted to start assembling – one that tests ansible live and for real, and is fully open source too!
To learn how to run the integration tests, see https://github.com/ansible/ansible/blob/devel/test/integration/README.md
As we grow, and especially as we refactor thigns, look for a lot of upgrades to the mock testing of the API as well, but the integration tests should provide a great framework for adding a lot of high-power real-world horsepower, that doesn’t require a rocket scientist to extend.
Comments and questions are welcome, but first person that recommends ServerSpec or Cucumber may get hit with a trout 
Fair warning!