I come from an Agile software development background in which test driven development (TDD) is the norm. As I write Ansible scripts, I’d like some way of testing them. In principle, I want to test every command in a playbook. For example, if one of my command changes the user permissions on a file, I want a test that independently confirms that it has in fact done so. I don’t see a “test” module but I may have missed it.
Is that something that Ansible may offer some day? I’m thinking of the Ansible equivalent to unit testing. I believe it would require the ability to execute arbitrary Python code in the test. The Java tests I have written could certainly be very complex.
I’m also curious what others do for testing using Ansible. What frameworks, etc.
Yes, this is super easy to already do today, basically just call your tests at the end as the last step of your playbook.
Executing arbitrary python code is possible, but you can use ansible modules like get_url and fail and so on.
If you want to push a python script, the ‘script’ module is awesome for that.
Many of users have tests integrated with their continuous deployment process so it will fail the rolling update block before moving on the to next, thus not taking more machines out of rotation.
However, if you feel you have to test the file module, seriously, you’re wasting time – if the file module doesn’t work for you, how good is the product? It would be much better to test instead for something functional, like whether your web service is operational, rather than duplicating all the basics of Ansible in, as you say, arbitrary python code just to make sure Ansible works.
There’s a difference between unit and integration testing, and also in testing a live deployment.
Unit tests are things you run on development code.
I see your point but I’m not sure I agree. “Unit testing” may not be the best term for it but it’s not too far off. Consider this case instead: installing and configuring a DHCP server. I have my senior admin write down exactly what the DHCP deployment should look like. This is how success will be measured. Testing this will include checking file permissions (which is also security issue), processes are running and with the right settings, logging , and a whole set of functionality based on the config file of the DHCP server.
Another admin (let’s say a junior admin) writes the Ansible code to accomplish this. When the code passes the test, it is done. Certainly this crosses into what is traditionally called functional testing. Testing IT infrastructure is somewhat different from testing a new software development. I would still test what is in the Ansible script for the DHCP server because the junior admin may have gotten it wrong and plus its still code and you always test code. Plus I would have the Ansible scripts initially written in a development environment with one suite of tests. These tests will be very obtrusive and are not the same I would use to test a successful deployment.
To be sure, the most important part is testing that the DHCP server works as expected (i.e., testing the config file) which is not an Ansible issue. Still I would test everything. The issue for me is what to use to write the tests? Cucumber? JBehave? Plain Python? I know plenty of tools designed to test custom apps/code, I don’t think the IT testing tools have caught up yet.
Maybe this isn’t even an Ansible issue. I was just speculating on how it might include automated TDD-like test process in the framework.
My point is if Ansible’s file module at that level must be verified you have greater problems
Definitely you can invoke tests of any kind from playbooks, script module, get_uri, fail module with when, stat module, etc, etc – or call another test system, locally or remotely
I too would find it useful to have the ability to “unit test” my tasks. However, I have opted to create a testing playbook [1] which handles integration testing. It is not perfect, but allows for TDD/BDD, and integrate into CI gating.
FYI, there are quite a few unit test already that verify that ‘the file module’ works as advertised, you can run ‘make tests’ in an ansible checkout to get them. There should be no need to do this per playbook.
not all modules or cases are covered but if you want to add tests at this level I suggest looking into the ‘test’ dir in the ansible checkout.
Do you tests all core java libraries when you deploy a war to tomcat?
Well phrased Brian, that was my point which I was having trouble trying to share.
The name “unit testing” versus “integration testing” (especially what happens in stage environments) or “monitoring” is vitally important.
You can trust that the “file” module works, so it’s great if you test whether the application is operational at the end and things like that.
We also have some integration tests for ansible modules that are going to be rolled into our buildsystem and project soon that are more intrusive than our core unit tests.
But what you want in production to make sure you have an app running before moving to the next rolling update slice is not a unit test, but something higher level
Ansible makes this easy by following a “fail first” mode, if something happens along the way, you don’t end up with a system that is only 45% configured and then returned
success because it evaluated part of the DAG – it makes you fix all your problems, and this makes it very very stable in a rolling update process.
The modules are all coded to fail when they hit error conditions.
I should clarify a bit because it looks like I have given you the wrong impression. I have confidence that the Ansible modules behave as advertised. My experience with them to date has been excellent. The Ansible code is not what I’m trying to test.
What I want to test is that the administrator has used Ansible to configure a system in a way that meets the spec. In other words, I want to test that the admin has written a correct Ansible script. Writing an Ansible playbook is like writing code. When you write code you have to test it externally. I intend exactly what Michael wrote above: testing outcome not implementation. If I don’t test the results of the playbook, how do I know the admin wrote the correct script?
I guess my trouble of understanding is how would this test be different than just the play itself. I’m not exactly an expert so forgive me, but given that plays are written in a declarative “This is what I want” style, wouldn’t the tests essentially be saying exactly the same?
In other words why have more granularity than “start apache; test port 80” which can be done with existing modules? Or am I missing this whole conversation altogether?
That is pretty good. I like the method you use to do the tests and t that you put them in a separate playbook. Following TDD style, the test playbook could be written first, then the implementing playbook. When you check in the implementing playbook in version control, the CI server can run the tests. Nice. I’ll give this a try.
I would disagree that it’s like writing code for the most part, it was designed to be very far from that, but there are definitely steps involved that happen in order.
You are meant to read the playbook and we made them very auditable for this reason.
Of course, monitoring is a very very good idea.
An example of this is what if a playbook does something you don’t have a test for, but you still don’t want… you can only get that by reading the playbook, which is why it’s designed to be so easy to read and follow changes in source control diffs.
I don’t think you need to write a test to say “foo is installed” as the Ansible module enforces that.
You do want to test that your web service is running ok, and not just at deploy time, but also in your monitoring.
Please find attached a small (and definitely imperfect/ugly/hacky) callback plugin that I wrote for ansible, It produces xUnit output from ansible playbook runs and integrates very nicely into Jenkins (xUnit → Check Version N/A http://check.sourceforge.net)
It should be copied to {{ ansible_dir }}/ansible/lib/ansible/callback_plugins/
I use it for TDD of my own infrastructure/code and use it in a pattern very similar to what John linked to below. Complex tests are implemented as scripts (rolename/files), and “test suites” are bundled into roles (roles/test_server_live, roles/test_server_logon, etc).
I’ve found that it works really well when integrating it into my build processes on Jenkins to catch errors and test for expected behaviour at an infrastructure/functional level. Of course it is up to you to creatively use ansible to do the tests. I use a combination of modules and scripts wrapping 3rd party tools, and rely on the return code to trigger “success” and “failure” in ansible.
Another note, I have ignore_errors peppered throughout the playbooks so that all relevant tests can run, ansible’s conditionals, etc make this a very powerful setup, here is a short example:
I hope that this is a step in the right direction, with any luck someone can polish the plugin and integrate it into the ansible codebase. Otherwise if there is interest, please indicate as much and I’ll try to get it into github for a proper pull request or to maintain it separately.
I use Severspec(introduced by shigeta) for TDD.
Now, Ansible has unit-test. but, as you say, I can not verify that the servers are configured correctly.(e.g. setup DHCP)
ServerSpec can verify that the servers are configured correctly!!
It’s popular software in Japan:)
ServerSpec says.
Serverspec tests your servers’ actual state through SSH access, so you don’t need to install any agent softwares on your servers and can use any configuration management tools, Puppet, Chef, CFEngine and so on.
So this is testing the product itself. That's cool and interesting, but what I'm really interested in is testing of my playbook changes themselves. Something I can run as a gate check on changes coming into my playbook repository. Syntax checking, referencing the right external files, etc...
Thanks for the link. serverspec looks like a good option since it has a nice syntax and already has several suitable modules. I’d like to see an Ansible module that could run these tests. Maybe have a ‘spec’ directory under each role and be able to call them from the role’s playbook. This would add a TDD capability to Ansible.
To those who are not convinced testing is needed in this case I would just add that testing is always needed. Whenever you change a table constraint in your databases you test it, when you change the VLAN config in your switch you test it, when you write a new function in your program you test it, when you change your DHCP config you test it. If you care about quality the issue is not whether to test, it is only how to test. The Agile, DevOps approach is to automate everywhere hence the need for automated CM tests like serverspec.
So I’m not a super expert on this or anything, but I think I’m going to disagree with you here. To illustrate my point I’m going to use the example from Serverspec.
describe package('httpd') do
it { should be_installed }
end
This is accomplished by the yum module. It will report changed, OK, or Failed.
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
service module…
describe port(80) do
it { should be_listening }
end
Pretty much anything in the Network category of modules would work. Or doing a local-action port connection with register
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
it { should contain "ServerName [www.example.jp](http://www.example.jp)" }
end
