ansible-testing

Archives Ansible Developer
1

Infrastructure testing is typically awkward and reliant on ruby gems (e.g. serverspec, cucumber).

One of the things I’ve often dreamed of is a test framework that allows tests to be run both on the server being tested (is a process running) and against the server being tested (is a port accessible). A bit like action and local_action.

From that, it wasn’t a big leap to adding some modules for testing infrastructure - check_tcp, check_process, etc. If you know Ansible, using this test suite will be trivial.

While Ansible can fix these kinds of things currently, having modules that don’t only observe and don’t change state has some benefits.

As such, I’ve started work on ansible-testing repo at http://github.com/willthames/ansible-testing

Ideally it’ll be merged into main Ansible under library/modules/testing - let me know if a pull request is desired for that. I’ve added a to be implemented section to the README, but I’m sure there’s further desirable behaviour. I’ll examine issues and pull requests while this remains separate to Ansible. (which, I repeat, I hope will not be for long)

Will

2

Good stuff, and I’d definitely like to see a category full of checks.

Initial thoughts is where something doesn’t require a remote test (i.e. variable comparisons), it should probably be an action plugin.

Brainstorming on ideas for check modules could be interesting!

(and of course, be sure they label themselves as supporting check mode – which is not the same thing :))

Maybe we should call these “test_” instead to avoid the name confusion :slight_smile:

I think “check_equals” could probably be done with the fail module:

  • fail: msg=“does not equal”
    when: asdf != foo

Except that you have to write the message yourself so it’s not super clean. But still, could it be done more simply as an assert I wonder?

  • assert: asdf != foo

The trick would be in showing the evaluated expression of course, and being able to pass in the explanation.

  • assert: condition=“asdf != foo” msg=“assert asdf equals foo”

??

Just thinking out loud mostly.

3

I’ve renamed the modules test_ rather than check_ and updated them to support check mode.

I’ll have a think about the action plugins - at the moment all but test_equals are useful remotely.

I do take your point about using fail rather than test_equals, but my preference was to avoid skipped tests, and have all tests return ok/fail. Also maintaining consistency of the test modules. But I’m not devoted to test_equals.

I assume assert is a theoretical module - your idea makes me realise a flaw in the existence of test_equals, which is that test_not_equal and many others might need to exist compared to a single module that can eval a condition. So perhaps test_equals won’t be long for this world!

Some of the suggestions I’ve had are:

  • test_command - checks status and stdout (should probably work with shell too)
  • test_connected - is a host connected to another host (e.g. is a connection to a DB established)
  • test_icmp / test_udp

Will

4

“* test_command - checks status and stdout (should probably work with shell too)”

This could be done with the assert style check and I would not want to duplicate command logic here.

I would disagree it’s entirely needed to use the monitoring infrastructure, as some of this would be very simple, though it wouldn’t hurt to have a nice module that could invoke a monitoring check too.

5

I’m not entirely clear what ‘it’ is in your second paragraph.

Are you saying none of the test_ modules are needed, or just test_command (I agree with your point on that - so presumably it just remains to write an assert module?)

What kind of nice module are you thinking of? What would its features be?

Will

6

test_equals has been replaced by assert and documentation updated to reflect.

I think assert would likely best be implemented by an action_plugin (presumably that means it uses lib/ansible/utils rather than lib/ansible/module_utils ?) and this wouldn’t be very tricky - while ansible-testing remains outside of ansible itself, it can remain a module.

Will

7

Right.

I’m not sure which paragraph you were referencing, but I meant don’t duplicate a module that can run commands like shell/command or script already does, as we’ll just have more to maintain and keep consistent.

Sounds good!

8

I somewhat forgot about this today and needed an assert for a test overhaul I was working. More on this later – it’s not quite ready for contribution yet.

This was mine:

  • assert: { that: “foo != bar” }

or:

  • assert: that=“foo != bar”

Though I somehow found the extra = in the output unsettling, so I wrote mine like the above. Both work of course.

The difference in the above is that the assert here is an action plugin, based heavily on Dag’s fail module, and is equivalent to this, mostly:

  • fail: msg=“…”
    when: foo != bar

There’s a little more quoting, but it’s a little more natural to me. When it dies it returns both the expression and what it evaluated to.

I found what I was playing with with stat a little cumbersome for the assert myself, so if you want to submit all the modules you have above (save assert, as I want this to be an action plugin) into the ‘utilities’ category, I think we’d like them – I can definitely see a reason for things like icmp being able to be issued from any post, so it doesn’t seem that would be an action plugin.