Hi Michael,
Thanks for the response. Some excellent points. I'll respond to a couple.
As such, I strongly favor integration tests run against stage environments to make sure
things work, and coupling that in a rolling update against a production environment as
a condition to decide whether to add something back to a load balanced environment
there as well - ideally using the same tests, but that's not always possible.
Agreed! Me too. I didn't address the downstream deployment stages, but
this would be consistent with my approach.
While more of a unit test thing, I personally find Cucumber to be wasted effort because
most product-management types (I guess I qualify) are not going to be the ones writing
test stubs, so it's usually fine to just go straight to tests.
Understood. On the contrary, I have found the type of tests I write
when using a natural language approach to be a great mechanism for
communicating requirements within a team. Though I wasn't sure, it
read like you are thinking of Cucumber as a unit testing framework. I
never think of Cucumber as a "unit" testing framework.
In any case, my point isn't about Cucumber in general, but rather
about the good practice of creating an acceptance test for a body of
work about to be embarked upon. I am glad for teams to use whatever
tool they think makes the most sense for their combination of team
members and customers. By acceptance test, I simply mean an automated
verification that the features we want actually do exist in the system
that a customer and engineer can collaborate on.
Good integration tests for ops are more important -- is this web service responsive?
Agreed.
Not things like "is the file permission of this file the same thing it is listed in the configuration
system - as that just duplicates configuration in two places.
Duplication is precisely the point, sort of. I like Uncle Bob's point
about unit testing being the double entry accounting of software
engineering. We express the functionality we want in two ways: code
and test. Then we automatically make sure our column sum equals our
row sum.
Over the long haul, I find the double entry approach to find problems
fast when they are introduced. In the short term, they alert me
quickly when I make mistakes.
I'm strongly not a fan of ServerSpec, because I think it fails to understand the basis of
configuration management in most places - that you do not have to check for what the
config tool already asserts.
I understand your point. Usually, I try to make sure that my
"functional tests" focus on testing my implementation. For example, I
testing that starting a service has the expected results. CM tools
can't do that generically, beyond asserting that they have in fact
done as instructed. For example, if I have a "service: name=foo
state=started" entry, all I can expect from Ansible natively is that
it verifies that there is a service named "foo", that asking for it's
status reports "running", and if not running, calling "start" puts it
into that state. But, I'm missing my test for foo listening on a
specific port and writing to a specific file, for example. That feels
like a good use case for Serverspec.
But, again, I don't care so much about the specific tool. I am happy
for teams to choose whatever works best for them. My workflow focuses
more applying TDD mentality to evolving infrastructure coding
alongside the rest of my application.
We've written about this here:
http://docs.ansible.com/test_strategies.html
Thanks for the reference! And thanks for writing that up! I like the
ideas a lot and they seem pretty consistent with how I like to
approach things. One thing I like to do is have the opportunity to run
tests in isolation from running the provisioning. That doesn't
preclude me from putting more of them in the Ansible files, of course.
I think the physical separation of test and implementation helps me
wear the two different hats I need to wear to write good tests, but
I'll gladly accept that as a personal preference.
While some of his posts are useful...
I'll skip the fun discussion about patterns for now. Interesting
points, and thank you! I assume that we can agree that "looking at
what we have implemented and finding opportunities to improve it,
non-functionally" is a good thing.
Here is the outline of my slides from my talk to the NYC Continuous Deployment group...
Looks like a great talk. Sorry I missed it!
Using Vagrant to push to AWS here seems weird to me, I'd probably just use the AWS
modules in Ansible directly from from Jenkins to trigger my tests towards AWS, rather
than kicking them off from a laptop.
I like the isolation of Ansible for configuration and Vagrant/Packer
for machine instantiation. It seems to make for a good isolation of
concerns.
As for the "from a laptop" part, I like to test stuff from my dev
environment, before I commit. So at that point it might be from my
laptop. But, when the CI/CD server is calling Vagrant, that's of
course not on a laptop.
And for your final points:
(A) try to keep it simple
(B) unit tests don't usually make sense in prod - integration tests DO matter, and are supremely important, but spend time writing tests for your application, not tests for the config tool
(C) monitoring is supremely important
(D) build -> qa/stage -> prod
We agree on every point here! Thanks for summarizing your position and
again for the thoughtful response!
- PJ
