# Infrastructure Development Workflow? **URL:** \ **Category:** Ansible Project\ **Tags:** aws\ **Created:** [September 24, 2014, 11:34am UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318 "2014-09-24T11:34:36Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Paul\_Julius](https://avatars.discourse-cdn.com/v4/letter/p/898d66/32.png) [@Paul\_Julius](https://forum.ansible.com/u/Paul_Julius)\ **Post date:** [September 24, 2014, 11:34am UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318/1 "2014-09-24T11:34:36Z")
Hi Ansible folks! Cross posting here from the Vagrant and Packer mailing lists, because I thought that people on the Ansible mailing list would probably have really great ideas to share with me. I am really enjoying my current workflow with Ansible. I love it because it models my Dev workflow, almost precisely, thereby getting me close to the “infrastructure as code” nirvana. We just wrapped up CITCON Zagreb [footnote:x] where I was talking to other DevOps folks about how they use Ansible. There were some interesting ideas! I wanted to ask on this mailing list what other people are doing. I would love any feedback. 1. Pickup story to automate deployment of something 2. Write broken acceptance test (in something like Cucumber) 3. Put acceptance test in “In Progress” bucket [footnote:xx]1. Write broken functional test (in something like Serverspec) 4. Write just enough code to make it pass - Vagrant + Ansible + Virtualbox 5. Refactor - Good sense and Fowler’s patterns 6. Run my pre-commit build - Packer + Ansible + AWS (or whatever target platform) 7. Commit/push - Git (or VCS of choice) 8. Go to step 3, until acceptance test passes 9. Review with customer, maybe go back to step 2 10. Move acceptance test into the “Complete” bucket 11. Story complete At step 7, of course, my CI server picks up the change and sends it through the following stages of my pipeline: 1. Checkout from Git 2. Runs Vagrant+Ansible+AWS 3. Executes functional tests - Serverspec - 0% tolerance for broken tests 4. Executes “Complete” Acceptance tests against the Vagrant instance - 0% tolerance for breakages 5. Executes “In Progress” Acceptance tests against the Vagrant instance - reporting on results and fail if a test passes [footnote:xxx]1. Runs Packer+Ansible+AWS 6. Execute functional tests - 0% tolerance for broken tests 7. Execute Acceptance tests - 0% tolerance for all “Complete” and “In Progress” acceptance tests [footnote:xxxx]1. Alerts the world that a fresh new AMI is ready to consume! There would be more Continuous Delivery related steps, but I will leave those off for brevity. Thoughts? Is this what other people are doing? Sincerely, PJ (Paul Julius) Footnotes: ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 24, 2014, 2:14pm UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318/2 "2014-09-24T14:14:43Z")
> Hi Ansible folks! > > Cross posting here from the Vagrant and Packer mailing lists, because I > thought that people on the Ansible mailing list would probably have really > great ideas to share with me. > > I am really enjoying my current workflow with Ansible. I love it because > it models my Dev workflow, almost precisely, thereby getting me close to > the "infrastructure as code" nirvana. > > We just wrapped up CITCON Zagreb [footnote:x] where I was talking to other > DevOps folks about how they use Ansible. There were some interesting ideas! > I wanted to ask on this mailing list what other people are doing. I would > love any feedback. > >    1. Pickup story to automate deployment of something >    2. Write broken acceptance test (in something like Cucumber) >    1. Put acceptance test in "In Progress" bucket [footnote:xx] If you're talking about your own unit tests, this is up to you here. What follows may be percieved as a bit of a rant, and I don't want it to be percieved as much, but I think most people who come from this uber-testing culture have made things incredibly too hard, create more work for themselves, and as a result move slower - not really breaking less - but doing extra work. Work that ansible (and declarative CM systems in general) are designed to not make to be a thing. As such, I strongly favor integration tests run against stage environments to make sure things work, and coupling that in a rolling update against a production environment as a condition to decide whether to add something back to a load balanced environment there as well - ideally using the same tests, but that's not always possible. While more of a unit test thing, I personally find Cucumber to be wasted effort because most product-management types (I guess I qualify) are not going to be the ones writing test stubs, so it's usually fine to just go straight to tests. That being said, I think there's a lot of niceness to come out of the Ruby testing community - I just never felt Cucumber was one of those things. Good integration tests for ops are more important -- is this web service responsive? Not things like "is the file permission of this file the same thing it is listed in the configuration system - as that just duplicates configuration in two places. >    1. Write broken functional test (in something like Serverspec) I'm strongly not a fan of ServerSpec, because I think it fails to understand the basis of configuration management in most places - that you do not have to check for what the config tool already asserts. I'm much more of a fan of checking to make sure a deployed application works. We've written about this here: [http://docs.ansible.com/test\_strategies.html](http://docs.ansible.com/test_strategies.html) >    1. >    2. Write just enough code to make it pass - Vagrant + Ansible + >    Virtualbox >    3. Refactor - Good sense and Fowler's patterns While some of his posts are useful, Fowler's refactoring suggests some rather silly things for code - change one thing, recompile, re-run tests, that would utterly sabotage development efficiency in most cases. He tries to make code design a bit too mechanical, IMHO. Unrelated, but somewhat on the Fowler-worship front: See somewhat related - [http://perl.plover.com/yak/design/samples/slide001.html](http://perl.plover.com/yak/design/samples/slide001.html) I'm also not really sure how Design Patterns apply so much for a configuration system 🙂 >    1. >    2. Run my pre-commit build - Packer + Ansible + AWS (or whatever >    target platform) >    3. Commit/push - Git (or VCS of choice) >    4. Go to step 3, until acceptance test passes >    5. Review with customer, maybe go back to step 2 >    6. Move acceptance test into the "Complete" bucket >    7. Story complete > > At step 7, of course, my CI server picks up the change and sends it > through the following stages of my pipeline: Here is the outline of my slides from my talk to the NYC Continuous Deployment group that suggests a good dev-\>stage-\>test workflow and how to incorporate tests into a CD pipeline: [https://gist.githubusercontent.com/brokenthumbs/7fd7992fc1af0cfcc63d/raw/e0c750e00aeb6e62da04fd680346516cb88f8ae5/gistfile1.txt](https://gist.githubusercontent.com/brokenthumbs/7fd7992fc1af0cfcc63d/raw/e0c750e00aeb6e62da04fd680346516cb88f8ae5/gistfile1.txt) >    1. Checkout from Git >    2. Runs Vagrant+Ansible+AWS >    1. Executes functional tests - Serverspec - 0% tolerance for broken >       tests >       2. Executes "Complete" Acceptance tests against the Vagrant >       instance - 0% tolerance for breakages >       3. Executes "In Progress" Acceptance tests against the Vagrant >       instance - reporting on results and fail if a test passes [footnote:xxx] Using Vagrant to push to AWS here seems weird to me, I'd probably just use the AWS modules in Ansible directly from from Jenkins to trigger my tests towards AWS, rather than kicking them off from a laptop. I guess TLDR is: (A) try to keep it simple (B) unit tests don't usually make sense in prod - integration tests DO matter, and are supremely important, but spend time writing tests for your application, not tests for the config tool (C) monitoring is supremely important (D) build -\> qa/stage -\> prod ---
**Author:** ![Paul\_Julius](https://avatars.discourse-cdn.com/v4/letter/p/898d66/32.png) [@Paul\_Julius](https://forum.ansible.com/u/Paul_Julius)\ **Post date:** [September 25, 2014, 4:50am UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318/3 "2014-09-25T04:50:09Z")
Hi Michael, Thanks for the response. Some excellent points. I'll respond to a couple. > As such, I strongly favor integration tests run against stage environments to make sure > things work, and coupling that in a rolling update against a production environment as > a condition to decide whether to add something back to a load balanced environment > there as well - ideally using the same tests, but that's not always possible. Agreed! Me too. I didn't address the downstream deployment stages, but this would be consistent with my approach. > While more of a unit test thing, I personally find Cucumber to be wasted effort because > most product-management types (I guess I qualify) are not going to be the ones writing > test stubs, so it's usually fine to just go straight to tests. Understood. On the contrary, I have found the type of tests I write when using a natural language approach to be a great mechanism for communicating requirements within a team. Though I wasn't sure, it read like you are thinking of Cucumber as a unit testing framework. I never think of Cucumber as a "unit" testing framework. In any case, my point isn't about Cucumber in general, but rather about the good practice of creating an acceptance test for a body of work about to be embarked upon. I am glad for teams to use whatever tool they think makes the most sense for their combination of team members and customers. By acceptance test, I simply mean an automated verification that the features we want actually do exist in the system that a customer and engineer can collaborate on. > Good integration tests for ops are more important -- is this web service responsive? Agreed. > Not things like "is the file permission of this file the same thing it is listed in the configuration > system - as that just duplicates configuration in two places. Duplication is precisely the point, sort of. I like Uncle Bob's point about unit testing being the double entry accounting of software engineering. We express the functionality we want in two ways: code and test. Then we automatically make sure our column sum equals our row sum. Over the long haul, I find the double entry approach to find problems fast when they are introduced. In the short term, they alert me quickly when I make mistakes. > I'm strongly not a fan of ServerSpec, because I think it fails to understand the basis of > configuration management in most places - that you do not have to check for what the > config tool already asserts. I understand your point. Usually, I try to make sure that my "functional tests" focus on testing my implementation. For example, I testing that starting a service has the expected results. CM tools can't do that generically, beyond asserting that they have in fact done as instructed. For example, if I have a "service: name=foo state=started" entry, all I can expect from Ansible natively is that it verifies that there is a service named "foo", that asking for it's status reports "running", and if not running, calling "start" puts it into that state. But, I'm missing my test for foo listening on a specific port and writing to a specific file, for example. That feels like a good use case for Serverspec. But, again, I don't care so much about the specific tool. I am happy for teams to choose whatever works best for them. My workflow focuses more applying TDD mentality to evolving infrastructure coding alongside the rest of my application. > We've written about this here: > > [http://docs.ansible.com/test\_strategies.html](http://docs.ansible.com/test_strategies.html) Thanks for the reference! And thanks for writing that up! I like the ideas a lot and they seem pretty consistent with how I like to approach things. One thing I like to do is have the opportunity to run tests in isolation from running the provisioning. That doesn't preclude me from putting more of them in the Ansible files, of course. I think the physical separation of test and implementation helps me wear the two different hats I need to wear to write good tests, but I'll gladly accept that as a personal preference. > While some of his posts are useful... I'll skip the fun discussion about patterns for now. Interesting points, and thank you! I assume that we can agree that "looking at what we have implemented and finding opportunities to improve it, non-functionally" is a good thing. > Here is the outline of my slides from my talk to the NYC Continuous Deployment group... Looks like a great talk. Sorry I missed it! > Using Vagrant to push to AWS here seems weird to me, I'd probably just use the AWS > modules in Ansible directly from from Jenkins to trigger my tests towards AWS, rather > than kicking them off from a laptop. I like the isolation of Ansible for configuration and Vagrant/Packer for machine instantiation. It seems to make for a good isolation of concerns. As for the "from a laptop" part, I like to test stuff from my dev environment, before I commit. So at that point it might be from my laptop. But, when the CI/CD server is calling Vagrant, that's of course not on a laptop. And for your final points: > (A) try to keep it simple > (B) unit tests don't usually make sense in prod - integration tests DO matter, and are supremely important, but spend time writing tests for your application, not tests for the config tool > (C) monitoring is supremely important > (D) build -\> qa/stage -\> prod We agree on every point here! Thanks for summarizing your position and again for the thoughtful response! - PJ ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 25, 2014, 12:01pm UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318/4 "2014-09-25T12:01:16Z")
> Hi Michael, > > Thanks for the response. Some excellent points. I'll respond to a couple. > > \> As such, I strongly favor integration tests run against stage > environments to make sure > \> things work, and coupling that in a rolling update against a production > environment as > \> a condition to decide whether to add something back to a load balanced > environment > \> there as well - ideally using the same tests, but that's not always > possible. > > Agreed! Me too. I didn't address the downstream deployment stages, but > this would be consistent with my approach. > > \> While more of a unit test thing, I personally find Cucumber to be wasted > effort because > \> most product-management types (I guess I qualify) are not going to be > the ones writing > \> test stubs, so it's usually fine to just go straight to tests. > > Understood. On the contrary, I have found the type of tests I write > when using a natural language approach to be a great mechanism for > communicating requirements within a team. Though I wasn't sure, it > read like you are thinking of Cucumber as a unit testing framework. I > never think of Cucumber as a "unit" testing framework. I guess what I'm trying to say is it's a language of defining "these are my tests", "someone go implement these". I would agree it's not specifically unit test driven, but I guess it's more of the statement that it doesn't provide a lot of value to me over well named functions and comments in the test file. Stubbed out test functions would do the same thing. A lot of users in ops land are new to development testing systems, so I want to make it clear this isn't required, and there's not a big gap that says you must incorporate 4 or 5 extra tools to use something as basic as Ansible in most cases. That's the important thing I'm trying to combat - the feeling that it's \*not easy\* and you need all this extra stuff. For those people that like it, great - I don't want them to stop doing that if that works for them for sure. > In any case, my point isn't about Cucumber in general, but rather > about the good practice of creating an acceptance test for a body of > work about to be embarked upon. I am glad for teams to use whatever > tool they think makes the most sense for their combination of team > members and customers. By acceptance test, I simply mean an automated > verification that the features we want actually do exist in the system > that a customer and engineer can collaborate on. > > \> Good integration tests for ops are more important -- is this web service > responsive? > > Agreed. > > \> Not things like "is the file permission of this file the same thing it > is listed in the configuration > \> system - as that just duplicates configuration in two places. > > Duplication is precisely the point, sort of. I like Uncle Bob's point > about unit testing being the double entry accounting of software > engineering. We express the functionality we want in two ways: code > and test. Then we automatically make sure our column sum equals our > row sum. This part I strongly disagree with. When configuration exists in two places, the maintaince costs double. A colleague raised a very good point WRT ServerSpec in that in many cases Puppet needs double checking more because it can fail and only partially evaluate a DAG, but Ansible will fail-first and make you fix a problem before proceeding. If it can't chmod something to a value, it will yell at you, and so forth. The checks you insert using get\_url and wait\_for and assert/stat, etc, can be part of the main playbook. Again the point is - don't make Ansible harder than it needs to be, a lot of the purpose of it is to allow you to quickly move on to other tasks. One of the pitfalls of "infrastructure is code" is it brought software development levels of complexity with it to infrastructure, when it should have brought only the idea of source control. This is what we're trying to change here 🙂 Hope that makes sense! ---
**Author:** ![Paul\_Julius](https://avatars.discourse-cdn.com/v4/letter/p/898d66/32.png) [@Paul\_Julius](https://forum.ansible.com/u/Paul_Julius)\ **Post date:** [September 26, 2014, 12:09am UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318/5 "2014-09-26T00:09:59Z")
Hi Michael, Thanks for sharing your perspective! This is great food for thought. Your passion for the topic oozes out of every sentence. It's inspiring. I have to take great issue with this assertion: > ...When configuration exists in two > places, the maintaince costs double. I have not found this to be true. Much like I have not found pair programming to double my feature development costs. It's an intuitive hypothesis, but not backed up by experiment after experiment after experiment. The double entry principle has saved my teams countless hours and problems. It is one factor in a series of principles related to Continuous Delivery that make complex systems easier to develop and maintain for years, by ever changing teams of people. I would be conservative in saying that well factored testing cuts the total cost of ownership in half, rather than doubling it. > The checks you insert using get\_url and wait\_for and assert/stat, etc, can > be part of the main playbook. > > Again the point is - don't make Ansible harder than it needs to be, a lot of > the purpose of it is to allow you to quickly move on to other tasks. I love this facet of Ansible. I love that mindset. Thank you, thank you. I think it has served the DevOps landscape very well. > One of the pitfalls of "infrastructure is code" is it brought software > development levels of complexity with it to infrastructure, when it should > have > brought only the idea of source control. > > This is what we're trying to change here 🙂 We clearly disagree on this point. Version control was the least of our worries, and we had it in many places before the notion of "infrastructure as code" ever came along. And certainly before Chef, Puppet or Ansible entered the picture. (As an example: People have done amazing things with version controlled Make for decades.) I strongly believe that we have gained much by applying the notion that configuring infrastructure is something that can be compiled once, tested throughout and run repeatably in many places, with a single button click/command. Ansible has done great things for the ops world. Ansible has been very well received by even the stodgiest of old school sys admins. But, I sincerely hope that Ansible's message does not become one that undoes the notions of testability, formality, and communication required to build complex systems, repeatably, reliably, consistently, maintainably, and easily that have been introduced in the last decade. I'll now circle back to an earlier comment that you made: > That's the important thing I'm trying to combat - the feeling that it's \*not > easy\* and you need all this extra stuff. > For those people that like it, great - I don't want them to stop doing that > if that works for them for sure. I can get on board with this type of thinking. What I am doing works really well for me. It works really well for the teams I have been on. It even works really well for teams I have advised from afar. But, like anything, it will certainly NOT work well for everyone. I am happy to accept that. I do feel compelled to give the enterprise sys ops folks something that I think will be sustainable. And doing things essentially the same way, except version controlled with an idempotent fail-fast tool, won't keep them employed at companies that have to innovate quickly enough to keep pace with their competitors. They must fundamentally change the way they interact with their product teams. All that said, I love Ansible as a tool. The Ansible team has done an amazing job making it a great framework! Your documentation is clean and understandable. You've helped open up DevOps to a segment that was too scared of the heavier weight alternatives. Keep up the good work! - PJ ---
**Author:** ![Michael\_DeHaan1](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@Michael\_DeHaan1](https://forum.ansible.com/u/Michael_DeHaan1)\ **Post date:** [September 26, 2014, 12:32pm UTC](https://forum.ansible.com/t/infrastructure-development-workflow/16318/6 "2014-09-26T12:32:00Z")
> The double entry principle has saved my teams countless hours and > problems. It is one factor in a series of principles related to > Continuous Delivery that make complex systems easier to develop and > maintain for years, by ever changing teams of people. I would be > conservative in saying that well factored testing cuts the total cost > of ownership in half, rather than doubling it. My suggestion is that the right things,not the wrong things, be tested. Test functionality, not implementation. > We clearly disagree on this point. Version control was the least of > our worries, and we had it in many places before the notion of > "infrastructure as code" ever came along. And certainly before Chef, > Puppet or Ansible entered the picture. (As an example: People have > done amazing things with version controlled Make for decades.) This is a logical fallacy to suggest I was suggesting version control is more important than validation. > I strongly believe that we have gained much by applying the notion > that configuring infrastructure is something that can be compiled > once, tested throughout and run repeatably in many places, with a > single button click/command. No one was suggesting not to test things. In fact, I've advocated for solid integration tests many times, including assertation that a system works before putting it back into a load balanced pool, etc. As well as stage environments. What I'm suggesting is that using serverspec to assert that /etc/motd has the right permissions when it's also specified in your ansible playbook is a waste of time. Focus on real tests that add value. > Ansible has done great things for the ops world. Ansible has been very > well received by even the stodgiest of old school sys admins. But, I > sincerely hope that Ansible's message does not become one that undoes > the notions of testability, formality, and communication required to > build complex systems, repeatably, reliably, consistently, > maintainably, and easily that have been introduced in the last decade. This part of the message is pretty content free and seems to having lack what was written above, and we have \*never\* said that. I'm just recoiling against a lot of people who don't understand testing, and say that "if testing is good, more testing is better", and they spend time testing the wrong things, like the permission of every file on the filesystem. Most of the serverspec content I've seen is an absolute waste of time, and the need for it came out of tools that did not fail fast, had unpredictable ordering, and other things, which are much strongly less of an issue in Ansible.