Source control of playbooks - how do you edit, test and perform version control....

Archives Ansible Project
,
1

I work in a Windows environment, pretty new to Ansible and keen to hear how you perform version control on your playbooks, how you edit and test them etc.

Do you work in a team where you have multiple engineers editing and creating new plays? You clone the master branch and then merge changes?

If you clone to your local machine - then how do you edit and test and deploy? Are you just copying the plays up to the Ansible host?

I’m about to start building some projects in AWX - these projects will connect to a git repo (GitLab) to pull the plays from a master branch.

2

I work in an windows environment too.

Mostly I use Windows Subsystem for Linux (WSL) also known as Bash on Windows. This lets me pull from our source code repo, edit and check in changes and run playbooks (ansible runs fine inside WSL), also you can use ansible-vault to encrypt your secrets so they never land in your source control.

Sometimes I target VMs running in VirtualBox, other times I target vms in our dev environment.

If you like vagrant you can run vagrant inside WSL and target virtualbox running on the windows 10 host after setting a couple of environment variables - see https://www.vagrantup.com/docs/other/wsl.html

Mostly we just pull and update source code and then push changes. This has worked fine for us. When switching to a new version of ansible, typically I branch, make changes and test them before letting others test and use the new version of ansible.

For test and production we have set up ansible in vms. Its relatively straightforward to create a playbook to create an ansible controller on a freshly provisioned vm, so you can bootstrap the other ansible controllers from your dev machine if you need to.

Hope this helps,

Jon

3

Thanks for your insight Jon. When you use WSL - are you pulling from your source code repo from within the Ansible controller? You then just test and edit the code directly on the controller?

What do you use for a code repo? Hosted Gitlab?

4

I’ll jump in with with what we are doing.

We have a Git server (GitHub on site, but it could be any sourcecode repository, local or hosted), and each of us have a copy of the repository in our test/dev environment. Development on the files (playbooks, roles, vars, hosts, etc) are modified/pulled/pushed/etc just like any other source code.

On my workstation I have my own hosts file that points to my small test environment (a few VMs spun up in the lab) that I can point the playbooks at. When I get the playbooks working there, I push the code to the repository. Then someone else pulls down the changes to their workstation and runs the playbooks against their test hosts. (Or I can do the same with their updates.)

When we’re good with what we’ve produced, someone from the ops team pulls down the repo to their system which has access to the QA (pre-production) environment, runs the playbook in check mode to validate, then runs the playbook during the next change window.

We also have Ansible Tower (RedHat supported version of AWX) which has jobs/templates setup that automatically check-out the repository to a temporary location on the Tower server, and run the playbook in that repository against a set of hosts with a specific set of credentials. This way teams can setup their own host inventories on Tower, we can safely store credentials for Ansible to use to login to these systems (developers don’t need the passwords in this case), and we have a job history log for review.

5

Thanks for all the helpful info Dan.

Are you creating a new branch when you develop/edit plays or are you just cloning the master?

6

We’re still pretty new in our process, so mainly it’s just clones of the master.

In the future I want to setup branches so that the production ops team uses “version 1.0” code, while we work on “version 2.0” playbooks. Then it is/should be a matter of updating their process to use the new version in the git checkout.

7

Yeah I can pull from our source code repo straight into my WSL bash session, edit with vim or nano and run playbooks at the command line.

I believe you can point host editors at files within WSL if you want but I’ve never bothered, WSL is similar enough to a terminal session on a linux box.

Using employer mandated mercurial for source control. No idea why mercurial isn’t more popular, but that’s another story.

8

Thanks for your replys. I think getting a process in place that everyone follows will be the biggest challenge.

9

Hi Dan. I’ve just started using AWX today. Do you mind giving me a bit of background on how you have set yours up?

We currently have no config mgmt for our server environment and for the project I’m on (+ any other project) we will use Ansible to manage the configuration changes. I’ve so far created a stack of playbooks, each within their own sub directory which includes a vars folder, scripts where necessary etc. I do however have a global inventory file that all plays reference outside of the playbook sub directory. Running the required plays via CLI has been fine for myself and I have all the playbook projects listed as subfolders within a gitlab repo called playbooks.

Now that I want to use AWX so other people can manage the plays I’m doubting whether my playbook structure is ideal, plus the fact from what I’ve seen each playbook directory (app/service - e.g GIS, SQL) directory requires an inventory file so AWX can reference it.

Do you have an inventory file in each playbook directory? All playbooks and inventory in single directory and no sub folders to separate the plays?

Thanks in advance. Excuse typos… typed on mobile!

10

Hi Lee, sorry for taking so long to respond.

To be honest our setup has grown quite organically in some cases, but we’re trying to get some guard-rails around things (e.g. consistent groups defined in common tools we use, access to credentials control, inventory maintenance, etc).

We currently have no config mgmt for our server environment and for the project
I’m on (+ any other project) we will use Ansible to manage the configuration changes.

IMHO, that’s a good start and mirrors how we got started. Our OS deployment was in bad shape and the legacy scripts that worked up to RHEL-6 were failing in RHEL-7. Plus our environment had grown so complex and diverse that the build scripts we used to automate configurations (DNS, NTP, LDAP, SUDO, Satellite/patching, etc) were not keeping up and their original intent was for a much more manual build process.

When we started the push to move to RHEL-7 so we could get off of vulnerable and un-supported RHEL-5 and older, we decided to make that the time to use Ansible. We didn’t have Tower (and AWX wasn’t open-sourced yet) so our VMWare build process called out to a script on a central server that kicked off the Ansible playbook. Our RHEL-7 templates are built by Packer (https://www.packer.io/) and have a build SSH key included. This SSH key is used by SSH on the central server to connect and execute the Ansible playbooks.

The playbook is made up of a number of Ansible roles that each perform a specific task:

The nice thing is that each role should actually be owned by the team responsible for that service. For instance, we use Zabbix for monitoring and the Zabbix team has kept up the setup_monitoring role files themselves building the configuration files, pointing to the install files, etc.

Breaking the project into roles also helped a lot to work around pain points. For instance, in the early version the roles for DNS, then hostname then Satellite were called but we got some intermittent failures. After working through the logs, we saw that the time on the new VM was incorrect and the SSL certificate that Satellite was providing wasn’t valid with the local clock. Rearranging the roles and getting NTP setup earlier resolved that quickly. It did mean that we had to include NTP in the base system, but that was a minor increase in size.

Converting all this to a playbook that is reusable by Tower/AWX is my next step, too. With the exception of the monitoring team, all other changes that come in are either an email or other non-git controlled change that I have to incorporate and help them test. This has required me to start learning the basics of Git branching so I can keep the master branch stable while we/I are working on fixes and features.

Inventories will be a challenge, too. On our older CLI environment, the individual users kept their own inventory files and would create the text inventory files - the most common was to name these files with the number of the trouble-ticket they were working on (to build new servers, fix issues, etc). Given the number of people that had access, it was safer to let each person keep their inventory files in their home directories. (Putting them all in one location would lead to someone accidentally running a playbook on “inventory.TKT98765” when it should have been “inventory.TKT98775”.)

In the Tower environment, the bigger project has been given the direction by our corporate designers/architects to have our development departments broken up into six organizations that we’ve mapped into Git and Tower (using LDAP). Management of the users is easy, but I’m a bit concerned that the size of those organizations might lead to teams inadvertently choosing the wrong inventory and making changes to systems they shouldn’t have. Part of it will be a learning and education experience, but I’d rather get the guard rails setup to reduce that chance. Right now most teams are building their own inventory files and that will probably continue to work. Within their organizations they have enough communication that the naming collision shouldn’t cause a problem. If you come up with a slick way to add inventory to a wider group but still maintain access controls on the servers I’m all ears.

At a future date I’d love to be able to have Tower query our CMDB or VMware infrastructure and get the inventory back dynamically. That’s a long way off, but it’s a goal I’m aiming at. This will permit me (and more importantly my users and their managers) the ability to adjust the inventory based on business need. (“Select all servers which handle CustomerX and ensure this patch is applied.” Or “Configure the ssh service on all systems in the DMZ to only permit logins from the internal management subnet.”)