How do you organize your private/internal/org/company Ansible projects?

Project Discussions
, ,
1

How do you organize your Ansible related content in your organizations?

A little background for why I’m asking: I am relatively new to my company, and have inherited a lot of Ansible technical debt. Even though I am new to the scene, I am pretty much the Ansible lead even though my only other Ansible team member is my Linux team lead (Linux team is much larger, but we’re on the Infra side).

Almost all of the existing Ansible content is in a single GitHub repository, and is all playbooks with import_playbook/include_tasks that would be better suited as roles. There’s a lot of things that are standard/common to all inventory hosts, and a lot of things that are application specific (and plenty of rhel 7/8/9 variants of the same thing, which again is better suited as a role that dynamically handles w/e differences there are between major OS releases).

I can write these roles just fine. That part is easy for me. I would also like to group related roles into collections. And, so far, I have two namespaces to place all of these collections under. One is for redoing legacy content, and the other is specifically for AWX CI pipelines that I’m working on.

How do you organize your Namespaces, Collections, Roles, Playbooks, and AWX “projects” in your git source control? Right now, I have a namespace level repo with all of the collections and subsequent roles, and added this repo as a git submodule to the legacy Ansible repo. Then I’ve started creating new playbooks to use those in the legacy repo to try and migrate old playbooks to the new ones. Unfortunately, I’ve grown to dislike this approach because it feels absolutely cluttered, but I’m not sure how I might reorganize this into something more manageable without simultaneously breaking the existing workflow.

Would it make more sense to have each collection in its own repo, so that I can actually add git release tags to match the collection version?

Edit: Just found How to arrange structure for the case multiple projects - Get Help - Ansible again, I saw this the other day and tried to find it again before posting here. I definitely appreciate @utoddl’s insights there. Made myself a bookmark this time.

Do we have any sort of FAQ/Guide/skeleton/overview for this sort of thing anywhere?

1 Like
2

This sounds similar to @romainquere 's situation that I responded to a few days ago:

The big problem […] [is that] everything seems to be “up a level” above where it ought to be. I’m guessing that your playbooks should probably be roles, and your roles (if you have any) should probably be task files within those roles.

But I’m not throwing stones at your predecessors’ work. When they were just getting started, not really knowing what they were doing, and got something to work, they likely thought, “Okay, that’s how you’re supposed to do it!” and did it the same way from then on. You can dig a really nice, deep hole before figuring out it’s in the wrong place.

I described our setup in that other thread, and as I indicated back in November I’ve started to wonder what we’re missing because we’re using a structure that pre-dates collections.

If you restructure, your choices are “nip and tuck” vs “wreck and rebuild”, although you don’t actually have to wreck; you can rebuild while the old stuff continues to run. But it is worth asking: If you could (or had to) start from scratch, how would you make it look? Being familiar with your current setup makes it really hard to avoid letting the old influence your answer. Only then, though, should you start the process of deciding how to get from where you are to where you want to be. Or to put it another way, be really clear where you want to go before coming up with a transition plan. Above all, don’t just keep tweaking things in the hope that eventually they’ll get better.

That’s way too general to be of practical use, but that’s a luxury I have by being unfamiliar with your setup. :slight_smile: I’m guessing most of us have only really done things one way most of the time, and I’m not confident enough in “our way” to say you should do it the way we do.

Here’s a guideline I’ll throw in for free. I may have come up with it myself, or I might be repeating something I heard way back; I honestly do not know. But it has to do with deciding whether to put something directly in a playbook vs. in some task file. And it’s a guideline rather than a rule, so you’re free to violate it as long as you do so with purpose. Here goes: Playbooks determine what assets to do things to; task files do the things. Not very deep, or maybe profound. I can’t tell! As an organizing principle, it my keep you from digging some holes in the wrong places.

Do follow-up and keep us posted on how you change things. (As my son says often, “I try to benefit from the mistakes of those who take my advice.”)

2 Likes
3

Well, for one thing so far, I’ve decided to move collections into their own git repositories. I’m keeping the namespace-level repo that had all of the collections before, but I am replacing the collection-level folders with git submodules to their newly created git repository. The namespace-level repository will remain a git submodule in the legacy ansible repository to use there. It’s a little tedius, but I can more easily do collection version control for the legacy repo.

In the meantime, anything new I think might go into a new AWX-centric repository, if not an application specific repository that is shared with other teams (DBA’s/Developers, etc). I want to keep it agnostic enough that I can run from local, but I want to keep in mind the leverage offered by AWX.

2 Likes
4

That sounds good. All our individual projects can run locally or in AWX. But of course AWX allows for workflows, too, which we don’t try to duplicate from our desktops or our ssh bastion.

1 Like
5

AWX allows for workflows, too, which we don’t try to duplicate from our desktops

Yeah that’s almost impossible. If you treat a playbook as a proper book, you can list plays in a sort of workflow, but there’s no success/fail logic to control which plays do or don’t run. Only a handful of my playbooks have multiple plays in them.

That said, I have an AWX workflow that builds… AWX.

image
image1896×809 69.8 KB

The two Galaxy Repository syncs are for prod/dev instances. The prod one has a firewall rule for the galaxy.ansible.com, the dev one doesn’t so it pulls from prod. This workflow builds both dev and production pipelines, so the awx-ee that gets built uses one private galaxy server or the other.

This wouldn’t be fun to squeeze into a single playbook, lol.

2 Likes
6

Wow. Sweet. We only have a few workflows, and they are simpler than that.

I wanted to ask about your inventory. How dynamic is it? Ours is quite static. In fact, we can go months without changes. We have some deeply nested groups – maybe 4 levels deep – but all our inventory host names are also host FQDNs. And I know of only 1 exception to our hosts showing up no more than once in our .ini inventory. I.e. a strict group hierarchy. No inventory variables; that’s all in host_vars and group_vars, so it’s pretty easy to find where something gets set for a host or group. Our inventory strictly is for group hierarchies.

It feels like we’re cheating. :slight_smile:

2 Likes
7

We have Red Hat Satellite as a “source of truth”, not to mention VCenter and even AD if we really wanted to for dynamic inventories. The legacy inventory is “static”, but used a shell script and hammer to generate an INI style inventory. Since I took over Satellite as well, I’ve stopped using that script and maintain the static inventory manually.

New stuff is using Satellite, though it’s quite slow on my sandbox even with a redis inventory cache. Works fabulously in AWX though, but I’ve yet to commit many job templates there since I don’t consider the AWX instances to be in a “production ready” state (I want AWX to be defined/deployed as code, but that isn’t done yet).

Most groups are defined by Satellite, which are not nested, even if they’re in nested Satellite hostgroups, but I can define static groups in projects if I include them as a source in the same AWX inventory. That’s where I intend define group_vars that are application specific.

Edit: I do use Satellite for a couple of Ansible Roles, and those role vars are defined in Satellite. They’re defined at Org level, but I could define them in a number of other places if it made sense. These vars get picked up by the dynamic inventory script, which can be handy.

2 Likes