Excluding jobs by priority, schedules

Get Help
, ,
1

We’re using AWX (looking at AAP, but I don’t believe it would change anything relevant here) with several organizations. One is systems which provisions boxes (VMs mostly) with patched OS and Ansible automation account already set up. Another is middleware which installs, configures, and patches applications. Each host OS gets patched and rebooted monthly on a schedule determined by the environment (dev, tst, prd), parity (“apphost0d” vs “apphost1d”) and data center of residence. All this work happens through jobs in the same AWX instance. Normal application upgrades avoid relevant patch-n-reboot windows.

However, “normal” patching and upgrading has taken a back seat to “urgent” patches as CVEs of increasing severity and frequency hit us. This leads to events like last night, where an out-of-band middleware application upgrade collided with an OS system patch-n-reboot. We have a friendly handshake agreement between the organizations to avoid such self-inflicted oopsies, but “don’t be stupid” makes for a fragile policy.

The only exclusion I’m aware of in AWX is the option to preclude simultaneous execution of multiple instances of a specific job template. There’s nothing that could do that at a higher level — across multiple projects or even multiple organizations.

Okay, so, what’s the question? That’s exactly my question! I’m trying to imagine a more robust mechanism than “don’t be stupid” to avoid these types of scheduling issues. I don’t really expect anybody to have a complete solution ready to apply overnight. Ideally what I’m hoping for is ideas about how to think about the problem, so that the relevant questions naturally pop out, questions that could be followed by implementation of the answers. The fact that we’ve been talking internally about this all morning and we haven’t been able to formulate a decent question indicates we aren’t thinking about it in the right way yet.

Thanks for any perspectives you’d care to share.

2

I have some ideas for this, but I’ll need to play around between now and EoD Monday to fully get the details ironed out. Long story short, I believe we can figure something out.

3

Sounds kind of like when two processes write to one file. So you could use something like a lock file.
At the start of a playbook check for a file. If it exists wait, if it doesn’t, make it. At the end of the playbook remove the file. All orgs need to implement, or you force it with workflows

4

My idea is very similar, except I’m using the host filesystem fact storage (this way it doesn’t matter at all about what is stored anywhere in AAP/AWX). This allows gather_facts to instantly have access to the lock value at any point. Mine also is designed to require a workflow job template so that it’s easier to enforce a “clear” at the end, and for any/all failures that happen in the middle, to unset the fact that was set at the beginning (so the host doesn’t stay in a locked state).