We’re using AWX (looking at AAP, but I don’t believe it would change anything relevant here) with several organizations. One is systems which provisions boxes (VMs mostly) with patched OS and Ansible automation account already set up. Another is middleware which installs, configures, and patches applications. Each host OS gets patched and rebooted monthly on a schedule determined by the environment (dev, tst, prd), parity (“apphost0d” vs “apphost1d”) and data center of residence. All this work happens through jobs in the same AWX instance. Normal application upgrades avoid relevant patch-n-reboot windows.
However, “normal” patching and upgrading has taken a back seat to “urgent” patches as CVEs of increasing severity and frequency hit us. This leads to events like last night, where an out-of-band middleware application upgrade collided with an OS system patch-n-reboot. We have a friendly handshake agreement between the organizations to avoid such self-inflicted oopsies, but “don’t be stupid” makes for a fragile policy.
The only exclusion I’m aware of in AWX is the option to preclude simultaneous execution of multiple instances of a specific job template. There’s nothing that could do that at a higher level — across multiple projects or even multiple organizations.
Okay, so, what’s the question? That’s exactly my question! I’m trying to imagine a more robust mechanism than “don’t be stupid” to avoid these types of scheduling issues. I don’t really expect anybody to have a complete solution ready to apply overnight. Ideally what I’m hoping for is ideas about how to think about the problem, so that the relevant questions naturally pop out, questions that could be followed by implementation of the answers. The fact that we’ve been talking internally about this all morning and we haven’t been able to formulate a decent question indicates we aren’t thinking about it in the right way yet.
Thanks for any perspectives you’d care to share.