Async task / queue management

utoddl
Fri, 10 Nov 2023 21:44:18 +0000
Get Help

I’ve hit a situation and would appreciate advice on how to proceed. At the point my playbook starts, there are some number of predefined “jobs,” each of which should run on a predetermined “cluster”, and there are several clusters. The playbook targets “localhost”; each job’s “cluster” is a parameter to a simple task that uses a proprietary connection/execution module to run the jobs, the details of which aren’t relevant. (I promise!)

What we’ve done until now is fire off all the jobs on each cluster asynchronously. That works fine as long as there are fewer than about 3 jobs on any given cluster. More than that, and users start to notice degraded performance. When this was first put together, more than 3 jobs wasn’t an issue; now it is.

What I’d like to implement is a per-cluster queue wherein 3 of N jobs can run simultaneously, and when any of the 3 running jobs completes we can start another, etc. until there are no more pending jobs, then each of the last jobs completes in turn and that cluster is done. Simultaneously, the other clusters are off jobbing their respective jobs the same way.

This all worked really well in my head this morning! That was before I (re)discovered (A) that until: can’t be used on include_tasks: and (B) that async_status requires a particular job ID. Ansible doesn’t seem to have the equivalent of bash’s wait which can wait for any child process completion, or for any process id from a list. Maybe such a thing could be implemented by snooping around the temporary async job task files in ~/.ansible_async/? I don’t know, but implementing job queues in vanilla Ansible is kind of tough with neither A nor B available.

It wouldn’t be nearly as hard (i.e. it might be possible) to use the batch filter and group the jobs into sets of 3 per cluster and run each batch to completion. But we’re trying to minimize time, and that approach leaves gaps in utilization.

So that’s were I sit at the end of the day with text buffers full of non-working code and a wall-sore head full of shattered ideas. I’d appreciate reading your suggestions.

6 posts - 4 participants

Read full topic