# AWX Job Workflow Management **URL:** \ **Category:** Get Help\ **Tags:** documentation, awx\ **Created:** [November 14, 2023, 2:29pm UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364 "2023-11-14T14:29:51Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sirjaren](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@sirjaren](https://forum.ansible.com/u/sirjaren)\ **Post date:** [November 14, 2023, 2:29pm UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364/1 "2023-11-14T14:29:51Z")
I would like to understand and address how Workflow Jobs are managed in AWX better. I made a comment to a Github issue where I thought it was appropriate, but I realize the community would have a hard time finding it. While the Github issue has the information and questions I have with an example architecture, I’m curious how others have managed to do maintenance in AWX on K8s while preventing unexpected job termination within workflows. Even if jobs from a workflow are on AWX K8s nodes that are not undergoing maintenance, bringing down a controller node in AWX can terminate those workflow jobs across said other nodes. Much more information in the relevant Github issue below: [https://github.com/ansible/awx/issues/13848](https://github.com/ansible/awx/issues/13848) Since I added the documentation tag, I figured I would include relevant docs I have been perusing which may apply here: > > ---
**Author:** ![TheRealHaoLiu](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/therealhaoliu/32/100_2.png) [@TheRealHaoLiu](https://forum.ansible.com/u/TheRealHaoLiu)\ **Post date:** [February 7, 2024, 7:48pm UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364/2 "2024-02-07T19:48:25Z")
Last year we made an attempt add adding more resiliency on AWX around k8s node maintenance I made a YouTube video about it [https://www.youtube.com/watch?v=EqYl2hDs90c](https://www.youtube.com/watch?v=EqYl2hDs90c) (like and subscribe and ring the bell 😉) This feature in conjunction with [Disruptions | Kubernetes](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) _should_ allow you to do a rolling upgrade of K8S cluster underneath without disruption to jobs I had a draft blog about how to do this (that I didn’t finish \>.\<)… ---
**Author:** ![sirjaren](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@sirjaren](https://forum.ansible.com/u/sirjaren)\ **Post date:** [February 7, 2024, 8:40pm UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364/3 "2024-02-07T20:40:51Z")
> (like and subscribe and ring the bell 😉) (already subscribed!) I appreciate the feedback! 😍 I have already seen the YouTube video (great video, btw!) and looked at pod disruption budgets. They all work great but do not prevent disruption, just manages expectations and tolerates some disruptions. The part I am having concerns with is while job templates are easy enough to handle without killing jobs (since all jobs are exposed as pods), the workflow jobs are not exposed in the same way. So, if you bring down an AWX node that is part of a workflow, it seems that it can kill the workflow job (which may contain other workflows or regular jobs). That’s the part where I haven’t been able to work around. ☹ ---
**Author:** ![TheRealHaoLiu](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/therealhaoliu/32/100_2.png) [@TheRealHaoLiu](https://forum.ansible.com/u/TheRealHaoLiu)\ **Post date:** [February 8, 2024, 2:59am UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364/4 "2024-02-08T02:59:38Z")
Gotcha. Thanks for pointing out the gap. Let me discuss with the team and see if we can figure something out. Can you put up an RFE and provide detail of how you configure your environment and the “minimal” reproducer ---
**Author:** ![sirjaren](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@sirjaren](https://forum.ansible.com/u/sirjaren)\ **Post date:** [February 8, 2024, 3:54am UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364/5 "2024-02-08T03:54:32Z")
> [@TheRealHaoLiu](#): > > Can you put up an RFE and provide detail of how you configure your environment and the “minimal” reproducer I did a fairly exhaustive breakdown in the Github issue here (including breaking down the architecture with examples): > > > \### Please confirm the following > > \- \[X\] I agree to follow this project's \[code o…f conduct\](https://docs.ansible.com/ansible/latest/community/code\_of\_conduct.html). > \- \[X\] I have checked the \[current issues\](https://github.com/ansible/awx/issues) for duplicates. > \- \[X\] I understand that AWX is open source software provided for free and that I might not receive a timely response. > > \### Feature type > > New Feature > > \### Feature Summary > > Scenario: > > - A job that sleeps for several minutes is launched on an execution node, so \`controller\_node\` != \`execution\_node\` > - The job successfully starts and starts producing events > - Mid-run, services are restarted on the job's controller node > > Right now, this job will be reaped. We \_hate\_ it when jobs get reaped. > > Details of proposed solution, subject to change: > > - replace the \`reap\` method called from \`cluster\_node\_heatbeat\` with a different reconciliation method called from \`awx\_receptor\_workunit\_reaper\`. This method has access to \`receptorctl status\`. > - in addition, this method will be given access to the process list, \`worker\_tasks\` > - in addition, this method may pull the list of running jobs from the database, as needed > - In the event that the database status is "running" but the process is missing (timing issues assumed to be worked out), it will send a message back to the dispatcher main process if the \_receptor\_ status is still active. > - When the dispatcher gets a message that an active job is orphaned, it will launch \`RunJob\` or its equivalent. > - Instead of starting the job, it will pick up \`process\`ing from the last line processed, which can be ascertained from the saved events or some other means of tracking. > > \### Select the relevant components > > \- \[\] UI > \- \[X\] API > \- \[\] Docs > \- \[\] Collection > \- \[\] CLI > \- \[\] Other > > \### Steps to reproduce > > See feature summary > > \### Current results > > Reaper message, job canceled > > \### Sugested feature result > > Jobs should never be reaped - just have processing resumed. > > Receptor becomes source of ultimate truth. > > \### Additional information > > \_No response\_ I’m happy to do an RFE if needed, let me know where to submit and I can work something out. Thanks again! ---
**Author:** ![TheRealHaoLiu](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/therealhaoliu/32/100_2.png) [@TheRealHaoLiu](https://forum.ansible.com/u/TheRealHaoLiu)\ **Post date:** [February 8, 2024, 6:18pm UTC](https://forum.ansible.com/t/awx-job-workflow-management/2364/6 "2024-02-08T18:18:49Z")
the one created by @AlanCoding works