# AWX Instance group workload management **URL:** \ **Category:** Get Help\ **Tags:** awx\ **Created:** [June 10, 2024, 9:46am UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271 "2024-06-10T09:46:58Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![prateektl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/prateektl/32/8985_2.png) [@prateektl](https://forum.ansible.com/u/prateektl)\ **Post date:** [June 10, 2024, 9:46am UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/1 "2024-06-10T09:46:58Z")
I have an Instance group having 2 ec2 instances attached to the group. When I execute the workflow template (Heavy load), I observer that the job template gets executed only one instance which is part of the instance group. Load is not being shared between the instances, rather the task templates are in waiting for the previous job template to be completed as its being executed only on 1 instance. How can we distribute the load, so that the overall execution time can be reduced. ---
**Author:** ![fosterseth](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/fosterseth/32/101_2.png) [@fosterseth](https://forum.ansible.com/u/fosterseth)\ **Post date:** [June 10, 2024, 3:22pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/2 "2024-06-10T15:22:10Z")
are the workflow nodes using the same template? if so, you will want to enable “concurrent jobs” on the job template Seth ---
**Author:** ![prateektl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/prateektl/32/8985_2.png) [@prateektl](https://forum.ansible.com/u/prateektl)\ **Post date:** [June 11, 2024, 1:45pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/3 "2024-06-11T13:45:09Z")
No the workflow template has different individual job templates ---
**Author:** ![alvaroc20](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/alvaroc20/32/3485_2.png) [@alvaroc20](https://forum.ansible.com/u/alvaroc20)\ **Post date:** [June 11, 2024, 4:29pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/4 "2024-06-11T16:29:22Z")
Hi Prateek! You should still try to have the ‘concurrent jobs’ option enabled in both templates, regardless of whether they are the same or not. ---
**Author:** ![prateektl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/prateektl/32/8985_2.png) [@prateektl](https://forum.ansible.com/u/prateektl)\ **Post date:** [June 11, 2024, 5:07pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/5 "2024-06-11T17:07:07Z")
Let me enable the ‘concurrent jobs’ option and test. Will share the results here. ---
**Author:** ![prateektl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/prateektl/32/8985_2.png) [@prateektl](https://forum.ansible.com/u/prateektl)\ **Post date:** [June 11, 2024, 5:21pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/6 "2024-06-11T17:21:36Z")
Below is the structure of the work flow template I’m using. ![image](https://us1.discourse-cdn.com/ansible/original/2X/d/db6be5007eaf8709b0983a071fe70b5531245090.png) I have enabled concurrent jobs option on all the individual job template. For the entire Workflow template I’m passing the inventory with host of 15. In the inventory i have selected the instance group, which will be applied to the entire work flow template. This instance group has 2 instances (A and B) But even after enabling concurrent jobs option as suggested, i see all the job templates in the work flow template gets executed against instance A. Where as the instance B is in idle state. Only observation is that the **instance A** specs are higher in terms of RAM, CPU and memory when compared to **instance B** ---
**Author:** ![fosterseth](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/fosterseth/32/101_2.png) [@fosterseth](https://forum.ansible.com/u/fosterseth)\ **Post date:** [June 11, 2024, 5:57pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/7 "2024-06-11T17:57:05Z")
a couple of things to check for 1. make sure Prevent Instance Group Fallback is disabled on JT (and inventory and organization if IGs are set there too) 2. You should see in the task logs information as to why pending jobs are not running e.g. “needs capacity” ```auto 2024-06-11 17:53:25,847 INFO [2f8865a6] awx.analytics.job_lifecycle job-7 needs capacity {"type": "job", "task_id": 7, "state": "needs_capacity", "work_unit_id": null, "task_name": "pause"} ``` what does it say for you? 1. when starting the workflow and things are running, what is the current capacity % of instance A and instance B? ---
**Author:** ![prateektl](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/prateektl/32/8985_2.png) [@prateektl](https://forum.ansible.com/u/prateektl)\ **Post date:** [June 13, 2024, 9:24am UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/8 "2024-06-13T09:24:23Z")
It took some time to validate all the above mentioned points and below are the results: 1. make sure Prevent Instance Group Fallback is disabled on JT (and inventory and organization if IGs are set there too) : **Disabled and tested. It didnt help** For the 2nd and 3rd questions below is the answers: Yes, when we triggered a Workflow template with more than 50+ nodes we are observing this “needs capacity” in the logs: ![image](https://us1.discourse-cdn.com/ansible/original/2X/c/c8deb6abc830bb3407503aec01b1e239530bda26.jpeg) But the surprising part in this testing is the load of the instance. Below is the screenshot taken of the instance at various time point. a\> Right at the point when the Work Flow Template was triggered: ![image](https://us1.discourse-cdn.com/ansible/original/2X/3/3bc89ef02865f62fd85fe39dc6d6dc8148e91a18.png) b\> Max load in terms of % (never crossed beyond this) ![image](https://us1.discourse-cdn.com/ansible/original/2X/0/09ec6f79f310270989803aee6f7e66ce54fe19b4.png) c\> When the used capacity was at 28%, below is the jobs in pending ![image](https://us1.discourse-cdn.com/ansible/original/2X/2/2ea5c720d28965c525284cd39c83016d32145017.png) Observation: Even if the log states **needs capacity** , second instance in the instance group is not utilized at all during the entire workflow template execution duration. ---
**Author:** ![fosterseth](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/fosterseth/32/101_2.png) [@fosterseth](https://forum.ansible.com/u/fosterseth)\ **Post date:** [June 13, 2024, 8:15pm UTC](https://forum.ansible.com/t/awx-instance-group-workload-management/6271/9 "2024-06-13T20:15:02Z")
your control node also has capacity that gets consumed in addition to the execution nodes - what does that report? Can you turn on log level debug and grep the task container logs for “capacity”? does it provide extra reasoning for why these jobs need capacity? Seth