In my team we are trying to limit the amount of forks that a given organization would be able to use when running a job.
Our main goal is to share AWX with other teams. The problem is that whenever a job is being executed, other systems may be comprised due to overload. After doing some research, we concluded that a way to avoid this problem would be by limiting the amount of forks that each organization may use.
Having said this, is there any way to limit the amount of forks per organization by the system administrator? This way, each team will be able to use AWX without conflicts.
If this is not possible using organizations, it would be helpful to know if there is any other way to achieve this goal.
IMO you’d be better off updating and using Instance Groups to manage and reserve capacity. In a clustered AWX environment, they let you assign an Organization to a single instance(s) of AWX and that’s all the capacity that will be available for that Organization. Other Orgs on different IGs will be unaffected even if one of the instances does get overloaded.
That being said, 1.0.6.8 is an old version of AWX and probably doesn’t have IGs or other features implemented. There are a few tools out there to export your resources and import them into a newer version. It might be worth the time to research them and come up with an upgrade path for yourself to get off an old version, especially since the AWX project does sometimes have broken upgrades.
Hi Uriel, thanks for your quick answer. I have followed your suggestion and established a maximum of 54 forks for a determined Instance Group. The problem is that this configuration can be overwritten by the user that runs the job, as can be seen in the following picture:
The capacity is exceeded by 31.5% and this is what we are trying to restrict. Under no circumstances should a project that belongs to this specific IG exceed 54 forks (as far as we are looking for). Do you know if there is any way to achieve this? Is this goal achievable?
Currently there’s no way to limit the number of forks that an organization (or any other resource like it) can use. You can reserve capacity, as shown above, but that’s more about limiting how many jobs can run simltaneously. If any individual job exceeds the capacity of any individual cluster instance to run then we’ll still assign it (though it may be the only job that can run across the cluster at that moment in time). Generally if you are seeing this kind of overcomitting of cluster resources then your cluster is under-provisioned.
For your core use case of limiting the max forks based on Organization, you should open a feature request on AWX for that.
I was thinking of a scenario where you have 2 AWX Instances on 2 different nodes. My understanding is if Instance A on node A goes over board with work, it shouldn’t affect the work Instance B on Node B is trying to do since their resources aren’t shared.
If you’re considering a scenario where you have 2 AWX Instances on a single node, I can see how Instance A can cause problems for the other instance because both instances would be competing for host resources.
Maybe you can enforce container resource limits to ensure the service/pod doesn’t eat up too much on the host? I don’t know how that will play with the percentages you see on the Instances tabs, but that will give you reasonable certainty that an AWX instance won’t eat more resources than you expect it to.
You can also more-or-less achieve this already in a clustered environment by assigning Instance to Instance Groups and then assigning that Instance Groups to particular organizations.
