HPA on Kubernetes in EKS?

[trippinnik (Nik)]

Horizontal Pod Autoscaler (HPA) - Ansible AWX Operator Documentation

task_manage_replicas: True

I have this in place using awx-operator 2.19.1. My expectation is that when I kick off a lot of jobs I would see the task pod increase in number. I have resource limits set.
Right now I am seeing the jobs pending and no additional task pods

[alvaroc20]

Hi!

I ran into the same issue initially—I expected it to handle scaling automatically, either by creating an HPA or through internal Operator management, but that’s not the case.

By default, this parameter is set to true, meaning it will ALWAYS use the values from replicas, web_management_replicas, or task_management_replicas. This was the previous behavior as well. The new feature now allows you to set it to false, which makes it ignore these three parameters. This way, you can create a custom HPA to scale the deployment based on your requests values and the chosen metric.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: awx-task-memory-hpa
  namespace: awx
spec:
  maxReplicas: 10
  metrics:
  - resource:
      name: memory
      target:
        averageUtilization: 75
        type: Utilization
    type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: deployment
    name: awx-task

This is an example where awx-task scales based on memory. You would need to create another one for awx-web, and it would work as you want since you won’t see an issue with CPU.

That percentage is based on the requests you’ve set in the CRD.

    task_resource_requirements:
      limits:
        cpu: "2"
        memory: 2Gi
      requests:
        cpu: 100m
        memory: 1Gi

For awx-web, I have 1.5GB in the memory request since, by default, my usage is always around 1.2GB. Still, check your own usage and adjust the values accordingly!

Hope this helps you!