AWX UI/API slowness

Hi all, we are running our production AWX 11.1.0 in a local Kubernetes cluster with two masters and ten worker nodes. This instance has its Postgres database in EC2. We also have a couple of single-server AWX docker-based installations running much older versions (1.0.x).

I have always found that the production AWX in Kubernetes (pre-11.1.0 AWX versions as well) is much less responsive when navigating the UI than the other instances. Viewing a job run or an inventory or even just hitting the settings page always takes five to ten seconds to complete. The same delays can be seen when making direct API calls. The same operations are generally done within a second or two on the other instances.

I’m aware this could be due to the way Kubernetes is configured, or something in the networking between the machines, or perhaps the vSphere hosts where they are running – but I wanted to ask here if anyone else had run into this and might be able to point me in the right direction to solve it.

The EC2 postgres instance never seems to be very busy, so I don’t get the impression that’s the reason for the slowness. Latency between our local network and AWS could play some role, but I’m not convinced it would result in these kinds of delays.

Thanks in advance for any advice.