Hey Folks,
I am observing huge performance degradation after moving to AWX v17.1.0 from AWX v3.0.1. I have spent days analysing the docker stats, vm stats etc to pin point what is causing the issue but no luck. Any help or pointers would be greatly appreciated.
Sharing some info into the perf issue mentioned. We have one env similar to what is mentioned under “ENV1” below and we are planning to upgrade it to “ENV2”. During our testing, we are seeing huge perf degradation even on running a simple job on “ENV2” when compared to “ENV1”. Sample job details mentioned below. We are observing similar degradation for all the job we run, i.e like atleast 2times increase in time taken.
ENV1:
AWX setup: AWX v3.1 containers (task, web, memcached v1.6, postgres v9.6 , rabiitmq v3.9.12 ) running as standalone docker containers
Other docker containers running on the VM: “vmcp-compute-mgmt” (A simple Ubuntu18.04 container with a bunch of python2.7 pkgs installed, basically acts as a forward proxy container to allow connectivity from awx task container to our other backend systems)
Ansible version: 2.7.4
Python version: 2.7
Ubuntu VM version: 16.04
VM size: 4cores, 32GB RAM
ENV2:
AWX setup: AWX v17.1.0 containers (task, web, redis v7, postgres v12 ) running as standalone docker containers
Other docker containers running on the VM: “vmcp-compute-mgmt” (A simple Ubuntu18.04 container with a bunch of python3.6 pkgs installed, basically acts as a forward proxy container to allow connectivity from awx task container to our other backend systems)
Ansible version: 4.10.0 and ansible-core==2.11.12
Python version: 3.6
Ubuntu VM version: 18.04
VM size: 50 cores, 128GB RAM
One sample job which started taking very long:
A simple job which runs on AWX and invokes “debug_connection.yml” playbook.
Debug_connecion.yml: