Hello! I was able to deploy AWX using kurokobo’s tutorial describe here: https://github.com/kurokobo/awx-on-k3s
Everything works fine, but I have notice that whole process of playbook’s execution takes a bit longer than my prior environment (awx 14.10.1 using docker-compose)
I consider this behavior normal and logical due the automation_job pod creation.
For example a job that before takes 30 sec, in this new environment takes about 1:30 min, I notice that playbooks execution per se, maintain the same times
of my prior environment (in this example about 30 sec both), but is the pod creation that takes between 1~1:20 min, which is the reason of the whole process takes longer to complete.
Due to intensive use of AWX at my workplace I need to short this times to minimal, so… With this in mind and looking for a way to speed up the whole process I have tried setup capacity values that best fit my env (https://github.com/ansible/awx/blob/devel/docs/capacity.md),
try using mitogen (https://github.com/mitogen-hq/mitogen/blob/master/docs/ansible_detailed.rst) and try to set up (without luck) an execution node (https://github.com/ansible/awx/blob/devel/docs/execution_nodes.md).
So my question: is there a way to make automation_job’s creation faster than what it is today? Or in general what do you suggest me to do in n this case?
My host environment is:
cpu: 4vcpus
memory: 8Gb RAM.
awx-operator: 0.29.0
awx: 21.6.0
k3s: v1.24.7+k3s1
Thanks in advance for any help or guidance you can give me.
Hello,
This is likely due to image pulls. We have a few questions for you:
Do you have pull always set in the execution environment?
Are you using a custom EE or are you using the latest EE image?
Where are you pulling the images from?
Please let us know how you are handling those particular instances and we can certainly further assist you.
Where are you pulling the images from?
A/ I import locally the image using “k3s ctr images import” command, so I have it cached on the container runtime for Kubernetes.
Hope this can clarify you, if there is something else needed please let me know.
Once again, thanks a lot!
Hello,
Thank you so much for providing that additional information!
Based on the information you provided regarding the manner in which you are pulling images, we do not feel that the timing is due to the image pull.
Could you check if there is a project sync for those jobs? Settings for project syncs can be adjusted, and this could help you. Please let us know if that is fruitful.
Does your project repository contain large files?
All files in project will be compressed, transfered from AWX to your EE for your job through Kubernetes API, and extracted. This happens every job run, so if your project contains some big file, it takes some time.
Yes, my repository has some large files in it. Following your idea, I did some tests with smaller repositories and as expected, the automation_pod was rapidly created and jobs run very fast.
Thanks a lot for clarify me!! I am good to go on what to do next.
Regards
PD: By the way @kurokobo, I cannot thanks enough your enourmous contribution and your amazing, detailed and always updated GitHub repository (which I use to deploy my AWX env).
Following your idea, I did some tests with smaller repositories and as expected, the automation_pod was rapidly created and jobs run very fast.
Good to hear that, thanks for updating. I don’t know what kind of files are required for your job but to eliminate the need to transfer lage files, placing large files on additional PV and mounting it from EE using Container Group might be one of the solution I guess.