Jobs failing with parallelism (forks > 1) on newer EE images

Get Help
, , ,
1

I have a working old Execution Environment based on the old awx-ee:latest base image, probably about 2+ years old. I was trying to update my Execution Environment, to pick up security and performance updates and bug fixes, and now I’m having trouble running jobs against many hosts with > 1 fork.

Note that I’m using a bastion/jump SSH server between AWX and the target hosts.

On the prior setup, I was able to safely run with 8 forks (ideally I’d do 16 or 32, but those would fail). Now, using the fedora-42 base image or the ee-minimal-rhel9 base image, with 32 hosts and 8 forks, I get seemingly arbitrary unreachable/connect failures on about half of the hosts (16 of 32).

Failed to connect to the host via ssh: Connection timed out during banner exchange

I haven’t changed the playbooks; just the EE. Has anyone else run into this, and do you have any guidance? I’m seeing mixed info about the recommended base image to use. I create custom EEs to support things like Google Cloud, AWS, HashiCorp, etc.

Thanks!

2

Digging into my configs a little more, I realized that with the old base_image, I had an ansible.cfg in place with pipelining=True, and the new build did not have the ansible.cfg

Once I added it back, parallel execution seems to work correctly.

Still seems a little odd that this seems to be required, but maybe my post will help someone else in the future:

ansible.cfg:

[defaults]
host_key_checking=False
interpreter_python=/usr/bin/python3
stdout_callback=debug
pipelining=True

timeout = 60
connect_timeout = 300
connect_retry_timeout = 200
command_timeout = 60

relevant sections of execution-environment.yml:

additional_build_files:
    - src: files/ansible.cfg
      dest: configs

additional_build_steps:
  prepend_final:
    - COPY _build/configs/ansible.cfg /etc/ansible/ansible.cfg