Ansible in packer - "failed to handshake" error in docker python::3.8 - works in AWS codebuild

Hey all, this is my first post here!

I’m having trouble running Ansible in a docker (python::3.8) image in Circle CI (which I’m hoping to migrate to outside of AWS). The exact same script works fine in AWS Codebuild, so this issue is environment specific. In the docker environment, I get “failed to handshake” before the playbook starts.

I’m using Ansible within packer to setup AMI’s.

I know the connection is otherwise fine with ssh because packer is provisioning the machine by itself up to this point where it fails, so it can establish an ssh connection fine without Ansible. I have also installed openssh-client and run apt update on the docker image. What other steps should I take to fix the error?

Here is an ansible log of the succesful command being run in codebuild:

e[0;32m    amazon-ebs.centos7-rendernode-ami: <127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=38535 -o 'IdentityFile="/tmp/ansible-key079502232"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="centos"' -o ConnectTimeout=10 -o IdentitiesOnly=yes -o 'ControlPath="/root/.ansible/cp/056e7ed182"' 127.0.0.1 '/bin/sh -c '"'"'rm -f -r /tmp/ansible-tmp-1702638099.8131983-2942-21960760045885/ > /dev/null 2>&1 && sleep 0'"'"''e[0m

And here is the log of the failed run in circle ci:

e[0;32m amazon-ebs.centos7-rendernode-ami: <127.0.0.1> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=39955 -o 'IdentityFile="/tmp/ansible-key970938106"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="centos"' -o ConnectTimeout=10 -o IdentitiesOnly=yes -o 'ControlPath="/root/.ansible/cp/008f59f972"' 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "echo /tmp"&& mkdir "echo /tmp/ansible-tmp-1702635204.0458648-3114-278143423596940" && echo ansible-tmp-1702635204.0458648-3114-278143423596940="echo /tmp/ansible-tmp-1702635204.0458648-3114-278143423596940" ) && sleep 0'"'"''e[0m e[1;31m==> amazon-ebs.centos7-rendernode-ami: failed to handshakee[0m

This is the relevent segment of the packer file:

File: firehawk-ami.pkr.hcl
740:   provisioner "ansible" { # Disable SELINUX for rendernodes until tested and working when enabled.
741:     playbook_file = "./ansible/selinux.yaml"
742:     user          = "centos"
743:     extra_arguments = [
744:       "-vvvv",
745:       "--extra-vars",
746:       "variable_host=default set_selinux=disabled package_python_interpreter=/usr/bin/python2.7"
747:     ]
748:     collections_path = "./ansible/collections"
749:     roles_path       = "./ansible/roles"
750:     ansible_env_vars = ["ANSIBLE_CONFIG=ansible/ansible.cfg"]
751:     galaxy_file      = "./requirements.yml"
752:     only = [
753:       "amazon-ebs.centos7-rendernode-ami"
754:     ]
755:   }

This is the ansible file:

File: selinux.yaml
01: ---
02: 
03: - hosts: "{{ variable_host | default('ansible_control') }}"
04:   remote_user: "{{ variable_connect_as_user | default(ansible_user) }}"
05:   gather_facts: "{{ variable_gather_facts | default('false') }}"
06:   become: true
07: 
08:   vars:
09:     set_selinux: unchanged
10: 
11:   tasks:
12:   - name: Disable SELinux # Disable SELINUX for some hosts not exposed to public internet - it can often cause issues with various systems like PDGMQ, licensing, and PCOIP.  May try permissive mode in future.
13:     selinux:
14:       state: "{{ set_selinux }}"
15:     when: set_selinux != "unchanged"
16:     vars:
17:       ansible_python_interpreter: "{{ package_python_interpreter }}"
18: 

This is the ansible -version info:

+ ansible --version
ansible [core 2.12.10]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /root/.local/bin/ansible
  python version = 3.8.18 (default, Nov 29 2023, 06:12:04) [GCC 12.2.0]
  jinja version = 3.1.2
  libyaml = True
+ ansible-playbook --version
ansible-playbook [core 2.12.10]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /root/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /root/.local/bin/ansible-playbook
  python version = 3.8.18 (default, Nov 29 2023, 06:12:04) [GCC 12.2.0]
  jinja version = 3.1.2
  libyaml = True

This is the command I use to install Ansible:

File: install-packages
83: python3.8 -m pip install --upgrade pip &&
84:     python3.8 -m pip install --user "ansible==5.9.0" &&
85:     python3.8 -m pip install --user boto3 botocore &&
86:     export PATH=$PATH:/root/.local/bin &&
87:     ansible --version