Ansible as a Packer provisioner hanging?

Trying to use ansible as a provisioner in packer. Ansible is hanging during the TASK [Gathering Facts] and I have no idea why

the provisioner is below:

{
"type": "ansible",
"command": "script/ansible.sh",
"playbook_file": "playbooks/testing_ami.yml",
"user": "{{ user `ssh_username` }}",
"groups": [
"aws_comap_com",
"linux_servers"
],
"extra_arguments": [
"-u ubuntu",
"-vvvv"
],
"only": [
"amazon-ebs"
]
}

The testing_ami.yml is below:

- hosts: all
vars:
keep_deb_src: true
keep_build_essentials: true
pre_tasks:
roles:
- { role: take-over, become: yes }
tasks:
post_tasks:

The script/ansible.sh

#!/bin/bash

ansible-playbook $@

I do not understand why a couple of the ssh calls to the packer AMI work and then last one fails.

I do not even know if this is an ansible problem as invoking the same playbook from the command line
works as expected. BUT the logs show the invocation of ansible-playbook like I would do via the command
line.

gist of the logs

https://gist.github.com/basictheprogram/017998412e0dc7aa068ee765cdd85ad5

I can ssh to the public IP address of the Packer AMI but Ansible being run as a provisioner in Packer shows this log:

amazon-ebs: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: ubuntu
amazon-ebs: <127.0.0.1> SSH: EXEC ssh -vvv -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o Port=50328 -o ‘IdentityFile=“/var/folders/y_/w_pbl7355vg3gv5xkmsld49m0000gn/T/ansible-key347004316”’ -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ‘User=“ubuntu”’ -o ConnectTimeout=10 127.0.0.1 ‘/bin/sh -c ‘"’"’/usr/bin/python3.6 && sleep 0’“'”‘’
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 SSH proxy: accepted connection
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 authentication attempt from 127.0.0.1:50333 to 127.0.0.1:50328 as ubuntu using none
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 authentication attempt from 127.0.0.1:50333 to 127.0.0.1:50328 as ubuntu using publickey
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 new env request: LANG=en_US.UTF-8
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 new exec request: /bin/sh -c ‘/usr/bin/python3.6 && sleep 0’
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 [INFO] 1234 bytes written for ‘stdin’
2019/07/31 22:18:52 [INFO] 1234 bytes written for ‘stdin’
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 [DEBUG] Opening new ssh session
2019/07/31 22:18:52 packer: 2019/07/31 22:18:52 [DEBUG] starting remote command: /bin/sh -c ‘/usr/bin/python3.6 && sleep 0’

As I understand it Packer running an ssh proxy

$ telnet localhost port
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
SSH-2.0-Go

So Packer (Go) is being a proxy to the Packer AMI but I never see any sort of login attempts in /var/log/syslog on the Packer AMI then I try to “ssh 127.0.0.1 port”

Even more weird, in a terminal windows, not trying to run python, just attempting to get a shell (complete log at https://gist.github.com/basictheprogram/3696f33543a0edc7eacdee76c54c8452)

shell request failed on channel 0

This is being tracked in https://github.com/ansible/ansible/issues/57542

The problem is that Ansible ssh pipelining does not work with the packer ssh proxy. To a further extent, the new python interpreter discovery in ansible 2.8 uses pipelining.

You can set ansible_python_interpreter=/usr/bin/python3.6 (or whatever interpreter you want to use), and then also ensure that you have pipelining turned off.