Ansible: How to distribute SSH sessions via load balancer?

I have servers set up on AWS that look something like the attached picture.

Ansible is installed on a single server which is connected to a load balancer.

When playbook on this server is executed, it tries to ssh into one of the build servers via the load balancer
and distribute web application build workload among the servers available in the auto scaling group at that moment.

What I want to accomplish is keep each SSH session persistent and execute the whole playbook at a time on the selected build server.

However, I constantly get the following error messages and fail halfway through executing the playbook.

[error message - 1]

fatal: [load-balancer]: FAILED! => {"failed": true, "msg": "ERROR! failed to transfer file to /home/user-name/.ansible/tmp/ansible-tmp-1525242792.1-136738294239036/file:\n\nWarning: Permanently added 'load-balancer,xx.xx.xx.xxx' (ECDSA) to the list of known hosts.\r\nscp: /home/user-name/.ansible/tmp/ansible-tmp-1525242792.1-136738294239036/file: No such file or directory\n"}

[error message - 2]

fatal: [load-balancer]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_name": "file"}, "module_stderr": "", "module_stdout": "/usr/bin/python: can't open file '/home/user-name/.ansible/tmp/ansible-tmp-1525242796.87-266579261792875/file': [Errno 2] No such file or directory\r\n", "msg": "MODULE FAILURE", "parsed": false}

What could be the cause of these errors?