First off, make sure you are using "-c ssh"
Also, I'd try editing your SSH config file, which Ansible will respect.
The ANSIBLE_SSH_ARGS thing is mostly there to override control master settings.
I use paramiko most of the time myself, so perhaps others with this setup can weigh in.
--Michael