A popular way to do it is to keep your passwords outside of the tree,
using something like:
vars_files:
- /some/dir/my_passwords.yml
You could keep these in a different git repo.
I think someone else was doing this using a *smudge* filter in git,
which I thought was pretty clever but haven't done this myself.