Handler not run when using python vs ansible-playbook

So I have a webhook listener python app that runs Ansible. It seems that when I run with ansible-playbook my handler is executed fine, but running the same with the python app results in the handler not running.

I have it all nicely formatted in an issue already to I will post that: https://github.com/ansible/ansible/issues/12316

Am I missing something in my app? do I need to manually include handlers?