What version of AWX are you using? You can find this information via the UI (in the upper right corner there’s an “i” inside of a circle; click on that) or at the /api/v2/config/ endpoint (search for “version”). The latest version has additional error messaging on failed email notification tests that give a bit more detail (specifically when you TEST a notification) on why a notification might be failing. Also, to double-check, do you have port 25 open? Sometimes SMTP defaults to 465, 587 or 2525.
Have you confirmed that postfix is definitely working outside of AWX? Barring postfix having an issue, I would recommend upgrading to the latest version of AWX and see if you can trigger a more detailed error message from the UI by running an email notification test.