Our group has traditionally been using Molecule+Testinfra to test our roles and playbooks. As Molecule has moved to having Ansible be the default verifier, we’ve migrated some roles and playbooks over to using Molecule testing via Ansible via asserts.
When Molecule used Testinfra via pytest, the setup for the tests and the output was spectacular, including being able to run multiple tests in parallel via pytest-xdist.
One of the best things about using pytest+testinfra was that when a test failed, the whole test harness didn’t puke and bail, returning an error and shutting down the entire workflow. Pytest+testinfra would continue running the rest of the tests and show a summary of passed and failed tests including some output from failed tests.
Does anyone know a way to do this when using the Ansible verifier? We have playbooks with extensive testing that takes time to run through and a lot of that testing is done in GitLab, running the entire test workflow on 10+ machines in parallel. Doing iterative fail/correct/pass development for each failing test is eating into developer time in a significant way.
We used to be able to get all the failed tests and correct as many as we could per commit.
Thoughts?
I have a dead discussion around this on Molecule’s GitHub discussion area: Any way to have Molecule run all Ansible tests and report list of fail/pass like Testinfra? · ansible/molecule · Discussion #4172 · GitHub