So that I can be sure that I don't need my systems to be
forwards-compatible...
I want to say:
- For some combination of server groups, applications and versions:
- Stop all the existing versions of applications on all the servers
they run on (*so I can be sure that old application versions won't
see messages produced by newer versions*)
That should be easy - by default Ansible finishes a task on all hosts before starting the next task, so having a playbook with:
- name: stop my app
service: name=my_app state=stopped
Will make sure that, for all the hosts, either my_app is stopped or the host has failed (my_app is missing, Ansible can't connect etc.).
If you want to stop the moment *any* of the hosts has problems, you should set max_fail_percentage on the play to 0
(at least I think so, I'm not sure if 0 is a special value here).
- When all services have stopped, deploy the new versions of the
services to the appropriate servers (all of our services deploy in an
identical way- download a tar from S3, unpack it, create an upstart service
that calls run.sh, provided inside the tar)
All of this, and even starting the stack up, can be made into a nice idempotent playbook that makes sure all the infrastructure has the desired configuration.
- Restart the stack
This an old-fashioned way of doing things but we don't need rolling
deployments. All of our services are asynchronous and tolerant of outages.
The cost of downtime during deployment is far less than the cost of having
to think about forwards compatibility everywhere. Backwards compatibility
we're happy to think about.
This doesn't fit my understanding of Ansible's model, where there's not
much coordination between playbooks.
There's not much coordination between ansible-playbook invocations, but playbooks can share inventories, group and host variables, secrets, imported variables etc. and even import each other - I think that's plenty of coordination 
Can anyone suggest a solution? Or maybe there's a better way to achieve the
original goal of avoiding forwards-compatibility?
I think I'd start with 2 playbooks.
First one for setting the whole site up (installing dependencies, apps, creating upstart configs and starting it all up).
Second one would stop the services, remove (or otherwise tag as unusable) current versions of apps, and then call the first one to setup the new version.