Validating performance expectations: 10x overhead on ssh?

Our “go off and do everything” playbook is now taking an uncomfortable number of minutes to run against a single host, even when there are no changes. This feels like a lot to me.

I have pipelining and ControlPersist enabled, and I’m pretty sure they’re working. The hosts are about 10ms away, according to ping, and high spec (lots of idle cpu and ram and ssds).

For an example, I was playing with the file module, just checking the same directory exists over and over again. A lot of our plays do this; copy directories or files one by one (but also loads of other things). Maybe with_items: would be faster.

In ssh:
% time (for i in {1…20}; do ssh -o ControlPath=/home/faux/.ansible/cp/%h-%r -o ControlMaster=auto -o ControlPersist=180s xmas mkdir -p /tmp/ansible/foo; done) 0.65s user 0.06s system 47% cpu 1.490 total

In ansible:
% ansible-playbook -vvvv -i short-hosts.lst perf-playbook.yml 4.73s user 1.65s system 45% cpu 13.953 total

Nearly ten times slower? Does this match other people’s expectations?