Performance considerations: consolidating with_items

Hello,

we have set up a relatively simple testing environment, and we are
hitting performance issues that I hope can be easily resolved, e.g.
because I am doing something wrong.

We are already using Fireball.

Pull-mode is not an option, unfortunately, because we need to be
able to access hostvars after all facts have been gathered from all
hosts.

For instance, a simple task that just removes files using a list:

    - name: remove files
      file: path={{item}} state=absent
      with_items:

takes about 2 seconds per file in SSH mode, and about 1,6 seconds
per file with Fireball.

The problem is even worse with e.g. APT, which might take 5 seconds
or more per package to be removed, installation would take even
longer, obviously.

The root cause I think is that the above task is executed completely
separately for each item in the list.

For each item in the list, ansible creates a local temporary
workspace, composes the module locally, creates a remote temporary
workspace, transfers the module to the remote machine, and executes
it.

Is there a way I am overlooking to do all of this just once and pass
the list of files?

If not, do you see this as a possible improvement? I.e. would it
even be possible given the current code-base?

Thanks,

So far the yum, apt, and openbsd_pkg modules group with_items things into a single list.

It is possible for other modules to also do this, though they require changes to make this happen.

In any event, apt taking 5 seconds is because apt is being slow, those already group together, it’s not a transport related thing.

Actually you seeing 2 seconds per task is a bit weird too, kind of like you are managing your cloud from far outside using a very slow network connection or something – recommend not doing that :slight_smile:

also sprach Michael DeHaan <michael@ansibleworks.com> [2013.06.19.1711 +0200]:

So far the yum, apt, and openbsd_pkg modules group with_items things into a
single list.

See my other post, it does not do this consistently, I think.

In any event, apt taking 5 seconds is because apt is being slow, those
already group together, it's not a transport related thing.

Yes, APT is very slow, I know.

However, when I am seeing something like this:

  ok: [cirrus.madduck.net] => (item=puppet)
  ok: [cirrus.madduck.net] => (item=puppet-common)

then it means that the APT module is invoked twice, not passed
a list…

Actually you seeing 2 seconds per task is a bit weird too, kind of
like you are managing your cloud from far outside using a very
slow network connection or something -- recommend not doing that :slight_smile:

We don't manage a cloud. We manage real servers in real racks that
are in various countries from days long before people started
equating computing with water vapour. :wink:

We use VPN, so yes, there's a bit of lag, but round-trip times are
usually less than 50ms. Since the connection is already set up
(ControlPersist, or Fireball), this should hardly matter.

I believe I’ve already replied to another thread about this, but I’ve reverted the patch that attempted to streamline this behavior and with_items reuse is back to normal now.