How do I shuffle/randomize long lists of hosts please?

Archives Ansible Project
1

How do I shuffle/randomize long lists of hosts? That is, without writing an executable hosts file that randomizes them and then spits them out, and then dealing with all the groupings and variables and other wonderfulness that I can include in the hosts file?

My inventory file is collected semi-dynamically and comprises long lists of servers that are all running on the same blade on a long list of blades. I run upto 40 servers per blade, and I have about a 80 blades (or more). The FIRST time I run an “update bash” (for instance) I’ll wind up hitting all of one blade first, effectively flat-lineing it. If I can shuffle my hosts list, I can have a larger fork/serial value and won’t flatline my blades and I can shorten my effective runtime.

I was hoping for a “–shuffle-hosts” variable but I couldn’t find it.

2

There’s no such setting.

However, I don’t understand why changing the order wouldn’t smakc your other blade, so this question seems orthogonal.

Sounds like you need to take it out of rotation if it’s already nearing 100% CPU, such as using our load balancing modules.

3

Changing the setting to random and the fork value to 40 gives me a reasonable chance that all 40 forks will be more or less equally distributed. With more than 40 blades, it’s a good bet only one fork per blade. Of course, being random, I could someday get all 40 forks going to one blade. It’s not that the blade is already at 100% CPU, it’s that hitting a single blade with 40 updates at the same time will not only flatline my cpu, but the ethernet sockets for the same blade as well. And on some of my blades, the system may try and move LPARs to different blades to re-equalize the load, which just then makes it take even longer.

This is a semi-common option, at least in the network scanning tools I have used. I was hoping it would already be here and I just wasn’t seeing it.

For instance,

 Nmap 6.47SVN ( http://nmap.org )
 Usage: nmap [Scan Type(s)] [Options] {target specification}
 TARGET SPECIFICATION:
   -iR <num hosts>: Choose random targets

And for Nessus
http://static.tenable.com/documentation/nessus_5.0_user_guide.pdf page 11

Avoid Sequential Scans

By default, Nessus scans a list of IP addresses in sequential order. If checked, Nessus
will scan the list of hosts in a random order. This is typically useful in helping to
distribute the network traffic directed at a particular subnet during large scans

We have to be careful when we scan our Z series boxes for instance with Nessus. Thank goodness for random host selection.

This is a scalability issue as far as I am concerned.

4

If you are collecting the inventory file semi-dynamically, why not randomize the order while you are collecting the hosts?

5

That’s what we’re apparently going to have to do :frowning:

Like I said though, the option exists in other tools that access a large number of hosts. I was hoping it would be in Ansible, just hidden where I couldn’t find it.

And the problem with doing ourselves is that then we lose order in the hosts file itself. I could write another script to shuffle the hosts and I’m going to have to do that I guess.

Can somebody tell me where can I submit this as a future enhancement request?

6

You can on github https://github.com/ansible/ansible

7

Eric Wedaa <csgsurferdude2003@gmail.com> napisał:

That's what we're apparently going to have to do :frowning:

Like I said though, the option exists in other tools that access a
large
number of hosts. I was hoping it would be in Ansible, just hidden
where I
couldn't find it.

And the problem with doing ourselves is that then we lose order in the
hosts file itself. I could write another script to shuffle the hosts
and
I'm going to have to do that I guess.

You can look at vars_plugins, IIRC they have access to inventory but are separate from it - you wouldn't even have to touch your inventory scripts then.

8

“Changing the setting to random and the fork value to 40 gives me a reasonable chance that all 40 forks will be more or less equally distributed. With more than 40 blades, it’s a good bet only one fork per blade”

So this seems to imply you have more than one VM running on a given blade (and not that you are talking about a blade chassis).

Ok I don’t think it’s appropriate for ansible to resolve to random chance to avoid too much activity on a device, and I’m not sure what the best way to express that in your configuration would be.

I’m interested in the discussion but don’t believe we’re ready for a feature proposal just yet - what you propose could still fail - what you really seem to want is a way to say “only target one of these set of machines at a time” inside some sort of micro groups.

Honestly, I’m really curious why your blade can’t handle the load and should explore that first.

9

We can have multiple VMs on a blade, for instance MANY guests on an esxi blade, or multiple LPARs on a p-blade. Each of these VMs don’t consume a lot of CPU/Networking.

We have tested 100 VMs on a single blade successfully. Not that they were doing a lot, but were available. Reality is more like 5-20, depending on what they are doing.

If I run a yum update all on all of those at the same time, it will flatline my blade.

I would take ““only target one of these set of machines at a time” inside some sort of micro groups.” if that was easy enough to do, but I’d even prefer “only target N of these set…” since the blades CAN take more than one yum update at a time. For instance, I have 16 cores on some of my blades, so I’d take 8 at a time and see how that goes.

I really thought --randomize-hosts would have been easier.

10

So I'v been thinking of a new play parameter for a while, sort:, which
will sort the host selection according to
inventory(default)|r_inventory|alpha|r_alpha|numeric|r_numeric|random

This would sort hosts for processing depending on this parameter.

11

YES!!!

I’ll even beta test the darn thing, that’s how important this is to us :slight_smile:

Of course, now I need to learn while loops…