/etc/ansible/hosts has 700+ entires, and I can’t hope to run ansible-playbook against it without an absurd amount of memory.
My plan to use Ansible was to have a ‘gold standard’ playbook for configurations I’d want on all machines, and to have larger more descriptive playbooks for various classes of servers. However, when running with such a large hosts file, I’m swapping (with 16GiB of ram) before my first task (yum) is finished.
/etc/ansible/hosts has 700+ entires, and I can't hope to run ansible-playbook against it without an absurd amount of memory.
I would be interested in how the removal of "host_vars" improves the situation, if any -- because this variable contains all of the facts for all of the hosts. It's useful for multimode but it comes at a reasonably high memory cost.
I suspect your playbooks are addressing very large groups also?
Check the size of your setup file as written to the host machines.
My plan to use Ansible was to have a 'gold standard' playbook for configurations I'd want on all machines, and to have larger more descriptive playbooks for various classes of servers. However, when running with such a large hosts file, I'm swapping (with 16GiB of ram) before my first task (yum) is finished.
There are two setup tasks that run before the yum task, FWIW.
--Michael
That’s right. Most are addressing less than 50 hosts, but one is ~380 hosts.
--Michael
Hi everyone,
I did some testing with this (255 systems in my case), default
paramiko type, default number of forks.
Things went mostly fine for me, but I'll explain what I found.
With both ohai and facter installed, each host generates about 8k in
facts (disk size, not in memory as a data structure), which lead to
about 500 MB of resident memory per top on the ansible machine. Yeah,
a bit high, but not terrible. The RAM consumption held steady after
clearing the setup module, so it's definitely the "setup cache" that
consumes RAM and nothing else.
If you had 860 hosts I can see it being rather large and at some
point, that's too much ram, sure. I didn't see any CPU problems.
The host_vars data was not a problem, so there is no N^2 behavior,
only linear behavior. This is excellent for us and means we don't
have any serious scaling issues, we are just moving a fair amount of
data around.
Easiest for this is to remove ohai and facter -- and maybe make it
configurable in the playbook what facts you want. (Don't think you
answered whether ohai and/or facter were installed on the nodes...
would love to know!)
The way to control this consumption, I'm fairly certain, is to
generate less facts in this case, and maybe make it controllable, so
maybe you could choose by telling the setup module to generate facts
only in certain categories. (Good idea? I'm going to open a
ticket...)
Also, definitely uninstall ohai and facter if you do not need them.
We could make that part of the thing that was switchable.
Let me know if this seems to match what you are seeing or not.
Hi everyone,
I did some testing with this (255 systems in my case), default
paramiko type, default number of forks.
My first "tests" (more like 'loose observations') were jumping between
<100 hosts and 300 hosts with 30 forks, and I noticed non-linear
scaling. The total time divided by hosts would jump from ~9 seconds to
~45 seconds. FWIW, this was before 0.4's release, and it feels (again,
with non-rigorous tests) that Ansible handles >100 hosts better now.
Easiest for this is to remove ohai and facter -- and maybe make it
configurable in the playbook what facts you want. (Don't think you
answered whether ohai and/or facter were installed on the nodes...
would love to know!)
Sorry about that 
I don't have facter and ohai installed.
The way to control this consumption, I'm fairly certain, is to
generate less facts in this case, and maybe make it controllable, so
maybe you could choose by telling the setup module to generate facts
only in certain categories. (Good idea? I'm going to open a
ticket...)
I like that. FWIW I have a few playbooks that aren't using facts, but
that could have been because of I was still learning Ansible, and not
because it was the best way to solve a particular problem. This might
be less important for mature Ansible users :3.
Or maybe it's style choice. If I'm writing a playbook for 800 hosts,
it behooves me to not use setup/fact generation.
> Hi everyone,
>
> I did some testing with this (255 systems in my case), default
> paramiko type, default number of forks.My first "tests" (more like 'loose observations') were jumping between
<100 hosts and 300 hosts with 30 forks, and I noticed non-linear
scaling. The total time divided by hosts would jump from ~9 seconds to
~45 seconds. FWIW, this was before 0.4's release, and it feels (again,
with non-rigorous tests) that Ansible handles >100 hosts better now.
Unsure. Could be the indeterminance could be because you were finally hitting swap with the larger case.
> Easiest for this is to remove ohai and facter -- and maybe make it
> configurable in the playbook what facts you want. (Don't think you
> answered whether ohai and/or facter were installed on the nodes...
> would love to know!)Sorry about that
That should reduce RAM by 2/3 had it been the case, but you still have a large amount of hosts, so the RAM consumption you saw still seems to roughly track, and yeah, things are slightly rearranged now so that could have also been a factor.
> The way to control this consumption, I'm fairly certain, is to
> generate less facts in this case, and maybe make it controllable, so
> maybe you could choose by telling the setup module to generate facts
> only in certain categories. (Good idea? I'm going to open a
> ticket...)I like that. FWIW I have a few playbooks that aren't using facts, but
that could have been because of I was still learning Ansible, and not
because it was the best way to solve a particular problem. This might
be less important for mature Ansible users :3.
Yep.
I very seldom have any use for facts, because I usually know top down what something should be, and don't need much info off the server.
I tend to think the central server, in most cases, should be as close to totally authoritative as possible.
They are useful for where you one server needs to know about the ip addresses of all the other servers, or something like that, where you only have the hostnames, but a lot of the data we return is not useful most of the time.
In those cases, I just want the network info.
Or maybe it's style choice. If I'm writing a playbook for 800 hosts,
it behooves me to not use setup/fact generation.
I don't think they'll hurt you if you just gather a few of them, but yeah. If you don't need them, you don't need them.
It might be interesting for you to fork the setup module and pair it down, and see how that affects your environment. I'll do the same just for a quick test, though if that proves true, it can be made configurable.
I really can't think of anything else building up data that would cause any problems.