started a branch to implement setup/fact cache, follows the spirit of other plugins
https://github.com/bcoca/ansible/tree/setup_cache
not working yet but added most of the infrastructure and doesn’t break current functionality (anymore).
comments appreciated, not sure if this is the best direction or how to add aditional config (ansible.cfg or per plugin .ini files?)
Excellent.
This is definitely something we’ll get in 1.5. It’s been a while and has gotten pushed back here and there, but the time is now.
As we’re getting the release out, I’ll probably have more comments after Thursday or so.
I can see this shipping with a default-enabled resource of some kind (bdb or sqlite or flatfiles? Hmm…) and then you can maybe select different types of plugins to use.
AWX will probably use a database or API based one, we should also support things like Riak and Redis and MongoDB. Should be super easy to configure, regardless, in ansible.cfg – we want to strongly
avoid any pain caused by a similar implementation in a configuration tool I know of 
Very very cool.
For those not familar with the idea – this has a few goals – be able to support not having to poll for facts between every ansible run, for instance, if talking to dbservers and you talked to webservers 5 minutes ago, just use that data. (B) Eventually use this to support deeper inventory about systems and also making it possible to build higher level tracking systems on top of this, (C) being able to do things like this:
which is https://github.com/ansible/ansible/issues/4341
To me, this is one of the last remaining big super important things I’d love to get in for 1.5, the other being deciding on how we’d like to implement the ‘ansible-vault’ idea and getting it done.
Two major things that will make this release very exciting.
(We’re also working on some cleanup to the cloud guide documentation for both ec2 and rackspace, among some other things, as a major part of this cycle).
Also early on in 1.5, we’ll be releasing Galaxy, which is our free website for sharing roles. It’s in the later stages of development and is just getting finished up.
Excerpts from Michael DeHaan's message of 2013-11-18 21:56:15 -0500:
I can see this shipping with a default-enabled resource of some kind (bdb
or sqlite or flatfiles? Hmm...) and then you can maybe select different
types of plugins to use.
It'd be very nice to have the default be something nice and simple and
potentially already installed (bdb, sqlite, and text files all meet that
criterion).
Also, early vote for a feature request: auto-magic population of the
cache for hosts that haven't been talked to yet. Perhaps this could rid
us of the (in my opinion) odd idiom of:
- hosts: all
tasks:
- hosts: stuff_I_actually_want
tasks:
- thing_I_actually_want_to_do: bar baz howdy howdy howdy
Just my 2¢.
“Also, early vote for a feature request: auto-magic population of the
cache for hosts that haven’t been talked to yet.”
It’s a good idea, but unlikely to happen – in many cases the pool of hosts someone had defined to is very very large, and while you could call a module just before referencing a fact for that host, it would lead to a lot of
inefficient behavior.
Thankfully with this mechanism you can gather facts on initial provisioning and avoid this, and even write a simple play to update all of the facts and nothing else too.
seems i didnt reply to list
So I ended up going with just dict subclass and now have the ‘read’ part of the cache working.
https://github.com/bcoca/ansible/tree/cache
The one issue I have questions about is when gather_facts: true, ansible ALWAYS goes to gather facts, this results in a merge with the local cached facts.
Should ansible avoid running setup if cache is valid (gather_facts explicit or action: setup cases)?
Should ansible ovewrite the cache when running setup?
Should ansible avoid running setup if cache is valid (gather_facts explicit or action: setup cases)?
No here, it should merge.
Should ansible ovewrite the cache when running setup?
It should merge, but there should be a way to clear it via at least some kind of task.
gather_facts: overwrite
or
setup: overwrite=True
(default false)
To overwrite the existing cache?
I’m just curious what kind of facts you would store that would not be found via “setup” ?
If all facts were discovered via the setup module, then it seems overwrite would make sense.
So we’re likely just going to have it overwrite each key, key by key, every time.
But examples include things like ec2_facts and rax_facts, or if the user wrote a custom module, or perhaps if we included our own diagnostic modules.
(BTW, if you didn’t notice my previous “Runner refactoring” thread, I’m currently waiting to apply some cleanup to Runner before we bring this back in. There’s probably absolutely nothing wrong with Brian’s patch, but I wanted memory caching to be a plugin, and there were some problems in my implementation – this will be easier to do once we cleanup some of the underlying bits, so that’s where this is currently at).