Custom inventory with group-based lookup

Archives Ansible Project
1

We have a database that is populated with host and group information. I have searched through various documents on the ansible site and looked at the inventory plugins, but I am not sure how to implement what I need.

We currently have a custom lookup script that does what we need, so I am just trying to pass a call to this from ansible.

example:

hostlookup role dbservers

db1
db2
db3

I want to be able to pass the “dbservers” role/group to my custom inventory script.

ansible -i lookup.py dbservers -m ping

I am unclear on what sys.argv/argument ansible passes, on how to get this passed to an internal function for my custom inventory script.

Thanks.

2

We have a nice guide for developing external inventory scripts: http://www.ansibleworks.com/docs/developing_inventory.html

The key piece for your question:

“When the external node script is called with the single argument ‘–list’, the script must return a JSON hash/dictionary of all the groups to be managed.”

3

Hi James,

I have checked out the document before, but I am under the impression that I am not able to pass a group to my custom inventory script through the ansible command without making modifications to the ansible inventory modules. The database we have already organizes our hosts by role and returns the list of hostnames. I only need to pass the groupname to my inventory script. This doesn’t seem available without making modifications.

4

When you say “pass a group to”, are you talking about when using the inventory script with ansible-playbook?

5

I’m unsure why you need to pass a groupname to your custom script, unless I am misunderstanding your use of groupname in this context. You can return structured JSON that groups hosts automatically, which you can then use in the ‘hosts’ specification in a playbook with ansible-playbook or from the command line with the ansible command.

However, with that aside, there are ways to get data from the command line into your inventory script. Many of the current inventory scripts do this using environment variables. An example would look like:

GROUP=foo ansible-playbook -i /path/to/my/inventory.py playbook.yml

Then in your custom inventory script, you would utilize something like:

os.getenv(‘GROUP’)

I do know for one that the rax.py inventory script does this with RAX_CREDS_FILE as well as RAX_REGION

6
7

Maybe I am making too much of an assumption here. But generally you would store the host groupings in some way in your custom inventory/cmdb.

Then, using that information (hosts + groupings) you return something from your inventory script that looks similar to http://www.ansibleworks.com/docs/developing_inventory.html#id2 where the top level keys of the json response are the groups, that contain a list of hosts.

Also, you can have hosts in more than a single group.

But from my previous response, if you need to pass info into your inventory, you need to use environment variables.

8

This is exactly what my inventory/cmdb does already. At the heart of my custom script is basically a call that looks similar to ansibles output, but I also format into a json dictionary:

From custom_inventory.py script, i have a function called get_hosts_by_role and it looks like:

/usr/local/bin/lookup_hosts group webservers

web1
web2
web3

Ansible Inventory plugin:

(ans-prod)/srv/ansible$ ./plugins/inventory/custom_inventory.py --list
{
“webservers”: {
“hosts”: [
“web1”,
“web2”,
“web3”,
“web4”
]
}
}

This works if I use the GROUP=“webservers”, then os.gentenv(‘GROUP’) and run the custom script manually with --list. But if I were to use a playbook or even a one-liner, how would I specify the group “webservers” and be able to pass that to the inventory or ansible to look up?

I feel like I am asking the same question over and over.

9

If your inventory script already provides these groups, you can simply use those groups wherever you are used to mentioning hosts or lists of hosts (in playbooks, on the command-line, ...)

Simply do:

     ansible webservers -a 'ls'
     ansible-playbook -l webservers playbook.yml

And you can also use them in set-theory compositions like:

     ansible-playbook -l webservers:\!web1

See:

     http://www.ansibleworks.com/docs/intro_patterns.html

BTW There is no need to make your inventory-script conditionally create groups. Just have your inventory-script create all possible groups you need all the time, and let ansible figure out what groups/hosts are needed simply by using them on the command-line or playbooks.

PS If you have lots of custom groups based around properties of hosts, you can make dynamic groups based on facts using the group_by module. This is quite powerful, however make sure the groups exist prior to using them (in all playbooks). You cannot use these dynamic groups on the command-line since they don't exist at that point (!).

10

When using ansible-playbook, if you only wanted to operate on that group and your inventory script can limit it, you basically run it like:

GROUP=webservers ansible-playbook -i /path/to/custom_inventory.py my-playbook.yml

In my-playbook.yml you specify “hosts: webservers” or “hosts: all”.

Generally, I just wouldn’t limit the output from the inventory script, and let the inventory script return all hosts and groups, still with the correct data structure with host groups.

From a one-liner, it would look like:

GROUP=webservers ansible webservers -i /path/to/custom_inventory.py -m ping

or

GROUP=webservers ansible all -i /path/to/custom_inventory.py -m ping

In the above one-line examples, using web servers or all would give you the same thing, since you are limiting the response from your inventory script to only the webservers group.

If you just returned all hosts and groups, you could skip the GROUP=whatever part and just do something like:

ansible webservers -i /path/to/custom_inventory.py -m ping

11

Thanks for all the help everyone. The GROUP os env variable being passed just makes for a confusing simple one-liner call. I was hoping to find something similar to how saltstack can do a in-line list call with -L: salt -L web1,web2,web3 test.ping

I guess I will put that in as a feature request.

12

I really do believe you are missing what we are saying. Either that or I am just uber confused about what you are doing that we are not providing you with an explanation for.

You do not need to limit the output of the inventory script itself, you do that filtering with the ansible/ansible-playbook commands. Your inventory should return all groups and hosts. The ansible and ansible-playbook commands offer the ability to filter the list of groups/hosts that you want to run the tasks on.

Then you would so something like:

ansible webserver -i /path/to/inventory.py -m ping

In that example, the 2nd argument there is the group you are targeting, or it can be a host, a pattern…

or

ansible-playbook -i /path/to/inventory.py --limit web server my-playbook.py

In this example --limit allows you to limit to a group, or hosts, etc… that you want to target

Also note that the ansible command also offers -l/–limit

You might want to look at http://www.ansibleworks.com/docs/intro_adhoc.html also

13

Hi Matt,

I appreciate your help, and I do understand what you are explaining. My inventory script does not return all my hosts with all respective groups. This is a limitation based on the number of hosts and how the api outputs a query on our end. I understand that ansible expects the inventory to return all hosts, but I was trying to circumnavigate the ansible call, and be able to pass through the group from the command arguments to get only a limited number of hosts.

Thanks again.

14

Sounds like you need to address the limitations of your API in your the inventory script, if it’s returning a page full of data, just walk through the pages, and cache if need be.

Anyway, this is really a topic for ansible-devel, since this is development related.

I should say, if you want your script to respond to environment variables it of course can. for instance, if you were using ACME_FOO_CLOUD you might have it filter by paying attention to a ACME_FOO_CLOUD_REGION variable.

This is all 100% up to you.

Also, the aforementioned tooling should really give proper credit for it’s origins as we invented host selectors for Func and that’s where they came from :slight_smile: