Developing a vars_plugin

Archives Ansible Project
1

Hi! Anyone have any advice on developing a vars_plugin? This may sound kinda screwy, but for a chef → ansible transition I need to access data from a chef data_bag in an ansible play . . . I’m able to get the chef data using pychef, but I’m having issues when trying to run the code within vars_plugin code:

import chef

def get_dbag_data():
chef_api = chef.autoconfigure()
dbag, dbag_item = ‘dbag’, ‘dbag_item’
bag = chef.DataBag(dbag) # some debugging shows the problem is here on the second run. . !
item = bag[dbag_item]
return item.to_dict()

The code above works outside of ansible, but when run in ansible it seems to get run twice, and fails on the “bag = chef.DataBag(dbag)” line the second time around.

Any thoughts would be greatly appreciated!

Thanks!
Guy

2

Perhaps you could show your vars_plugin? There is the potential that it could be run twice, depending on where you call that code from.

The 3 following methods are all run 1 time in ansible.inventory.Inventory:

plugin.get_group_vars
plugin.run
plugin.get_host_vars

Also, as a heads up, this is probably a better discussion for ansible-devel.

3

Matt,
Thanks for the reply! Here’s the full code

import chef

def get_dbag_data():
chef_api = chef.autoconfigure()
dbag, dbag_item = ‘twelve_factor_environment’, ‘scratch-gmatz-sandbox-app-web’
bag = chef.DataBag(dbag)
item = bag[dbag_item]
return item.to_dict()

class VarsModule(object):
def init(self, inventory):
self.inventory = inventory
self.group_cache = {}
def run(self, host, vault_password=None):
inventory = self.inventory
results = get_dbag_data()
return results

Thanks!

4

migrated to ansible-devel . . . thanks!

5

​Please note that run() was kept around for backwards compatibility. Before
1.7, this was the only exposed function in vars_plugins​, and it was
supposed to return *all* calculated vars for a particular host.

See lib/ansible/inventory/vars_plugins/noop.py for the signatures.

For newly written plugins, just use get_group/host_vars. Those are called
just once for every group/host.

6

​Hi Guy,
​

import chef

def get_dbag_data():
    chef_api = chef.autoconfigure()
    dbag, dbag_item = 'twelve_factor_environment',
'scratch-gmatz-sandbox-app-web'
    bag = chef.DataBag(dbag)
    item = bag[dbag_item]
    return item.to_dict()

class VarsModule(object):
    def __init__(self, inventory):
        self.inventory = inventory
        self.group_cache = {}
    def run(self, host, vault_password=None):
        inventory = self.inventory
        results = get_dbag_data()
        return results

​So, you are using the run() function here, which gets a host as a
parameter​ for which you are supposed tro return the its vars.

I have no clue what the chef code does, but this sounds like it always
returns data for that sandbox web host?
How does ""bag = chef.DataBag(dbag)"" fail at the second run?
(It runs - at least twice - so you have - at least - two hosts in your
inventory).

The code in the VarsModule constructor is not used, also the inventory var
in run() - you might leave those out.

You'll need to give me some more info on that chef code, what it does, and
what exactly you try to retrieve, and how you expect to put this in ansible.

  Serge

7

Serge,
Thanks for the reply! The chef code retrieves some information form the chef server that I want to make available as vars (either host vars or group vars, or whatever) to the ansible run.

Yeah, I see what you mean about some of the vars in my code . . . what you’re seeing is a hacked up attempt to get something working . . . here it is again, a bit cleaner (prints are included to show flow)
~/Code/mdansible⟫ cat inventory/vars_plugins/chefdb.py

import chef

#print(“In chefdb.py”)

class VarsModule(object):

def init(self, inventory):

“”" constructor “”"

Apparently nothing needed here

##print(“In init with %s” % dir(inventory) )
pass

def run(self, host, vault_password=None):
#print("Starting run . … ")
api = chef.autoconfigure()
#print(“In init, api = %s” % api )
dbag, dbag_item = ‘twelve_factor_environment’, ‘scratch-gmatz-sandbox-app-web’
#print(“dbag = %s” % dbag)
bag = chef.DataBag(dbag)
#print bag
item = bag[dbag_item]
return item.to_dict()

if name == ‘main’:
v = VarsModule(‘inv’)
print v.run(‘hostname’)

Here it is failing when run by ansible:

⟫ ansible-playbook -i inventory site.yml
Traceback (most recent call last):
File “/opt/ansible/bin/ansible-playbook”, line 324, in
sys.exit(main(sys.argv[1:]))
File “/opt/ansible/bin/ansible-playbook”, line 160, in main
inventory = ansible.inventory.Inventory(options.inventory, vault_password=vault_pass)
File “/opt/ansible/lib/ansible/inventory/init.py”, line 149, in init
host.vars = utils.combine_vars(host.vars, self.get_host_variables(host.name, vault_password=self._vault_password))
File “/opt/ansible/lib/ansible/inventory/init.py”, line 445, in get_host_variables
self._vars_per_host[hostname] = self._get_host_variables(hostname, vault_password=vault_password)
File “/opt/ansible/lib/ansible/inventory/init.py”, line 457, in _get_host_variables
vars_results = [ plugin.run(host, vault_password=vault_password) for plugin in self._vars_plugins if hasattr(plugin, ‘run’)]
File “/home/gmatz/Code/mdansible/inventory/vars_plugins/chefdb.py”, line 20, in run
bag = chef.DataBag(dbag)
File “/usr/local/lib/python2.7/dist-packages/chef/base.py”, line 58, in init
data = self.api[self.url]
TypeError: ‘NoneType’ object has no attribute ‘getitem’

8

Apologies!! Turns out to be a problem when I make my call to the chef DB and it has nothing to return . . . If I wrap the code that could fail in a try block it works:

def get_host_vars(self, host, vault_password=None):
print(“Getting vars for host %s” % host.name)
data = {}
try:
api = chef.autoconfigure()
dbag, dbag_item = ‘twelve_factor_environment’, ‘scratch-gmatz-sandbox-app-web’
bag = chef.DataBag(dbag)
except Exception as e:
return data
#raise errors.AnsibleError(‘Unable to get chef data: %s’ % e.message)
for k, v in bag[dbag_item].raw_data.items():
data[k] = v
return data

Two more question, though! :frowning:

  1. Do I need to return the data in a certain format? I don’t seem to be able to access it . . . is there a good way to test this? I’m just dumping everything I can think of in a template
  2. Does anyone know if I can restrict this var lookup to the hosts in my play? The chef query takes a few seconds

Thanks again,
Guy

9

1. Do I need to return the data in a certain format? I don't seem to be
able to access it . . . is there a good way to test this? I'm just
dumping everything I can think of in a template

​Should be a dict holding all variables for the host/group called for.
​

2. Does anyone know if I can restrict this var lookup to the hosts in my
play? The chef query takes a few seconds

​That is how it worked before 1.7, but the downside there was ​you had
repeated calls for every host, re-parsing the same groups over and over.

Depending how the Chef code works, you could maybe do 1 call in the
constructor to retrieve all data in 1 call, then return it per host and or
group in the other functions.
Or try to limit calls to get data for big groups only; or implement some
caching.

I also have this utility that you can use to dump your data in a json,
which you can then retrieve via an ansible inventory script:

https://github.com/ginsys/ansible-plugins/blob/devel/bin/ansible-inventory

  Serge

10

Serge,
Thank you for your effort here! So, do you have any idea how I can get at that data in a template?

Regards,
Guy

11

A fact module to query your databag could be one thing, though it seems like you should just shoot it in the head and switch over to vault.

Your inventory script could also choose to provide this data perhaps, if it’s something dynamic.

12

​
That's what he's trying to do, and what we are troubleshooting :slight_smile:

14

“That’s what he’s trying to do, and what we are troubleshooting :)”

I was replying to the discussion of lookup plugins, which I would suggest against.

I don’t feel vars_plugins are ideal either.

15

Sorry to resurrect a zombie thread.

I recently wrote a simple lookup plugin to pull down chef databags, using pychef’s search functionality. I found this to be (at least in my case) simpler and more performant than accessing databags directly in pychef. In theory, it should also allow you to query a chef server for more than just databags, however, this doesn’t seem to work in my testing.

Thoughts? Comments? Criticisms?

`
from ansible.errors import AnsibleError
from ansible.plugins.lookup import LookupBase
try:
import chef
except ImportError:
AnsibleError(“Please pip install pychef to use this module”)

class LookupModule(LookupBase):

def run (self, terms, variables, **kwargs):

api = chef.autoconfigure()

ret =
for term in terms:

chef_results =
for result in chef.Search(term, **kwargs):
chef_results.append(result[‘raw_data’])

ret.append(chef_results)

return ret
`

Thank you!
Gabriel Burkholder