How to have stateful python class during play execution? (singleton?

Archives Ansible Developer
1

Hi guys

I would like to parse specifc JSON files, create objects from them and store them in memory for further lookup (for my usecase I can’t use jmespath).
So I plan to read once the whole json file into dictionary with key = ipAdress, value = class.
Using a task I would like to run a lookup(ip) and retrieve the class i.e. as dictionary.

In pure python it works pretty well. But I can not figure out how I do embed it in ansible.
First I tried with FilterPlugin but it seems I can not use my pydantic classes and putting the code in subdirectory as module was not working.
Then I tried as a AnsibleModule. Which works but I can not reuse the class holding my dictionary of parsed itesm.

I assume it is because Modules are made for exection on the target host and not the core itself…
Anyway, in python a tried an approach using subfolder/module (init.py) and global variable to use singleton pattern.

Somthing like this should also work for ansible. But I really don’t know what is best approach for this?
Mabye LookupPlugin? But how can I initially load the json data? Does it accept to call other methods like a Module?

Hope you can help me or even better give me an example how to achieve that.

Many thanks in advance for your help!

Best regards,
Thomas

2

Hi Thomas.

You obviously have some very specific use-case in mind, but I was not
able to follow your thought process based on the description in your
post. So my advice is a bit generic:

1. I don't know of anything in Ansible that will help you to keep a
single Python object in memory and use it from different tasks. Even
if there is some trickery by which you could accomplish this goal
directly, it doesn't seem to fit how Ansible works, so I wouldn't
recommend it.

2. I assume the reason you want to do this is that the JSON input
file(s) are large, and you don't want to keep re-parsing everything.

3. Here are some ideas you could consider instead of the singleton approach:

(a) Store the JSON input as an Ansible fact and pass it to
json_query/a custom lookup plugin (may use a lot of memory, but will
give you convenient access to everything).

(b) Write a module that parses the input and returns the result of not
just one but many key lookups at once, and store the results as an
Ansible fact. This would use less memory, but would work only if you
have—or can build—a list of all the keys you want to retrieve.

(c) Write a separate process that parses the input and responds to
questions over a network socket, and make your lookup plugin (or
module) ask it questions and return the results. This could be just a
few-line program, or you could potentially even load your data into
something like Postgres or redis.

It's up to you to decide, based on the size and nature of the input
data and the pattern of lookups, which of these approaches would work
best for you.

-- Abhijit

3

Hello Abhijit

Thank you for the reply! And sorry if my original post was not clear enough.
I thought that it doesn’t matter to put all the code or examples in the post because I would to solve an issue in general that custom python classes get constructed on every task.

I assumed that there should be a possibility as a plugin not a module because they are used on the ansible controller not the target host…
During the execution of play that should be one process and I really do not understand why I class should not live for the time the playbook runs but maybe my ansible knowledge is not deep enough.

Yes, the usecase is bit special. Is is single target playbook with CIsco Security Manager (firewall mgmg) application as target using xml files in request and response.
So basically I do:
1.) parsexml2json using Filter
2.) load the a huge json string into class (as there is only a GET-ALL method)
2.1.) parse the json file into nice python classes to easily use with an IDE (i.e. Visual Studio Code)
2.2.) take from class one attribute (ip-address) to build a hashmap/dictionary { key = ipAdress, value = classObject }
3.) lookup one specific ipAddress according to given variables and retreive one single entry…
(this alreay works using a custom ansible modul, If I pass the fullJSON string arguement and a lookup(ip) arguement it works like a charm)

More or less I had the same ideas as you suggested. Best for performance would be database but then I have another dependency.
As we are planing to use a tower cluster it will not be single machine so local files , local databases, local caching or seperate process etc. can’t be used.

Anyway, this makes it just a bit too complicated and I would use a shared database only as last option if performance is not good enough.

If there is really no other way I will go with your suggestion to store one fact for all parsed items and to lookup on that dictionary.
But this is kinda very limited and works not for complex use cases where we have not an ip-adress as key but a netmask instead.
As you might can imagine some firewall rules do not use a specific ip-adress but a whole range (netmask).

Anyway, I am still learning Ansible from day to day and I am happy to get further insights how such problems can be solved!

Best regards,
Thomas

4

Let me share some example and what I wanted to achieve:

raw json data
(one sample for each ipData notation)

{
“protVersion”: 1,
“policyObject”: {
“networkPolicyObject”: [
{
“gid”: “00000000-0000-0000-0000-180388630238”,
“name”: “Network Object dummy1”,
“lastUpdateTime”: “2013-09-10T09:55:32.743Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “dns”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“isGroup”: false,
“ipData”: “131.117.1.125”
},
{
“gid”: “00000000-0000-0000-0012-206297055671”,
“name”: “Network Object dummy2”,
“lastUpdateTime”: “2020-06-24T06:08:08.82Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “sample comment”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“isGroup”: false,
“ipData”: [
“10.7.190.8”,
“10.7.48.65”,
“10.7.49.41”,
“10.7.50.42”,
“10.7.190.73”,
“10.7.45.188”,
“10.7.45.189”,
“10.7.190.144”
]
},
{
“gid”: “00000000-0000-0000-0000-180388630208”,
“name”: “Network Object dummy3”,
“lastUpdateTime”: “2013-06-10T07:34:40.56Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “logging”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“isGroup”: false,
“ipData”: “10.1.57.52”
},
{
“gid”: “00000000-0000-0000-0001-765231559040”,
“name”: “Network Object dummy4”,
“lastUpdateTime”: “2010-10-29T11:58:06.496Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “nms”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“isGroup”: false,
“ipData”: “10.194.8.0/255.255.255.0”
},
{
“gid”: “00000000-0000-0000-0010-758893077356”,
“name”: “Network Object dummy5”,
“lastUpdateTime”: “2016-10-12T12:21:40.51Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “abc range”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“subType”: “NR”,
“isGroup”: false,
“ipData”: “131.117.80.0-131.117.127.255”
}
]
}
}

target structure

now, as a first step I build a map like this

{
“131.117.1.125”: {
“gid”: “00000000-0000-0000-0000-180388630238”,
“name”: “Network Object dummy1”,
“lastUpdateTime”: “2013-09-10T09:55:32.743Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “dns”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“isGroup”: false,
“ipData”: “131.117.1.125”
}
},
“131.117.1.126”: …

or better said in memory not using dictionary:

{
“131.117.1.125” : networkobject_class_ref1,

“131.117.1.126” : networkobject_class_ref2
}

…

{
“ranges”: [
{
“10.194.8.0/255.255.255.0”: {
“gid”: “00000000-0000-0000-0001-765231559040”,
“name”: “Network Object dummy4”,
“lastUpdateTime”: “2010-10-29T11:58:06.496Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “nms”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“isGroup”: false,
“ipData”: “10.194.8.0/255.255.255.0”
}
},
{
“131.117.80.0-131.117.127.255”: {
“gid”: “00000000-0000-0000-0010-758893077356”,
“name”: “Network Object dummy5”,
“lastUpdateTime”: “2016-10-12T12:21:40.51Z”,
“parentGID”: “00000000-0000-0000-0000-000000000000”,
“type”: “NetworkPolicyObject”,
“comment”: “abc range”,
“nodeGID”: “00000000-0000-0000-0000-000000000001”,
“isProperty”: false,
“subType”: “NR”,
“isGroup”: false,
“ipData”: “131.117.80.0-131.117.127.255”
}
}
]
}

and of course a lookup- Method to check wheather there is an entry that matches this given ip-address:

lookup.yml (using custom module/plugin ‘csm_network_objects’)

  • name: load data in memory

csm_network_objects:

loadJSONfull: “{{ content_json_networkobjects }}”

  • name: do ip lookup first time

csm_network_objects:

lookupIP: “131.117.1.125”

  • name: do ip lookup second time

csm_network_objects:

lookupIP: “131.117.1.125”