Ever since ansible-core v2.19 we’ve seen a major (>10x) performance regression in using inventory cache.
There is this existing issue on GitHub:
I have used cPython to profile real world usage of 2.18.19 (fast) vs. 2.21.3 (slow), but I’d like to not share that info publicly in case it contains data that should not be shared publicly.
@shertel: How can I share this data with you or all the contributors in a non-public way?
Good news, I don’t think more data is necessary. Profiling data is what I used as the basis for my current draft, which wasn’t great since the cause wasn’t apparent from profiling data.
I’m exploring possible options and need to discuss more with @Core, but the 3rd LLM-generated reproducer you most recently shared contains a comment about the probable cause, which is ansible_modules/plugins/inventory/nb_inventory.py at e8ec5859095b6bfd79a230fba30f9029f7999adf · netbox-community/ansible_modules · GitHub. This explains why the performance issue is only affecting users of that inventory plugin, since using the self.loader object to load random JSON payloads (instead of the json lib) is not recommended.
I’ll update my draft when I can. Thanks for your effort in helping me reproduce the issue!
How are the discussions going?
Should be fast to test, if I understand correctly there is just one change from self.loader.load() to json.loads() on that line?
Edit: Made a draft PR here https://github.com/netbox-community/ansible_modules/pull/1602, will test it once I’m back to work next week.
Yeah, seems like the plugin should probably be using json.loads(). I have no idea why it’s using self.loader.load there.
In terms of ansible-core, I am still investigating solutions and don’t have an update. Once I identify possible options I will discuss which the team prefers and update my draft. But anyone is welcome to take a stab at improving it, it doesn’t need to wait on me.
It’s not clear to me if your reproducer is the same as performance regression when reading from inventory cache · Issue #86292 · ansible/ansible · GitHub, because you said jsonfile: add option to configure the cache format by s-hertel · Pull Request #87479 · ansible/ansible · GitHub didn’t help with the issue, but that toggle would work around this specific issue for the jsonfile cache plugin since it discards the tags from self.loader.load instead of caching them.
Using json.loads() made no improvement on cache read and parse performance. Looks like that code path is about reading HTTP response, not cache?
Hm, I see a large performance difference. When testing, I delete the cache, populate it, then run the command a second time to see how long it takes to read the cache. This reproducer isolating the cache behavior takes about a minute to read from the cache due to setting the cache with tagged data from self.loader. If I update it to not set the cache using tagged data from self.loader, reading from the cache takes <1 second.