# AWX | Inventory Proxmox Error - hangs at "Processing JSON output..."

**URL:** <https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994>\
**Category:** Get Help\
**Tags:** awx\
**Created:** [April 17, 2024, 12:21pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994 "2024-04-17T12:21:15Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![bortastv](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bortastv/32/2916_2.png) [@bortastv](https://forum.ansible.com/u/bortastv)\
**Post date:** [April 17, 2024, 12:21pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/1 "2024-04-17T12:21:15Z")

</div>

Hi together,

we want to create a proxmox inventory with:

```auto
plugin: community.general.proxmox
url: https://10.0.40.1:8006/
user: ansible@pve
password: 
validate_certs: false
want_facts: true

want_proxmox_nodes_ansible_host: true
# compose:
# ansible_host: proxmox_ipconfig0.ip | default(proxmox_net0.ip) | ipaddr('address')

keyed_groups:
  - key: proxmox_ostype
    separator: "_"
    prefix: ix_dev-2024

```

This whole Setup is running in a K3s Cluster and has the Version of awx:20.1.0.

I got the problem that the Inventory Sync get a Error:

```auto
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
script declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
Loading collection community.general from /runner/requirements_collections/ansible_collections/community/general
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.proxmox' to process inventory source '/runner/project/ix-devcluster-2024.proxmox.yml'
Parsed /runner/project/ix-devcluster-2024.proxmox.yml inventory source with auto plugin
   14.097 INFO Processing JSON output...
   14.097 DEBUG Loaded group: all
   14.098 DEBUG Loaded group: proxmox_all_lxc
   14.098 DEBUG Adding child group proxmox_all_lxc to parent all
   14.099 DEBUG Loaded group: proxmox_all_qemu
   14.099 DEBUG Adding child group proxmox_all_qemu to parent all
   14.099 DEBUG Loaded group: proxmox_all_running
   14.099 DEBUG Adding child group proxmox_all_running to parent all
   14.100 DEBUG Loaded group: proxmox_all_stopped
   14.100 DEBUG Adding child group proxmox_all_stopped to parent all
   14.100 DEBUG Loaded group: proxmox_nodes
   14.100 DEBUG Adding child group proxmox_nodes to parent all
   14.101 DEBUG Loaded group: proxmox_proxmoxsm40_lxc
   14.101 DEBUG Adding child group proxmox_proxmoxsm40_lxc to parent all
   14.101 DEBUG Loaded group: proxmox_proxmoxsm40_qemu
   14.101 DEBUG Adding child group proxmox_proxmoxsm40_qemu to parent all
   14.101 DEBUG Loaded group: proxmox_proxmoxsm41_lxc
   14.102 DEBUG Adding child group proxmox_proxmoxsm41_lxc to parent all
   14.102 DEBUG Loaded group: proxmox_proxmoxsm41_qemu
   14.102 DEBUG Adding child group proxmox_proxmoxsm41_qemu to parent all

```

Last time it workes, that was the core Version “ansible-inventory [core 2.15.10rc1]”  
But i dont know why this was autoupdated ?!

The Problem is, i dont see why it does not work. I try to get the ansible user more rights at the Proxmox, but this seems not the Problem.

I see in the access log from Proxmox that the AWX reached the machine:

```auto
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes HTTP/1.1" 200 424
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm40/network HTTP/1.1" 200 367
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm40/lxc HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm40/qemu HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm41/network HTTP/1.1" 200 366
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm41/lxc HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm41/qemu HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/pools HTTP/1.1" 200 11
::ffff:10.0.28.254 - - [17/04/2024:13:17:05 +0200] "GET / HTTP/1.1" 200 889

```

I hope for any tipps where i can search for the error.

Best Regards  
Chris

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [April 17, 2024, 1:17pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/2 "2024-04-17T13:17:41Z")

</div>

> [@bortastv](#):
>
> This whole Setup is running in a K3s Cluster and has the Version of awx:20.1.0.

Are you using awx-ee:latest? That would explain why it’s using ansible-core==2.15.10rc1 instead of ansible-core~=2.12.0

Unfortunately, `quay.io/ansible/awx-ee` only has matching awx tags going back to 21.11.0, but since you were fine until recently, that might work okay.

---

<div class="post-metadata">

**Author:** ![bortastv](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bortastv/32/2916_2.png) [@bortastv](https://forum.ansible.com/u/bortastv)\
**Post date:** [April 18, 2024, 7:05am UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/3 "2024-04-18T07:05:58Z")

</div>

Hey Denney-tech,

thanks for your answer. Yes we use awx-ee:latest.  
Should i change it to avoid the problem ?  
I Inherited the project so sorry for the dump question ^^

Best Regards  
Chris

Edit: I changed the Execution Environments to Version 23.9.0 and 22.0.0 but i stuck at the same error.

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [April 18, 2024, 1:29pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/4 "2024-04-18T13:29:17Z")

</div>

> [@bortastv](#):
>
> we use awx-ee:latest.  
> Should i change it to avoid the problem ?  
> Edit: I changed the Execution Environments to Version 23.9.0 and 22.0.0 but i stuck at the same error.

I consider it best practice to keep the awx-ee version the same as the awx version, but in this case I was hoping your inventory would just “work” on a previous version. There could have been a change that broke your inventory due to deprecated settings or something.

Can you share the actual error message from the inventory sync? You posted some debug output, but it shows the inventory successfully creating groups from proxmox. It doesn’t show any hosts being added or any error messages so I assume it’s been truncated.

---

<div class="post-metadata">

**Author:** ![bortastv](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bortastv/32/2916_2.png) [@bortastv](https://forum.ansible.com/u/bortastv)\
**Post date:** [April 18, 2024, 4:03pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/5 "2024-04-18T16:03:00Z")

</div>

Hey Denney-tech,

ok, we plan a update next Week from the AWX - maybe this will fix it.  
I get no other Logs, i changed the logging to Verbose and Post it in my first Post.  
Or do you know any other “click” where i can get more logs ?

Best Regards  
Chris

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [April 18, 2024, 4:19pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/6 "2024-04-18T16:19:36Z")

</div>

> [@bortastv](#):
>
> ok, we plan a update next Week from the AWX - maybe this will fix it.

Be mindful of the postgres upgrade(s).

> [@ATTENTION - do NOT upgrade to AWX operator to 2.13.0](http://forum.ansible.com/t/attention-do-not-upgrade-to-awx-operator-to-2-13-0/4363):
>
> We’ve discovered a bug that would cause you to lose your data. If you already upgraded to 2.13.0 and didn’t make a backup - do NOT stop / restart your postgres 15 container. The database data is living only in that container, and not in the mounted in volume. We’ll update this post with instructions for copying that data out to preserve it.

* * *

> [@bortastv](#):
>
> I get no other Logs, i changed the logging to Verbose and Post it in my first Post.  
> Or do you know any other “click” where i can get more logs ?

Where do you see an error though? A screenshot might help. The verbose logs you posted appear to be from where we would expect to see an error (you’re looking in the right place), but the log doesn’t look complete. The error message(s) most likely would be at the bottom of the log page, but all that’s there is the first few lines of the inventory being processed. If the output is being terminated in the middle of processing the inventory and without any stderr, that would be… intriguing.

---

<div class="post-metadata">

**Author:** ![bortastv](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bortastv/32/2916_2.png) [@bortastv](https://forum.ansible.com/u/bortastv)\
**Post date:** [April 18, 2024, 4:51pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/7 "2024-04-18T16:51:10Z")

</div>

Hi Denney,  
thanks for the warning - we will make a Backup - when its crashed - so we will restore it, hopefully 😃 Attached the screenshoot.

 ![image](https://us1.discourse-cdn.com/ansible/original/2X/c/c7582249262d054dcf943abe4829f084a6d7afc6.png)

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [April 18, 2024, 5:21pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/8 "2024-04-18T17:21:48Z")

</div>

Okay cool, but that’s still just the top of the output. Can you scroll to the bottom? Or click the `download output` button on the top right and upload the file somewhere or paste the entire contents?

 ![image](https://us1.discourse-cdn.com/ansible/original/2X/8/84c557a4b3f9ffd4e6ecfac8a99cb8b3e01bfe1a.png)

---

<div class="post-metadata">

**Author:** ![bortastv](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/bortastv/32/2916_2.png) [@bortastv](https://forum.ansible.com/u/bortastv)\
**Post date:** [April 19, 2024, 4:14am UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/9 "2024-04-19T04:14:10Z")

</div>

Hi Denney,

this was scrolled down - attached the downloadet log.

```auto
ansible-inventory [core 2.15.9]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
script declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
Loading collection community.general from /runner/requirements_collections/ansible_collections/community/general
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.proxmox' to process inventory source '/runner/project/ix-devcluster-2024.proxmox.yml'
Parsed /runner/project/ix-devcluster-2024.proxmox.yml inventory source with auto plugin
   13.994 INFO Processing JSON output...
   13.995 DEBUG Loaded group: all
   13.996 DEBUG Loaded group: proxmox_all_lxc
   13.997 DEBUG Adding child group proxmox_all_lxc to parent all
   13.998 DEBUG Loaded group: proxmox_all_qemu
   13.998 DEBUG Adding child group proxmox_all_qemu to parent all
   13.998 DEBUG Loaded group: proxmox_all_running
   13.999 DEBUG Adding child group proxmox_all_running to parent all
   13.999 DEBUG Loaded group: proxmox_all_stopped
   13.999 DEBUG Adding child group proxmox_all_stopped to parent all
   14.000 DEBUG Loaded group: proxmox_nodes
   14.000 DEBUG Adding child group proxmox_nodes to parent all
   14.000 DEBUG Loaded group: proxmox_proxmoxsm41_lxc
   14.000 DEBUG Adding child group proxmox_proxmoxsm41_lxc to parent all
   14.001 DEBUG Loaded group: proxmox_proxmoxsm41_qemu
   14.001 DEBUG Adding child group proxmox_proxmoxsm41_qemu to parent all
   14.001 DEBUG Loaded group: proxmox_proxmoxsm40_lxc
   14.001 DEBUG Adding child group proxmox_proxmoxsm40_lxc to parent all
   14.002 DEBUG Loaded group: proxmox_proxmoxsm40_qemu
   14.002 DEBUG Adding child group proxmox_proxmoxsm40_qemu to parent all

```

---

<div class="post-metadata">

**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\
**Post date:** [April 19, 2024, 4:36am UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/10 "2024-04-19T04:36:41Z")

</div>

This could have also been triggered through an update of community.general. The inventory plugins contained in its latest version are triggering a bug in awx. The log above does not show the error, and I have no idea where to find the real error, but some folks provided it here: [Reddit - Dive into anything](https://www.reddit.com/r/Proxmox/comments/1bwdoue/dynamic_inventory_error_in_ansible_awxtower/)

Issue in c.g: [Dynamic inventory error in Ansible AWX - Nmap Plugin · Issue #8212 · ansible-collections/community.general · GitHub](https://github.com/ansible-collections/community.general/issues/8212)

The problem is that while awx requested that unsafe values are wrapped in dicts ([Unsafe values not marked as unsafe by JSON encoder, ansible-inventory · Issue #47295 · ansible/ansible · GitHub](https://github.com/ansible/ansible/issues/47295), [Properly JSON encode AnsibleUnsafe, using a pre-processor by sivel · Pull Request #60602 · ansible/ansible · GitHub](https://github.com/ansible/ansible/pull/60602); see also [`ansible-inventory --list` JSON output shows unsafe values as dictionaries with `__ansible_unsafe` key · Issue #82999 · ansible/ansible · GitHub](https://github.com/ansible/ansible/issues/82999)), awx does not seem to properly handle that in all cases, like when `ansible_host` happens to be marked as unsafe.

There is a PR for community.general which tries to avoid that awx bug ([inventory plugins: make wrapping variables as unsafe smarter to avoid triggering an AWX bug by felixfontein · Pull Request #8225 · ansible-collections/community.general · GitHub](https://github.com/ansible-collections/community.general/pull/8225)), if someone could test it with actual awx that would be great 🙂

---

<div class="post-metadata">

**Author:** ![Denney-tech](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/denney-tech/32/1556_2.png) [@Denney-tech](https://forum.ansible.com/u/Denney-tech)\
**Post date:** [April 19, 2024, 12:59pm UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/11 "2024-04-19T12:59:19Z")

</div>

Ah, thank you @felixfontein , I thought one of us was going to go crazy trying to find an actual error message, but seems there’s an actual bug preventing us from seeing stderr in the job output?

---

<div class="post-metadata">

**Author:** ![felixfontein](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/felixfontein/32/7530_2.png) [@felixfontein](https://forum.ansible.com/u/felixfontein)\
**Post date:** [April 20, 2024, 8:13am UTC](https://forum.ansible.com/t/awx-inventory-proxmox-error-hangs-at-processing-json-output/4994/12 "2024-04-20T08:13:35Z")

</div>

I have no idea where that error message should show up, I also don’t use awx so I cannot try it myself. In any case I merged a PR for community.general that only marks values as unsafe that contain a `{` or `}`, which will hopefully improve the situation.

(It would be ever better if awx fixes the bug, but I have no idea how long that will take, and it is likely easier for users to switch to a newer community.general version than to upgrade awx.)