AWX | Inventory Proxmox Error - hangs at "Processing JSON output..."

bortastv · April 17, 2024, 12:21pm

Hi together,

we want to create a proxmox inventory with:

plugin: community.general.proxmox
url: https://10.0.40.1:8006/
user: ansible@pve
password: 
validate_certs: false
want_facts: true

want_proxmox_nodes_ansible_host: true
# compose:
#   ansible_host: proxmox_ipconfig0.ip | default(proxmox_net0.ip) | ipaddr('address')

keyed_groups:
  - key: proxmox_ostype
    separator: "_"
    prefix: ix_dev-2024

This whole Setup is running in a K3s Cluster and has the Version of awx:20.1.0.

I got the problem that the Inventory Sync get a Error:

  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
script declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
Loading collection community.general from /runner/requirements_collections/ansible_collections/community/general
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.proxmox' to process inventory source '/runner/project/ix-devcluster-2024.proxmox.yml'
Parsed /runner/project/ix-devcluster-2024.proxmox.yml inventory source with auto plugin
   14.097 INFO     Processing JSON output...
   14.097 DEBUG    Loaded group: all
   14.098 DEBUG    Loaded group: proxmox_all_lxc
   14.098 DEBUG    Adding child group proxmox_all_lxc to parent all
   14.099 DEBUG    Loaded group: proxmox_all_qemu
   14.099 DEBUG    Adding child group proxmox_all_qemu to parent all
   14.099 DEBUG    Loaded group: proxmox_all_running
   14.099 DEBUG    Adding child group proxmox_all_running to parent all
   14.100 DEBUG    Loaded group: proxmox_all_stopped
   14.100 DEBUG    Adding child group proxmox_all_stopped to parent all
   14.100 DEBUG    Loaded group: proxmox_nodes
   14.100 DEBUG    Adding child group proxmox_nodes to parent all
   14.101 DEBUG    Loaded group: proxmox_proxmoxsm40_lxc
   14.101 DEBUG    Adding child group proxmox_proxmoxsm40_lxc to parent all
   14.101 DEBUG    Loaded group: proxmox_proxmoxsm40_qemu
   14.101 DEBUG    Adding child group proxmox_proxmoxsm40_qemu to parent all
   14.101 DEBUG    Loaded group: proxmox_proxmoxsm41_lxc
   14.102 DEBUG    Adding child group proxmox_proxmoxsm41_lxc to parent all
   14.102 DEBUG    Loaded group: proxmox_proxmoxsm41_qemu
   14.102 DEBUG    Adding child group proxmox_proxmoxsm41_qemu to parent all

Last time it workes, that was the core Version “ansible-inventory [core 2.15.10rc1]”
But i dont know why this was autoupdated ?!

The Problem is, i dont see why it does not work. I try to get the ansible user more rights at the Proxmox, but this seems not the Problem.

I see in the access log from Proxmox that the AWX reached the machine:

::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes HTTP/1.1" 200 424
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm40/network HTTP/1.1" 200 367
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm40/lxc HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm40/qemu HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm41/network HTTP/1.1" 200 366
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm41/lxc HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/nodes/proxmoxsm41/qemu HTTP/1.1" 200 11
::ffff:10.30.0.109 - ansible@pve [17/04/2024:13:16:08 +0200] "GET /api2/json/pools HTTP/1.1" 200 11
::ffff:10.0.28.254 - - [17/04/2024:13:17:05 +0200] "GET / HTTP/1.1" 200 889

I hope for any tipps where i can search for the error.

Best Regards
Chris

Denney-tech · April 17, 2024, 1:17pm

Are you using awx-ee:latest? That would explain why it’s using ansible-core==2.15.10rc1 instead of ansible-core~=2.12.0

Unfortunately, quay.io/ansible/awx-ee only has matching awx tags going back to 21.11.0, but since you were fine until recently, that might work okay.

bortastv · April 18, 2024, 7:05am

Hey Denney-tech,

thanks for your answer. Yes we use awx-ee:latest.
Should i change it to avoid the problem ?
I Inherited the project so sorry for the dump question ^^

Best Regards
Chris

Edit: I changed the Execution Environments to Version 23.9.0 and 22.0.0 but i stuck at the same error.

Denney-tech · April 18, 2024, 1:29pm

I consider it best practice to keep the awx-ee version the same as the awx version, but in this case I was hoping your inventory would just “work” on a previous version. There could have been a change that broke your inventory due to deprecated settings or something.

Can you share the actual error message from the inventory sync? You posted some debug output, but it shows the inventory successfully creating groups from proxmox. It doesn’t show any hosts being added or any error messages so I assume it’s been truncated.

bortastv · April 18, 2024, 4:03pm

Hey Denney-tech,

ok, we plan a update next Week from the AWX - maybe this will fix it.
I get no other Logs, i changed the logging to Verbose and Post it in my first Post.
Or do you know any other “click” where i can get more logs ?

Best Regards
Chris

Denney-tech · April 18, 2024, 4:19pm

Be mindful of the postgres upgrade(s).

Where do you see an error though? A screenshot might help. The verbose logs you posted appear to be from where we would expect to see an error (you’re looking in the right place), but the log doesn’t look complete. The error message(s) most likely would be at the bottom of the log page, but all that’s there is the first few lines of the inventory being processed. If the output is being terminated in the middle of processing the inventory and without any stderr, that would be… intriguing.

bortastv · April 18, 2024, 4:51pm

Hi Denney,
thanks for the warning - we will make a Backup - when its crashed - so we will restore it, hopefully Attached the screenshoot.

Denney-tech · April 18, 2024, 5:21pm

Okay cool, but that’s still just the top of the output. Can you scroll to the bottom? Or click the download output button on the top right and upload the file somewhere or paste the entire contents?

bortastv · April 19, 2024, 4:14am

Hi Denney,

this was scrolled down - attached the downloadet log.

ansible-inventory [core 2.15.9]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
  ansible collection location = /runner/requirements_collections:/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.9.18 (main, Jan 24 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
  jinja version = 3.1.3
  libyaml = True
No config file found; using defaults
setting up inventory plugins
Loading collection ansible.builtin from 
host_list declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
script declined parsing /runner/project/ix-devcluster-2024.proxmox.yml as it did not pass its verify_file() method
Loading collection community.general from /runner/requirements_collections/ansible_collections/community/general
Using inventory plugin 'ansible_collections.community.general.plugins.inventory.proxmox' to process inventory source '/runner/project/ix-devcluster-2024.proxmox.yml'
Parsed /runner/project/ix-devcluster-2024.proxmox.yml inventory source with auto plugin
   13.994 INFO     Processing JSON output...
   13.995 DEBUG    Loaded group: all
   13.996 DEBUG    Loaded group: proxmox_all_lxc
   13.997 DEBUG    Adding child group proxmox_all_lxc to parent all
   13.998 DEBUG    Loaded group: proxmox_all_qemu
   13.998 DEBUG    Adding child group proxmox_all_qemu to parent all
   13.998 DEBUG    Loaded group: proxmox_all_running
   13.999 DEBUG    Adding child group proxmox_all_running to parent all
   13.999 DEBUG    Loaded group: proxmox_all_stopped
   13.999 DEBUG    Adding child group proxmox_all_stopped to parent all
   14.000 DEBUG    Loaded group: proxmox_nodes
   14.000 DEBUG    Adding child group proxmox_nodes to parent all
   14.000 DEBUG    Loaded group: proxmox_proxmoxsm41_lxc
   14.000 DEBUG    Adding child group proxmox_proxmoxsm41_lxc to parent all
   14.001 DEBUG    Loaded group: proxmox_proxmoxsm41_qemu
   14.001 DEBUG    Adding child group proxmox_proxmoxsm41_qemu to parent all
   14.001 DEBUG    Loaded group: proxmox_proxmoxsm40_lxc
   14.001 DEBUG    Adding child group proxmox_proxmoxsm40_lxc to parent all
   14.002 DEBUG    Loaded group: proxmox_proxmoxsm40_qemu
   14.002 DEBUG    Adding child group proxmox_proxmoxsm40_qemu to parent all

felixfontein · April 19, 2024, 4:36am

This could have also been triggered through an update of community.general. The inventory plugins contained in its latest version are triggering a bug in awx. The log above does not show the error, and I have no idea where to find the real error, but some folks provided it here: Reddit - Dive into anything

Issue in c.g: Dynamic inventory error in Ansible AWX - Nmap Plugin · Issue #8212 · ansible-collections/community.general · GitHub

The problem is that while awx requested that unsafe values are wrapped in dicts (Unsafe values not marked as unsafe by JSON encoder, ansible-inventory · Issue #47295 · ansible/ansible · GitHub, Properly JSON encode AnsibleUnsafe, using a pre-processor by sivel · Pull Request #60602 · ansible/ansible · GitHub; see also `ansible-inventory --list` JSON output shows unsafe values as dictionaries with `__ansible_unsafe` key · Issue #82999 · ansible/ansible · GitHub), awx does not seem to properly handle that in all cases, like when ansible_host happens to be marked as unsafe.

There is a PR for community.general which tries to avoid that awx bug (inventory plugins: make wrapping variables as unsafe smarter to avoid triggering an AWX bug by felixfontein · Pull Request #8225 · ansible-collections/community.general · GitHub), if someone could test it with actual awx that would be great

Denney-tech · April 19, 2024, 12:59pm

Ah, thank you @felixfontein , I thought one of us was going to go crazy trying to find an actual error message, but seems there’s an actual bug preventing us from seeing stderr in the job output?

felixfontein · April 20, 2024, 8:13am

I have no idea where that error message should show up, I also don’t use awx so I cannot try it myself. In any case I merged a PR for community.general that only marks values as unsafe that contain a { or }, which will hopefully improve the situation.

(It would be ever better if awx fixes the bug, but I have no idea how long that will take, and it is likely easier for users to switch to a newer community.general version than to upgrade awx.)