AWX error satellite6 inventory sync

Get Help
1

Hello.
I have a problem with the inventory sync from satellite.

Until a few days ago it was working correctly.
Now it fails returning the following error:

ansible-inventory [core 2.15.13]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.11.11 (main, Dec  3 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/usr/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
Using inventory plugin 'ansible_collections.theforeman.foreman.plugins.inventory.foreman' to process inventory source '/runner/inventory/foreman.yml'
/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'satellite.example.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
[WARNING]:  * Failed to parse /runner/inventory/foreman.yml with auto plugin:
Error scheduling inventory report on foreman. Please check foreman logs!
  File "/usr/local/lib/python3.11/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.11/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 690, in parse
    self._populate()
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 447, in _populate
    self._populate_report_api()
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 479, in _populate_report_api
    host_data = copy.deepcopy(self._post_request())
                              ^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 427, in _post_request
    raise Exception("Error scheduling inventory report on foreman. Please check foreman logs!")
[WARNING]: Unable to parse /runner/inventory/foreman.yml as an inventory source
ERROR! No inventory was parsed, please check your configuration and options.

What can be the cause?
This is the job log when it was working:

ansible-inventory [core 2.15.13]
  config file = None
  configured module search path = ['/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.11.10 (main, Sep  9 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/usr/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True
No config file found; using defaults
Using inventory plugin 'ansible_collections.theforeman.foreman.plugins.inventory.foreman' to process inventory source '/runner/inventory/foreman.yml'
/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'satellite.example.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
[WARNING]: Failed to use Reports API, falling back to Hosts API: Error
scheduling inventory report on foreman. Please check foreman logs!
Parsed /runner/inventory/foreman.yml inventory source with auto plugin
  661.212 INFO     Processing JSON output...
  661.618 INFO     Loaded 2440 groups, 3930 hosts
  702.634 INFO     Inventory import completed for satellite server in 41.4s```
2

Hi to all
I am resuming this topic as I still have not found a solution. :frowning:
The problem would seem to be related to the version of python used (3.11.11) by the sync job, but on all AWX PODs the version appears to be 3.9.18), even on the pod created when sync starts

Where is this version of python taken from?

AWX was some time ago installed using the indications given here:

On another installation of AWX the sync is done using version 3.9.X and everything works
Does anyone have any ideas?

3

I solved with the puthon version problem. It was related to the EE used for sync. It now results as 3.9.18, but the problem persists.

Loading collection theforeman.foreman from /usr/share/ansible/collections/ansible_collections/theforeman/foreman
Using inventory plugin 'ansible_collections.theforeman.foreman.plugins.inventory.foreman' to process inventory source '/runner/inventory/foreman.yml'
/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host 'satellite.xxx.xxx.xx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
[WARNING]:  * Failed to parse /runner/inventory/foreman.yml with auto plugin:
Error scheduling inventory report on foreman. Please check foreman logs!
  File "/usr/local/lib/python3.9/site-packages/ansible/inventory/manager.py", line 293, in parse_source
    plugin.parse(self._inventory, self._loader, source, cache=cache)
  File "/usr/local/lib/python3.9/site-packages/ansible/plugins/inventory/auto.py", line 59, in parse
    plugin.parse(inventory, loader, path, cache=cache)
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 690, in parse
    self._populate()
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 447, in _populate
    self._populate_report_api()
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 479, in _populate_report_api
    host_data = copy.deepcopy(self._post_request())
  File "/usr/share/ansible/collections/ansible_collections/theforeman/foreman/plugins/inventory/foreman.py", line 427, in _post_request
    raise Exception("Error scheduling inventory report on foreman. Please check foreman logs!")
[WARNING]: Unable to parse /runner/inventory/foreman.yml as an inventory source
ERROR! No inventory was parsed, please check your configuration and options.