Issue with import hostvars Option in Ansible Dynamic Inventory Parsing
Description : When using the import hostvars option with Ansible dynamic inventory (Azure RM Inventory Plugin), Rundeck fails to correctly parse and identify hosts. Without this option, the inventory is parsed and hosts are identified correctly.
Steps to Reproduce:
- Configure Azure RM Inventory Plugin in Ansible with import hostvars option.
- Run the Ansible inventory command via Rundeck.
- Observe that hosts are not identified correctly.
Expected Behavior: Hosts should be correctly identified and parsed with the import hostvars option.
Environment:
Rundeck Version: 5.3.0
Ansible Version: 2.16.7
Ansible Inventory Plugin: azure.azcollection.azure_rm
OS: [Ubuntu 22.04.4 LTS]
Python Version: 3.10.12
Additional Information: The same configuration works correctly without the import hostvars option.
Dynamic Inventory file which rundeck uses to create its own inventory file.
plugin: azure.azcollection.azure_rm # Specifies the Azure dynamic inventory plugin
cloud_environment : "AzureCloud"
auth_source: auto # Automatically determines the authentication source
include_vm_resource_groups: # List of Azure resource groups to include
- "decops-aero"
regions: # List of Azure regions to include
- centralindia
plain_host_names: yes # Use plain host names instead of the FQDN
strict_permissions: false # Do not enforce strict permissions
allow_duplicated_hosts: false # Do not allow duplicated hosts
# Conditional groups based on VM tags and other properties
conditional_groups:
logstash_hosts: '"logstash_hosts" in (tags and tags["tools_groups"])'
kafka_hosts: '"kafka_hosts" in (tags and tags["tools_groups"])'
cra_hosts: '"cra_hosts" in (tags and tags["tools_groups"])' #
httpd_hosts: '"httpd_hosts" in (tags and tags["tools_groups"])'
java_hosts: '"java_hosts" in (tags and tags["tools_groups"])'
cra_core: '"cra_core" in (tags and tags["service_groups"])'
cra_webapps: '"cra_webapps" in (tags and tags["service_groups"])'
cra_analytics: '"cra_analytics" in (tags and tags["service_groups"])'
cra_agents_hosts: '"cra_agents_hosts" in (tags and tags["service_groups"])'
cra_sme_ai_hosts: '"cra_sme_ai_hosts" in (tags and tags["service_groups"])'
cra_datastats_hosts: '"cra_datastats_hosts" in (tags and tags["service_groups"])'
crate_hosts: '"crate_hosts" in (tags and tags["db_groups"])'
mysqldb_hosts: '"mysqldb_hosts" in (tags and tags["db_groups"])'
cra_database_hosts: '"cra_database_hosts" in (tags and tags["db_groups"])'
# Filters to apply to the inventory
filters:
powerState:
- running # Include only running VMs
tags.env:
- dev # Include only VMs with the 'dev' environment tag
use_extra_vars: true # Allow use of extra variables in the inventory
# Compose variables to set for each host
compose:
ansible_host: public_ip_address[0].ipv4_address
ansible_user: "'azureuser'"
zookeeper_id: tags.zookeeper_id # Set the zookeeper_id from the tags
kafka_broker_id: tags.kafka_broker_id # Set the kafka_broker_id from the tags
crate_node_master: tags.crate_node_master # Set the crate_node_master from the tags
cluster_name: "'cracluster'"
opt_dir: "'/opt'"
data_dir: "'/data'"
spark_dir: "'/opt/spark'"
cra_dir: "'/opt/cra'"
cra_nexus_repo_url: "'https://jfrog.independent.com/repository'"
cra_local_repo_url: "'/opt/cra/artifacts'"
ansible_managed: "'ansible managed file'"
service_file_path: "'/etc/systemd/system/'"
cra_schema_name: "'cra'"
cra_env_files_path: "'/etc/sysconfig'"
java_17_home: "'/usr/lib/jvm/temurin-17-jre'"
java_8_home: "'/usr/lib/jvm/temurin-8-jre'"
deploymentType: "'cra-services'"
platform_tools: "'cra-rpms'"
mysql_user: "'root'"
mysql_password: "'Pass2024'"
mysql_port: 3306
mysql_schema_name: "'cra'"
configureFirewall: true
db_script_location: "'/opt/cra/database-scripts'"
crate_heap_size: "'2G'"
crate_file_max: 500000
crate_max_map_count: 262144
logstash_jvm_options: |
'-Xms500m -Xmx500m'
- Working fine without using import host vars option .
Result Output
ansible-inventory [core 2.16.7]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/var/lib/rundeck/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /var/lib/rundeck/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible-inventory
python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
jinja version = 3.0.3
libyaml = True
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from
redirecting (type: inventory) ansible.builtin.azure_rm to azure.azcollection.azure_rm
Loading collection azure.azcollection from /var/lib/rundeck/.ansible/collections/ansible_collections/azure/azcollection
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (2.0.7) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
Loading collection amazon.aws from /usr/lib/python3/dist-packages/ansible_collections/amazon/aws
host_list declined parsing /var/lib/rundeck/.azure/gurucul_azure_rm.yml as it did not pass its verify_file() method
script declined parsing /var/lib/rundeck/.azure/gurucul_azure_rm.yml as it did not pass its verify_file() method
Using inventory plugin 'ansible_collections.azure.azcollection.plugins.inventory.azure_rm' to process inventory source '/var/lib/rundeck/.azure/gurucul_azure_rm.yml'
Parsed /var/lib/rundeck/.azure/gurucul_azure_rm.yml inventory source with auto plugin
{
"_meta": {
"hostvars": {
"devops-aero1": {
"ansible_host": "89.112.55.23",
"ansible_managed": "ansible managed file",
"ansible_user": "azureuser",
"availability_zone": null,
"cluster_name": "cracluster",
"computer_name": "devops-aero1",
"configureFirewall": true,
"cra_dir": "/opt/cra",
"cra_env_files_path": "/etc/sysconfig",
"cra_local_repo_url": "/opt/cra/artifacts",
"cra_nexus_repo_url": "https://jfrog.independent.com/repository",
"cra_schema_name": "cra",
"crate_file_max": "500000",
"crate_heap_size": "2G",
"crate_max_map_count": "262144",
"crate_node_master": "true",
"creation_time": "2024-06-07T10:13:08.1213518+00:00",
"data_dir": "/data",
"data_disks": [],
"db_script_location": "/opt/cra/database-scripts",
"default_inventory_hostname": "devops-aero1",
"deploymentType": "cra-services",
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Compute/virtualMachines/devops-aero1",
"image": {
"offer": "centos-8-3-free",
"publisher": "cognosys",
"sku": "centos-8-3-free",
"version": "latest"
},
"java_17_home": "/usr/lib/jvm/temurin-17-jre",
"java_8_home": "/usr/lib/jvm/temurin-8-jre",
"kafka_broker_id": "7",
"license_type": "Unknown",
"location": "centralindia",
"logstash_jvm_options": "-Xms500m -Xmx500m",
"mac_address": [
"7C-1E-52-0E-AD-53"
],
"mysql_password": "Pass2024",
"mysql_port": "3306",
"mysql_schema_name": "cra",
"mysql_user": "root",
"name": "devops-aero1",
"network_interface": [
"devops-aero1236"
],
"network_interface_id": [
"/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/networkInterfaces/devops-aero1236"
],
"opt_dir": "/opt",
"os_disk": {
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Compute/disks/devops-aero1_OsDisk_1_4451a11b6db64cfa87e6c6f68deddeb2",
"name": "devops-aero1_OsDisk_1_4451a11b6db64cfa87e6c6f68deddeb2",
"operating_system_type": "linux"
},
"os_profile": {
"system": "linux"
},
"plan": null,
"platform_tools": "cra-rpms",
"powerstate": "running",
"private_ipv4_addresses": [
"10.0.0.4"
],
"provisioning_state": "succeeded",
"public_dns_hostnames": [],
"public_ip_address": [
{
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/publicIPAddresses/devops-aero1-ip",
"ipv4_address": "52.172.198.97",
"name": "devops-aero1-ip"
}
],
"public_ipv4_address": [
"52.172.198.97"
],
"resource_group": "decops-aero",
"resource_type": "Microsoft.Compute/virtualMachines",
"security_group": [
"devops-aero1-nsg"
],
"security_group_id": [
"/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/networkSecurityGroups/devops-aero1-nsg"
],
"service_file_path": "/etc/systemd/system/",
"spark_dir": "/opt/spark",
"subnet": [
{
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/virtualNetworks/devops-aero-vnet/subnets/default"
}
],
"tags": {
"Name": "devops-aero1",
"crate_node_master": "true",
"db_groups": "crate_hosts",
"env": "dev",
"kafka_broker_id": "7",
"service_groups": "cra_core",
"tools_groups": "kafka_hosts",
"zookeeper_id": "7"
},
"virtual_machine_size": "Standard_D2s_v3",
"vmid": "b6bdef57-2abf-4790-aa8d-c13abc99000e",
"vmss": {},
"zookeeper_id": "7"
}
}
},
"all": {
"children": [
"ungrouped",
"kafka_hosts",
"crate_hosts"
]
},
"crate_hosts": {
"hosts": [
"devops-aero1"
]
},
"kafka_hosts": {
"hosts": [
"devops-aero1"
]
}
}
- Not-Working with using import host vars option .
Result Output
Below output is with that inventory which rundeck generates at runtime using above inventory file
ansible-inventory [core 2.16.7]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/var/lib/rundeck/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /var/lib/rundeck/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible-inventory
python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3)
jinja version = 3.0.3
libyaml = True
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Loading collection ansible.builtin from
redirecting (type: inventory) ansible.builtin.azure_rm to azure.azcollection.azure_rm
Loading collection azure.azcollection from /var/lib/rundeck/.ansible/collections/ansible_collections/azure/azcollection
/usr/lib/python3/dist-packages/requests/__init__.py:87: RequestsDependencyWarning: urllib3 (2.0.7) or chardet (5.2.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
Loading collection amazon.aws from /usr/lib/python3/dist-packages/ansible_collections/amazon/aws
host_list declined parsing /tmp/rundeck/ansible-inventory646200268921537314.json as it did not pass its verify_file() method
script declined parsing /tmp/rundeck/ansible-inventory646200268921537314.json as it did not pass its verify_file() method
auto declined parsing /tmp/rundeck/ansible-inventory646200268921537314.json as it did not pass its verify_file() method
Not replacing invalid character(s) "{'{', '"', '-', ':'}" in group name ({"name":"devops-aero1")
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
**Not replacing invalid character(s) "{'"', ':'}" in group name ("tools_groups":"kafka_hosts")**
Not replacing invalid character(s) "{'"', ':'}" in group name ("env":"dev")
Not replacing invalid character(s) "{'"', ':'}" in group name ("db_groups":"crate_hosts")
Not replacing invalid character(s) "{'"', ':'}" in group name ("zookeeper_id":"7")
Not replacing invalid character(s) "{'}', '"', ':'}" in group name ("kafka_broker_id":"7"})
Not replacing invalid character(s) "{'"', ':'}" in group name ("crate_node_master":"true")
Not replacing invalid character(s) "{'"', ':'}" in group name ("service_groups":"cra_core")
Parsed /tmp/rundeck/ansible-inventory646200268921537314.json inventory source with yaml plugin
{
"\"crate_node_master\":\"true\"": {
"hosts": [
"devops-aero1"
]
},
"\"db_groups\":\"crate_hosts\"": {
"hosts": [
"devops-aero1"
]
},
"\"env\":\"dev\"": {
"hosts": [
"devops-aero1"
]
},
"\"kafka_broker_id\":\"7\"}": {
"hosts": [
"devops-aero1"
]
},
"\"service_groups\":\"cra_core\"": {
"hosts": [
"devops-aero1"
]
},
"\"tools_groups\":\"kafka_hosts\"": {
"hosts": [
"devops-aero1"
]
},
"\"zookeeper_id\":\"7\"": {
"hosts": [
"devops-aero1"
]
},
"_meta": {
"hostvars": {
"devops-aero1": {
"ansible_host": "52.172.198.97",
"bios_date": "12/07/2018",
"bios_version": "090008",
"cluster_name": "cracluster",
"computer_name": "devops-aero1",
"configureFirewall": true,
"cra_dir": "/opt/cra",
"cra_env_files_path": "/etc/sysconfig",
"cra_local_repo_url": "/opt/cra/artifacts",
"cra_nexus_repo_url": "https://jfrog.independent.com/repository",
"cra_schema_name": "cra",
"crate_file_max": 500000,
"crate_heap_size": "2G",
"crate_max_map_count": 262144,
"crate_node_master": true,
"creation_time": "2024-06-07T10:13:08.1213518+00:00",
"data_dir": "/data",
"data_disks": [],
"db_script_location": "/opt/cra/database-scripts",
"default_inventory_hostname": "devops-aero1",
"deploymentType": "cra-services",
"description": "CentOS 8.3",
"distribution": "CentOS",
"distribution_major_version": 8,
"distribution_release": "NA",
"distribution_version": 8.3,
"domain": "internal.cloudapp.net",
"fips": false,
"form_factor": "Desktop",
"hostname": "52.172.198.97",
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Compute/virtualMachines/devops-aero1",
"image": {
"offer": "centos-8-3-free",
"publisher": "cognosys",
"sku": "centos-8-3-free",
"version": "latest"
},
"java_17_home": "/usr/lib/jvm/temurin-17-jre",
"java_8_home": "/usr/lib/jvm/temurin-8-jre",
"kafka_broker_id": 7,
"license_type": "Unknown",
"location": "centralindia",
"logstash_jvm_options": "-Xms500m -Xmx500m",
"mac_address": [
"7C-1E-52-0E-AD-53"
],
"machine_id": "6cc37d7585ce476d8909603a266fb56f",
"memtotal_mb": 7960,
"mysql_password": "Pass2024",
"mysql_port": 3306,
"mysql_schema_name": "cra",
"mysql_user": "root",
"name": "devops-aero1",
"network_interface": [
"devops-aero1236"
],
"network_interface_id": [
"/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/networkInterfaces/devops-aero1236"
],
"nodename": "devops-aero1",
"opt_dir": "/opt",
"osArch": "x86_64",
"osFamily": "RedHat",
"osVersion": "4.18.0-240.1.1.el8_3.x86_64",
"os_disk": {
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Compute/disks/devops-aero1_OsDisk_1_4451a11b6db64cfa87e6c6f68deddeb2",
"name": "devops-aero1_OsDisk_1_4451a11b6db64cfa87e6c6f68deddeb2",
"operating_system_type": "linux"
},
"os_profile": {
"system": "linux"
},
"pkg_mgr": "dnf",
"platform_tools": "cra-rpms",
"powerstate": "running",
"private_ipv4_addresses": [
"10.0.0.4"
],
"processor0": 0,
"processor1": "GenuineIntel",
"processor_cores": 1,
"processor_count": 1,
"processor_threads_per_core": 2,
"processor_vcpus": 2,
"product_name": "Virtual Machine",
"product_serial": "0000-0013-1679-4404-3159-5703-20",
"product_version": 7.0,
"provisioning_state": "succeeded",
"public_dns_hostnames": [],
"public_ip_address": [
{
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/publicIPAddresses/devops-aero1-ip",
"ipv4_address": "52.172.198.97",
"name": "devops-aero1-ip"
}
],
"public_ipv4_address": [
"52.172.198.97"
],
"resource_group": "decops-aero",
"resource_type": "Microsoft.Compute/virtualMachines",
"security_group": [
"devops-aero1-nsg"
],
"security_group_id": [
"/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/networkSecurityGroups/devops-aero1-nsg"
],
"service_file_path": "/etc/systemd/system/",
"service_mgr": "systemd",
"spark_dir": "/opt/spark",
"subnet": [
{
"id": "/subscriptions/ab5ec2e0-e4bb-4cae-a308-80cf8bcff59a/resourceGroups/decops-aero/providers/Microsoft.Network/virtualNetworks/devops-aero-vnet/subnets/default"
}
],
"swaptotal_mb": 0,
"system_vendor": "Microsoft Corporation",
"tags": {
"Name": "devops-aero1",
"crate_node_master": "true",
"db_groups": "crate_hosts",
"env": "dev",
"kafka_broker_id": "7",
"service_groups": "cra_core",
"tools_groups": "kafka_hosts",
"zookeeper_id": "7"
},
"tz": "UTC",
"tz_offset": "+0000",
"username": "azureuser",
"userspace_architecture": "x86_64",
"userspace_bits": 64,
"virtual_machine_size": "Standard_D2s_v3",
"virtualization_role": "guest",
"virtualization_type": "VirtualPC",
"vmid": "b6bdef57-2abf-4790-aa8d-c13abc99000e",
"vmss": {},
"zookeeper_id": 7
}
}
},
"all": {
"children": [
"ungrouped",
"{\"name\":\"devops-aero1\"",
"\"tools_groups\":\"kafka_hosts\"",
"\"env\":\"dev\"",
"\"db_groups\":\"crate_hosts\"",
"\"zookeeper_id\":\"7\"",
"\"kafka_broker_id\":\"7\"}",
"redhat",
"\"crate_node_master\":\"true\"",
"\"service_groups\":\"cra_core\""
]
},
"redhat": {
"hosts": [
"devops-aero1"
]
},
"{\"name\":\"devops-aero1\"": {
"hosts": [
"devops-aero1"
]
}
}
Issue : conditional_groups not get created and tagged group popup with backslash