gcloud SSH succeeds in AWX, but ControlPersist/SSH multiplexing settings are not applied
Dear Community,
We built a custom wrapper script to connect to GCP VMs from AWX, as no native plugin supports this.
SSH works, but a new connection is created for every task, causing slowness in playbook execution. We tried adding SSH multiplexing (ControlMaster/ControlPersist), but it doesn’t seem to work.
We built the wrapper script for ssh and below is the snippet responsible for connecting.
# --- Setup SSH multiplexing ---
CONTROL_DIR="/tmp/gcloud-ssh-sockets"
mkdir -p "$CONTROL_DIR"
# Hash user@host:port + key for uniqueness
SOCKET_HASH=$(echo -n "${host}_${SSH_KEY_PATH}_22" | md5sum | awk '{print $1}')
CONTROL_PATH="${CONTROL_DIR}/cm-${SOCKET_HASH}"
# --- Final SSH Call ---
exec gcloud compute ssh "$host" \
"${opts[@]}" \
--tunnel-through-iap \
--ssh-key-file="$SSH_KEY_PATH" \
--ssh-flag="-o ControlMaster=auto" \
--ssh-flag="-o ControlPersist=10m" \
--ssh-flag="-o ControlPath=${CONTROL_PATH}" \
--strict-host-key-checking=no \
--quiet \
--command="$final_cmd"
Below is a sample verbosity 3 log showing repeated connections for each task.
ansible-playbook [core 2.15.13]
config file = /runner/project/ansible.cfg
configured module search path = ['/runner/project/my_modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /runner/requirements_collections:/root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible-playbook
python version = 3.9.17 (main, Aug 9 2023, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True
Using /runner/project/ansible.cfg as config file
host_list declined parsing /runner/inventory/hosts as it did not pass its verify_file() method
Parsed /runner/inventory/hosts inventory source with script plugin
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
redirecting (type: modules) ansible.builtin.sysctl to ansible.posix.sysctl
redirecting (type: modules) ansible.builtin.filesystem to community.general.filesystem
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
PLAYBOOK: initialize_kafka_node.yml ********************************************
2 plays in playbooks/initialize_kafka_node.yml
PLAY [Initialize and configure a new kafka node.] ******************************
TASK [Gathering Facts] *********************************************************
task path: /runner/project/playbooks/initialize_kafka_node.yml:11
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/setup.py
Pipelining is enabled.
<XXXX-host> ESTABLISH SSH CONNECTION FOR USER: root
<XXXX-host> SSH: EXEC /etc/ansible/scripts/gcp-ssh-wrapper.sh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/ssh-key-ansible-sa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 --tunnel-through-iap --zone=XXXX-zone --project=XXXX-project-o 'ControlPath="/runner/cp/7f8f72306d"' XXXX-host '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<XXXX-host> (0, b'Activating GCP service account using: /runner/env/tmp_37i7ghc\\n\\n{"invocation": {"module_args": {"filter": [], "gather_subset": ["all"], "fact_path": "/etc/ansible/facts.d", "gather_timeout": 10}}, "ansible_facts": {"ansible_fibre_channel_wwn": [], "module_setup": true, "ansible_distribution_version": "7.9", "ansible_board_name": "Google Compute Engine", "ansible_distribution_file_variety": "RedHat", "ansible_env": {"LANG": "en_US.UTF-8", "SHELL": "/bin/bash", "XDG_RUNTIME_DIR": "/run/user/3004348419", "MAIL": "/var/mail/sa_107611174234308979642", "SHLVL": "1", "JAVA_HOME": "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.372.b07-1.el7_9.x86_64", "SSH_CLIENT": "35.235.240.112 43701 22", "SELINUX_LEVEL_REQUESTED": "", "PWD": "/home/sa_107611174234308979642", "SELINUX_ROLE_REQUESTED": "", "SELINUX_USE_CURRENT_RANGE": "", "LOGNAME": "sa_107611174234308979642", "USER": "sa_107611174234308979642", "PATH": "/liveperson/code/kafka/3.8.1/bin:/usr/local/bin:/usr/bin", "HOME": "/home/sa_107611174234308979642", "KAFKA_HOME": "/liveperson/code/kafka/3.8.1", "LESSOPEN": "||/usr/bin/lesspipe.sh %s", "XDG_SESSION_ID": "1582", "_": "/usr/bin/python", "SSH_CONNECTION": "35.235.240.112 43701 10.201.192.107 22"}, "ansible_locally_reachable_ips": {"ipv4": ["10.201.192.107", "10.201.192.120", "127.0.0.0/8", "127.0.0.1"], "ipv6": []}, "ansible_userspace_bits": "64", "ansible_architecture": "x86_64", "ansible_default_ipv4": {"macaddress": "42:01:0a:c9:c0:6b", "network": "10.201.192.107", "prefix": "32", "mtu": 1460, "broadcast": "10.201.192.107", "alias": "eth0", "netmask": "255.255.255.255", "address": "10.201.192.107", "interface": "eth0", "type": "ether", "gateway": "10.201.192.1"}, "ansible_swapfree_mb": 0, "ansible_default_ipv6": {}, "ansible_chassis_vendor": "Google", "ansible_ssh_host_key_ed25519_public_keytype": "ssh-ed25519", "ansible_selinux": {"status": "enabled", "policyvers": 31, "type": "targeted", "mode": "enforcing", "config_mode": "enforcing"}, "ansible_userspace_architecture": "x86_64", "ansible_product_uuid": "NA", "ansible_pkg_mgr": "yum", "ansible_distribution": "CentOS", "ansible_iscsi_iqn": "", "ansible_all_ipv6_addresses": [], "ansible_uptime_seconds": 2154265, "ansible_kernel": "3.10.0-1160.83.1.el7.x86_64", "ansible_system_capabilities_enforced": "True", "ansible_python": {"executable": "/usr/bin/python", "version": {"micro": 5, "major": 2, "releaselevel": "final", "serial": 0, "minor": 7}, "type": "CPython", "has_sslcontext": true, "version_info": [2, 7, 5, "final", 0]}, "ansible_is_chroot": true, "ansible_hostnqn": "nqn.2014-08.org.nvmexpress:uuid:6335cf4e-2a7a-423c-aa90-38022ed65355", "ansible_board_asset_tag": "6445A4AB-E04B-AA82-6190-AA9F7F30C9BE", "ansible_user_shell": "/bin/bash", "ansible_product_serial": "NA", "ansible_form_factor": "Other", "ansible_distribution_file_parsed": true, "ansible_fips": false, "ansible_user_id": "sa_107611174234308979642", "ansible_selinux_python_present": true, "ansible_board_serial": "NA", "ansible_kernel_version": "#1 SMP Wed Jan 25 16:41:43 UTC 2023", "ansible_local": {}, "ansible_processor_vcpus": 30, "ansible_processor": ["0", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "1", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "2", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "3", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "4", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "5", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "6", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "7", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "8", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "9", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "10", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "11", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "12", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "13", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "14", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "15", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "16", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "17", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "18", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "19", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "20", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "21", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "22", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "23", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "24", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "25", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "26", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "27", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "28", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz", "29", "GenuineIntel", "INTEL(R) XEON(R) PLATINUM 8581C CPU @ 2.10GHz"], "ansible_ssh_host_key_ecdsa_public": "AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLZd1f6cVHm+PQpMYRGUA/ymXJ1n6tZV1MZUtgBJl5FRlZvLiovNgiBjEOKPsuK61vUw7CjqE92LjNMZsVkgNC0=", "ansible_user_gid": 3004348419, "ansible_system_vendor": "Google", "ansible_swaptotal_mb": 0, "ansible_loadavg": {"5m": 1.92, "15m": 1.8, "1m": 1.84}, "ansible_distribution_major_version": "7", "ansible_real_group_id": 3004348419, "ansible_lsb": {"release": "7.9.2009", "major_release": "7", "codename": "Core", "id": "CentOS", "description": "CentOS Linux release 7.9.2009 (Core)"}, "ansible_machine": "x86_64", "ansible_ssh_host_key_rsa_public_keytype": "ssh-rsa", "ansible_user_gecos": "", "ansible_processor_threads_per_core": 2, "ansible_eth0": {"macaddress": "42:01:0a:c9:c0:6b", "features": {"tx_checksum_ipv4": "off [fixed]", "generic_receive_offload": "on", "tx_checksum_ipv6": "off [fixed]", "tx_scatter_gather_fraglist": "off [fixed]", "rx_all": "off [fixed]", "highdma": "on", "rx_fcs": "off [fixed]", "tx_lockless": "off [fixed]", "tx_tcp_ecn_segmentation": "on", "rx_udp_tunnel_port_offload": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "off", "tx_checksumming": "on", "vlan_challenged": "off [fixed]", "loopback": "off [fixed]", "fcoe_mtu": "off [fixed]", "tx_checksum_sctp": "off [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_tcp_segmentation": "on", "rx_vlan_stag_filter": "off [fixed]", "large_receive_offload": "off [fixed]", "tx_checksum_ip_generic": "on", "rx_checksumming": "on", "rx_gro_hw": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "busy_poll": "off [fixed]", "generic_segmentation_offload": "on", "tx_udp_tnl_segmentation": "off [fixed]", "tcp_segmentation_offload": "on", "l2_fwd_offload": "off [fixed]", "rx_vlan_offload": "off [fixed]", "ntuple_filters": "off [fixed]", "rx_vlan_filter": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_sctp_segmentation": "off [fixed]", "udp_fragmentation_offload": "off [fixed]", "scatter_gather": "on", "tx_sit_segmentation": "off [fixed]", "tx_checksum_fcoe_crc": "off [fixed]", "hw_tc_offload": "off [fixed]", "tx_scatter_gather": "on", "netns_local": "off [fixed]", "tx_nocache_copy": "off", "tx_gre_csum_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "receive_hashing": "on", "tx_gre_segmentation": "off [fixed]"}, "pciid": "0000:00:03.0", "module": "gve", "mtu": 1460, "active": true, "promisc": false, "timestamping": ["rx_software", "software"], "ipv4": {"broadcast": "10.201.192.107", "network": "10.201.192.107", "netmask": "255.255.255.255", "prefix": "32", "address": "10.201.192.107"}, "device": "eth0", "type": "ether", "hw_timestamp_filters": []}, "ansible_system": "Linux", "ansible_processor_nproc": 30, "ansible_bios_vendor": "Google", "ansible_all_ipv4_addresses": ["10.201.192.107"], "ansible_python_version": "2.7.5", "ansible_virtualization_tech_guest": ["kvm"], "ansible_product_version": "NA", "ansible_service_mgr": "systemd", "ansible_memory_mb": {"real": {"total": 64258, "free": 415, "used": 63843}, "swap": {"cached": 0, "total": 0, "used": 0, "free": 0}, "nocache": {"used": 11988, "free": 52270}}, "ansible_user_dir": "/home/sa_107611174234308979642", "gather_subset": ["all"], "ansible_real_user_id": 3004348419, "ansible_virtualization_role": "guest", "ansible_dns": {"nameservers": ["169.254.169.254"], "search": ["c.lpgprj-b2b-p-dbeng-eu-1.internal", "XXXX-zone.c.lpgprj-b2b-p-dbeng-eu-1.internal", "google.internal"]}, "ansible_effective_group_id": 3004348419, "ansible_lo": {"features": {"tx_checksum_ipv4": "off [fixed]", "generic_receive_offload": "on", "tx_checksum_ipv6": "off [fixed]", "tx_scatter_gather_fraglist": "on [fixed]", "rx_all": "off [fixed]", "highdma": "on [fixed]", "rx_fcs": "off [fixed]", "tx_lockless": "on [fixed]", "tx_tcp_ecn_segmentation": "on", "rx_udp_tunnel_port_offload": "off [fixed]", "tx_gso_robust": "off [fixed]", "tx_ipip_segmentation": "off [fixed]", "tx_tcp_mangleid_segmentation": "on", "tx_checksumming": "on", "vlan_challenged": "on [fixed]", "loopback": "on [fixed]", "fcoe_mtu": "off [fixed]", "tx_checksum_sctp": "on [fixed]", "tx_vlan_stag_hw_insert": "off [fixed]", "rx_vlan_stag_hw_parse": "off [fixed]", "tx_gso_partial": "off [fixed]", "tx_tcp_segmentation": "on", "rx_vlan_stag_filter": "off [fixed]", "large_receive_offload": "off [fixed]", "tx_checksum_ip_generic": "on [fixed]", "rx_checksumming": "on [fixed]", "rx_gro_hw": "off [fixed]", "tx_fcoe_segmentation": "off [fixed]", "busy_poll": "off [fixed]", "generic_segmentation_offload": "on", "tx_udp_tnl_segmentation": "off [fixed]", "tcp_segmentation_offload": "on", "l2_fwd_offload": "off [fixed]", "rx_vlan_offload": "off [fixed]", "ntuple_filters": "off [fixed]", "rx_vlan_filter": "off [fixed]", "tx_tcp6_segmentation": "on", "tx_udp_tnl_csum_segmentation": "off [fixed]", "tx_sctp_segmentation": "on", "udp_fragmentation_offload": "on", "scatter_gather": "on", "tx_sit_segmentation": "off [fixed]", "tx_checksum_fcoe_crc": "off [fixed]", "hw_tc_offload": "off [fixed]", "tx_scatter_gather": "on [fixed]", "netns_local": "on [fixed]", "tx_nocache_copy": "off [fixed]", "tx_gre_csum_segmentation": "off [fixed]", "tx_vlan_offload": "off [fixed]", "receive_hashing": "off [fixed]", "tx_gre_segmentation": "off [fixed]"}, "hw_timestamp_filters": [], "mtu": 65536, "active": true, "promisc": false, "timestamping": ["rx_software", "software"], "ipv4": {"broadcast": "", "network": "127.0.0.0", "netmask": "255.0.0.0", "prefix": "8", "address": "127.0.0.1"}, "device": "lo", "type": "loopback"}, "ansible_memtotal_mb": 64258, "ansible_device_links": {"masters": {}, "labels": {"nvme0n1p2": ["root"]}, "ids": {"nvme0n1p1": ["google-XXXX-host-disk1-part1", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000001-part1", "nvme-nvme_card-pd_nvme_card-pd-part1"], "nvme0n1p2": ["google-XXXX-host-disk1-part2", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000001-part2", "nvme-nvme_card-pd_nvme_card-pd-part2"], "nvme0n3": ["google-XXXX-host-disk3", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000003"], "nvme0n2": ["google-XXXX-host-disk2", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000002", "nvme-nvme_card-pd_nvme_card-pd"], "nvme0n1": ["google-XXXX-host-disk1", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000001"]}, "uuids": {"nvme0n1p1": ["2CD7-416A"], "nvme0n1p2": ["78e589b6-2c25-4a3e-ba57-e06505f65a66"], "nvme0n3": ["bba92e4d-7cdf-49dc-95c7-7a8c60251bbb"], "nvme0n2": ["ffadead6-d3d8-44bd-9275-edb66187dfd1"]}}, "ansible_apparmor": {"status": "disabled"}, "ansible_board_vendor": "Google", "ansible_proc_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1160.83.1.el7.x86_64", "elevator": "noop", "net.ifnames": "0", "biosdevname": "0", "crashkernel": "auto", "console": "ttyS0,38400n8", "scsi_mod.use_blk_mq": "Y", "ro": true, "root": "UUID=78e589b6-2c25-4a3e-ba57-e06505f65a66"}, "ansible_memfree_mb": 415, "ansible_board_version": "NA", "ansible_cmdline": {"BOOT_IMAGE": "/boot/vmlinuz-3.10.0-1160.83.1.el7.x86_64", "elevator": "noop", "net.ifnames": "0", "biosdevname": "0", "crashkernel": "auto", "console": "ttyS0,38400n8", "scsi_mod.use_blk_mq": "Y", "ro": true, "root": "UUID=78e589b6-2c25-4a3e-ba57-e06505f65a66"}, "ansible_virtualization_tech_host": ["kvm"], "ansible_processor_count": 1, "ansible_hostname": "XXXX-host", "ansible_interfaces": ["lo", "eth0"], "ansible_machine_id": "c52f13ebb78204879fbb035ad6b26011", "ansible_fqdn": "XXXX-host.XXXX-zone.c.lpgprj-b2b-p-dbeng-eu-1.internal", "ansible_mounts": [{"block_used": 143782, "uuid": "ffadead6-d3d8-44bd-9275-edb66187dfd1", "size_total": 107321753600, "block_total": 26201600, "mount": "/liveperson/code", "block_available": 26057818, "size_available": 106732822528, "fstype": "xfs", "inode_total": 52428800, "inode_available": 52428523, "device": "/dev/nvme0n2", "inode_used": 277, "block_size": 4096, "options": "rw,seclabel,relatime,attr2,inode64,noquota"}, {"block_used": 2867, "uuid": "2CD7-416A", "size_total": 209489920, "block_total": 51145, "mount": "/boot/efi", "block_available": 48278, "size_available": 197746688, "fstype": "vfat", "inode_total": 0, "inode_available": 0, "device": "/dev/nvme0n1p1", "inode_used": 0, "block_size": 4096, "options": "rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro"}, {"block_used": 1383733, "uuid": "78e589b6-2c25-4a3e-ba57-e06505f65a66", "size_total": 80308387840, "block_total": 19606540, "mount": "/", "block_available": 18222807, "size_available": 74640617472, "fstype": "xfs", "inode_total": 39218176, "inode_available": 39106116, "device": "/dev/nvme0n1p2", "inode_used": 112060, "block_size": 4096, "options": "rw,seclabel,relatime,attr2,inode64,noquota"}, {"block_used": 149447806, "uuid": "bba92e4d-7cdf-49dc-95c7-7a8c60251bbb", "size_total": 3756261376000, "block_total": 917056000, "mount": "/liveperson/data", "block_available": 767608194, "size_available": 3144123162624, "fstype": "xfs", "inode_total": 367001600, "inode_available": 366987395, "device": "/dev/nvme0n3", "inode_used": 14205, "block_size": 4096, "options": "rw,seclabel,relatime,attr2,inode64,noquota"}], "ansible_ssh_host_key_rsa_public": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDHmgur6f3rZT4pVPSs30XPekRAcX9l5QGW96iAGR0zuPGH3E1oRX0I91yp1DJ15HL3nGJ5XqNlmmAUdRsugTSIf3PlMELxknC1kg6WksDo9reBxzu6rYRD5RGAL3P3/05UKTmxJFHXvFZmpMhzegpUvWadoQPOPTcEFBlnNqj8Ut0dQn1N62AEnxPADAt3zbNrX23hEp6SNTN72IJYr7mRwQbvrIo7abLKN3K+Z202u6BDUIqDgb1pwODp8EHGOyvrj3GjmPSFp2fH0ASU5eqH8jhM9/5Qdh5D8a+S4PpiRD7D1qbzSoijOSR97DGGwx/qDJKz5fb+YvNnpOqs2CtJ", "ansible_nodename": "XXXX-host", "ansible_lvm": "N/A", "ansible_domain": "XXXX-zone.c.lpgprj-b2b-p-dbeng-eu-1.internal", "ansible_distribution_file_path": "/etc/redhat-release", "ansible_virtualization_type": "kvm", "ansible_ssh_host_key_ed25519_public": "AAAAC3NzaC1lZDI1NTE5AAAAIG/mhxnPr+Yqnl73l4B3P0JkRxBula4EBAtDW4KnVf/9", "ansible_processor_cores": 15, "ansible_chassis_serial": "NA", "ansible_bios_version": "Google", "ansible_date_time": {"weekday_number": "2", "iso8601_basic_short": "20250930T050314", "tz": "UTC", "weeknumber": "39", "hour": "05", "time": "05:03:14", "epoch_int": "1759208594", "tz_offset": "+0000", "month": "09", "epoch": "1759208594", "iso8601_micro": "2025-09-30T05:03:14.043302Z", "weekday": "Tuesday", "iso8601_basic": "20250930T050314043302", "year": "2025", "date": "2025-09-30", "iso8601": "2025-09-30T05:03:14Z", "tz_dst": "UTC", "day": "30", "minute": "03", "second": "14"}, "ansible_distribution_release": "Core", "ansible_os_family": "RedHat", "ansible_effective_user_id": 3004348419, "ansible_ssh_host_key_ecdsa_public_keytype": "ecdsa-sha2-nistp256", "ansible_product_name": "Google Compute Engine", "ansible_devices": {"nvme0n3": {"scheduler_mode": "none", "rotational": "0", "vendor": null, "sectors": "7340032000", "links": {"masters": [], "labels": [], "ids": ["google-XXXX-host-disk3", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000003"], "uuids": ["bba92e4d-7cdf-49dc-95c7-7a8c60251bbb"]}, "sas_device_handle": null, "sas_address": null, "virtual": 1, "host": "", "sectorsize": "512", "removable": "0", "support_discard": "512", "model": "nvme_card-pd", "size": "3.42 TB", "holders": [], "partitions": {}}, "nvme0n2": {"scheduler_mode": "none", "rotational": "0", "vendor": null, "sectors": "209715200", "links": {"masters": [], "labels": [], "ids": ["google-XXXX-host-disk2", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000002", "nvme-nvme_card-pd_nvme_card-pd"], "uuids": ["ffadead6-d3d8-44bd-9275-edb66187dfd1"]}, "sas_device_handle": null, "sas_address": null, "virtual": 1, "host": "", "sectorsize": "512", "removable": "0", "support_discard": "512", "model": "nvme_card-pd", "size": "100.00 GB", "holders": [], "partitions": {}}, "nvme0n1": {"scheduler_mode": "none", "rotational": "0", "vendor": null, "sectors": "157286400", "links": {"masters": [], "labels": [], "ids": ["google-XXXX-host-disk1", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000001"], "uuids": []}, "sas_device_handle": null, "sas_address": null, "virtual": 1, "host": "", "sectorsize": "512", "removable": "0", "support_discard": "512", "model": "nvme_card-pd", "size": "75.00 GB", "holders": [], "partitions": {"nvme0n1p1": {"sectorsize": 512, "uuid": "2CD7-416A", "sectors": "409600", "links": {"masters": [], "labels": [], "ids": ["google-XXXX-host-disk1-part1", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000001-part1", "nvme-nvme_card-pd_nvme_card-pd-part1"], "uuids": ["2CD7-416A"]}, "start": "2048", "holders": [], "size": "200.00 MB"}, "nvme0n1p2": {"sectorsize": 512, "uuid": "78e589b6-2c25-4a3e-ba57-e06505f65a66", "sectors": "156872800", "links": {"masters": [], "labels": ["root"], "ids": ["google-XXXX-host-disk1-part2", "nvme-nvme.1ae0-6e766d655f636172642d7064-6e766d655f636172642d7064-00000001-part2", "nvme-nvme_card-pd_nvme_card-pd-part2"], "uuids": ["78e589b6-2c25-4a3e-ba57-e06505f65a66"]}, "start": "411648", "holders": [], "size": "74.80 GB"}}}}, "ansible_user_uid": 3004348419, "ansible_bios_date": "06/12/2025", "ansible_chassis_version": "NA", "ansible_system_capabilities": [""], "ansible_chassis_asset_tag": "NA"}}\\n', b"Activated service account credentials for: [lpgsac-p-svc-awx@lpgprj-gss-p-seracc-gl-01.iam.gserviceaccount.com]\\nWarning: Permanently added 'compute.633288219271272956' (ED25519) to the list of known hosts.\\r\\nSSH Login banner\\n")
ok: [XXXX-host]
TASK [kafka_os_setup : Remove repo files which are not needed] *****************
task path: /runner/project/roles/kafka_os_setup/tasks/main.yml:4
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/file.py
Pipelining is enabled.
<XXXX-host> ESTABLISH SSH CONNECTION FOR USER: root
<XXXX-host> SSH: EXEC /etc/ansible/scripts/gcp-ssh-wrapper.sh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/ssh-key-ansible-sa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 --tunnel-through-iap --zone=XXXX-zone --project=XXXX-project-o 'ControlPath="/runner/cp/7f8f72306d"' XXXX-host '/bin/sh -c '"'"'become=True become_user=root become_method=sudo become_flags=-H /usr/bin/python && sleep 0'"'"''
<XXXX-host> (0, b'\\n{"invocation": {"module_args": {"src": null, "serole": null, "force": false, "modification_time": null, "unsafe_writes": false, "selevel": null, "seuser": null, "recurse": false, "state": "absent", "access_time_format": "%Y%m%d%H%M.%S", "_original_basename": null, "setype": null, "follow": true, "_diff_peek": null, "modification_time_format": "%Y%m%d%H%M.%S", "access_time": null, "owner": null, "group": null, "path": "/etc/yum.repos.d/CentOS-x86_64-kernel.repo", "attributes": null, "mode": null}}, "path": "/etc/yum.repos.d/CentOS-x86_64-kernel.repo", "state": "absent", "changed": false}\\n', b'')
ok: [XXXX-host] => ------
}
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/file.py
Pipelining is enabled.
<XXXX-host> ESTABLISH SSH CONNECTION FOR USER: root
<XXXX-host> SSH: EXEC /etc/ansible/scripts/gcp-ssh-wrapper.sh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/ssh-key-ansible-sa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 --tunnel-through-iap --zone=XXXX-zone --project=XXXX-project-o 'ControlPath="/runner/cp/7f8f72306d"' XXXX-host '/bin/sh -c '"'"'become=True become_user=root become_method=sudo become_flags=-H /usr/bin/python && sleep 0'"'"''
<XXXX-host> (0, b'\\n{"invocation": {"module_args": {"src": null, "serole": null, "force": false, "modification_time": null, "unsafe_writes": false, "selevel": null, "seuser": null, "recurse": false, "state": "absent", "access_time_format": "%Y%m%d%H%M.%S", "_original_basename": null, "setype": null, "follow": true, "_diff_peek": null, "modification_time_format": "%Y%m%d%H%M.%S", "access_time": null, "owner": null, "group": null, "path": "/etc/yum.repos.d/CentOS-Vault.repo", "attributes": null, "mode": null}}, "path": "/etc/yum.repos.d/CentOS-Vault.repo", "state": "absent", "changed": false}\\n', b'')
ok: [XXXX-host] => (-----
}
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/file.py
Pipelining is enabled.
<XXXX-host> ESTABLISH SSH CONNECTION FOR USER: root
<XXXX-host> SSH: EXEC /etc/ansible/scripts/gcp-ssh-wrapper.sh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/ssh-key-ansible-sa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 --tunnel-through-iap --zone=XXXX-zone --project=XXXX-project-o 'ControlPath="/runner/cp/7f8f72306d"' XXXX-host '/bin/sh -c '"'"'become=True become_user=root become_method=sudo become_flags=-H /usr/bin/python && sleep 0'"'"''
<XXXX-host> (0, b'\\n{"invocation": {"module_args": {"src": null, "serole": null, "force": false, "modification_time": null, "unsafe_writes": false, "selevel": null, "seuser": null, "recurse": false, "state": "absent", "access_time_format": "%Y%m%d%H%M.%S", "_original_basename": null, "setype": null, "follow": true, "_diff_peek": null, "modification_time_format": "%Y%m%d%H%M.%S", "access_time": null, "owner": null, "group": null, "path": "/etc/yum.repos.d/CentOS-Sources.repo", "attributes": null, "mode": null}}, "path": "/etc/yum.repos.d/CentOS-Sources.repo", "state": "absent", "changed": false}\\n', b'')
ok: [XXXX-host] => (----
}
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/file.py
Pipelining is enabled.
<XXXX-host> ESTABLISH SSH CONNECTION FOR USER: root
<XXXX-host> SSH: EXEC /etc/ansible/scripts/gcp-ssh-wrapper.sh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/ssh-key-ansible-sa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 --tunnel-through-iap --zone=XXXX-zone --project=XXXX-project-o 'ControlPath="/runner/cp/7f8f72306d"' XXXX-host '/bin/sh -c '"'"'become=True become_user=root become_method=sudo become_flags=-H /usr/bin/python && sleep 0'"'"''
<XXXX-host> (0, b'\\n{"invocation": {"module_args": {"src": null, "serole": null, "force": false, "modification_time": null, "unsafe_writes": false, "selevel": null, "seuser": null, "recurse": false, "state": "absent", "access_time_format": "%Y%m%d%H%M.%S", "_original_basename": null, "setype": null, "follow": true, "_diff_peek": null, "modification_time_format": "%Y%m%d%H%M.%S", "access_time": null, "owner": null, "group": null, "path": "/etc/yum.repos.d/CentOS-Media.repo", "attributes": null, "mode": null}}, "path": "/etc/yum.repos.d/CentOS-Media.repo", "state": "absent", "changed": false}\\n', b'')
ok: [XXXX-host] => (----
I’m guessing this might be the case: according to the official documentation, “If your control machine uses an older OpenSSH version that doesn’t support ControlPersist, Ansible falls back to a Python implementation called ‘paramiko’.”
Could this be what’s happening here? Any guidance would be appreciated.