AWX to GCP VM : Establish and Maintain IAP Tunnel for Subsequent Playbook Execution

SaravanaSelvaraj · April 25, 2025, 3:30pm

AWX to GCP SSH connectivity is getting closed for subsequent playbook Execution.
Dear community,
My project

---
- name: Establish GCP connectivity using impersonated SA
  hosts: all
  gather_facts: false
  any_errors_fatal: true
  tasks:
    - name: Establish SSH tunnel via gcloud
      block:
        - name: Run GCP connectivity role
          include_role:
            name: gcp_connectivity_role
          vars:
            gcp_project: "{{ hostvars[inventory_hostname]['project'] }}"
            gcp_zone: "{{ hostvars[inventory_hostname]['zone'] }}"
            gcp_instance_name: "{{ hostvars[inventory_hostname]['name'] }}"
            gcp_impersonate_sa: "{{ hostvars[inventory_hostname]['gcp_impersonate_sa'] }}"
      delegate_to: localhost

- name: Run shell commands on connected GCP hosts
  hosts: all
  gather_facts: false
  tasks:
    - name: Run uname -a
      command: uname -a

    - name: Run uptime
      command: uptime

    - name: Show disk usage
      command: df -h

And my role is

---
- name: Run GCP tasks via gcloud
  block:
    - name: Set facts from credential and vars
      ansible.builtin.set_fact:
        gcp_sa_json_file: "{{ lookup('env', 'gcp_sa_json') }}"
        gcp_impersonate_sa: "{{ lookup('env', 'gcp_sa_user_id') }}"
        gcp_project: "{{ hostvars[inventory_hostname]['project'] }}"
        gcp_zone: "{{ hostvars[inventory_hostname]['zone'] }}"
        gcp_instance_name: "{{ hostvars[inventory_hostname]['name'] }}"
        gcp_commands: "{{ gcp_commands | default([]) }}"

    - name: Authenticate once
      shell: gcloud auth activate-service-account --key-file={{ gcp_sa_json_file }}
      #when: current_auth.stdout.strip() != gcp_impersonate_sa
      register: auth_result
      failed_when: auth_result.rc != 0
      run_once: true
      delegate_to: localhost

    - name: Run gcloud SSH with commands
      shell: >
        yes y | gcloud compute ssh {{ gcp_instance_name }} \
        --zone={{ zone }} \
        --project={{ project }} \
        --impersonate-service-account={{ gcp_impersonate_sa }} \
        --tunnel-through-iap \
        --quiet \
        --verbosity debug
      delegate_to: localhost
      register: gcp_ssh_result
      failed_when: gcp_ssh_result.rc != 0
      changed_when: false

I am using the role gcp_connectivity_role to establish the gcp connectivity. And then in the project I am running commands in the project starting from Run shell commands on connected GCP hosts . As expected the connection closes after the role finished its job and then the commands does not run on the hosts i am intended to do. Can you please check if I am following the correct process here ? Please help.

Thanks and Regards
Saravana Selvaraj

SaravanaSelvaraj · April 25, 2025, 3:42pm

@kurokobo , @wayt and @bcoca tagging you here . Please help if you have any idea in this topic.

Thanks and in advance.

SaravanaSelvaraj · April 26, 2025, 5:16am

@iamroddo I see you have been working on GCP and AWX. Could you please share your thoughts here if possible ? In simple words,
Do you have a working example of connecting from AWX to GCP hosts (Internal IP only) and running some commands on it?

wayt · April 27, 2025, 6:59pm

I setup a constrained example consisting of 2 VMs hosted in GCE and a Linux PC acting as the control node. The VMs are setup using metadata based authentication with an SSH key setup at the project level and are internal-only. The VMs have outbound connectivity via a NAT router. The control node will use the feature of the GCloud CLI to establish a tunnel through the IAP service to connect to the VMs on port 22 which is allowed by the default firewall rules. This same setup should work for Windows VMs using WinRM by forwarding the port for WinRM instead of 22 through the IAP service and allowing it to the VM from IAP’s well known IP range as the source.

An inventory like:

[project]
managed-node-1
managed-node-2

[project:vars]
gce_project = project

[us_central1:children]
project

[us_central1:vars]
gce_zone = us-central1

[all:vars]
ansible_ssh_common_args="-o ‘ProxyCommand gcloud compute start-iap-tunnel %h 22 --listen-on-stdin --project {{ gce_project }} --zone {{ gce_zone }}’"
ansible_user = selvaraj

A playbook like:

---
- name: Activate the `gcloud` CLI
  hosts: localhost

  tasks:
    - name: Activate the `gcloud` CLI
      ansible.builtin.command:
        argv:
          - gcloud
          - auth
          - activate-service-account
          - --key-file
          - '{{ lookup("ansible.builtin.env", "GCE_CREDENTIALS_FILE_PATH") }}'
      changed_when: false

- name: Take action on GCP based managed nodes
  hosts: all

  tasks:
    - name: Print hostname
      ansible.builtin.debug:
        var: ansible_hostname

ansible-navigator configured like ( ansible-navigator.yml ):

The mode is set to stdout really just so I could copy and paste the play output easier

---
ansible-navigator:
  execution-environment:
    environment-variables:
      set:
        ANSIBLE_HOST_KEY_CHECKING: false
        GCE_CREDENTIALS_FILE_PATH: /.gce.json
    image: localhost/selvaraj:2
    volume-mounts:
      - src: ~/.gce.json
        dest: /.gce.json
        options: Z
  mode: stdout
  playbook-artifact:
    enable: false

Ran via:

ansible-navigator run playbook.yml --inventory hosts

Results in:

$ ansible-navigator run playbook.yml --inventory hosts

PLAY [Activate the `gcloud` CLI] *************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Activate the `gcloud` CLI] *************************************************************************************************************************************************************************************************************
ok: [localhost]

PLAY [Take action on GCP based managed nodes] ************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
[WARNING]: Platform linux on host managed-node-1 is using the discovered Python interpreter at /usr/bin/python3.11, but future installation of another Python interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
ok: [managed-node-1]
[WARNING]: Platform linux on host managed-node-2 is using the discovered Python interpreter at /usr/bin/python3.11, but future installation of another Python interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
ok: [managed-node-2]

TASK [Print hostname] ************************************************************************************************************************************************************************************************************************
ok: [managed-node-1] => {
    "ansible_hostname": "managed-node-1"
}
ok: [managed-node-2] => {
    "ansible_hostname": "managed-node-2"
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
managed-node-1             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
managed-node-2             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

SaravanaSelvaraj · April 28, 2025, 10:19am

Thanks a lot @wayt for this. I will try this one. It was a great help.

Best regards
Saravana Selvaraj

wayt · April 28, 2025, 1:35pm

Glad to have helped. Good luck!

SaravanaSelvaraj · April 28, 2025, 2:06pm

@wayt , Thanks again. While you are still here, I want some help on adding a custom variable as hostvars in the inventory. But none of the below values are getting populated in the hostvars for my hosts. Am I doing anything wrong ? Please help .

compose:
   gcp_compute_ssh_flags: "--tunnel-through-iap --no-user-output-enabled --quiet --ssh-key-file=/root/.ssh/google_compute_engine --project={{ project }}"
  gcp_compute_scp_flags: "--tunnel-through-iap --quiet --ssh-key-file=/root/.ssh/google_compute_engine --project={{ project }}"
  ansible_python_interpreter: /usr/bin/python3

Topic		Replies	Views
Delegate_to: localhost is not working as expected Get Help awx , playbook , gcp , ansible	12	78	April 30, 2025
How to use GCP service account for ssh authentication in AWX Get Help awx , ansible-core , gcp	2	279	April 27, 2025
Alternate SSH using ssh_executable AWX Project awx , gcp	1	23	July 15, 2020
GCP Tunnel through IAP Ansible Project gcp	2	66	February 9, 2021
Unable to SSH from Ansible AWX to GCP VM Instance Get Help awx , gcp	5	56	April 22, 2025

AWX to GCP VM : Establish and Maintain IAP Tunnel for Subsequent Playbook Execution

Related topics