Dynamically give Ansible a private key from an Infisical vault/Terraform

Hi everyone,

I am learning to use Ansible but I am running into a bit of a wall with regards to authentication. It could be that I am going about it all wrong in the first place or that I am somehow missing something relatively small. Regardless of which one it is, I could use some help.

So, I’ve deployed two vsphere virtual machines using Terraform. That part works as it should, I’ve verified this through connecting to them manually. Now, with security in mind, I want to make it so that Ansible can access the machines using a key. I’ve created a key pair using Terraform’s TLS module and stored both the private and public key in a vault (Infisical). My Ansible is deployed from gitlab and can obtain the secrets from Infisical by authenticating through environment variables. The public key should be on the deployed VM’s, but I cannot figure out how to get the private key in a position where Ansible can connect with my machines.

I’ve tried to look for an option to parse the key directly like one would a password, but failed to find such an option. Now, I’m trying to put the key in a file on the local runner like this:

tasks:
    - name: Create the .ssh directory
      ansible.builtin.shell: mkdir -p /home/ansible/.ssh/
      delegate_to: 127.0.0.1

    - name: Put private key on local runner
      ansible.builtin.shell: cat "{{ ansible_private_key }}" > /home/ansible/.ssh/id_rsa
      delegate_to: 127.0.0.1

    - name: Put public key on local runner
      ansible.builtin.shell: cat "{{ ansible_public_key }}" > /home/ansible/.ssh/id_rsa.pub
      delegate_to: 127.0.0.1

    - name: List all entries of the .ssh directory
      ansible.builtin.shell: ls -lah ~.ssh/
      delegate_to: 127.0.0.1
      register: ls
    - debug: var=ls.stdout_lines

I’ve also tried to use local_action in a similar fashion, but neither of them works. I’m not even getting the debug output I expected, only that it cannot find the file that it is supposed to make:

$ ansible-playbook --inventory inventory.yml playbook.yml $ANSIBLE_VARS
PLAY [Install the necessary software on the remote hosts] **********************
TASK [Gathering Facts] *********************************************************
fatal: [webserver01]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '<IP_address>:' (ED25519) to the list of known hosts.\r\nno such identity: /home/ansible/.ssh/id_rsa: No such file or directory\r\nubuntu@<IP_address>:: Permission denied (publickey).", "unreachable": true}
fatal: [webserver02]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '<IP_address>:' (ED25519) to the list of known hosts.\r\nno such identity: /home/ansible/.ssh/id_rsa: No such file or directory\r\nubuntu@<IP_address>: Permission denied (publickey).", "unreachable": true}
PLAY RECAP *********************************************************************
webserver01                : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   
webserver02                : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

I’ve manually exchanged the actual IP addresses for <IP_address> for security.

So, I guess the overarching question is, how do I get my private key from where it is made into the Ansible container that runs in gitlab? Am I on the right track? Given that I am still learning, I am also happy with a hint or link to the documentation that contains the solution.

Thank you in advance for your help!

Hello Alejandro,

Please repost this to Get Help category instead and lets discuss there.

2 Likes

Welcome Alejandro, I’ve moved this post to the Get Help category for you.

Thanks for flagging that @kristianheljas

1 Like

You seem to be almost on the right track.

My assumption is that you have either too broad permissions (due to just piping into the file) on the ssh private key file or you are not running the ansible-playbook command with the ansible user.

Have you verified ssh connection manually?

ssh -i /home/ansible/.ssh/id_rsa ubuntu@remote-host

Also, there are ansible modules you could use instead of the shell module.

I would suggest to rewrite the ssh key play like so, including correct permissions for the key files:

    - name: Create .ssh directory
      ansible.builtin.file:
        path: /home/ansible/.ssh
        state: directory
        mode: u=rwx
        owner: ansible
        group: ansible

    - name: Copy private key
      ansible.builtin.copy:
        path: /home/ansible/.ssh/id_rsa
        content: "{{ ansible_private_key }}"
        mode: u=rw
        owner: ansible
        group: ansible

    - name: Copy public key
      ansible.builtin.copy:
        path: /home/ansible/.ssh/id_rsa.pub
        content: "{{ ansible_public_key }}"
        mode: u=rw,go=r
        owner: ansible
        group: ansible

1 Like

I think this may be the issue. Initially I had the path specified as ~/.ssh/id_rsa which resulted in Ansible looking for /root/.ssh/id_rsa/ instead, indicating it is running as root rather than as ansible. So it seems like I will need to find a way to either let the docker container run with the user ansible or find/make a different container that does. I will look into that as well as the different modules. Thank you for the suggestion!

PS: I was in the Get Help section and clicked to look at the guides/faq section to see if this is a common issue, then forgot to switch back when I was making the post. Apologies!

1 Like

All right, I’ve got the connection to work. Turns out my issue was missing a key part of knowledge that thankfully one of the coaches here realized. I hope you all don’t mind the long reply as I lay out the issue and the solution.

So from what I’ve learned, when Ansible starts its run, it will first check its resources and attempt to connect to the specified server. In my case, this was the remote server, which is why I attempted to use delegate_to: 127.0.0.1 and local_action. I had expected the script to (locally) parse the keys and then connect to the host, since this is the order layed out in the playbook. However, since Ansible checks the resources first, it attempted to connect using the non-existend key before getting to the part that made said key.

The solution I ended up with was to make two separate playbooks; one to run the local keyparsing job and one to make the remote changes. I successfully managed to parse the keys using the following playbook:

# playbook_key.yml
--- 
- name: Prepare the key on the local machine
  hosts: localhost
  connection: local 

  vars:
    infisical_cid: "{{ lookup('ansible.builtin.env', 'INFISICAL_CID') }}"
    infisical_pid: "{{ lookup('ansible.builtin.env', 'INFISICAL_PID') }}"
    infisical_cs: "{{ lookup('ansible.builtin.env', 'INFISICAL_CS') }}"
    ansible_private_key: "{{ lookup('infisical.vault.read_secrets', universal_auth_client_id=infisical_cid, universal_auth_client_secret=infisical_cs, project_id=infisical_pid, path='/', env_slug='dev', secret_name='VM_KEY_PRIVATE').value }}"
    ansible_public_key: "{{ lookup('infisical.vault.read_secrets', universal_auth_client_id=infisical_cid, universal_auth_client_secret=infisical_cs, project_id=infisical_pid, path='/', env_slug='dev', secret_name='VM_KEY_PUBLIC').value }}"

  tasks:
    - name: Copy private key 
      ansible.builtin.copy:
        dest: ~/.ssh/id_rsa
        content: "{{ ansible_private_key }}"
        mode: 0600      # equals chmod 600, the leading 0 is necessary to clarify that it is an octal number
        owner: ansible
        group: ansible

    - name: Copy public key
      ansible.builtin.copy:
        dest: ~/.ssh/id_rsa.pub
        content: "{{ ansible_public_key }}"
        mode: 0644
        owner: ansible
        group: ansible

In short, this playbook connects to my Infisical vault using the authentication provided in environment variables and parses the keys through to the files. Note the .value at the end of the Infisical lookups, which was necessary due to the multi-line format of the text. I changed the symbolic permissions to numerical, which works better in my case.

From there, I was successfully able to connect to my VMs and apply a test change, verified through manual connection:

# playbook_vm.yml
---
- name: Install the necessary software on the remote hosts
  hosts: webservers
  remote_user: ubuntu

  vars:
    infisical_cid: "{{ lookup('ansible.builtin.env', 'INFISICAL_CID') }}"
    infisical_pid: "{{ lookup('ansible.builtin.env', 'INFISICAL_PID') }}"
    infisical_cs: "{{ lookup('ansible.builtin.env', 'INFISICAL_CS') }}"
    ip_server_01: "{{ lookup('infisical.vault.read_secrets', universal_auth_client_id=infisical_cid, universal_auth_client_secret=infisical_cs, project_id=infisical_pid, path='/', env_slug='dev', secret_name='VM_IP_01') }}"
    ip_server_02: "{{ lookup('infisical.vault.read_secrets', universal_auth_client_id=infisical_cid, universal_auth_client_secret=infisical_cs, project_id=infisical_pid, path='/', env_slug='dev', secret_name='VM_IP_02') }}"
    client_public_key: "{{ lookup('infisical.vault.read_secrets', universal_auth_client_id=infisical_cid, universal_auth_client_secret=infisical_cs, project_id=infisical_pid, path='/', env_slug='dev', secret_name='SSHKEY_ABAARS_SOGYO').value }}"
    user: ubuntu

  tasks:
    - name: Test the connection
      ansible.builtin.shell:
        cmd: |
          echo "I am connected!" > ~/process.begin.message
# inventory.yml
webservers:
  hosts:
    webserver01:
      ansible_host: "{{ ip_server_01.value }}"
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
      ansible_ssh_user: "{{ user }}"
      http_port: 22
    webserver02:
      ansible_host: "{{ ip_server_02.value }}"
      ansible_ssh_private_key_file: ~/.ssh/id_rsa
      ansible_ssh_user: "{{ user }}"
      http_port: 22

As with regards to the permission issue, I was able to dig up a workaround to specify the user that Ansible should run with within the docker environment. We can specify the docker’s entrypoint:

# gitlab-ci.yml
variables:
  DOCKER_ANSIBLE: serversideup/ansible:8.7.0-alpine3.20-python3.12

stages:
  - verify
  - deploy

verify_ansible_playbook:
  stage: verify
  image: 
    name: $DOCKER_ANSIBLE
    entrypoint: ['sh', '-c', 'exec su ansible -c sh']
  before_script:
    - pip install infisicalsdk
    - ansible-galaxy collection install infisical.vault
    - cp ansible_launch_app/.ansible.cfg ~/.ansible.cfg
  script:
    - ansible-playbook --syntax-check ansible_get_key/playbook_key.yml
    - ansible-playbook --check ansible_get_key/playbook_key.yml
    - ansible-playbook --inventory ansible_launch_app/inventory.yml --syntax-check ansible_launch_app/playbook_vm.yml
    - ansible-playbook --inventory ansible_launch_app/inventory.yml --check ansible_launch_app/playbook_vm.yml
  allow_failure: true

deploy_minesweeper:
  stage: deploy
  image: 
    name: $DOCKER_ANSIBLE
    entrypoint: ['sh', '-c', 'exec su ansible -c sh']
  rules:
    - if: $CI_COMMIT_BRANCH
      when: manual
  before_script:
    - pip install infisicalsdk
    - ansible-galaxy collection install infisical.vault
    - cp ansible_launch_app/.ansible.cfg ~/.ansible.cfg
  script:
    - ansible-playbook ansible_get_key/playbook_key.yml
    - ansible-playbook -v --inventory ansible_launch_app/inventory.yml ansible_launch_app/playbook_vm.yml
  needs: ["verify_ansible_playbook"]

As a sidenote, the allow_failure: true is necessary in this configuration because the ansible-playbook --check will throw an error as it cannot find the specified key file (because it is not made yet).

So, there’s most of the knowledge I’ve obtained these past few hours. Thank you again for the suggestions as they have certainly put me on the right track!