Vmware ssl certs (works in one module, not the other!)

[vmadmin]

Somewhat new to Ansible and connecting it to vsphere, but I threw together a simple playbook to test some modules. I have imported the vcenter root CA certificate and verified it is trusted via openssl. The task that calls the cluster info connects via SSL and returns output just fine, but the last task fetching the content library info returns an error with validate_certs on or off (see more below):

---
- name: host in cluster info demo
  hosts: localhost
  become: false
  gather_facts: false
  collections:
    - community.vmware
  pre_tasks:
    - include_vars: vars/vpc_vault.yml
    - include_vars: vars/vpc.yml
  tasks:
    - name: Gather info about all ESXi Hosts in given Cluster
      community.vmware.vmware_cluster_info:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        #validate_certs: "{{ vcenter_validate_certs }}"
        validate_certs: true
        cluster_name: "{{ cluster_name }}"
        schema: vsphere
        properties:
          - name
      delegate_to: localhost
      register: cluster_info
    - name: print cluster info
      ansible.builtin.debug:
        var: cluster_info
    - name: Content Library info
      community.vmware.vmware_content_library_info:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        validate_certs: true
      delegate_to: localhost

Returns:

PLAY [host in cluster info demo] ************************************************************************************************************************************************************************************************************

TASK [include_vars] *************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [include_vars] *************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Gather info about all ESXi Hosts in given Cluster] ************************************************************************************************************************************************************************************
ok: [localhost]

TASK [print cluster info] *******************************************************************************************************************************************************************************************************************
ok: [localhost] => {
    "cluster_info": {
        "changed": false,
        "clusters": {
            "redacted": {
                "name": "redacted"
            }
        },
        "failed": false
    }
}

TASK [Content Library info] *****************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to connect to vCenter or ESXi API at XXXXXredacted:443 due to SSL verification failure : HTTPSConnectionPool(host='XXXXXXXredacted', port=443): Max retries exceeded with url: /api (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))"}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

And if I set validate_certs to false on the content library task:

I get a huge error message with some bits like

Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings\n  warnings.warn(\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc":

and

fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "/home/XXXXX/.local/lib/python3.10/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'XXXXredacted'. Adding certificate verification is strongly advised.

Essentially: I know validate_certs: true works because the cluster task completes. But the content lib task fails on this.

Furthermore, I tried forcing http (and also “http”) instead of default https in the content library module parameter but still the same error. I can confirm all my ESXi hosts and vcenter have valid certs (self-signed though). The connection between esxi and vcenter is trusted. So, while I’m new to vsphere SDK, I’m beginning to think is a bug maybe.

[mariolenz]

We’re also seeing this issue in the CI pipeline of the collection, but only recently. We’ve started to investigate.

[mariolenz]

If I understand you correct there might be two different issues here with community.vmware.vmware_content_library_info.

1. Why do you get an error when you define validate_certs: false
2. Why doesn’t the module accept the certificate while community.vmware.vmware_cluster_info does?

Would it be possible for you to test if this PR fixes at least the first one?

[vmadmin]

Hi Mario,

Yes on both 1. and 2. I really appreciate you looking into this.

Unfortunately, I am not terribly familiar with git. If you could tell me how to pull that down that specific branch/pull request I think I could get this module in a virtual env and test.

[ptn (Pierre TOURON)]

Hi,

If you could tell me how to pull that down that specific branch/pull request

Simplest way IMO would be to clone 20240114_validate_certs branch from mariolenz’s community.vmware fork: git clone git@github.com:mariolenz/community.vmware.git -b 20240114_validate_certs, then run your tests targeting this checkout.

Edit: An even simplest option would be to directly install collection from this fork: ansible-galaxy collection install git+https://github.com/mariolenz/community.vmware,20240114_validate_certs -p /tmp/collections/

[mariolenz]

I guess there are several ways. One way would be to clone my repo as @ptn suggested. But it should also be possible to get the code from the PR directly:

$ mkdir -p  ~/.ansible/collections/ansible_collections/community
$ git clone https://github.com/ansible-collections/community.vmware ~/.ansible/collections/ansible_collections/community/vmware
$ cd ~/.ansible/collections/ansible_collections/community/vmware
$ git fetch origin pull/1969/head:20240114_validate_certs
$ git checkout 20240114_validate_certs

The important thing is to have it somewhere where ansible can find it. I often use ~/.ansible/collections since I don’t have anything installed in my home. And then I remove it afterwards so I have clean environment again.

[vmadmin]

@mariolenz

$ mkdir -p ~/.ansible/collections/ansible_collections/community
$ git clone GitHub - ansible-collections/community.vmware: Ansible Collection for VMware ~/.ansible/collections/ansible_collections/community/vmware
$ cd ~/.ansible/collections/ansible_collections/community/vmware
$ git fetch origin pull/1969/head:20240114_validate_certs
$ git checkout 20240114_validate_certs

Thank you Mario and @ptn for your responses. I have done the above. However, I’m not entirely sure I’m calling the new collection properly (apologies, but I am just that new to Ansible)

I have put the playbook into:

~/.ansible/collections/ansible_collections/community/vmware/playbooks

And the playbook still has:

collections: - community.vmware

When I run it, I’m still getting:

TASK [Content Library info] ***************************************************************************************************************************************************************************************************************** fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to connect to vCenter or ESXi API at redacted:443 due to SSL verification failure : HTTPSConnectionPool(host='redacted', port=443): Max retries exceeded with url: /api (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))"}

Whereas the first task in the playbook validates the SSL like before.

[mariolenz]

This looks like you’ve tested with validate_certs: true, right?

What happens if you use validate_certs: false?

[vmadmin]

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: com.vmware.vapi.std.errors_client.Unauthorized: {messages : [LocalizableMessage(id='com.vmware.vdcs.vsphere-auth-lib.permission.denied', default_message='Permission to perform this operation was denied.', args=[], params=None, localized=None)], data : None, error_type : UNAUTHORIZED}

This happens with the original playbook too on the original module. I can copy the whole dump or even with -vvv if needed but it’s showing a permissions error (even though the first trivial task authenticates and works with valid certs)

[vmadmin]

worth noting from ‘ansible-config’ dump

COLLECTIONS_PATHS(default) = [‘/home/nick/.ansible/collections’, ‘/usr/share/ansible/collections’]

I am assuming that putting the PR into the directory structure I pasted in previous pos with the playbook directory in there was done correctly and that I’m running off the new PR and not the existing collection.

[mariolenz]

@vmadmin

Permission to perform this operation was denied.

While the SSL issues look like problems in the collection, this one doesn’t. This message should come from the vCenter and tells you that you (that is the user in the playbook) doesn’t have the required permissions,

BTW:

@vmadmin

I have put the playbook into:

~/.ansible/collections/ansible_collections/community/vmware/playbooks

You don’t have to put your playbook there. The important thing is to have community.vmware (that is: the changes from my PR) there so ansible can find it.

[vmadmin]

I switched the vcenter user to the administrator@vsphere.local account (had a separate ansible service account w/ admin privs but not global permission) and it works as long as I have validate_certs to false.

It throws the clear SSL verify error when validate_certs is true

` certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)’

but we have a work around and that works for me. I get the same results with existing collection or the one from the PR now.

[mariolenz]

Since my PR seems to fix your issue with validate_certs: false I’ve merged it and will try to release it next week.

I’ll try to find out why the module fails with validate_certs: true. One problem at a time

I don’t think I’m able to fix it in the next collection release, though…