AWX failing to clone repo from Azure Devops Server 2019

Get Help
, ,
1

I am currently evaluating AWX to help manage infrastructure provisioning at my company.

for context, I have a rocky Linux VM running k3s and AWX installed using the awx-operator which is deployed via helm

My issue is when AWX attempts to sync our project it fails to do so stating authentication issues and it comes down to two issues:

  1. For https clones the builtin git module does not support the way Azure Devops Server expects you to pass through credentials
  2. For ssh I require some additional ssh config

the logic option for me is to get SSH working as I cannot see any possible path for https. I attempted to change the ssh config using the following addition based on the ansible module documentation
https://docs.ansible.com/ansible/latest/collections/ansible/builtin/git_module.html#parameter-ssh_opts

    task_extra_env: |
      - name: GIT_SSH_COMMAND
        value: "ssh -o StrictHostKeyChecking=no -o PubkeyAcceptedAlgorithms=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa PubkeyAcceptedKeyTypes=+ssh-rsa"
    web_extra_env: |
      - name: GIT_SSH_COMMAND
        value: "ssh -o StrictHostKeyChecking=no -o PubkeyAcceptedAlgorithms=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa PubkeyAcceptedKeyTypes=+ssh-rsa"
    rsyslog_extra_env: |
      - name: GIT_SSH_COMMAND
        value: "ssh -o StrictHostKeyChecking=no -o PubkeyAcceptedAlgorithms=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa PubkeyAcceptedKeyTypes=+ssh-rsa"
    ee_extra_env: |
      - name: GIT_SSH_COMMAND
        value: "ssh -o StrictHostKeyChecking=no -o PubkeyAcceptedAlgorithms=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa PubkeyAcceptedKeyTypes=+ssh-rsa"

I can confirm that these envs are inside the containers but for some reason when I run sync it still doesn’t pick them up, I was able to replicate the sync playbook on my local machine but cannot for the life of me get it working in AWX

Please let me know if I have missed any information and any help would be appreciated.

FYI I did come across Using Azure DevOps repo as SCM for AWX project fails to authenticate - #2 by wmarbach which helped with diagnosing the issue but the solution has not worked for me.

2

i performed some more testing

i created a playbook locally with the which has the following

- name: Test
  hosts: localhost
  tasks:
    - name: Git
      ansible.builtin.git:
        repo: 'ssh://<GIT_URL>'
        dest: ./repo
        accept_hostkey: true

and i execute

export GIT_SSH_COMMAND="ssh -o PubkeyAcceptedAlgorithms=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa" -o IdentityFile=~/.ssh/other_keys/id_rsa"

which works fine and the repo clones

the error i get when specific SSH_GIT_COMMAND via extra_env is

fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/git ls-remote 'ssh:********@<GIT URL>' -h refs/heads/HEAD", "msg": "ssh_dispatch_run_fatal: Connection to <IP_ADDRESS> port 22: error in libcrypto\\r\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.", "rc": 128, "stderr": "ssh_dispatch_run_fatal: Connection to <IP_ADDRESS> port 22: error in libcrypto\\r\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.\\n", "stderr_lines": ["ssh_dispatch_run_fatal: Connection to 192.168.202.81 port 22: error in libcrypto", "fatal: Could not read from remote repository.", "", "Please make sure you have the correct access rights", "and the repository exists."], "stdout": "", "stdout_lines": []}

but when i remove the extra_env i get

Unable to negotiate with <IP_ADDRESS> port 22: no matching host key type found. Their offer: ssh-rsa\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n

so it is clearly picking up the ssh config but something else is going on inside AWX that is preventing it from cloning,

My current thought/process is maybe setting GIT_SSH_COMMAND is overwriting AWX setting the credential file but i am unsure how this works at the moment.

3

Some further research, it seems to be an issue with the awx-ee control plane execution environment, running the same test above but inside the container does not work and gives the same error.

4

so after some further testing with the image, obviously, a newer version of OpenSSL has dropped support for older encryptions but the options I am passing through to GIT_SSH_COMMAND and also the ssh config seem to be causing this error.

the only reliable method I have found to get it working is to run update-crypto-policies --set LEGACY I would assume this is a bug but I cannot find any other reports on this issue