How can i get ssh from AWX to Github private repo working trough proxy

Get Help
,
23 
 kubectl exec -it -n awx awx-web-77c4d969dc-ccg7p -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
Warning: Identity file /var/lib/awx/.ssh/id_rsa_proxy not accessible: No such file or directory.
The authenticity of host 'github.com (140.82.121.3)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
git@github.com: Permission denied (publickey).
command terminated with exit code 255
[root@ansible-awx awx-on-k3s]#
24

We only added the volume to the task pod, and that should be the only one you need. If you try using your git credential with github in the AWX UI, does it work?

25

It works from the AWX UI but traffic is not going trough the proxy. I’m testing in my home lab and AWX VM itself also has access to internet and GitHub. The proxy is on a separate VM and i’m checking with tcpdump there. I see the traffic is sent via the proxy if i test with command but it does not work from the UI. On my production site AWX has no internet access.
This works

kubectl exec -it -n awx awx-task-7cbbcfd9ff-nb4bf  -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
Warning: Permanently added 'squid-lxc-1.home' (ED25519) to the list of known hosts.
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Hi pemca! You've successfully authenticated, but GitHub does not provide shell access.
26

Here i try with the web container. Actually it grabs the ProxyJump config in AWX because it jumps to the gituser@squid-lxc-1.home VM, but it cannot read the key.

kubectl exec -it -n awx awx-web-5957c8c495-cd2xs  -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
Warning: Permanently added 'squid-lxc-1.home' (ED25519) to the list of known hosts.
Load key "/var/lib/awx/.ssh/id_rsa_proxy": Permission denied
gituser@squid-lxc-1.home's password:

Maybe there are some logging so i can see what happens in the UI.

27

Hi,

I haven’t been reading the previous posts very carefully, but could you please specify ee_extra_volume_mounts rather than task_extra_volume_mounts ?

Also, you should try ssh command inside the awx-ee container in awx-task pod, instead of awx-task container in the same pod. Add -c awx-ee to your kubectl exec command:

kubectl exec -it -n awx awx-task-... -c awx-ee -- ssh -i ...
                                     ^^^^^^^^^

Technical background:

  • awx-task pod has multiple containers such as awx-task, awx-ee, etc.
  • Actual git command is invoked in the awx-ee container instead of awx-task container.
  • The commands which are passed through kubectl exec for awx-task pod will be invoked in the awx-task container by default (by design). If we want to run commands in awx-ee container, we have to specify it by -c explicitly.
1 Like
28

Hi kurokobo, thank you for joining to this. My AWX instance is deployed from your github kurokobo/awx-on-k3s.
1 : i changed config to use ee_extra_volume_mounts and restarted pods

  ee_extra_volume_mounts: |
    - name: awx-proxy-config
      mountPath: "/var/lib/awx/.ssh"
      readOnly: false
  ee_resource_requirements: {}
  extra_volumes: |
    - name: awx-proxy-config
      projected:
        sources:
          - configMap:
              name: awx-proxy-ssh-config
              items:
                - key: config
                  path: config
          - secret:
              name: awx-proxy-ssh-key
              items:
                - key: id_rsa_proxy
                  path: id_rsa_proxy
                  mode: 600

This still works and traffic is going trough proxy

kubectl exec -it -n awx awx-task-685d64b878-wkrpz -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
Warning: Permanently added 'squid-lxc-1.home' (ED25519) to the list of known hosts.
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Hi pemca! You've successfully authenticated, but GitHub does not provide shell access.
command terminated with exit code 1

But using -c awx-ee does not work

 kubectl exec -it -n awx awx-task-685d64b878-wkrpz -c awx-ee -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
Warning: Identity file /var/lib/awx/.ssh/id_rsa_proxy not accessible: No such file or directory.
git@github.com: Permission denied (publickey).
command terminated with exit code 255

And Testing with sync to github from AWX UI still works but sends the traffic directly and not via the proxy

Thank You.

29

Simply restarting the pod will not reflect those changes.

You need to wait until the changes are applied to the AWX resources and the Operator modifies the actual Pod definition accordingly.

1 Like
30

Same behaveur after 1.5 hours. Can i make a check printout ?

31

You can set it in “Extra Environment Variables”:

https://ansible.readthedocs.io/projects/awx/en/24.6.1/administration/configure_awx.html

{
“http_proxy”: "http…,
“https_proxy”: “http…”,
“no_proxy”: “localhost,127.0.0.1,.example.com”
}

32

To get it working in AWX credentials i need to use ssh via proxy.

33

If you have deployed the AWX following my guide, the steps to modify the AWX resources are as follows:

  1. Modify base/awx.yaml.
  2. Execute kubectl apply -k base.
  3. Wait for the Operator to apply the changes.

To confirm that the changes are complete, the following steps are helpful:

  • Use kubectl -n awx get awx awx -o yaml or kubectl -n awx describe awx awx to verify that the specified values are retained. In your case, find ee_extra_volume_mounts.
  • Check the logs of the Operator to ensure that the logs stop with failed=0 in the PLAY RECAP state
    $ kubectl -n awx logs -f deployments/awx-operator-controller-manager
...
----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx/awx) -----
PLAY RECAP *********************************************************************
localhost                  : ok=90   changed=0    unreachable=0    failed=0    skipped=83   rescued=0    ignored=1
  • Confirm that all Pods are Running with RESTART 0 and are stable: kubectl -n awx get pod
  • Look at the definition of the awx-task pod to verify that the desired information appears in the Mounts of the awx-ee container: kubectl -n awx describe deployment/awx-task
2 Likes
34

Thank you, kurokobo, i will try that this evening and come cack.

35

Oh. SSH via proxy (sorry, missed that) If all else fails you can roll it into a custom EE and use that. Also lets you run the container locally to test it’s working.

36

Hi
Thank you all for your help. I did not manage to solve the problem. My conclution is that AWX is a perfect tool but too complicated to manage and maintain without good knowlegde to kubernetes. I have found a simpler solution (semaphore) that can cover our needs for now.
thank you.