I have installed newest AWX on K8s. I have created Git ssh credentials and a test project where i connect to my private Github ansible repo. All that works, i can sync the repo. I want to make the same AWX in Production, but here AWX and K8s will be behind a outbound proxy. How do i set up the ssh to go trough the proxy ? - I have to use ssh credential because it is a private repo. Hope someone can help. Thanks.
I think you could create an ssh config and credential secret in k8s, and mount that as an extra volume for AWX. You’ll want to create a unique name for your ssh private key and specify its use for the proxy in your ssh config, that way it doesn’t get used accidentally for anything other than the proxy and only when the proxy is needed.
Many thanks Denney-tech for suggestions, but I don’t have the knowledge to set this up. And I also don’t know how to get AWX to use this sshconfig. When AWX makes the ssh connection to GitHub, it is not via the VM’s normal ssh, but from within AWX itself. A tcpdump in the VM on port 22 shows nothing.
Thank you.
The exact steps depend a little bit on how you deployed AWX. Did you use Helm or the AWX-Operator? If you used Helm, I don’t know exactly what to tell you there. However, there is some documentation for the operator on adding extra volume mounts in k8s.
Custom Volume and Volume Mount Options - Ansible AWX Operator Documentation
In your case, you’ll want to have an ssh config as a configmap, and the private key as a secret. I’ll provide some examples below, but make sure that your ssh config and proxy key work like this on a local machine first.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: awx-proxy-ssh-config
namespace: awx
data:
config: |
Host my_proxy
HostName proxy.example.com
User your_username
IdentityFile ~/.ssh/id_rsa_proxy
Host github.com
ProxyJump my_proxy
Host *.github.com
ProxyJump my_proxy
---
apiVersion: v1
kind: Secret
metadata:
name: awx-proxy-ssh-key
namespace: awx
stringData:
id_rsa_proxy: <literal-ssh-private-key-content>
type: Opaque
Now you need to add the extra_volumes and task_extra_volume_mounts to the AWX spec.
---
spec:
...
extra_volumes: |
- name: awx-proxy-config
projected:
sources:
- configMap:
name: awx-proxy-ssh-config
items:
- key: config
path: config
- secret:
name: awx-proxy-ssh-key
items:
- key: id_rsa_proxy
path: id_rsa_proxy
mode: 400
task_extra_volume_mounts: |
- name: awx-proxy-config
mountPath: "/var/lib/awx/.ssh"
This will mount the ssh config and key under the .ssh folder of awx’s home directory, but only on the task container where it will be used. And the ssh config only uses the key whenever the container needs to connect to github.com or its subdomains.
Thanks a lot 
I’ll try it out and come back.
Should this contain the key ?
Yes, in this example, you would paste the plaintext content of the private key file.
I’m on my mobile right now, so I can’t give specific instruction at the moment, but alternatively, you can use kubectl to create the secret with --from-file. Or you can properly format the yaml with the base64 encoded file.
In any case, do not commit the yaml secret to your code repository with the private key embedded plainly like this. If you want to automate the changes with Ansible, you could use kubernetes.core.k8s (iirc) to create the secret from a file or template and/or a lookup plugin/vaulted variable to provide the private key data.
Peter, “proxy” as in “http proxy”?
yes it is an http (squid) proxy but i will ProxyJump with ssh trough it
so you aren’t actually using the Squid proxy service but just using the same server as SSH jump host, got it
Hi again
i already deployed before i saw your last message.
I deployed the Configmap and Secret with succes but i get error on the last one with the extra volumes
kubectl apply -f awx-extra-volumes-config.yaml
error: error validating “awx-extra-volumes-config.yaml”: error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false
---
spec:
extra_volumes: |
- name: awx-proxy-config
projected:
sources:
- configMap:
name: awx-proxy-ssh-config
items:
- key: config
path: config
- secret:
name: awx-proxy-ssh-key
items:
- key: id_rsa_proxy
path: id_rsa_proxy
mode: 400
task_extra_volume_mounts: |
- name: awx-proxy-config
mountPath: "/var/lib/awx/.ssh"
The extra volumes needs to be spliced into the main AWX CRD so the operator knows what to do.
I did not understand that ?
The snippet of code for the extra volumes needs to be added to the kind: awx custom resource definition that the awx-operator uses to deploy AWX in the first place. You need to merge the spec: sections yourself.
I think i got it, i insertedunder specs section here
kubectl edit awx awx -n awx
1 Like
[root@ansible-awx awx-on-k3s]# kubectl get pods -n awx
NAME READY STATUS RESTARTS AGE
awx-migration-24.6.1-7p5pz 0/1 Completed 0 5d
awx-operator-controller-manager-687b856498-tggkz 2/2 Running 6 (2d9h ago) 5d1h
awx-postgres-15-0 1/1 Running 3 (2d9h ago) 5d
awx-task-7566c66bf4-7k8xl 4/4 Running 0 13m
awx-web-969877586-2ckv4 3/3 Running 0 13m
[root@ansible-awx awx-on-k3s]# kubectl exec -it -n awx awx-task-7566c66bf4-7k8xl -- ls -l /var/lib/awx/.ssh
total 0
lrwxrwxrwx. 1 root 1000 13 Jan 3 20:47 config -> ..data/config
lrwxrwxrwx. 1 root 1000 19 Jan 3 20:47 id_rsa_proxy -> ..data/id_rsa_proxy
kubectl exec -it -n awx awx-task-7566c66bf4-7k8xl -- ssh -T git@github.com
The authenticity of host 'squid-lxc-1.home (192.168.0.66)' can't be established.
ED25519 key fingerprint is SHA256:Zsyp0i4GfMKBTx99O8F90+KO3DO/Qd/LZJtu1pFQbDI.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Failed to add the host to the list of known hosts (/var/lib/awx/.ssh/known_hosts).
The authenticity of host 'github.com (<no hostip for proxy command>)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])?
Host key verification failed.
command terminated with exit code 255[root@ansible-awx awx-on-k3s]# kubectl exec -it -n awx awx-task-7566c66bf4-7k8xl -- cat /var/lib/awx/.ssh/id_rsa_proxy
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
NhAAAAAwEAAQAAAYEAqQr878CTPY5IIl4+5pyYzD/UiOLyoSnJkiNSKPKKuC8VABQLQ2OX
v9iUMHBHau+QyuD/ppOJ4uId1zVZxnkrefJFE8SChyHa2eclra3OFtgmG4/3XLeDqZTGod
kubectl exec -it -n awx awx-task-7566c66bf4-7k8xl -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
The authenticity of host 'squid-lxc-1.home (192.168.0.66)' can't be established.
ED25519 key fingerprint is SHA256:Zsyp0i4GfMKBTx99O8F90+KO3DO/Qd/LZJtu1pFQbDI.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Failed to add the host to the list of known hosts (/var/lib/awx/.ssh/known_hosts).
The authenticity of host 'github.com (<no hostip for proxy command>)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Failed to add the host to the list of known hosts (/var/lib/awx/.ssh/known_hosts).
Hi pemca! You've successfully authenticated, but GitHub does not provide shell access.
command terminated with exit code 1
It is nearly working except for write permission here
Failed to add the host to the list of known hosts (/var/lib/awx/.ssh/known_hosts)
Hi pemca! You’ve successfully authenticated
I changed here
---
apiVersion: v1
kind: ConfigMap
metadata:
name: awx-proxy-ssh-config
namespace: awx
data:
config: |
Host git_proxy
HostName squid-lxc-1.home
User gituser
IdentityFile ~/.ssh/id_rsa_proxy
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
Host github.com
ProxyJump git_proxy
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
Host *.github.com
ProxyJump git_proxy
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
kubectl exec -it -n awx awx-task-75b47c6679-l5bgd -- ssh -i /var/lib/awx/.ssh/id_rsa_proxy -T git@github.com
Warning: Permanently added 'squid-lxc-1.home' (ED25519) to the list of known hosts.
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Hi pemca! You've successfully authenticated, but GitHub does not provide shell access.
command terminated with exit code 1
It works from the awx ssh root cli (awx-task), but not from the awx web.
maybe it should have been done for awx-web container ??
NAME READY STATUS RESTARTS AGE
awx-migration-24.6.1-7p5pz 0/1 Completed 0 5d1h
awx-operator-controller-manager-755577f7c-pf8ln 2/2 Running 0 16m
awx-postgres-15-0 1/1 Running 3 (2d10h ago) 5d1h
awx-task-75b47c6679-l5bgd 4/4 Running 0 16m
awx-web-77c4d969dc-ccg7p 3/3 Running 0 16m