Ability to allow inbound connection to AWX receptor mesh on Kubernetes

TheRealHaoLiu · November 16, 2023, 6:33pm

@kurokobo @tanganellilore we got a good bits of the work done in
feature_awx-mesh-ingress branch on awx and awx-operator

right now if you try to use it it should work

for the awx-operator change we mostly focused on OCP with Route as the ingress

If anyone is interested would u like to contribute the bits needed to get this to work well on non OCP K8S?

kurokobo · November 17, 2023, 4:38am

Thank you for informing me, I’ll give it a little try in a few days.

kurokobo · November 21, 2023, 1:17pm

github.com/ansible/awx-operator

[AWXMeshIngress] The port for internal receptor address should be 27199 instead of 443

opened 01:09PM - 21 Nov 23 UTC

kurokobo

needs_triage community

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx-operator/issues) for duplicates. - [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response. ### Bug Summary The Service resource for AWXMeshIngress listens 27199 instead of 443, so controle node cannnot establish backend connection to `wss://<service name>:443`. https://github.com/ansible/awx-operator/blob/b0233cd7d4356d2005ca2abd0b8c727b7a91acda/roles/mesh_ingress/tasks/creation.yml#L125-L130 ### AWX Operator version feature_awx-mesh-ingress ### AWX version feature_awx-mesh-ingress ### Kubernetes platform kubernetes ### Kubernetes/Platform version k3s version v1.27.6+k3s1 ### Modifications no ### Steps to reproduce Deploy following AWXMeshIngress CR: ```yaml --- apiVersion: awx.ansible.com/v1alpha1 kind: AWXMeshIngress metadata: namespace: awx name: awxmeshingress spec: deployment_name: awx external_hostname: meshingress.ansible.internal external_ipaddress: x.x.x.x ingress_type: none ``` Review `conf` file and logs for `awx-ee`. ### Expected results `awx-ee` can establish backend connection to AWXMeshIngress. ### Actual results ```bash # Listens port 27199 $ kubectl -n awx get svc awxmeshingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE awxmeshingress ClusterIP 10.43.56.143 <none> 27199/TCP 3m59s # Points 443 $ kubectl -n awx exec -it deployment/awx-task -c awx-ee -- cat /etc/receptor/receptor.conf ... - ws-peer: address: wss://awxmeshingress:443 tls: tlsclient # Backend connection cannnot be established $ kubectl -n awx logs -f deployment/awx-task -c awx-ee INFO 2023/11/21 13:06:05 Running control service control INFO 2023/11/21 13:06:05 Initialization complete WARNING 2023/11/21 13:08:20 Backend connection failed (will retry): dial tcp 10.43.56.143:443: connect: connection timed out ``` ### Additional information _No response_ ### Operator Logs @TheRealHaoLiu @fosterseth F.Y.I.

github.com/ansible/awx

[AWXMeshIngress] "Something went wrong" in Peers tab for Instances

opened 12:56PM - 21 Nov 23 UTC

kurokobo

type:bug component:api component:ui needs_triage community

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx/issues) for duplicates. - [X] I understand that AWX is open source software provided for free and that I might not receive a timely response. - [X] I am **NOT** reporting a (potential) security vulnerability. (These should be emailed to `security@ansible.com` instead.) ### Bug Summary Peers page for any Instances seems to be broken. ![image](https://github.com/ansible/awx/assets/2920259/53848c3c-3303-4533-b9cd-ce5cb10b1cfb) ```text GET api/v2/instances/1/peers/ 400 ReceptorAddress has no field named 'hostname' ``` ### AWX version feature_awx-mesh-ingress ### Select the relevant components - [X] UI - [ ] UI (tech preview) - [X] API - [ ] Docs - [ ] Collection - [ ] CLI - [ ] Other ### Installation method kubernetes ### Modifications no ### Ansible version _No response_ ### Operating system _No response_ ### Web browser _No response_ ### Steps to reproduce 1. Deploy fresh AWX with minimal configuration 2. Open Peers tab for any control plane nodes ### Expected results Peers can be displayed ### Actual results Got "Something went wrong" ### Additional information @TheRealHaoLiu @fosterseth F.Y.I.

github.com/ansible/awx

[AWXMeshIngress] Adding new instance that has AWXMeshIngress node as peer causes 400 error

opened 01:36PM - 21 Nov 23 UTC

kurokobo

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx/issues) for duplicates. - [X] I understand that AWX is open source software provided for free and that I might not receive a timely response. - [X] I am **NOT** reporting a (potential) security vulnerability. (These should be emailed to `security@ansible.com` instead.) ### Bug Summary ![image](https://github.com/ansible/awx/assets/2920259/abf05452-c4ab-4d21-a92c-3e312c4d06ae) Cannot add new instance that has internal hop node as peer. Seems the API does not support it yet. ### AWX version feature_awx-mesh-ingress ### Select the relevant components - [ ] UI - [ ] UI (tech preview) - [X] API - [ ] Docs - [ ] Collection - [ ] CLI - [ ] Other ### Installation method kubernetes ### Modifications no ### Ansible version _No response_ ### Operating system _No response_ ### Web browser _No response_ ### Steps to reproduce 1. Deploy AWX in branch `feature_awx-mesh-ingress` using AWX Operator in branch `feature_awx-mesh-ingress` 2. Deploy AWXMeshIngress CR 3. Add new instance that has internal hop node as peer ### Expected results New instance can be added ### Actual results Got 400 error with the body `{"peers":["Incorrect type. Expected pk value, received str."]}` #### Request payload ```json {"hostname":"exec01.ansible.internal","description":"","node_type":"execution","node_state":"installed","listener_port":27199,"enabled":true,"managed_by_policy":true,"peers_from_control_nodes":false,"peers":["awxmeshingress"]} ``` ### Response ```json {"peers":["Incorrect type. Expected pk value, received str."]} ``` ### Additional information _No response_

kurokobo · November 21, 2023, 2:46pm

@TheRealHaoLiu @fosterseth
Is it feasible for AWX Operator to support deployment of IngressRouteTCP for Traefik in addition to Route and Ingress?
I can send PR for this, but It would also be possible to deploy AWXMeshIngress CR with ingress_type: none and have users create their own IngressRouteTCP (In this case AWX Operator does not support IngressRouteTCP).

Traefik (default ingress controller for k3s) requires a TCP Router for TLS passthrough, which consists of a CR called IngressRouteTCP rather than the usual Ingress resource.

Technically, I already have an implementation in my environment that allows jobs to run on Executon Node via Internal Hop Node using IngressRouteTCP.

Draft PR: wip: add ingress and ingressroutetcp for awxmeshingress by kurokobo · Pull Request #1646 · ansible/awx-operator · GitHub