Ability to allow inbound connection to AWX receptor mesh on Kubernetes

TheRealHaoLiu · November 16, 2023, 6:33pm

@kurokobo @tanganellilore we got a good bits of the work done in
feature_awx-mesh-ingress branch on awx and awx-operator

right now if you try to use it it should work

for the awx-operator change we mostly focused on OCP with Route as the ingress

If anyone is interested would u like to contribute the bits needed to get this to work well on non OCP K8S?

kurokobo · November 17, 2023, 4:38am

Thank you for informing me, I’ll give it a little try in a few days.

kurokobo · November 21, 2023, 1:17pm

github.com/ansible/awx-operator

[AWXMeshIngress] The port for internal receptor address should be 27199 instead of 443

opened 01:09PM - 21 Nov 23 UTC

kurokobo

needs_triage community

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx-operator/issues) for duplicates. - [X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response. ### Bug Summary The Service resource for AWXMeshIngress listens 27199 instead of 443, so controle node cannnot establish backend connection to `wss://<service name>:443`. https://github.com/ansible/awx-operator/blob/b0233cd7d4356d2005ca2abd0b8c727b7a91acda/roles/mesh_ingress/tasks/creation.yml#L125-L130 ### AWX Operator version feature_awx-mesh-ingress ### AWX version feature_awx-mesh-ingress ### Kubernetes platform kubernetes ### Kubernetes/Platform version k3s version v1.27.6+k3s1 ### Modifications no ### Steps to reproduce Deploy following AWXMeshIngress CR: ```yaml --- apiVersion: awx.ansible.com/v1alpha1 kind: AWXMeshIngress metadata: namespace: awx name: awxmeshingress spec: deployment_name: awx external_hostname: meshingress.ansible.internal external_ipaddress: x.x.x.x ingress_type: none ``` Review `conf` file and logs for `awx-ee`. ### Expected results `awx-ee` can establish backend connection to AWXMeshIngress. ### Actual results ```bash # Listens port 27199 $ kubectl -n awx get svc awxmeshingress NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE awxmeshingress ClusterIP 10.43.56.143 <none> 27199/TCP 3m59s # Points 443 $ kubectl -n awx exec -it deployment/awx-task -c awx-ee -- cat /etc/receptor/receptor.conf ... - ws-peer: address: wss://awxmeshingress:443 tls: tlsclient # Backend connection cannnot be established $ kubectl -n awx logs -f deployment/awx-task -c awx-ee INFO 2023/11/21 13:06:05 Running control service control INFO 2023/11/21 13:06:05 Initialization complete WARNING 2023/11/21 13:08:20 Backend connection failed (will retry): dial tcp 10.43.56.143:443: connect: connection timed out ``` ### Additional information _No response_ ### Operator Logs @TheRealHaoLiu @fosterseth F.Y.I.

github.com/ansible/awx

[AWXMeshIngress] "Something went wrong" in Peers tab for Instances

opened 12:56PM - 21 Nov 23 UTC

kurokobo

type:bug component:api component:ui needs_triage community

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx/issues) for duplicates. - [X] I understand that AWX is open source software provided for free and that I might not receive a timely response. - [X] I am **NOT** reporting a (potential) security vulnerability. (These should be emailed to `security@ansible.com` instead.) ### Bug Summary Peers page for any Instances seems to be broken. ![image](https://github.com/ansible/awx/assets/2920259/53848c3c-3303-4533-b9cd-ce5cb10b1cfb) ```text GET api/v2/instances/1/peers/ 400 ReceptorAddress has no field named 'hostname' ``` ### AWX version feature_awx-mesh-ingress ### Select the relevant components - [X] UI - [ ] UI (tech preview) - [X] API - [ ] Docs - [ ] Collection - [ ] CLI - [ ] Other ### Installation method kubernetes ### Modifications no ### Ansible version _No response_ ### Operating system _No response_ ### Web browser _No response_ ### Steps to reproduce 1. Deploy fresh AWX with minimal configuration 2. Open Peers tab for any control plane nodes ### Expected results Peers can be displayed ### Actual results Got "Something went wrong" ### Additional information @TheRealHaoLiu @fosterseth F.Y.I.

github.com/ansible/awx

[AWXMeshIngress] Adding new instance that has AWXMeshIngress node as peer causes 400 error

opened 01:36PM - 21 Nov 23 UTC

kurokobo

### Please confirm the following - [X] I agree to follow this project's [code o…f conduct](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html). - [X] I have checked the [current issues](https://github.com/ansible/awx/issues) for duplicates. - [X] I understand that AWX is open source software provided for free and that I might not receive a timely response. - [X] I am **NOT** reporting a (potential) security vulnerability. (These should be emailed to `security@ansible.com` instead.) ### Bug Summary ![image](https://github.com/ansible/awx/assets/2920259/abf05452-c4ab-4d21-a92c-3e312c4d06ae) Cannot add new instance that has internal hop node as peer. Seems the API does not support it yet. ### AWX version feature_awx-mesh-ingress ### Select the relevant components - [ ] UI - [ ] UI (tech preview) - [X] API - [ ] Docs - [ ] Collection - [ ] CLI - [ ] Other ### Installation method kubernetes ### Modifications no ### Ansible version _No response_ ### Operating system _No response_ ### Web browser _No response_ ### Steps to reproduce 1. Deploy AWX in branch `feature_awx-mesh-ingress` using AWX Operator in branch `feature_awx-mesh-ingress` 2. Deploy AWXMeshIngress CR 3. Add new instance that has internal hop node as peer ### Expected results New instance can be added ### Actual results Got 400 error with the body `{"peers":["Incorrect type. Expected pk value, received str."]}` #### Request payload ```json {"hostname":"exec01.ansible.internal","description":"","node_type":"execution","node_state":"installed","listener_port":27199,"enabled":true,"managed_by_policy":true,"peers_from_control_nodes":false,"peers":["awxmeshingress"]} ``` ### Response ```json {"peers":["Incorrect type. Expected pk value, received str."]} ``` ### Additional information _No response_

kurokobo · November 21, 2023, 2:46pm

@TheRealHaoLiu @fosterseth
Is it feasible for AWX Operator to support deployment of IngressRouteTCP for Traefik in addition to Route and Ingress?
I can send PR for this, but It would also be possible to deploy AWXMeshIngress CR with ingress_type: none and have users create their own IngressRouteTCP (In this case AWX Operator does not support IngressRouteTCP).

Traefik (default ingress controller for k3s) requires a TCP Router for TLS passthrough, which consists of a CR called IngressRouteTCP rather than the usual Ingress resource.

Technically, I already have an implementation in my environment that allows jobs to run on Executon Node via Internal Hop Node using IngressRouteTCP.

Draft PR: wip: add ingress and ingressroutetcp for awxmeshingress by kurokobo · Pull Request #1646 · ansible/awx-operator · GitHub

TheRealHaoLiu · February 13, 2024, 10:15pm

Thanks everyone! the feature landed in AWX now!

check it out!

AWX: 23.8.0
AWX-Operator: 2.12.0

yoonmin1030 · April 15, 2024, 10:18am

The remote EE node from outside of k8s cluster, how to reach the hop node in the k8s cluster with port 443 ? does the loadbalancer in k8s cluster tell ?
I have installed bundle to remote EE node, but running health check tells the remote EE node is not in the receptor mesh.

what i understand is

in the k8s cluster, awxmeshingress takes pods to service.
outside of k8s cluster, ingress (ingress-nginx) takes the remote EE node to service.

please take a look what I am missing.

this is my service for awxmeshingress

apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: ‘{“apiVersion”:“v1”,“kind”:“Service”,“metadata”:{“name”:“mesh-ingress-1”,“namespace”:“awx”},“spec”:{“ports”:[{“name”:“ws”,“port”:27199,“targetPort”:27199}],“selector”:{“app.kubernetes.io/name":“mesh-ingress-1”},“type”:"ClusterIP”}}’
creationTimestamp: “2024-04-12T07:43:01Z”
name: mesh-ingress-1
namespace: awx
ownerReferences:

apiVersion: awx.ansible.com/v1alpha1
kind: AWXMeshIngress
name: mesh-ingress-1
uid: xxx
resourceVersion: “21687088”
uid: xxx
spec:
clusterIP: 10.x.x.x
clusterIPs:
10.x.x.x
internalTrafficPolicy: Cluster
ipFamilies:
IPv4
ipFamilyPolicy: SingleStack
ports:
name: ws
port: 27199
protocol: TCP
targetPort: 27199
selector:
app.kubernetes.io/name: mesh-ingress-1
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}

===============================================
following is service for ingress-nginx-controller

apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{“apiVersion”:“v1”,“kind”:“Service”,“metadata”:{“annotations”:{},“labels”:{“app.kubernetes.io/component":“controller”,“app.kubernetes.io/instance”:“ingress-nginx”,“app.kubernetes.io/name”:“ingress-nginx”,“app.kubernetes.io/part-of”:“ingress-nginx”,“app.kubernetes.io/version”:“1.10.0”},“name”:“ingress-nginx-controller”,“namespace”:“ingress-nginx”},“spec”:{“externalTrafficPolicy”:“Local”,“ipFamilies”:[“IPv4”],“ipFamilyPolicy”:“SingleStack”,“ports”:[{“appProtocol”:“http”,“name”:“http”,“port”:80,“protocol”:“TCP”,“targetPort”:“http”},{“appProtocol”:“https”,“name”:“https”,“port”:443,“protocol”:“TCP”,“targetPort”:“https”}],“selector”:{“app.kubernetes.io/component”:“controller”,“app.kubernetes.io/instance”:“ingress-nginx”,“app.kubernetes.io/name”:“ingress-nginx”},“type”:"LoadBalancer”}}
creationTimestamp: “2024-04-03T09:20:23Z”
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.10.0
name: ingress-nginx-controller
namespace: ingress-nginx
resourceVersion: “18611647”
uid: xxx
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 10.x.x.x
clusterIPs:

10.x.x.x
externalTrafficPolicy: Local
healthCheckNodePort: 31209
internalTrafficPolicy: Cluster
ipFamilies:
IPv4
ipFamilyPolicy: SingleStack
ports:
appProtocol: http
name: http
nodePort: 31789
port: 80
protocol: TCP
targetPort: http
appProtocol: https
name: https
nodePort: 32148
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer: {}

kurokobo · April 15, 2024, 11:36am

@yoonmin1030
Hi, could you please create a new topic on Get Help category?

yoonmin1030 · April 15, 2024, 1:51pm

I made it as a topic.

please help me to get more understanding of the topic “awxmeshingress and receptor”

Ability to allow inbound connection to AWX receptor mesh on Kubernetes

this is my service for awxmeshingress

=============================================== following is service for ingress-nginx-controller

===============================================
following is service for ingress-nginx-controller