Question about adding remote EE node to AWX k8s cluster

The remote EE node from outside of k8s cluster, how to reach the hop node in the k8s cluster with port 443 ? does the loadbalancer in k8s cluster tell ?
I have installed bundle to remote EE node, but running health check tells the remote EE node is not in the receptor mesh.

what i understand is

  1. in the k8s cluster, awxmeshingress takes pods to service.
  2. outside of k8s cluster, ingress (ingress-nginx) takes the remote EE node to service.

please take a look what I am missing.

this is my service for awxmeshingress

apiVersion: v1
kind: Service
metadata:
annotations:
[kubectl.kubernetes.io/last-applied-configuration:](http://kubectl.kubernetes.io/last-applied-configuration:) ‘{“apiVersion”:“v1”,“kind”:“Service”,“metadata”:{“name”:“mesh-ingress-1”,“namespace”:“awx”},“spec”:{“ports”:[{“name”:“ws”,“port”:27199,“targetPort”:27199}],“selector”:{“[app.kubernetes.io/name":“mesh-ingress-1”},“type”:"ClusterIP](http://app.kubernetes.io/name%22:%22mesh-ingress-1%22%7D,%22type%22:%22ClusterIP)”}}’
creationTimestamp: “2024-04-12T07:43:01Z”
name: mesh-ingress-1
namespace: awx
ownerReferences:

* apiVersion: [awx.ansible.com/v1alpha1](http://awx.ansible.com/v1alpha1)
kind: AWXMeshIngress
name: mesh-ingress-1
uid: xxx
resourceVersion: “21687088”
uid: xxx
spec:
clusterIP: 10.x.x.x
clusterIPs:
* 10.x.x.x
internalTrafficPolicy: Cluster
ipFamilies:
* IPv4
ipFamilyPolicy: SingleStack
ports:
* name: ws
port: 27199
protocol: TCP
targetPort: 27199
selector:
[app.kubernetes.io/name:](http://app.kubernetes.io/name:) mesh-ingress-1
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}

following is service for ingress-nginx-controller

apiVersion: v1
kind: Service
metadata:
annotations:
[kubectl.kubernetes.io/last-applied-configuration:](http://kubectl.kubernetes.io/last-applied-configuration:) |
{“apiVersion”:“v1”,“kind”:“Service”,“metadata”:{“annotations”:{},“labels”:{“[app.kubernetes.io/component":“controller”,“app.kubernetes.io/instance”:“ingress-nginx”,“app.kubernetes.io/name”:“ingress-nginx”,“app.kubernetes.io/part-of”:“ingress-nginx”,“app.kubernetes.io/version”:“1.10.0”},“name”:“ingress-nginx-controller”,“namespace”:“ingress-nginx”},“spec”:{“externalTrafficPolicy”:“Local”,“ipFamilies”:[“IPv4”],“ipFamilyPolicy”:“SingleStack”,“ports”:[{“appProtocol”:“http”,“name”:“http”,“port”:80,“protocol”:“TCP”,“targetPort”:“http”},{“appProtocol”:“https”,“name”:“https”,“port”:443,“protocol”:“TCP”,“targetPort”:“https”}],“selector”:{“app.kubernetes.io/component”:“controller”,“app.kubernetes.io/instance”:“ingress-nginx”,“app.kubernetes.io/name”:“ingress-nginx”},“type”:"LoadBalancer](http://app.kubernetes.io/component%22:%22controller%22,%22app.kubernetes.io/instance%22:%22ingress-nginx%22,%22app.kubernetes.io/name%22:%22ingress-nginx%22,%22app.kubernetes.io/part-of%22:%22ingress-nginx%22,%22app.kubernetes.io/version%22:%221.10.0%22%7D,%22name%22:%22ingress-nginx-controller%22,%22namespace%22:%22ingress-nginx%22%7D,%22spec%22:%7B%22externalTrafficPolicy%22:%22Local%22,%22ipFamilies%22:%5B%22IPv4%22%5D,%22ipFamilyPolicy%22:%22SingleStack%22,%22ports%22:%5B%7B%22appProtocol%22:%22http%22,%22name%22:%22http%22,%22port%22:80,%22protocol%22:%22TCP%22,%22targetPort%22:%22http%22%7D,%7B%22appProtocol%22:%22https%22,%22name%22:%22https%22,%22port%22:443,%22protocol%22:%22TCP%22,%22targetPort%22:%22https%22%7D%5D,%22selector%22:%7B%22app.kubernetes.io/component%22:%22controller%22,%22app.kubernetes.io/instance%22:%22ingress-nginx%22,%22app.kubernetes.io/name%22:%22ingress-nginx%22%7D,%22type%22:%22LoadBalancer)”}}
creationTimestamp: “2024-04-03T09:20:23Z”
labels:
[app.kubernetes.io/component:](http://app.kubernetes.io/component:) controller
[app.kubernetes.io/instance:](http://app.kubernetes.io/instance:) ingress-nginx
[app.kubernetes.io/name:](http://app.kubernetes.io/name:) ingress-nginx
[app.kubernetes.io/part-of:](http://app.kubernetes.io/part-of:) ingress-nginx
[app.kubernetes.io/version:](http://app.kubernetes.io/version:) 1.10.0
name: ingress-nginx-controller
namespace: ingress-nginx
resourceVersion: “18611647”
uid: xxx
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 10.x.x.x
clusterIPs:

* 10.x.x.x
externalTrafficPolicy: Local
healthCheckNodePort: 31209
internalTrafficPolicy: Cluster
ipFamilies:
* IPv4
ipFamilyPolicy: SingleStack
ports:
* appProtocol: http
name: http
nodePort: 31789
port: 80
protocol: TCP
targetPort: http
* appProtocol: https
name: https
nodePort: 32148
port: 443
protocol: TCP
targetPort: https
selector:
[app.kubernetes.io/component:](http://app.kubernetes.io/component:) controller
[app.kubernetes.io/instance:](http://app.kubernetes.io/instance:) ingress-nginx
[app.kubernetes.io/name:](http://app.kubernetes.io/name:) ingress-nginx
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer: {}

Hi, first of all please correct markdown syntax with code blocks (```) to make your YAML easy to read.

For your questions, there are good official docs about how AWXMeshIngress works and how we can deploy it so read these first carefully:

Basically, it should work as expected if you can do the following:

  • AWXMeshIngress is deployed with correct external_hostname
  • Execution nodes and hop nodes have configured with correct hostname that resolvable and accessible from peers
  • The peers for any control nodes, execution nodes, and hop nodes are properly configured and followed your design
  • Any external nodes are deployed by updated install bundle
  • Any firewalls between nodes and clusters are properly configured

What topology did you design in the AWX Web UI? Could you please share your screen on Topoligy page?

And could you please provide output from following commands? If the output contains any information you don’t want to share in public, replace it with a dummy string.

kubectl -n awx get awx,awxmeshingress,service,ingress -o yaml
1 Like

Thank you for your comments.
However, I have followed the official docs you mentioned already.

following topology is what I want to have. (eventually the dotted line has to be solid line)
image

in the mesh-ingress-1 of ingress object, the backends of the external_hostname :27199 configured. where could I find external_hostname:443 what my remote EE node wants to reach. (all the firewalls or 443 is opened, by the way)

following is the output of the command you asked.

apiVersion: v1
items:
- apiVersion: awx.ansible.com/v1beta1
  kind: AWX
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"awx.ansible.com/v1beta1","kind":"AWX","metadata":{"annotations":{},"name":"awx","namespace":"awx"},"spec":{"auto_upgrade":false,"control_plane_ee_image":"local-repo/ansible/awx-ee:24.1.0","ee_images":[{"image":"local-repo/ansible/awx-ee:24.1.0","name":"octa-custom-awx-ee"}],"extra_volumes":"- name: static-data\n  persistentVolumeClaim:\n    claimName: static-data-pvc\n    readOnly: false\n","image":"local-repo/ansible/awx","image_version":"24.1.0","init_container_image":"local-repo/ansible/awx-ee","init_container_image_version":"24.1.0","init_projects_container_image":"local-repo/centos/centos:stream9","postgres_configuration_secret":"awx-postgres-configuration","postgres_data_path":"/var/lib/pgsql/data/userdata","postgres_image":"local-repo/sclorg/postgresql-15-c9s","postgres_image_version":"latest","postgres_storage_class":"datapostgress","postgres_storage_requirements":{"requests":{"storage":"20Gi"}},"projects_persistence":true,"projects_storage_access_mode":"ReadWriteMany","redis_image":"local-repo/redis","redis_image_version":"7","secret_key_secret":"awx-secret-key","service_type":"nodeport","web_extra_volume_mounts":"- name: static-data\n  mountPath: /var/lib/projects\n"}}
    creationTimestamp: "2024-03-27T10:36:40Z"
    generation: 1
    labels:
      app.kubernetes.io/component: awx
      app.kubernetes.io/managed-by: awx-operator
      app.kubernetes.io/operator-version: 2.14.0
      app.kubernetes.io/part-of: awx
    name: awx
    namespace: awx
    ownerReferences:
    - apiVersion: awx.ansible.com/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: AWX
      name: awx
      uid: 75ef1d4c-61c0-45d1-ac07-e65de9559f60
    resourceVersion: "22488999"
    uid: 75ef1d4c-61c0-45d1-ac07-e65de9559f60
  spec:
    admin_user: admin
    auto_upgrade: false
    control_plane_ee_image: local-repo/ansible/awx-ee:24.1.0
    create_preload_data: true
    ee_images:
    - image: local-repo/ansible/awx-ee:24.1.0
      name: octa-custom-awx-ee
    extra_volumes: |
      - name: static-data
        persistentVolumeClaim:
          claimName: static-data-pvc
          readOnly: false
    garbage_collect_secrets: false
    image: local-repo/ansible/awx
    image_pull_policy: IfNotPresent
    image_version: 24.1.0
    init_container_image: local-repo/ansible/awx-ee
    init_container_image_version: 24.1.0
    init_projects_container_image: local-repo/centos/centos:stream9
    ipv6_disabled: false
    loadbalancer_class: ""
    loadbalancer_ip: ""
    loadbalancer_port: 80
    loadbalancer_protocol: http
    metrics_utility_cronjob_gather_schedule: '@hourly'
    metrics_utility_cronjob_report_schedule: '@monthly'
    metrics_utility_enabled: false
    metrics_utility_pvc_claim_size: 5Gi
    no_log: true
    postgres_configuration_secret: awx-postgres-configuration
    postgres_data_path: /var/lib/pgsql/data/userdata
    postgres_image: local-repo/sclorg/postgresql-15-c9s
    postgres_image_version: latest
    postgres_keepalives: true
    postgres_keepalives_count: 5
    postgres_keepalives_idle: 5
    postgres_keepalives_interval: 5
    postgres_storage_class: datapostgress
    postgres_storage_requirements:
      requests:
        storage: 20Gi
    projects_persistence: true
    projects_storage_access_mode: ReadWriteMany
    projects_storage_size: 8Gi
    redis_image: local-repo/redis
    redis_image_version: "7"
    replicas: 1
    route_tls_termination_mechanism: Edge
    secret_key_secret: awx-secret-key
    service_type: nodeport
    set_self_labels: true
    task_liveness_failure_threshold: 3
    task_liveness_initial_delay: 5
    task_liveness_period: 0
    task_liveness_timeout: 1
    task_privileged: false
    task_readiness_failure_threshold: 3
    task_readiness_initial_delay: 20
    task_readiness_period: 0
    task_readiness_timeout: 1
    web_extra_volume_mounts: |
      - name: static-data
        mountPath: /var/lib/projects
    web_liveness_failure_threshold: 3
    web_liveness_initial_delay: 5
    web_liveness_period: 0
    web_liveness_timeout: 1
    web_readiness_failure_threshold: 3
    web_readiness_initial_delay: 20
    web_readiness_period: 0
    web_readiness_timeout: 1
  status:
    adminPasswordSecret: awx-admin-password
    adminUser: admin
    broadcastWebsocketSecret: awx-broadcast-websocket
    conditions:
    - lastTransitionTime: "2024-03-27T10:37:40Z"
      reason: ""
      status: "False"
      type: Failure
    - lastTransitionTime: "2024-03-27T10:37:36Z"
      reason: Successful
      status: "True"
      type: Running
    - lastTransitionTime: "2024-04-14T15:40:34Z"
      reason: Successful
      status: "True"
      type: Successful
    image: local-repo/ansible/awx:24.1.0
    postgresConfigurationSecret: awx-postgres-configuration
    secretKeySecret: awx-secret-key
    version: 24.1.0
- apiVersion: awx.ansible.com/v1alpha1
  kind: AWXMeshIngress
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"awx.ansible.com/v1alpha1","kind":"AWXMeshIngress","metadata":{"annotations":{},"name":"mesh-ingress-1","namespace":"awx"},"spec":{"deployment_name":"awx","external_hostname":"mesh-ingress-1.ansible.internal","ingress_class_name":"nginx","ingress_controller":"nginx","ingress_type":"Ingress"}}
    creationTimestamp: "2024-04-16T06:28:49Z"
    finalizers:
    - awx.ansible.com/awx-mesh-ingress-finalizer
    generation: 1
    name: mesh-ingress-1
    namespace: awx
    resourceVersion: "23044177"
    uid: 7d80e166-4a03-41da-8c6e-1705d42a8860
  spec:
    deployment_name: awx
    external_hostname: mesh-ingress-1.ansible.internal
    ingress_class_name: nginx
    ingress_controller: nginx
    ingress_type: Ingress
  status:
    conditions:
    - lastTransitionTime: "2024-04-16T06:29:14Z"
      message: ""
      reason: ""
      status: "False"
      type: Failure
    - ansibleResult:
        changed: 3
        completion: 2024-04-16T06:29:32.198552
        failures: 0
        ok: 20
        skipped: 4
      lastTransitionTime: "2024-04-16T06:28:49Z"
      message: Awaiting next reconciliation
      reason: Successful
      status: "True"
      type: Running
    - lastTransitionTime: "2024-04-16T06:29:32Z"
      message: Last reconciliation succeeded
      reason: Successful
      status: "True"
      type: Successful
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"control-plane":"controller-manager"},"name":"controller-manager-metrics-service","namespace":"awx"},"spec":{"ports":[{"name":"https","port":8443,"protocol":"TCP","targetPort":"https"}],"selector":{"control-plane":"controller-manager"}}}
    creationTimestamp: "2024-03-27T10:36:40Z"
    labels:
      control-plane: controller-manager
    name: controller-manager-metrics-service
    namespace: awx
    resourceVersion: "16272602"
    uid: 2fd0ac75-d420-4915-bd53-0837c6d8bf15
  spec:
    clusterIP: 10.x.x.x
    clusterIPs:
    - 10.x.x.x
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: https
      port: 8443
      protocol: TCP
      targetPort: https
    selector:
      control-plane: controller-manager
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"Service","metadata":{"name":"mesh-ingress-1","namespace":"awx"},"spec":{"ports":[{"name":"ws","port":27199,"targetPort":27199}],"selector":{"app.kubernetes.io/name":"mesh-ingress-1"},"type":"ClusterIP"}}'
    creationTimestamp: "2024-04-16T06:28:57Z"
    name: mesh-ingress-1
    namespace: awx
    ownerReferences:
    - apiVersion: awx.ansible.com/v1alpha1
      kind: AWXMeshIngress
      name: mesh-ingress-1
      uid: 7d80e166-4a03-41da-8c6e-1705d42a8860
    resourceVersion: "23044006"
    uid: fc4fbde0-9727-41d9-96a8-b9a48676c1c9
  spec:
    clusterIP: 10.x.x.x
    clusterIPs:
    - 10.x.x.x
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: ws
      port: 27199
      protocol: TCP
      targetPort: 27199
    selector:
      app.kubernetes.io/name: mesh-ingress-1
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"Service","metadata":{"labels":{"app.kubernetes.io/component":"database","app.kubernetes.io/instance":"postgres-15-awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/name":"postgres-15","app.kubernetes.io/operator-version":"2.14.0","app.kubernetes.io/part-of":"awx"},"name":"awx-postgres-15","namespace":"awx"},"spec":{"clusterIP":"None","ports":[{"port":5432}],"selector":{"app.kubernetes.io/component":"database","app.kubernetes.io/instance":"postgres-15-awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/name":"postgres-15","app.kubernetes.io/part-of":"awx"}}}'
    creationTimestamp: "2024-03-27T10:37:07Z"
    labels:
      app.kubernetes.io/component: database
      app.kubernetes.io/instance: postgres-15-awx
      app.kubernetes.io/managed-by: awx-operator
      app.kubernetes.io/name: postgres-15
      app.kubernetes.io/operator-version: 2.14.0
      app.kubernetes.io/part-of: awx
    name: awx-postgres-15
    namespace: awx
    ownerReferences:
    - apiVersion: awx.ansible.com/v1beta1
      kind: AWX
      name: awx
      uid: 75ef1d4c-61c0-45d1-ac07-e65de9559f60
    resourceVersion: "16272777"
    uid: 43d0cd15-5ad6-4e33-b2ac-db7de0e1f9a8
  spec:
    clusterIP: None
    clusterIPs:
    - None
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - port: 5432
      protocol: TCP
      targetPort: 5432
    selector:
      app.kubernetes.io/component: database
      app.kubernetes.io/instance: postgres-15-awx
      app.kubernetes.io/managed-by: awx-operator
      app.kubernetes.io/name: postgres-15
      app.kubernetes.io/part-of: awx
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"Service","metadata":{"labels":{"app.kubernetes.io/component":"awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/operator-version":"2.14.0","app.kubernetes.io/part-of":"awx"},"name":"awx-service","namespace":"awx"},"spec":{"ports":[{"name":"http","port":80,"protocol":"TCP","targetPort":8052}],"selector":{"app.kubernetes.io/component":"awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/name":"awx-web"},"type":"NodePort"}}'
    creationTimestamp: "2024-03-27T10:37:30Z"
    labels:
      app.kubernetes.io/component: awx
      app.kubernetes.io/managed-by: awx-operator
      app.kubernetes.io/operator-version: 2.14.0
      app.kubernetes.io/part-of: awx
    name: awx-service
    namespace: awx
    ownerReferences:
    - apiVersion: awx.ansible.com/v1beta1
      kind: AWX
      name: awx
      uid: 75ef1d4c-61c0-45d1-ac07-e65de9559f60
    resourceVersion: "16274386"
    uid: c175153b-b6b2-4fc2-ac6f-6bed8e0cdb53
  spec:
    clusterIP: 10.x.x.x
    clusterIPs:
    - 10.x.x.x
    externalTrafficPolicy: Cluster
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: http
      nodePort: 32363
      port: 80
      protocol: TCP
      targetPort: 8052
    selector:
      app.kubernetes.io/component: awx
      app.kubernetes.io/managed-by: awx-operator
      app.kubernetes.io/name: awx-web
    sessionAffinity: None
    type: NodePort
  status:
    loadBalancer: {}
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"name":"mesh-ingress-1","namespace":"awx"},"spec":{"ingressClassName":"nginx","rules":[{"host":"mesh-ingress-1.ansible.internal","http":{"paths":[{"backend":{"service":{"name":"mesh-ingress-1","port":{"number":27199}}},"path":"/","pathType":"Prefix"}]}}]}}'
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    creationTimestamp: "2024-04-16T06:28:53Z"
    generation: 1
    name: mesh-ingress-1
    namespace: awx
    ownerReferences:
    - apiVersion: awx.ansible.com/v1alpha1
      kind: AWXMeshIngress
      name: mesh-ingress-1
      uid: 7d80e166-4a03-41da-8c6e-1705d42a8860
    resourceVersion: "23043986"
    uid: 8276ccfa-babb-40e3-b14f-9beed553f83c
  spec:
    ingressClassName: nginx
    rules:
    - host: mesh-ingress-1.ansible.internal
      http:
        paths:
        - backend:
            service:
              name: mesh-ingress-1
              port:
                number: 27199
          path: /
          pathType: Prefix
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""

Thanks for updating.

If you haven’t modified default HTTPS port on Nginx Ingress Controller, your AWXMeshIngress should be accessed via mesh-ingress-1.ansible.internal:443 since the annotation nginx.ingress.kubernetes.io/ssl-passthrough: true is added for the ingress mesh-ingress-1.
However even if the annotation is specified at the ingress resource, SSL Passthrough is disabled by default at the Ingress Controller level, so you should ensure SSL Passthrough is enabled on your controller first: TLS/HTTPS - Ingress-Nginx Controller

Then you should check logs of receptor on your remote execution node, and configuration file for receptor.
Ensure your receptor is up and running by systemctl status receptor, and gather following files on your exec node. There might be helpful logs.

  • /var/log/receptor/receptor.log
  • /etc/receptor/receptor.conf
1 Like

Thanks for the reply.
I have gotten more understanding for this topic.

I haven’t make any changes on ingress contoller and I have set --enable-ssl-passthrough: true on pod/ingress-nginx controller.

however, I have the same problem.

following /var/log/receptor/receptor.log from remote execution node.

DEBUG 2024/04/17 13:11:11 ee-awx-node-1 added service control to listener registry
INFO 2024/04/17 13:11:11 Running control service control
DEBUG 2024/04/17 13:11:11 Running Websocket peer connection wss://mesh-ingress-1.ansible.internal:443
INFO 2024/04/17 13:11:11 Initialization complete
DEBUG 2024/04/17 13:11:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:11:16.160454201 +0000 UTC m=+5.012143012 2 map[type:Control Service] [{ansible-runner true}]}
DEBUG 2024/04/17 13:12:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:12:16.164841523 +0000 UTC m=+65.016530374 2 map[type:Control Service] [{ansible-runner true}]}
DEBUG 2024/04/17 13:13:04 Client connected to control service @
DEBUG 2024/04/17 13:13:04 Control service closed
DEBUG 2024/04/17 13:13:04 Client disconnected from control service @
DEBUG 2024/04/17 13:13:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:13:16.169159221 +0000 UTC m=+125.020848072 2 map[type:Control Service] [{ansible-runner true}]}
WARNING 2024/04/17 13:13:21 Backend connection failed (will retry): dial tcp <node01_IP which is running ingress controller container>:443: connect: connection timed out
DEBUG 2024/04/17 13:14:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:14:16.174132676 +0000 UTC m=+185.025821528 2 map[type:Control Service] [{ansible-runner true}]}
DEBUG 2024/04/17 13:15:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:15:16.174603251 +0000 UTC m=+245.026292111 2 map[type:Control Service] [{ansible-runner true}]}
WARNING 2024/04/17 13:15:36 Backend connection failed (will retry): dial tcp <node01_IP>:443: connect: connection timed out
DEBUG 2024/04/17 13:16:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:16:16.179509861 +0000 UTC m=+305.031198713 2 map[type:Control Service] [{ansible-runner true}]}
DEBUG 2024/04/17 13:17:16 Sending service advertisement: &{ee-awx-node-1 control 2024-04-17 13:17:16.179728047 +0000 UTC m=+365.031416917 2 map[type:Control Service] [{ansible-runner true}]}

following is /etc/receptor/receptor.conf

---
- node:
    id: ee-awx-node-1


- work-verification:
    publickey: /etc/receptor/work_public_key.pem

- log-level: debug

- control-service:
    service: control
    filename: /var/run/receptor/receptor.sock
    permissions: 0660
    tls: tls_server
- tls-server:
    name: tls_server
    cert: /etc/receptor/tls/ee-awx-node-1.crt
    key: /etc/receptor/tls/ee-awx-node-1.key
    clientcas: /etc/receptor/tls/ca/mesh-CA.crt
    requireclientcert: true
    mintls13: False

- tls-client:
    name: tls_client
    cert: /etc/receptor/tls/ee-awx-node-1.crt
    key: /etc/receptor/tls/ee-awx-node-1.key
    rootcas: /etc/receptor/tls/ca/mesh-CA.crt
    insecureskipverify: false
    mintls13: False

- ws-peer:
    address: wss://mesh-ingress-1.ansible.internal:443
    redial: true
    tls: tls_client

- work-command:
    worktype: ansible-runner
    verifysignature: True
    params: worker
    command: ansible-runner
    allowruntimeparams: True

what could I look into more to get websocket connection from remote execute node to hop node ? on AWX UI when I run check health for the remote execute node , then I get => Instance ee-awx-node-1 is not in the receptor mesh

thank you!!

Just to be sure, have you confirmed that your Nginx Ingress Controller and the ingress resources using it are functioning properly, before setting up AWX and AWXMeshIngress?
I mean, is the ability to route HTTP requests with FQDNs from outside the cluster to the pod working?

I don’t know which kubernetes distribution is used and where, but since there is no IP address in the status of your ingress resource, I suspect that the ingress controller was not set up to be accessible from outside the cluster in the first place.

- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    ...
    name: mesh-ingress-1
    namespace: awx
    ...
  spec:
    ...
  status:
    loadBalancer: {}     ✅

Since the error is connection timed out, I assume that the TCP traffic is not reaching the pod for AWXMeshIngress in the first place.

The possibilities are:

  • The ingress and ingress controller itself is not routing trafifunctioning properly
  • The IP address resolved by DNS is incorrect
  • No reachability from Exec Node to IP address (routing, firewall, etc.)
1 Like

Hello,
Thanks for your reply.
Yesterday, I found the ingress controller was not config properly for my k8s cluster.
So, I made a correct like follows. the type of service/ingress controller did set LoadBalancer, but NO LB config at all (which was not working at all), I made changed the service type to NodePort like following.

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             **NodePort**    10.x.x.x   <none>        80:30490/TCP,443:30382/TCP   14h
service/ingress-nginx-controller-admission   ClusterIP   10.x.x.x    <none>        443/TCP                      14h

This is ingress yaml output.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/ssl-passthrough":"true"},"name":"mesh-ingress-1","namespace":"awx"},"spec":{"ingressClassName":"nginx","rules":[{"host":"mesh-ingress-1.ansible.internal","http":{"paths":[{"backend":{"service":{"name":"mesh-ingress-1","port":{"number":27199}}},"path":"/","pathType":"Prefix"}]}}]}}'
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  creationTimestamp: "2024-04-17T12:45:22Z"
  generation: 1
  name: mesh-ingress-1
  namespace: awx
  ownerReferences:
  - apiVersion: awx.ansible.com/v1alpha1
    kind: AWXMeshIngress
    name: mesh-ingress-1
    uid: 488d036c-9556-4fe6-9893-bfee1efb2779
  resourceVersion: "23477662"
  uid: 5cbd088f-c584-42ca-bff1-7566b7a44be0
spec:
  ingressClassName: nginx
  rules:
  - host: mesh-ingress-1.ansible.internal
    http:
      paths:
      - backend:
          service:
            name: mesh-ingress-1
            port:
              number: 27199
        path: /
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 172.x.x.x  # This is the node IP  which is the ingress controller pod running.

Following is logs from pod/ingress-controller. It seems service/mesh-ingress-1 registered.

I0417 12:45:26.365039       8 controller.go:190] "Configuration changes detected, backend reload required"
I0417 12:45:26.397939       8 controller.go:210] "Backend successfully reloaded"
I0417 12:45:26.398148       8 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-74fb798b87-jsnt6", UID:"a338f433-251e-4396-958e-381d4e61ae45", APIVersion:"v1", ResourceVersion:"23473311", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0417 12:45:46.314741       8 main.go:107] "successfully validated configuration, accepting" ingress="awx/mesh-ingress-1"
I0417 12:46:10.558519       8 status.go:304] "updating Ingress status" namespace="awx" ingress="mesh-ingress-1" currentValue=null newValue=[{"ip":"172.x.x.x"}]
I0417 12:46:10.565361       8 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"awx", Name:"mesh-ingress-1", UID:"5cbd088f-c584-42ca-bff1-7566b7a44be0", APIVersion:"networking.k8s.io/v1", ResourceVersion:"23477662", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0417 12:46:14.760524       8 main.go:107] "successfully validated configuration, accepting" ingress="awx/mesh-ingress-1"

  • The ingress and ingress controller itself is not routing trafifunctioning properly
    => from the ingress controller logs, I think controller seems like functioning properly. Is there any other way I can confirm the working?
  • The IP address resolved by DNS is incorrect
    => The mesh-ingress-1.ansible.internal has to be resolved with the IP 172.x.x.x which is from ingress? yes, remote execute node can resolve it.
  • No reachability from Exec Node to IP address (routing, firewall, etc.)
    => remote execute node can reach the IP address (172.x.x.x) there is no firewall, routing issue. since I am very new to k8s, is there anything I can check with k8s routing? it seems like the 443 port is not pointing to ingress-controller.

I have found the solution that I would like to share.

The service/ingress-controller NodePort is 443:30382, So, mesh-ingress-1.ansible.com has to be port 30382.

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.x.x.x   <none>        80:30490/TCP,443:30382/TCP   20h
service/ingress-nginx-controller-admission   ClusterIP   10.x.x.x    <none>        443/TCP                      20h

or I guess I could change 443:30382 to 443:443.
However, the service/ingress-controller NodePort has to be 443:443 ? or is it okay to be any port number?

Thank you very much for helping me to get more understanding.

Normally, such an operation should not be necessary.

Try deploying simple “hello world” web application.

$ kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
deployment.apps/web created

$ kubectl expose deployment web --port 8080
service/web exposed

$ cat <<EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
    - host: hello-world.info
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080
EOF
ingress.networking.k8s.io/example-ingress created

Then access this web app over FQDN, from the outside of the cluster e.g. different server or workstation. Replace 203.0.113.1 to your actual IP address.

$ curl --resolve hello-world.info:80:203.0.113.1 http://hello-world.info
Hello, world!
Version: 1.0.0
Hostname: web-57f46db77f-2fdl5

Usually, there is an LB in front of the Ingress Controller, and the LB listens for 443. Have you set up the equivalent of “Ingress-managed load balancer” in the diagram in the official documentation?: Ingress | Kubernetes
The MetalLB is the famous one for on-prem kubernetes cluster.

I don’t know where your cluster is located, how many nodes you have, and which distribution of kubernetes are you using, but first of all, try to get the hello world described above working.

1 Like