I have deployed the latest AWX using the Operator on a k8s cluster.
It is simple (for now) with one replica of both Operator and AWX instance, the PG DB is external.
AWX is exposed via https using Traefik (IngressRoute) for the WebUI/API part.
I would like to implement Execution Node(s) on remote site(s) but so far haven’t been able to make it work. I have tried (many iterations of):
creating a new instance via the UI and deploy this package on my remote execution node
deploy receptor and ansible-runner manually on the exec. node
As Stefan has been discussing in other posts (I have been following his adventures and trying to replicate), I understand we need to expose the receptor outside the k8s cluster so that execution nodes can reach it (ie. create a load-balancer service which maps to the receptor tcp port of the controller node ), but by default the “tcp-listener” part of the receptor config isn’t present in the configmap which is used for /etc/receptor/receptor.conf (on awx_ee container I think) so I imagine it isn’t listening?
What is the proper way to go?
enable receptor in the control node, using an extra configmap or the operator?
create a k8s service pointing to the default (27199) or custom (configmap) port for receptor (since receptor traffic isn’t on the web/api port)
create an install package from the UI for the new exec. node and deploy it (using the listener port configured above)
wait for the exec. node to call home and register in the UI (via the exposed k8s service)
schedule jobs on the new exec. node
Bonus question: can the exec. node be installed manually (no UI install package) and run fully in containers (receptor and ansible-runner)?
Thank you in advance for any help (or link) that would clarify the steps!
I understand we need to expose the receptor outside the k8s cluster so that execution nodes can reach it
Only the connection from awx-ee container to execution node is required, not the other way around.
Therefore, the awx-ee container does not need to have a tcp-listener, nor does it need to expose the port of the awx-ee container to the outside of the cluster.
Ensure the port 27199/tcp (default) on execution node is reachable from awx-ee container, e.g. configure firewalld on execution node to allow inbound connection to 27199.
Perhaps there is a different reason why your execution node is not working properly.
Can you provide screenshots or logs that you’ve faced? I think the logs of the awx-ee container would be helpful.
Thanks for the answer! It’s good news that I don’t need to expose the K8s side
We have strict firewall rules and it take ages to open a new port, can I use a custom port that I know is already opened (like 8080) for the exec. node(s)?
If so, do I specify the custom port when building the install package (when adding a new instance in AWX UI)?
Do you know if this method is the only one or if we can deploy receptor manually and have it call AWX to register?
can I use a custom port that I know is already opened (like 8080) for the exec. node(s)?
You can specify custom port for Receptor on execution node when you add new instance through Web UI.
Once you specify the port, the group_vars/all.yaml file in the generated install bundle contains your custom port as variable “receptor_port”. This variable will be used to generate configuration file for Receptor on the execution node.
Do you know if this method is the only one or if we can deploy receptor manually and have it call AWX to register?
Using install bundle is the simplest way since execution node has to have valid certificates for Receptor that generated by AWX. These certificates are generated by AWX, and included in the generated install bundle. The playbook in the install bundle places these certs on the correct path on the execution node, and generate configuration file for Receptor to use these certs, during installation.
Technically you can generate these certs manually by invoking some commands inside the awx-ee container and also you can deploy/configure Receptor on execution node manually, but it’s a complicated way.
If you want to deploy them manually, you should know what tasks are contained in the playbook inside the install bundle, and how to generate certs for Receotor,.
I tried deploying the bundle with port 8080, it worked but when I look for open ports on the receptor host only tcp6 is open for 8080, awx_ee cannot reach it… is there somewhere I need to enable tcp4?
when I look for open ports on the receptor host only tcp6 is open for 8080
“IPv4-mapped IPv6” may look like that.
In such case, even if you only see tcp6, you should still be able to connect with IPv4.
You should test the TCP connection to 8080 port for IPv4 by nc or telnet on the execution node.
I think your IPv4 is reachable but if you want to use specific IPv4 address to listen on,
you can add “bindaddr: ” under “tcp-listener” and restart receptor.
I have tried a fresh install of the exec. node using the bundle on a Rocky 9 machine… the playbook fails but after adding pip on the host the playbook installs properly, receptor runs but the port shows as tcp6 only (via netstat -tlpn):
receptor.service - Receptor
Loaded: loaded (/usr/lib/systemd/system/receptor.service; enabled; vend>
Drop-In: /etc/systemd/system/receptor.service.d
└─override.conf
Active: active (running) since Thu 2023-03-23 14:26:05 EDT; 6s ago
Main PID: 15563 (receptor)
Tasks: 7 (limit: 100428)
Memory: 9.2M
CPU: 97ms
CGroup: /system.slice/receptor.service
└─15563 /usr/bin/receptor -c /etc/receptor/receptor.conf
netstat -tlpn
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::8080 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
If it try to connect to port 8080 it fails: nc -vz host.domain.local 8080
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: No route to host.
I have also edited the receptor.conf file and added the bind address (bindaddr) but no luck.
On another host, I have deployed receptor 1.3.0 using docker-compose and this one seems to work on port 8080:
nc -vz host2.domain.local 8080
host2.domain.local (xx.xx.xx.xx:8080) open
Reg. the awx_ee logs, the only thing relevant is this, coming over and over:
WARNING 2023/03/23 18:41:34 Backend connection failed (will retry): dial tcp xx.xx.xx.xx:8080: connect: no route to host
WARNING 2023/03/23 18:41:37 Backend connection failed (will retry): tls: first record does not look like a TLS handshake
Again, I think this is not an issue. As I mentioned before this is usually “IPv4-mapped IPv6” address and can be connected via IPv4.
I’ve deployed Rocky 9, add intance on AWX with port 8080, and run installer for it. I got following output after installation.
This node can be connected via IPv4 and works well.
[root@exec01 ~]# netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 909/sshd: /usr/sbin
tcp6 0 0 :::8080 :::* LISTEN 6561/receptor
tcp6 0 0 :::22 :::* LISTEN 909/sshd: /usr/sbin
Could it be the bundle version the issue?
I don’t think so, “no route to host” is simply network-related issue, not receptor issue.
Installer do nothing to the host network e.g. iptables, nftables, firewalld.
Could you please double check the firewalld on your exec node, or some firewalls in your network configured correctly.
Oops, I feel so bad I didn’t check the host’s firewall (I thought it was disabled)… it did fix the issue and I can now see my Exec. Node! Thank you so much for the support!
On another note, I followed your excellent use case on Github to try to play with my receptor running with docker-compose…
I managed to create the necessary files (hostname.crt and hostname.key) with the awx_ee container and add them to my receptor, as well as the mesh CA.
Since the signature is enabled for the worker, I need to provide the public key (work_public_key.pem): where can I find this one or generate it?
I am glad to hear that your exec node works properly
If anything, I want installer to configure firewalld automatically. It may be good topic to create new issue as an RFE.
On another note, I followed your excellent use case on Github to try to play with my receptor running with docker-compose…
I managed to create the necessary files (hostname.crt and hostname.key) with the awx_ee container and add them to my receptor, as well as the mesh CA.
Since the signature is enabled for the worker, I need to provide the public key (work_public_key.pem): where can I find this one or generate it?
Hope you enjoy my repo
As you know “work_public_key.pem” is contained install bundle,
and you can dig how install bundle is generated inside the AWX here: https://github.com/ansible/awx/blob/devel/awx/api/views/instance_install_bundle.py
This code works on awx-web container, and you can see “work-public-key.pem” is copy of “/etc/receptor/signing/work-public-key.pem” (Line 66).
So, this is the file you want.
As a side note, this file is generated by AWX Operator and is stored as a Secret resource on K8s,
AWX Operator makes awx-web container in AWX pod to mount that Secret as “/etc/receptor/signing/work-public-key.pem”.
There are two keys; public key is for awx-web container, and private key is for awx-ee and awx-task container.
