AWX VM Executor node is showing as unavailable status

Get Help
1

The receptor error work unit expired issue on the VM executor node. could anyone please provide the remediation steps for this issue.
Error: Receptor error from VM, detail:
Work unit expired on Fri Apr 19 11:01:00

2

Hi team, could anyone of you please provide the insights for the above queries on the AWX VM Executor Node.

3

Have you tried running a health check on the Execution Node?

image
image1845Ă—667 49.2 KB

I’m not familiar with “work unit expired”, but generally speaking there are only 2 things to worry about with Execution Nodes.

  1. Make sure you can connect to it (routing, firewall, etc)
  2. Make sure your node has enough CPU/RAM resources.

In regards to point 2, Execution Nodes spawn jobs in Execution Environment containers, so the EN needs to have enough resources for all of the containers that may run on it. This may require you to do performance tuning on AWX by limiting the number of concurrent forks/jobs in your Instance Groups.

4

Thank you @Denney-tech for your valuable inputs.

I’m writing below the remediation steps that I followed to bring the executor node into ready status.

  1. Health Check: Attempted to run a health check from AWX platform, but it didn’t provide a resolution to the issue.

  2. Node Status Check: Checked the status of the executor node, which appeared to be reachable and accessible.

  3. Receptor Service Status: SSH into the executor node and inspected the status of the Receptor service, which was active. Attempted to restart the Receptor service, but this action did not resolve the issue.

  4. Error Logs: Upon examining the logs for the Pods, the following error was encountered:
    “ERROR 2024/04/22 02:26:06 Backend error: ********: rejected connection with node ****** because it connected using a node ID we are already connected to.”

  5. Resolution: Collaborated with the respective Azure Kubernetes Service team & they restarted the Pods associated with AWX and after that I restarted the receptor service again, that successfully rectified the problem.

Thanks,
Sakshi.

1 Like