kubectl logs returns "Forbidden" on k3s with AWX Operator – cannot retrieve container logs

Get Help
,
1

I’m running a single-node k3s cluster (on Amazon Linux 2023) and have deployed the AWX Operator. However, whenever I try to retrieve logs from the awx-operator-controller-manager deployment, I get a Forbidden error, like this:

$ kubectl logs -n awx deployments/awx-operator-controller-manager -c awx-manager
Error from server: Get “https://26.56.xxx.xxx:10250/containerLogs/awx/awx-operator-controller-manager-687b856498-xxxxx/awx-manager”: Forbidden

Environment

OS: Amazon Linux 2023
k3s version: v1.31.5+k3s1 (for example)
Single-node cluster (control-plane + worker on the same instance)
AWX Operator installed and running (Pods are in Running status)
Node INTERNAL-IP is 26.56.xxx.xxx

Troubleshooting Steps Taken

NO_PROXY configuration
    curl -k https://26.56.xxx.xxx:10250/ now returns 404, so we’re no longer blocked by a corporate proxy or firewall (previously we saw a 403 from the proxy).
    So the network path to port 10250 is open.

RBAC
    Confirmed I can get pods/log as the maintain user (kubectl auth can-i get pods/log -n awx --as=maintain → yes).
    Also tried running kubectl logs with cluster-admin credentials via sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl logs ..., still Forbidden.

Kubelet Configuration
    Attempted to set anonymous-auth=false and authentication-token-webhook=true in /etc/rancher/k3s/config.yaml and via systemd drop-in (--kubelet-arg=...).
    Restarted k3s multiple times, but the setting might not actually take effect; the logs remain Forbidden.

kubeconfig Permission
    Overcame permission denied by adjusting file perms or using sudo, so that’s no longer an issue.
    But the Forbidden error persists.

k3s logs (journalctl -u k3s)
    Whenever the Forbidden error occurs, the k3s log only shows:

    E... status.go:71] "Unhandled Error" err="apiserver received an error: ...Get \"https://26.56.xxx.xxx:10250/...\": Forbidden"

    No clear indication about whether anonymous-auth or token webhook is actually enabled.

Question / Request for Help

Despite configuring NO_PROXY, confirming RBAC privileges, and updating the Kubelet settings (authentication-token-webhook=true), the kubectl logs request remains Forbidden.
I’d prefer not to fully reinstall k3s if possible. Are there any additional debug steps or configurations I should check?
If anyone has encountered a similar issue with k3s returning Forbidden for logs, please let me know how you resolved it.

I’m happy to provide more info if needed. Thanks in advance for your help!