Hi all,
I don’t know if this is an awx issue or a k3s issue or some combination of both.
Several months ago I created a couple of single node k3s clusters using https://github.com/kurokobo/awx-on-k3s each with 4 vCPUs.
What I’ve noticed over time is that some of the iptables rules are being duplicated over and over again.
For example
~# iptables -L -n | sort | uniq -c | sort -k1n | tail -4
# Warning: iptables-legacy tables present, use iptables-legacy to see them
25 target prot opt source destination
21931 RETURN udp – 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
21932 RETURN tcp – 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
65790 ACCEPT all – 0.0.0.0/0 0.0.0.0/0 mark match 0x20000/0x20000 /* rule to explicitly ACCEPT traffic that comply to network policies */
When I run top, I can see 4 iptables consuming most of the CPU resource.
The net effect of this is that some awx jobs fail with timeout errors.
Stopping the k3s service, running k3s-killall.sh and starting the k3s service resolves the issue, for a while.
Any ideas about what’s happening here.