# AWX crashes suddenly

**URL:** <https://forum.ansible.com/t/awx-crashes-suddenly/3266>\
**Category:** Get Help\
**Tags:** ubuntu, awx, ee\
**Created:** [January 11, 2024, 10:15am UTC](https://forum.ansible.com/t/awx-crashes-suddenly/3266 "2024-01-11T10:15:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tapankb2024](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tapankb2024/32/3522_2.png) [@tapankb2024](https://forum.ansible.com/u/tapankb2024)\
**Post date:** [January 11, 2024, 10:15am UTC](https://forum.ansible.com/t/awx-crashes-suddenly/3266/1 "2024-01-11T10:15:50Z")

</div>

I have AWX running on a VM with the following spec:

- 8 vCPU
- 16 GB of RAM

OS - Ubuntu 22.04.1 LTS  
AWX - AWX 23.5.0  
running on K3S node :v1.28.4+k3s2

The AWX crashes suddenly even sometime with one jobs with low resource error,could you please suggest how can I increase resources for all the pods

Warning Evicted 9m3s kubelet The node was low on resource: ephemeral-storage. Threshold quantity: 536346631, available: 1062996Ki. Container redis was using 112Ki, request is 0, has larger consumption of ephemeral-storage. Container awx-task was using 180332Ki, request is 0, has larger consumption of ephemeral-storage. Container awx-ee was using 84764Ki, request is 0, has larger consumption of ephemeral-storage. Container awx-rsyslog was using 24Ki, request is 0, has larger consumption of ephemeral-storage

pod/awx-task-847b58df79-jt5v7 0/4 ContainerStatusUnknown 12 (42m ago) 17h  
pod/awx-operator-controller-manager-66d787886f-9s8gs 0/2 Error 9 (42m ago) 17h  
pod/awx-task-847b58df79-9jrnt 0/4 ContainerStatusUnknown 3 26m  
pod/automation-job-1816-mzcx7 0/1 Error 0 12m  
pod/awx-task-847b58df79-tqzdq 0/4 Init:1/2 0 12m  
pod/awx-operator-controller-manager-66d787886f-7jcck 0/2 Error 1 26m

Can we do it without reinstalling.

---

<div class="post-metadata">

**Author:** ![kurokobo](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/kurokobo/32/182_2.png) [@kurokobo](https://forum.ansible.com/u/kurokobo)\
**Post date:** [January 11, 2024, 11:10am UTC](https://forum.ansible.com/t/awx-crashes-suddenly/3266/2 "2024-01-11T11:10:46Z")

</div>

Hi, increasing free space for `/var/lib/rancher` may help you.

On k3s node, ephemeral storage consumes under this path.

---

<div class="post-metadata">

**Author:** ![tapankb2024](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/tapankb2024/32/3522_2.png) [@tapankb2024](https://forum.ansible.com/u/tapankb2024)\
**Post date:** [January 11, 2024, 12:21pm UTC](https://forum.ansible.com/t/awx-crashes-suddenly/3266/3 "2024-01-11T12:21:37Z")

</div>

Thanks,I have extended filesystem and will monitor it.

---

<div class="post-metadata">

**Author:** ![Dustin](https://sea1.discourse-cdn.com/ansible/user_avatar/forum.ansible.com/dustin/32/2573_2.png) [@Dustin](https://forum.ansible.com/u/Dustin)\
**Post date:** [January 11, 2024, 12:48pm UTC](https://forum.ansible.com/t/awx-crashes-suddenly/3266/4 "2024-01-11T12:48:15Z")

</div>

This. I’ve had to increase my ephemeral storage a few times.

For live troubleshooting, use “kubectl get pods” to find your task pod name. Use the -n argument for that command if you configured AWX in a namespace.

Then use “kubectl describe pod \<pod\_name\>” to get details about the pod, again use -n if you used a namespace.

This is the error I see when I have ephemeral storage issues:  
`0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.`