AWX- Filesystem space issue because of "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/" directory

Get Help
,
1

AWX- Filesystem space issue because of “/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/” directory.

  • Our AWX running with AWX 24.6.1 version and Filesystem filled very frequently. I have searched other forums but no luck.

I have referred following site to install AWX on my environment.

  • version details
    kubectl version
    Client Version: v1.29.6+k3s2
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.6+k3s2
    AWX 24.6.1
    Our gitlab project size is around 7GB, and Filesystem size is 120GB.

  • other details

[root@server ~]# df -h /
Filesystem                        Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-RootVol00  120G   79G   41G  66% /
[root@server ~]# du -sh /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
60G     /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/
[root@server ~]#
[root@server data]# kubectl events
LAST SEEN              TYPE      REASON                OBJECT                                     MESSAGE
19s (x35 over 170m)    Warning   FreeDiskSpaceFailed   Node/server                       (combined from similar events): Failed to garbage collect required amount of images. Attempted to free 39541062041 bytes, but only found 0 bytes eligible to free.
8s (x881 over 3h40m)   Warning   ProvisioningFailed    PersistentVolumeClaim/awx-projects-claim   storageclass.storage.k8s.io "awx-projects-volume" not found
[root@server data]#

@kurokobo