Loki PVC eating too much inodes
I was noticed by alertmanager that 85% of all inodes are used:
root@oas:~# df -hi|head
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 2.0M 346 2.0M 1% /dev
tmpfs 2.0M 1.7K 2.0M 1% /run
/dev/vda1 3.8M 3.2M 600K 85% /
...
I found that loki PVC was the culprit:
root@oas:/var/lib/OpenAppStack/local-storage# du -sch --inodes *
57 pvc-16d96851-fbdc-41a7-820b-afb4b8162b5c_varac_thelounge
1007 pvc-88394a20-2547-4bdb-8d08-8b9546dfe26a_oas_single-sign-on-userbackend
31 pvc-9ac75d98-79a3-4464-af62-9e1c3a09b8ba_oas_grafana
4 pvc-b8ee7900-388d-484f-b61f-e497394720d6_oas_alertmanager-kube-prometheus-stack-alertmanager-db-alertmanager-kube-prometheus-stack-alertmanager-0
55 pvc-c13960b9-4fee-4e0a-b16b-7398b8757383_oas_prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
3.0M pvc-faa37bef-e8a1-41a5-8ec5-a8f8c19937ea_oas_storage-loki-0
3.0M total