Wrong NodeFilesystemSpaceFillingUp prometheus alert
I get this alert on my personal cluster:
{
"labels": {
"alertname": "NodeFilesystemSpaceFillingUp",
"device": "/dev/xvda1",
"endpoint": "metrics",
"fstype": "xfs",
"instance": "213.108.108.227:9100",
"job": "node-exporter",
"mountpoint": "/",
"namespace": "oas",
"pod": "monitoring-prometheus-node-exporter-kbwf8",
"service": "monitoring-prometheus-node-exporter",
"severity": "warning"
},
"annotations": {
"description": "Filesystem on /dev/xvda1 at 213.108.108.227:9100 has only 0.19% available space left and is filling up.",
"runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-nodefilesystemspacefillingup",
"summary": "Filesystem is predicted to run out of space within the next 24 hours."
},
"state": "pending",
"activeAt": "2019-12-18T08:26:16.026362406Z",
"value": "1.869477436492428e-01"
},
But investigating the real disk usage:
root@oas:~# df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 20G 17G 3.8G 82% /
root@oas:~# df -hi /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 8.0M 445K 7.6M 6% /
root@oas:~#