Admin message

Due to spam, we had to disable registrations and GitHub login. Please reach out to us at support@greenhost.net if you want to contribute.

Wrong NodeFilesystemSpaceFillingUp prometheus alert

I get this alert on my personal cluster:

  {
    "labels": {
      "alertname": "NodeFilesystemSpaceFillingUp",
      "device": "/dev/xvda1",
      "endpoint": "metrics",
      "fstype": "xfs",
      "instance": "213.108.108.227:9100",
      "job": "node-exporter",
      "mountpoint": "/",
      "namespace": "oas",
      "pod": "monitoring-prometheus-node-exporter-kbwf8",
      "service": "monitoring-prometheus-node-exporter",
      "severity": "warning"
    },
    "annotations": {
      "description": "Filesystem on /dev/xvda1 at 213.108.108.227:9100 has only 0.19% available space left and is filling up.",
      "runbook_url": "https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-nodefilesystemspacefillingup",
      "summary": "Filesystem is predicted to run out of space within the next 24 hours."
    },
    "state": "pending",
    "activeAt": "2019-12-18T08:26:16.026362406Z",
    "value": "1.869477436492428e-01"
  },

But investigating the real disk usage:

root@oas:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G   17G  3.8G  82% /

root@oas:~# df -hi /
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       8.0M  445K  7.6M    6% /
root@oas:~# 
Assignee Loading
Time tracking Loading