Nextcloud cron job failure alert storm on oas.gh
Currently we get an alert storm from oas.gh (~7 mails/hour) like these:
alertname = KubeJobCompletion
container = kube-state-metrics
endpoint = http
instance = 10.42.0.194:8080
job = kube-state-metrics
job_name = nc-nextcloud-cron-27117035
namespace = oas-apps
pod = kube-prometheus-stack-kube-state-metrics-58fc6778f9-2mgrf
prometheus = oas/kube-prometheus-stack-prometheus
service = kube-prometheus-stack-kube-state-metrics
severity = warning
Annotations
description = Job oas-apps/nc-nextcloud-cron-27117035 is taking more than 12 hours to complete.
runbook_url = https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubejobcompletion
summary = Job did not complete in time