Hydra and Kratos serviceMonitors are failing
We've updated the Hydra and Kratos charts. I think the updates included a new feature: serviceMonitors.
I don't remember why, but we disable the serviceMonitor in Velero and we disable serviceMonitorSelectorNilUsesHelmValues
in kube-prometheus-stack. I'm not sure if we should enable Kratos/Hydra's serviceMonitor, but at least we know they don't work, because we're getting alerts:
[2] Firing
Labels
alertname = service_down
container = hydra
endpoint = http
instance = 10.42.0.99:4445
job = hydra-admin
namespace = stackspin
pod = hydra-6b44c7cb87-872xh
prometheus = stackspin/kube-prometheus-stack-prometheus
service = hydra-admin
severity = page
Annotations
description = 10.42.0.99:4445 of job hydra-admin has been down for more than 5 minutes.
summary = Instance 10.42.0.99:4445 down
In Prometheus we can see that they never get up:
The simple solution (at least to get rid of the alerts):
hydra:
serviceMonitor:
enabled: false