Loki 2.4.0: panic: runtime error: invalid memory address or nil pointer dereference
It's the pod with the most restarts:
❯ kc -n stackspin get pod --sort-by='{.status.containerStatuses[0].restartCount}'
NAME READY STATUS RESTARTS AGE
kube-prometheus-stack-grafana-7fd789567d-6t5zl 3/3 Running 0 5h31m
single-sign-on-kratos-courier-0 1/1 Running 0 23h
kube-prometheus-stack-kube-state-metrics-7ff96d49db-5kskd 1/1 Running 0 5h31m
single-sign-on-create-oauth2-clients-6r6gz 0/6 Completed 0 23h
single-sign-on-create-admin-dnln5 0/1 Completed 0 23h
kube-prometheus-stack-operator-b4bd847d5-h7d2p 1/1 Running 0 5h31m
single-sign-on-login-6754d989f8-7vbt7 1/1 Running 1 2d6h
prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 2 3d5h
single-sign-on-postgresql-0 1/1 Running 1 15d
kube-prometheus-stack-prometheus-node-exporter-2m5tl 1/1 Running 1 8d
alertmanager-kube-prometheus-stack-alertmanager-0 2/2 Running 2 3d5h
dashboard-c77dffb6d-hdchj 2/2 Running 2 15d
ingress-nginx-controller-64d7cb6996-j9tqd 1/1 Running 1 15d
single-sign-on-hydra-maester-ddc54cd9d-5p9wp 1/1 Running 2 30d
eventrouter-84c56d96b-4g2b6 1/1 Running 2 35d
single-sign-on-kratos-5cfd7f57f5-ql6sr 1/1 Running 4 15d
promtail-wgnjl 1/1 Running 4 35d
single-sign-on-hydra-5d97f5755b-l7zjd 1/1 Running 5 15d
loki-0 1/1 Running 34 5h32mLog from previous container:
level=info ts=2021-12-16T16:18:39.827355803Z caller=checkpoint.go:342 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.010444
level=info ts=2021-12-16T16:18:44.825043268Z caller=compactor.go:337 msg="compactor startup delay completed"
level=info ts=2021-12-16T16:18:44.825152196Z caller=compactor.go:376 msg="compactor started"
level=info ts=2021-12-16T16:18:44.825171457Z caller=marker.go:178 msg="mark processor started" workers=150 delay=2h0m0s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1af803c]
goroutine 3210 [running]:
github.com/grafana/loki/pkg/validation.(*Overrides).AllByUserID(0x41580c)
/src/loki/pkg/validation/limits.go:254 +0x1c
github.com/grafana/loki/pkg/storage/stores/shipper/compactor/retention.findSmallestRetentionPeriod({0x26f81e8, 0xc000031020})
/src/loki/pkg/storage/stores/shipper/compactor/retention/expiration.go:126 +0x82
github.com/grafana/loki/pkg/storage/stores/shipper/compactor/retention.(*expirationChecker).MarkPhaseStarted(0xc000d049f0)
/src/loki/pkg/storage/stores/shipper/compactor/retention/expiration.go:60 +0x45
github.com/grafana/loki/pkg/storage/stores/shipper/compactor.(*expirationChecker).MarkPhaseStarted(0xc000d029a0)
/src/loki/pkg/storage/stores/shipper/compactor/compactor.go:516 +0x29
github.com/grafana/loki/pkg/storage/stores/shipper/compactor.(*Compactor).RunCompaction(0xc00050c300, {0x26f7500, 0xc000afa080})
/src/loki/pkg/storage/stores/shipper/compactor/compactor.go:409 +0xb6
github.com/grafana/loki/pkg/storage/stores/shipper/compactor.(*Compactor).runCompactions.func1()
/src/loki/pkg/storage/stores/shipper/compactor/compactor.go:342 +0x29
github.com/grafana/loki/pkg/storage/stores/shipper/compactor.(*Compactor).runCompactions.func2()
/src/loki/pkg/storage/stores/shipper/compactor/compactor.go:350 +0x9e
created by github.com/grafana/loki/pkg/storage/stores/shipper/compactor.(*Compactor).runCompactions
/src/loki/pkg/storage/stores/shipper/compactor/compactor.go:348 +0x434Edited by Varac