Possible loki mem leak

An image speaks more than 1000 words (from grafana.stackspin.net :

Since a few days we're also getting ContainerMemoryUsage prom alerts for loki.

added 1 deleted label

See also Ingester memory leak version 2.4.1 #4713. This seems to reflect our findings, with 2 possible work arounds:

• downgrade loki from 2.4.1 to 2.4.0
• decreasing index_cache_validity to 5m, but this is already the case for us:

  / $ wget -q localhost:3100/config -O - | grep index_cache_validity
  index_cache_validity: 5m0s

Also, searching loki issues for "memory leak" show a lot of issues :/

created merge request !681 (merged) to address this issue

mentioned in merge request !681 (merged)

changed the description

mentioned in commit 52489642

changed milestone to %0.8.0

assigned to @varac

added QA label and removed 1 deleted label

For stackspin.net I manually restarted the stackspin statefulset in order to release some mem.

Same for init.stackspin.net, I also restarted loki there:

mentioned in commit 0e9e3660

closed with commit 52489642

closed with merge request !681 (merged)

mentioned in issue #1094 (closed)

marked this issue as related to #1094 (closed)

mentioned in issue #1111 (closed)