Merge branch '200-research-central-logging-options' into 'master'

Add Loki + promtail for log aggregation

Closes #200

See merge request openappstack/openappstack!208

ansible/roles/apps/templates/monitoring-settings.yaml

@@ -136,3 +136,9 @@ grafana:
     enabled: true
     size: 2Gi
     accessModes: ["ReadWriteOnce"]
+
+  additionalDataSources:
+  - name: Loki
+    url: http://loki-stack.oas.svc.cluster.local:3100
+    type: loki
+    access: proxy

docs/maintenance.md

 # Maintaining an Openappstack cluster
 
+## Logging
+
+Logs from pods and containers can be read in different ways:
+
+* In the cluster filesystem at `/var/log/pods/` or `/var/logs/containers/`.
+* Using [kubectl logs](https://kubectl.docs.kubernetes.io/pages/container_debugging/container_logs.html).
+* Querying aggregated logs with grafana, see below.
+
+## Central log aggregation
+
+We use [promtail](https://github.com/grafana/loki/tree/master/docs/clients/promtail),
+[Loki](https://grafana.com/oss/loki/) and [grafana](https://grafana.com/) for
+easy access of aggregated logs.
+The [Loki documentation](https://github.com/grafana/loki#documentation) is a
+good starting point how this setup works, and the [Using Loki in Grafana](https://grafana.com/docs/grafana/latest/features/datasources/loki/)
+gets you started with querying your cluster logs with grafana.
+
 ## Backup
 
 Please take care to backup the following locations:

flux/loki-stack.yml

+---
+apiVersion: helm.fluxcd.io/v1
+kind: HelmRelease
+metadata:
+  name: loki-stack
+  namespace: oas
+  annotations:
+    flux.weave.works/automated: "false"
+spec:
+  releaseName: loki-stack
+  chart:
+    repository: https://grafana.github.io/loki/charts
+    name: loki-stack
+    version: 0.24.0