maintenance.md

Maintaining an Openappstack cluster

Logging

Logs from pods and containers can be read in different ways:

• In the cluster filesystem at /var/log/pods/ or /var/logs/containers/.
• Using kubectl logs.
• Querying aggregated logs with grafana, see below.

Central log aggregation

We use promtail, Loki and grafana for easy access of aggregated logs. The Loki documentation is a good starting point how this setup works, and the Using Loki in Grafana gets you started with querying your cluster logs with grafana.

You will find the loki grafana integration on your cluster at https://grafana.oas.example.org/explore together with some generic query examples.

LogQL query examples

Please also refer to the LogQL documentation.

Flux

Flux is responsible for installing applications. It used helm-operator to deploy the desired helm releases.

Query all messages from flux:

{app="flux"}

Query all messages from flux and helm-operator:

{app=~"(flux|helm-operator)"}

flux messages containing wordpress:

{app = "flux"} |= "wordpress"

flux messages containing wordpress without unchanged events (to only show the installation messages):

{app = "flux"} |= "wordpress" != "unchanged"

Filter out redundant flux messages:

{ app = "flux" } !~ "(unchanged | event=refreshed | method=Sync | component=checkpoint)"

Debug oauth2 single sign-on with rocketchat:

{container_name=~"(hydra|rocketchat)"}

Cert-manager

Cert manager is responsible for requesting Let's Encrypt TLS certificates.

Query cert-manager messages containing chat:

{app="cert-manager"} |= "chat"

Hydra

Hydra is the single sign-on system.

Show only warnings and errors from hydra:

{container_name="hydra"} != "level=info"

Backup

Please take care to backup the following locations:

On your provisioning machine

• Your cluster config directory, located in the top-level sub-directory clusters in your clone of the openappstack git repository. Here you can find all the files generated during the create and install commands of the CLI, together with the generated secrets that are stored during installation.

On your cluster

• The local storage directories under /var/lib/OpenAppStack/local-storage. This is the place all persistant volumes are stored. Some are more important than others, if you want to hand-pick what volumes to backup, use kubectl get pvc --all-namespaces to see which volumes are used by what application. The prometheus and alertmanager volume contain metrics, so you could choose to not back those up to save space.
• The rke directory /var/lib/OpenAppStack/rke where the rke config and state file of your cluster is stored.
• At this moment, recurring, automated etcd snapshots are not configured. Please refer to the rke etc snapshot documentation if you like to backup etcd.

If you don't care about your backup disk usage too much, the easiest way is to backup the whole /var/lib/OpenAppStack/ directory.

Restore

Restore instructions will follow, please reach out to us if you need assistance.

Change the IP of your cluster

• rke etcd snapshot-save --config /var/lib/OpenAppStack/rke/cluster.yml --name test
• Change IP in /var/lib/OpenAppStack/rke/cluster.yml
• /usr/local/bin/rke up --config=/var/lib/OpenAppStack/rke/cluster.yml
• rke etcd snapshot-restore --config /var/lib/OpenAppStack/rke/cluster.yml --name test
• /usr/local/bin/rke up --config=/var/lib/OpenAppStack/rke/cluster.yml