How to recover from pod eviction

docs/maintenance.md

@@ -115,3 +115,16 @@ if you need assistance.
 * `/usr/local/bin/rke up --config=/var/lib/OpenAppStack/rke/cluster.yml`
 * `rke etcd snapshot-restore --config /var/lib/OpenAppStack/rke/cluster.yml --name test`
 * `/usr/local/bin/rke up --config=/var/lib/OpenAppStack/rke/cluster.yml`
+
+
+## Delete evicted pods
+
+In case your cluster disk usage is over 80%, kubernetes [taints](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
+the node with `DiskPressure`. Then it tries to evict pods, which is pointless in
+a single node setup but still happened anyway. Sometimes hundreds of pods will end
+up in `evicted` state but still showed up after `DiskPressure` recovered.
+See also the [out of resource handling with kubelet](https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/) documentation.
+
+You can delete all evicted pods with this:
+
+    kubectl get pods --all-namespaces -ojson | jq -r '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) | .metadata.name + " " + .metadata.namespace' | xargs -n2 -l bash -c 'kubectl delete pods $0 --namespace=$1'