Problems shutting down pod
Kubernetes started evicting our ONLYOFFICE pod because the node was low on disk space. Although the node should not have been low on disk space (we received at least 50 alerts about it over 2 days, but I didn't solve it in time), the events show an interesting story:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Evicted 3h21m kubelet The node was low on resource: ephemeral-storage. Container nextcloud-onlyoffice was using 642220Ki, which exceeds its request of 0.
Normal Killing 3h21m kubelet Stopping container nextcloud-onlyoffice
Warning ExceededGracePeriod 3h21m kubelet Container runtime did not kill the pod within specified grace period.
Warning FailedPreStopHook 3h20m kubelet Exec lifecycle hook ([documentserver-prepare4shutdown.sh]) for Container "nextcloud-onlyoffice" in Pod "nc-onlyoffice-documentserver-5fd8b77c85-mmj5s_oas-apps(bd259d19-c6ff-49b6-9022-9b681ab7a6e8)" failed - error: command 'documentserver-prepare4shutdown.sh' exited with 137: , message: "Preparing for shutdown, it can take a lot of time, please wait..."
Warning FailedKillPod 3h20m kubelet error killing pod: failed to "KillPodSandbox" for "bd259d19-c6ff-49b6-9022-9b681ab7a6e8" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"2e8a4fb0b77d1c8039424170f5ecd1120ced696291061df0fd98856f642b8a42\": failed to Statfs \"/var/run/netns/cni-3230c59e-b924-d637-7f85-59921115bcd3\": no such file or directory"
What needs to happen:
-
Container nextcloud-onlyoffice was using 642220Ki, which exceeds its request of 0.
-- apparently we need to request more disk space - We need to allow more time for
documentserver-prepare4shutdown.sh
, because the pod was killed before the script finished.