k3s template causes some "Terminating" pods
I don't think this is a problem, but it would be nice to solve it any way. If you log into a machine made by the CI, it has some lingering pods from when the pipeline VPS image was created.
root@772-no-image-tag-for-cert-manager-job:~# kubectl get nodes
kubectNAME STATUS ROLES AGE VERSION
maarten-test.openappstack.net NotReady control-plane,master 10d v1.20.4+k3s1
772-no-image-tag-for-cert-manager-job Ready control-plane,master 5h2m v1.20.4+k3s1
root@772-no-image-tag-for-cert-manager-job:~# kubectl get pods
No resources found in default namespace.
root@772-no-image-tag-for-cert-manager-job:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
...
kube-system coredns-854c77959c-vv7fc 1/1 Terminating 0 10d
kube-system metrics-server-86cbb8457f-q8sdm 1/1 Terminating 0 10d
...
oas prometheus-stack-prometheus-node-exporter-f8qv6 0/1 Terminating 0 4h50m
oas loki-stack-0 1/1 Running 0 4h51m
oas prometheus-stack-prometheus-node-exporter-v4bz8 0/1 Pending 0 4h31m
oas prometheus-stack-kube-prom-operator-7dcb856d4d-z5ggj 1/1 Running 0 4h31m
oas prometheus-stack-prometheus-node-exporter-bvsg2 1/1 Running 0 4h31m
oas prometheus-stack-kube-state-metrics-6f787f797d-tj565 1/1 Running 0 4h31m
...
As you can see the node from when I created the VPS image still exists. The pods that are stuck Terminating
are trying to bind to that node. You can also see that the prometheus operator made a node-exporter
for that node that's currently stuck on Pending
.
I haven't seen any performance issues linked to this yet, but it would be nice not to have random Terminating pods and an unreachable Node in the image.
I've manually solved it once with a kubectl delete node maarten-test.openappstack.net
. Perhaps it's possible to run this command before we create the k3s image.