Increase gitlab.stackspin.net CPU + mem
k9s reports CPU is between 110-150% and Mem 74% ( Warning Memory level!
):
helm install jobs time out, most likely because of host resource overcommitment, see https://open.greenhost.net/stackspin/nextcloud/-/jobs/274355
Related rabbitmq Pod nc261-rabbitmq-0 shows health check timeouts:
❯ kc -n nc261 describe pod nc261-rabbitmq-0
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8h default-scheduler Successfully assigned nc261/nc261-rabbitmq-0 to gitlab.stackspin.net
Normal Pulled 8h kubelet Container image "docker.io/bitnami/rabbitmq:3.10.7-debian-11-r4" already present on machine
Normal Created 8h kubelet Created container rabbitmq
Normal Started 8h kubelet Started container rabbitmq
Warning Unhealthy 8h kubelet Readiness probe failed: Error:
RabbitMQ on node rabbit@nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local is not running or has not fully booted yet (check with is_booting)
Warning Unhealthy 8h kubelet Readiness probe failed: Error: unable to perform an operation on node 'rabbit@nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit@nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: ['rabbit@nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local']
rabbit@nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local:
* connected to epmd (port 4369) on nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local
* epmd reports: node 'rabbit' not running at all
no other nodes on nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local
* suggestion: start the node
Current node details:
* node name: 'rabbitmqcli-164-rabbit@nc261-rabbitmq-0.nc261-rabbitmq-headless.nc261.svc.cluster.local'
* effective user's home directory: /opt/bitnami/rabbitmq/.rabbitmq
* Erlang cookie hash: 6PZf2Nlz+Zhdx+o88WFK4Q==
Warning Unhealthy 11m (x7 over 18m) kubelet Liveness probe failed: command "/bin/bash -ec rabbitmq-diagnostics -q ping" timed out
Warning Unhealthy 3m17s (x40 over 8h) kubelet Readiness probe failed: command "/bin/bash -ec rabbitmq-diagnostics -q check_running && rabbitmq-diagnostics -q check_local_alarms" timed out