Skip to content
Snippets Groups Projects

Cronjobs should wait longer on failure

    • Closed Issue created by Varac @varac

    Currently the NC cronjob is retried a couple times in intervals of ~10s. When there's a longer outage the job is failing and creates alert noise

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    Activity

    • Varac marked this issue as related to #983 (closed)

      marked this issue as related to #983 (closed)

    • Varac marked this issue as related to #968 (closed)

      marked this issue as related to #968 (closed)

    • Varac added User feedback label

      added User feedback label

    • Varac
      Varac @varac ·
      Author Contributor

      A quick summary of *nextcloud-cron* alerts in my inbox from info@stackspin.net:

      ❯ notmuch search nextcloud-cron info@stackspin.net 
thread:000000000002d156     April 20 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.144:8080 nc-nextcloud-cron-27507510 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-4vwxf stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002d149     April 20 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.144:8080 nc-nextcloud-cron-27507510 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-4vwxf stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cdd7     April 07 [1/1] info@stackspin.net; [RESOLVED] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.57:8080 stackspin-apps kube-prometheus-stack-kube-state-metrics-6ccdf9f5bd-srstz stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cdce     April 07 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.57:8080 stackspin-apps kube-prometheus-stack-kube-state-metrics-6ccdf9f5bd-srstz stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cdc2     April 07 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.223:8080 nc-nextcloud-cron-27488785 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-zdmbz stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cda3     April 06 [1/1] info@stackspin.net; [RESOLVED] kube-state-metrics (kube-state-metrics http 10.42.0.74:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-8xwjp stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cda2     April 06 [1/1] info@stackspin.net; [RESOLVED] kube-state-metrics (kube-state-metrics http 10.42.0.243:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-297jn stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cda1     April 06 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (kube-state-metrics http 10.42.0.74:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-8xwjp stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cda0     April 06 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (kube-state-metrics http 10.42.0.243:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-297jn stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd87     April 06 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (kube-state-metrics http 10.42.0.74:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-8xwjp stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd86     April 06 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (kube-state-metrics http 10.42.0.243:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-297jn stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd84     April 06 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (kube-state-metrics http 10.42.0.74:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-8xwjp stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd83     April 06 [1/1] info@stackspin.net; [FIRING:2] kube-state-metrics (kube-state-metrics http 10.42.0.243:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-297jn stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd81     April 05 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.74:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-8xwjp stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd80     April 05 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.243:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-297jn stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd71     April 05 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.74:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-8xwjp stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
thread:000000000002cd70     April 05 [1/1] info@stackspin.net; [FIRING:1] kube-state-metrics (KubeJobFailed true kube-state-metrics http 10.42.0.243:8080 nc-nextcloud-cron-27486020 stackspin-apps kube-prometheus-stack-kube-state-metrics-5bddd48fc7-297jn stackspin/kube-prometheus-stack-prometheus kube-prometheus-stack-kube-state-metrics warning) (new)
    • Maarten de Waard
      Maarten de Waard @maarten ·
      Contributor

      Proposed solution: run a separate container that does the requests to the cron endpoint regularly instead of using the Kubernetes CronJob

    • Maarten de Waard changed milestone to %2.1

      changed milestone to %2.1

    • Varac
      Varac @varac ·
      Author Contributor

      Duplicate of stackspin#1228 (closed) (fixed!)

    • Varac closed

      closed

    • Varac marked this issue as related to stackspin#1228 (closed)

      marked this issue as related to stackspin#1228 (closed)

    Please register or sign in to reply