Skip to content
Snippets Groups Projects

High node CPU usage by k3s and iptables procs

    • Closed Issue created by Varac @varac

    This is from oas.gh:

    image

    And this is the current top output:

    image

    There's also quite a lot of "iptables ChainExists" traces in the journal like

    Jul 13 13:39:40 oas.greenhost.net k3s[22500]: I0713 13:39:40.082583   22500 trace.go:205] Trace[196394245]: "iptables ChainExists" (13-Jul-2021 13:39:37.908) (total time: 2174ms):
    root@oas:~# journalctl --since=today | grep 'iptables ChainExists' | wc -l
1361
    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    Activity

    • Varac added Bug label

      added Bug label

    • Varac
      Varac @varac ·
      Author Contributor

      holy shit, tailing the eventrouter pod logs shows a storm of cronjob updates from jobs that are even a few days old:

      {"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102045-5ts45.169125fc3f18a2bb","namespace":"oas","uid":"75e0c4bb-5d98-4e34-8620-0ab71fffab4f","resourceVersion":"3888758","creationTimestamp":"2021-07-12T20:45:05Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-12T20:45:05Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102045-5ts45","uid":"ab76b657-962c-4634-b201-0d396144af78","apiVersion":"v1","resourceVersion":"3888733","fieldPath":"spec.containers{user-panel}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 701.53442ms","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-12T20:45:05Z","lastTimestamp":"2021-07-12T20:45:05Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102045-5ts45.169125fc3f18a2bb","namespace":"oas","uid":"75e0c4bb-5d98-4e34-8620-0ab71fffab4f","resourceVersion":"3888758","creationTimestamp":"2021-07-12T20:45:05Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-12T20:45:05Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102045-5ts45","uid":"ab76b657-962c-4634-b201-0d396144af78","apiVersion":"v1","resourceVersion":"3888733","fieldPath":"spec.containers{user-panel}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 701.53442ms","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-12T20:45:05Z","lastTimestamp":"2021-07-12T20:45:05Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102250-bv5c2.1691312df8b3f9b7","namespace":"oas","uid":"2dfdc77a-ee46-4316-bb93-c0c1bf872993","resourceVersion":"3934561","creationTimestamp":"2021-07-13T00:10:13Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T00:10:13Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102250-bv5c2","uid":"a9655a94-c520-4e8e-b446-a507e21a6dd1","apiVersion":"v1","resourceVersion":"3934499","fieldPath":"spec.containers{rocketchat}"},"reason":"Created","message":"Created container rocketchat","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T00:10:13Z","lastTimestamp":"2021-07-13T00:10:13Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102250-bv5c2.1691312df8b3f9b7","namespace":"oas","uid":"2dfdc77a-ee46-4316-bb93-c0c1bf872993","resourceVersion":"3934561","creationTimestamp":"2021-07-13T00:10:13Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T00:10:13Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102250-bv5c2","uid":"a9655a94-c520-4e8e-b446-a507e21a6dd1","apiVersion":"v1","resourceVersion":"3934499","fieldPath":"spec.containers{rocketchat}"},"reason":"Created","message":"Created container rocketchat","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T00:10:13Z","lastTimestamp":"2021-07-13T00:10:13Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"nc-nextcloud-cron-27102555-ksbdr.169141d13e2cde30","namespace":"oas-apps","uid":"25694947-b1a8-4343-8443-b1dfc85523bb","resourceVersion":"4001749","creationTimestamp":"2021-07-13T05:15:07Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T05:15:07Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27102555-ksbdr","uid":"14382f2f-801e-417e-8037-baa0ba7f387c","apiVersion":"v1","resourceVersion":"4001708","fieldPath":"spec.containers{nextcloud}"},"reason":"Started","message":"Started container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T05:15:07Z","lastTimestamp":"2021-07-13T05:15:07Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"nc-nextcloud-cron-27102555-ksbdr.169141d13e2cde30","namespace":"oas-apps","uid":"25694947-b1a8-4343-8443-b1dfc85523bb","resourceVersion":"4001749","creationTimestamp":"2021-07-13T05:15:07Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T05:15:07Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27102555-ksbdr","uid":"14382f2f-801e-417e-8037-baa0ba7f387c","apiVersion":"v1","resourceVersion":"4001708","fieldPath":"spec.containers{nextcloud}"},"reason":"Started","message":"Started container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T05:15:07Z","lastTimestamp":"2021-07-13T05:15:07Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102490-kdd9f.16913e47e49022ad","namespace":"oas","uid":"cfdf16ef-fe5c-483d-ab3a-509da2e8166c","resourceVersion":"3987441","creationTimestamp":"2021-07-13T04:10:18Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T04:10:18Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102490-kdd9f","uid":"ff0aebd6-8eed-49de-b5ce-bdb3a61a63d2","apiVersion":"v1","resourceVersion":"3987373","fieldPath":"spec.containers{grafana}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 797.936714ms","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T04:10:18Z","lastTimestamp":"2021-07-13T04:10:18Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102490-kdd9f.16913e47e49022ad","namespace":"oas","uid":"cfdf16ef-fe5c-483d-ab3a-509da2e8166c","resourceVersion":"3987441","creationTimestamp":"2021-07-13T04:10:18Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T04:10:18Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102490-kdd9f","uid":"ff0aebd6-8eed-49de-b5ce-bdb3a61a63d2","apiVersion":"v1","resourceVersion":"3987373","fieldPath":"spec.containers{grafana}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 797.936714ms","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T04:10:18Z","lastTimestamp":"2021-07-13T04:10:18Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102655-lb2b4.169147474ce58388","namespace":"oas","uid":"5f0272f0-d93f-40ea-a0c5-785ed37a81a1","resourceVersion":"4023726","creationTimestamp":"2021-07-13T06:55:12Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T06:55:12Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102655-lb2b4","uid":"12002339-59b2-4478-86fd-0325ebf6a80a","apiVersion":"v1","resourceVersion":"4023691","fieldPath":"spec.containers{user-panel}"},"reason":"Started","message":"Started container user-panel","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T06:55:12Z","lastTimestamp":"2021-07-13T06:55:12Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27102655-lb2b4.169147474ce58388","namespace":"oas","uid":"5f0272f0-d93f-40ea-a0c5-785ed37a81a1","resourceVersion":"4023726","creationTimestamp":"2021-07-13T06:55:12Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T06:55:12Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27102655-lb2b4","uid":"12002339-59b2-4478-86fd-0325ebf6a80a","apiVersion":"v1","resourceVersion":"4023691","fieldPath":"spec.containers{user-panel}"},"reason":"Started","message":"Started container user-panel","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T06:55:12Z","lastTimestamp":"2021-07-13T06:55:12Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27100375-zsmsk.1690cadb9a9af2be","namespace":"oas","uid":"87bd8a2f-6bb6-42cc-975e-70c2731aec96","resourceVersion":"3530297","creationTimestamp":"2021-07-11T16:55:10Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-11T16:55:10Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27100375-zsmsk","uid":"3b534d89-f270-4f1b-b2c2-cd2acb802f1b","apiVersion":"v1","resourceVersion":"3530253","fieldPath":"spec.containers{rocketchat}"},"reason":"Started","message":"Started container rocketchat","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-11T16:55:10Z","lastTimestamp":"2021-07-11T16:55:10Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27100375-zsmsk.1690cadb9a9af2be","namespace":"oas","uid":"87bd8a2f-6bb6-42cc-975e-70c2731aec96","resourceVersion":"3530297","creationTimestamp":"2021-07-11T16:55:10Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-11T16:55:10Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27100375-zsmsk","uid":"3b534d89-f270-4f1b-b2c2-cd2acb802f1b","apiVersion":"v1","resourceVersion":"3530253","fieldPath":"spec.containers{rocketchat}"},"reason":"Started","message":"Started container rocketchat","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-11T16:55:10Z","lastTimestamp":"2021-07-11T16:55:10Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27100395-kx785.1690cbf29fcf5078","namespace":"oas","uid":"e8541917-4a11-4cbd-9488-bc73ef17968d","resourceVersion":"3534398","creationTimestamp":"2021-07-11T17:15:08Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-11T17:15:08Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27100395-kx785","uid":"9215bece-e166-4b9c-a356-83db0255c567","apiVersion":"v1","resourceVersion":"3534368","fieldPath":"spec.containers{wordpress}"},"reason":"Pulling","message":"Pulling image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\"","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-11T17:15:08Z","lastTimestamp":"2021-07-11T17:15:08Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27100395-kx785.1690cbf29fcf5078","namespace":"oas","uid":"e8541917-4a11-4cbd-9488-bc73ef17968d","resourceVersion":"3534398","creationTimestamp":"2021-07-11T17:15:08Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-11T17:15:08Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27100395-kx785","uid":"9215bece-e166-4b9c-a356-83db0255c567","apiVersion":"v1","resourceVersion":"3534368","fieldPath":"spec.containers{wordpress}"},"reason":"Pulling","message":"Pulling image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\"","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-11T17:15:08Z","lastTimestamp":"2021-07-11T17:15:08Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"nc-nextcloud-cron-27100515-2d4mq.1690d30f029e97d2","namespace":"oas-apps","uid":"edd4bf9c-6f4a-4a1f-a35f-b6e4ff8e75a9","resourceVersion":"3562089","creationTimestamp":"2021-07-11T19:25:26Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-11T19:25:26Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27100515-2d4mq","uid":"a31ba6df-af88-447a-af95-593cb92903b5","apiVersion":"v1","resourceVersion":"3562077","fieldPath":"spec.containers{nextcloud}"},"reason":"Created","message":"Created container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-11T19:25:26Z","lastTimestamp":"2021-07-11T19:25:26Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"nc-nextcloud-cron-27100515-2d4mq.1690d30f029e97d2","namespace":"oas-apps","uid":"edd4bf9c-6f4a-4a1f-a35f-b6e4ff8e75a9","resourceVersion":"3562089","creationTimestamp":"2021-07-11T19:25:26Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-11T19:25:26Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27100515-2d4mq","uid":"a31ba6df-af88-447a-af95-593cb92903b5","apiVersion":"v1","resourceVersion":"3562077","fieldPath":"spec.containers{nextcloud}"},"reason":"Created","message":"Created container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-11T19:25:26Z","lastTimestamp":"2021-07-11T19:25:26Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
{"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27101880-mx8jj.16911cffd375e3f6","namespace":"oas","uid":"e0fd0162-e660-4c49-b9d4-31c472379e12","resourceVersion":"3852333","creationTimestamp":"2021-07-12T18:00:25Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-12T18:00:25Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27101880-mx8jj","uid":"34b74af3-11a4-4247-9c6c-d3db63daac79","apiVersion":"v1","resourceVersion":"3852239","fieldPath":"spec.containers{rocketchat}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 959.507293ms","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-12T18:00:25Z","lastTimestamp":"2021-07-12T18:00:25Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27101880-mx8jj.16911cffd375e3f6","namespace":"oas","uid":"e0fd0162-e660-4c49-b9d4-31c472379e12","resourceVersion":"3852333","creationTimestamp":"2021-07-12T18:00:25Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-12T18:00:25Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27101880-mx8jj","uid":"34b74af3-11a4-4247-9c6c-d3db63daac79","apiVersion":"v1","resourceVersion":"3852239","fieldPath":"spec.containers{rocketchat}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 959.507293ms","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-12T18:00:25Z","lastTimestamp":"2021-07-12T18:00:25Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
      • Varac
        Varac @varac ·
        Author Contributor

        So we're talking about 6-40 UPDATED job events per second:

        ❯ kc -n oas logs eventrouter-7f9fbbb44c-w4xfn -f | ts | tee /tmp/events.log
... let it record for a few seconds, then:

❯ grep 15:56:00 /tmp/events.log | wc -l
6

❯ grep 15:55:59 /tmp/events.log | wc -l
10

❯ grep 15:55:58 /tmp/events.log | wc -l
23

❯ grep 15:55:57 /tmp/events.log | wc -l
19

❯ grep 15:55:56 /tmp/events.log | wc -l
40
      • Maarten de Waard
        Maarten de Waard @maarten ·
        Contributor

        I can't reproduce this. If I tail the eventrouter log not a lot happens.

        Did you wait for the old logs to be printed before you started grepping timestamps?

      • Varac
        Varac @varac ·
        Author Contributor

        I don't remember. Why I do the same cmd now, I see every 5m many ADDED and a few UPDATED events:

        

Jul 13 16:45:02 {"verb":"UPDATED","event":{"metadata":{"name":"nc-nextcloud-cron.168d129ba170f182","namespace":"oas-apps","uid":"9fa6660f-c148-4ad4-a874-10c79a398b09","resourceVersion":"4125092","creationTimestamp":"2021-07-12T09:50:02Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-12T09:50:02Z"}]},"involvedObject":{"kind":"CronJob","namespace":"oas-apps","name":"nc-nextcloud-cron","uid":"3317cda4-6db9-48f8-aa2e-b3a6f148f88c","apiVersion":"batch/v1","resourceVersion":"3743546"},"reason":"SuccessfulCreate","message":"(combined from similar events): Created job nc-nextcloud-cron-27103125","source":{"component":"cronjob-controller"},"firstTimestamp":"2021-06-29T14:05:00Z","lastTimestamp":"2021-07-13T14:45:01Z","count":3193,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"nc-nextcloud-cron.168d129ba170f182","namespace":"oas-apps","uid":"9fa6660f-c148-4ad4-a874-10c79a398b09","resourceVersion":"4124014","creationTimestamp":"2021-07-12T09:50:02Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-12T09:50:02Z"}]},"involvedObject":{"kind":"CronJob","namespace":"oas-apps","name":"nc-nextcloud-cron","uid":"3317cda4-6db9-48f8-aa2e-b3a6f148f88c","apiVersion":"batch/v1","resourceVersion":"3743546"},"reason":"SuccessfulCreate","message":"(combined from similar events): Created job nc-nextcloud-cron-27103120","source":{"component":"cronjob-controller"},"firstTimestamp":"2021-06-29T14:05:00Z","lastTimestamp":"2021-07-13T14:40:01Z","count":3192,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:03 {"verb":"ADDED","event":{"metadata":{"name":"nc-nextcloud-cron-27103125.169160eae7e7f0d6","namespace":"oas-apps","uid":"f2e715bf-0f8d-4d0a-bf7f-a5eb16772b9a","resourceVersion":"4125097","creationTimestamp":"2021-07-13T14:45:03Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:03Z"}]},"involvedObject":{"kind":"Job","namespace":"oas-apps","name":"nc-nextcloud-cron-27103125","uid":"2ae54c1b-d0b4-41e1-95af-df632a9e6ef1","apiVersion":"batch/v1","resourceVersion":"4125087"},"reason":"SuccessfulCreate","message":"Created pod: nc-nextcloud-cron-27103125-dkx7v","source":{"component":"job-controller"},"firstTimestamp":"2021-07-13T14:45:02Z","lastTimestamp":"2021-07-13T14:45:02Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:04 {"verb":"ADDED","event":{"metadata":{"name":"nc-nextcloud-cron-27103125-dkx7v.169160eaf67733aa","namespace":"oas-apps","uid":"188620c5-5aee-4cdf-bba4-3a803d096852","resourceVersion":"4125098","creationTimestamp":"2021-07-13T14:45:02Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"events.k8s.io/v1","time":"2021-07-13T14:45:02Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27103125-dkx7v","uid":"fc6b251e-10c8-4b2d-8813-6d5256518f42","apiVersion":"v1","resourceVersion":"4125091"},"reason":"Scheduled","message":"Successfully assigned oas-apps/nc-nextcloud-cron-27103125-dkx7v to oas.greenhost.net","source":{},"firstTimestamp":null,"lastTimestamp":null,"type":"Normal","eventTime":"2021-07-13T14:45:02.731064Z","action":"Binding","reportingComponent":"default-scheduler","reportingInstance":"default-scheduler-oas.greenhost.net"}}
Jul 13 16:45:04 {"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients.168d0f9b4c03f1c7","namespace":"oas","uid":"e4fcc77d-8f9c-46fe-9924-15d9a1a51f7f","resourceVersion":"4125106","creationTimestamp":"2021-06-29T13:10:00Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-06-29T13:10:00Z"}]},"involvedObject":{"kind":"CronJob","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients","uid":"c8fb3f35-04d7-4626-afa8-0bfb5b03a58f","apiVersion":"batch/v1","resourceVersion":"13623"},"reason":"SuccessfulCreate","message":"(combined from similar events): Created job single-sign-on-recreate-oauth2-clients-27103125","source":{"component":"cronjob-controller"},"firstTimestamp":"2021-06-29T13:10:00Z","lastTimestamp":"2021-07-13T14:45:03Z","count":4052,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients.168d0f9b4c03f1c7","namespace":"oas","uid":"e4fcc77d-8f9c-46fe-9924-15d9a1a51f7f","resourceVersion":"4124038","creationTimestamp":"2021-06-29T13:10:00Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-06-29T13:10:00Z"}]},"involvedObject":{"kind":"CronJob","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients","uid":"c8fb3f35-04d7-4626-afa8-0bfb5b03a58f","apiVersion":"batch/v1","resourceVersion":"13623"},"reason":"SuccessfulCreate","message":"(combined from similar events): Created job single-sign-on-recreate-oauth2-clients-27103120","source":{"component":"cronjob-controller"},"firstTimestamp":"2021-06-29T13:10:00Z","lastTimestamp":"2021-07-13T14:40:07Z","count":4051,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:05 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125.169160eb4fafced2","namespace":"oas","uid":"ed8e69f0-6c57-4ed7-9239-2889f1640687","resourceVersion":"4125108","creationTimestamp":"2021-07-13T14:45:04Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:04Z"}]},"involvedObject":{"kind":"Job","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125","uid":"4584db1f-07a3-4511-9540-b82819eedfd9","apiVersion":"batch/v1","resourceVersion":"4125088"},"reason":"SuccessfulCreate","message":"Created pod: single-sign-on-recreate-oauth2-clients-27103125-xkphc","source":{"component":"job-controller"},"firstTimestamp":"2021-07-13T14:45:04Z","lastTimestamp":"2021-07-13T14:45:04Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:06 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160ebbb844a5b","namespace":"oas","uid":"ad713336-8170-4e2f-8da2-a7986c6e7127","resourceVersion":"4125116","creationTimestamp":"2021-07-13T14:45:06Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"events.k8s.io/v1","time":"2021-07-13T14:45:06Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125102"},"reason":"Scheduled","message":"Successfully assigned oas/single-sign-on-recreate-oauth2-clients-27103125-xkphc to oas.greenhost.net","source":{},"firstTimestamp":null,"lastTimestamp":null,"type":"Normal","eventTime":"2021-07-13T14:45:06.037034Z","action":"Binding","reportingComponent":"default-scheduler","reportingInstance":"default-scheduler-oas.greenhost.net"}}
Jul 13 16:45:06 {"verb":"UPDATED","event":{"metadata":{"name":"kube-prometheus-stack-prometheus-node-exporter-72qz8.168dae357c25586f","namespace":"oas","uid":"a3988172-f84e-41c2-a8be-0d31a732441f","resourceVersion":"4125117","creationTimestamp":"2021-07-01T13:36:25Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-01T13:36:25Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"kube-prometheus-stack-prometheus-node-exporter-72qz8","uid":"b1e54836-e33c-4cdc-8d8d-065263fadfad","apiVersion":"v1","resourceVersion":"306579","fieldPath":"spec.containers{node-exporter}"},"reason":"Unhealthy","message":"Liveness probe failed: Get \"http://213.108.108.57:9100/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-01T13:36:25Z","lastTimestamp":"2021-07-13T14:45:05Z","count":493,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""},"old_event":{"metadata":{"name":"kube-prometheus-stack-prometheus-node-exporter-72qz8.168dae357c25586f","namespace":"oas","uid":"a3988172-f84e-41c2-a8be-0d31a732441f","resourceVersion":"4117792","creationTimestamp":"2021-07-01T13:36:25Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-01T13:36:25Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"kube-prometheus-stack-prometheus-node-exporter-72qz8","uid":"b1e54836-e33c-4cdc-8d8d-065263fadfad","apiVersion":"v1","resourceVersion":"306579","fieldPath":"spec.containers{node-exporter}"},"reason":"Unhealthy","message":"Liveness probe failed: Get \"http://213.108.108.57:9100/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-01T13:36:25Z","lastTimestamp":"2021-07-13T14:10:25Z","count":489,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:09 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160ec686f36aa","namespace":"oas","uid":"c19de548-6d91-45c0-b190-f3dba309c862","resourceVersion":"4125125","creationTimestamp":"2021-07-13T14:45:08Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:08Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{user-panel}"},"reason":"Pulling","message":"Pulling image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\"","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:08Z","lastTimestamp":"2021-07-13T14:45:08Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:09 {"verb":"ADDED","event":{"metadata":{"name":"nc-nextcloud-cron-27103125-dkx7v.169160ec80738eba","namespace":"oas-apps","uid":"f158eb6d-211a-46b3-9bc1-3e12204bc60e","resourceVersion":"4125128","creationTimestamp":"2021-07-13T14:45:09Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:09Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27103125-dkx7v","uid":"fc6b251e-10c8-4b2d-8813-6d5256518f42","apiVersion":"v1","resourceVersion":"4125093","fieldPath":"spec.containers{nextcloud}"},"reason":"Pulled","message":"Container image \"nextcloud:19.0.3-apache\" already present on machine","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:09Z","lastTimestamp":"2021-07-13T14:45:09Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:10 {"verb":"ADDED","event":{"metadata":{"name":"nc-nextcloud-cron-27103125-dkx7v.169160ecc6245594","namespace":"oas-apps","uid":"d43d0c62-a569-4cd4-8a4d-c20088709bd3","resourceVersion":"4125135","creationTimestamp":"2021-07-13T14:45:10Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:10Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27103125-dkx7v","uid":"fc6b251e-10c8-4b2d-8813-6d5256518f42","apiVersion":"v1","resourceVersion":"4125093","fieldPath":"spec.containers{nextcloud}"},"reason":"Created","message":"Created container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:10Z","lastTimestamp":"2021-07-13T14:45:10Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:10 {"verb":"ADDED","event":{"metadata":{"name":"nc-nextcloud-cron-27103125-dkx7v.169160ecd66f2a13","namespace":"oas-apps","uid":"c34d1f2a-50d7-48af-8cf7-e66071034a2e","resourceVersion":"4125136","creationTimestamp":"2021-07-13T14:45:10Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:10Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas-apps","name":"nc-nextcloud-cron-27103125-dkx7v","uid":"fc6b251e-10c8-4b2d-8813-6d5256518f42","apiVersion":"v1","resourceVersion":"4125093","fieldPath":"spec.containers{nextcloud}"},"reason":"Started","message":"Started container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:10Z","lastTimestamp":"2021-07-13T14:45:10Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:13 {"verb":"ADDED","event":{"metadata":{"name":"nc-nextcloud-cron-27103125.169160ed5757dc20","namespace":"oas-apps","uid":"2a08853d-efd3-4aeb-8b0a-2853710fbaa1","resourceVersion":"4125148","creationTimestamp":"2021-07-13T14:45:13Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:13Z"}]},"involvedObject":{"kind":"Job","namespace":"oas-apps","name":"nc-nextcloud-cron-27103125","uid":"2ae54c1b-d0b4-41e1-95af-df632a9e6ef1","apiVersion":"batch/v1","resourceVersion":"4125101"},"reason":"Completed","message":"Job completed","source":{"component":"job-controller"},"firstTimestamp":"2021-07-13T14:45:12Z","lastTimestamp":"2021-07-13T14:45:12Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:15 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160eddb88c810","namespace":"oas","uid":"c9dc5fa1-d86b-4519-a01c-67f7ad9fdfa1","resourceVersion":"4125159","creationTimestamp":"2021-07-13T14:45:15Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:15Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{user-panel}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 6.226012885s","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:15Z","lastTimestamp":"2021-07-13T14:45:15Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:16 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160ee2467c49f","namespace":"oas","uid":"bf30cb80-952e-430b-88ca-55ec397ad354","resourceVersion":"4125165","creationTimestamp":"2021-07-13T14:45:16Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:16Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{user-panel}"},"reason":"Created","message":"Created container user-panel","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:16Z","lastTimestamp":"2021-07-13T14:45:16Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:17 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160ee69ed5091","namespace":"oas","uid":"c2b94f43-6d91-4306-958a-3b5484f12ee0","resourceVersion":"4125169","creationTimestamp":"2021-07-13T14:45:17Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:17Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{user-panel}"},"reason":"Started","message":"Started container user-panel","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:17Z","lastTimestamp":"2021-07-13T14:45:17Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:18 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160ee6e4e5e9d","namespace":"oas","uid":"63a17541-b83d-4a43-88a4-1de4bc9fd8a4","resourceVersion":"4125173","creationTimestamp":"2021-07-13T14:45:17Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:17Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{nextcloud}"},"reason":"Pulling","message":"Pulling image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\"","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:17Z","lastTimestamp":"2021-07-13T14:45:17Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:22 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160ef8978bcb7","namespace":"oas","uid":"0375e1d9-8fbf-483b-bdf4-47e5aa85e4be","resourceVersion":"4125187","creationTimestamp":"2021-07-13T14:45:22Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:22Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{nextcloud}"},"reason":"Pulled","message":"Successfully pulled image \"open.greenhost.net:4567/openappstack/user-panel/backend:master\" in 4.750706981s","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:22Z","lastTimestamp":"2021-07-13T14:45:22Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:25 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160efd4833721","namespace":"oas","uid":"5fa5f03a-f417-4323-9f17-4ccbc9ca96b6","resourceVersion":"4125191","creationTimestamp":"2021-07-13T14:45:24Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:24Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{nextcloud}"},"reason":"Created","message":"Created container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:23Z","lastTimestamp":"2021-07-13T14:45:23Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
Jul 13 16:45:26 {"verb":"ADDED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc.169160f038b445b5","namespace":"oas","uid":"d2a600f6-defe-425a-905d-d61a0be6cd5c","resourceVersion":"4125194","creationTimestamp":"2021-07-13T14:45:25Z","managedFields":[{"manager":"k3s","operation":"Update","apiVersion":"v1","time":"2021-07-13T14:45:25Z"}]},"involvedObject":{"kind":"Pod","namespace":"oas","name":"single-sign-on-recreate-oauth2-clients-27103125-xkphc","uid":"13c7227c-9a2d-4ab1-8ace-c225372bd31e","apiVersion":"v1","resourceVersion":"4125107","fieldPath":"spec.containers{nextcloud}"},"reason":"Started","message":"Started container nextcloud","source":{"component":"kubelet","host":"oas.greenhost.net"},"firstTimestamp":"2021-07-13T14:45:25Z","lastTimestamp":"2021-07-13T14:45:25Z","count":1,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
      • Varac
        Varac @varac ·
        Author Contributor

        See the timestamp of this old job (2021-06-29):

        Jul 13 16:45:04 {"verb":"UPDATED","event":{"metadata":{"name":"single-sign-on-recreate-oauth2-clients.168d0f9b4c03f1c7","namespace":"oas","uid":"e4fcc77d-8f9c-46fe-9924-15d9a1a51f7f","resourceVersion":"4125106","creationTimestamp":"2021-06-29T13:10:00Z","
      • Varac
        Varac @varac ·
        Author Contributor

        @maarten

        I can't reproduce this. If I tail the eventrouter log not a lot happens

        just wait a bit, events keep on coming still, even not so brutal as before.

      • Please register or sign in to reply
    • Varac changed the description

      changed the description

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Contributor

      We are working on a theory that Kubernetes jobs are actually pretty heavy for the things we use them for. Especially with the nextcloud helm chart, we're in a situation that's pretty similar to this issue: https://open.greenhost.net/openappstack/wordpress-helm/-/issues/85

      • Maarten de Waard
        Maarten de Waard @maarten ·
        Contributor

        The iptables -L list has this line 35054 times

        Chain KUBE-ROUTER-INPUT (1 references)
target     prot opt source               destination      
RETURN     all  --  anywhere             10.43.0.0/16         /* allow traffic to cluster IP - M66LPN4N3KB5HTJR */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */
RETURN     tcp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL TCP traffic to node ports - LR7XO7NXDBGQJD2M */
RETURN     udp  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL multiport dports 30000:32767 /* allow LOCAL UDP traffic to node ports - 76UCBPIZNGJNWNUZ */

...
        Edited by Maarten de Waard
      • Maarten de Waard
        Maarten de Waard @maarten ·
        Contributor

        Seems related:

        https://github.com/k3s-io/k3s/issues/3117

      • Maarten de Waard
        Maarten de Waard @maarten ·
        Contributor

        I'm 99% this causes iptables to go haywire (35000 lines is insane)

        According to this comment we can solve it by using k3s's shipped IPtables:

        actually it looks like it's not even a kernel thing - it's just a bug in the version of the nftables package that Debain is shipping. If you apt remove iptables nftables -y and reboot the node, K3s will use its packaged version of the iptables/nftables tools which work properly:

      • Varac
        Varac @varac ·
        Author Contributor

        do I have your permission to hit oas.greenhost.net with a hammer as suggested in the comment I posted here

        YES please! :hammer_pick:

      • Varac
        Varac @varac ·
        Author Contributor

        I was already wondering if having iptables rules setup during boot like we have might cause issues, so please remove the netfilter-persistent package (iptables-persistent and nftables are half-removed, we should purge them completely):

        root@oas:~# dpkg -l nfttables
rc  nftables                       0.9.0-2                      amd64        Program to control packet filtering rules by Netfilter project

root@oas:~# dpkg -l | grep iptab
rc  iptables-persistent            1.0.11+deb10u1               all          boot-time loader for netfilter rules, iptables plugin
      • Mart van Santen
        Mart van Santen @mart ·
        Maintainer

        35000 rules is of course insane, but actually, the kernel can still process that in a quite decent time. The top shows the IPtables process, not the rules themselves in the kernel. Iptables is just a tool to configure the rules in the kernel. They seem to use iptables-restore, which rebuild the full table (and not just inserting one/two rules). This is a heavy task with so many rules also risky, as during that process there is no proper fire walling in place. That process is, AFAIK not atomic.

        In my opinion, the correct behavior is that there is only a limited set of rules and that k8s removes/adds the rules based on a container. You would expect that k8s removes all container related rules once a container is removed. That seems not to be the case here. We see a lot of duplicated rules. Apparently k8s uses comments on the rules to regonize them.

        It can be that the bundeled iptables works a bit different and indeed can update and the debian based does not work to recognize existing rules. . IIRC the (new) debian bundled iptables automatically converts rules to nftables if the kernel support nftables. So maybe something is going wrong there.

        But this is a lot of guessing all :-)

      • Maarten de Waard
        Maarten de Waard @maarten ·
        Contributor

        What you say sounds to me a lot like what the people in the K3s Github issue are saying as well. k3s is the "lightweight k8s" we use.

      • Simon
        Simon @simon ·
        Owner

        Please mind that when removing iptables-persistent and netfilter-persistent you are removing your default firewall configuration, exposing all machine ports. For instance the machine metrics are now world readable on:

        http://oas.greenhost.net:9100/metrics

        Other exposed ports are:

        PORT      STATE    SERVICE
22/tcp    open     ssh
80/tcp    open     http
443/tcp   open     https
6443/tcp  open     k3s server
7168/tcp  open     cosmos2d
7472/tcp  open     speaker (metallb)
7946/tcp  open     speaker (metallb)
9100/tcp  open     prometheus node exporter
10250/tcp open     k3s server
30924/tcp open     k3s server
30964/tcp open     k3s server
31451/tcp open     k3s server
31713/tcp open     k3s server

        From the above ports I would say you only want 22, 80, 443 and maybe limited 6443 open. Maybe we can combine iptables-persistent with the iptables binary of k3s by symlinking/update-alternatives.

        Edited by Simon
      • Varac
        Varac @varac ·
        Author Contributor

        Hi @simon you're completely right, unfortunatly we discussed this on the MR and not here: https://open.greenhost.net/openappstack/openappstack/-/merge_requests/447#note_26903

        So it's sth we need to look into with more time, that's why this got moved to %"0.7.1"

      • Varac
        Varac @varac ·
        Author Contributor

        Maybe we can combine iptables-persistent with the iptables binary of k3s by symlinking/update-alternatives.

        That would be an interesting approach. Until now I thought the best way would be to bind all services only to localhost which doesn't need to be exposed, then we would not need a firewall config anymore.

      • Please register or sign in to reply
    • Maarten de Waard created merge request !447 (closed) to address this issue

      created merge request !447 (closed) to address this issue

    • Maarten de Waard mentioned in merge request !447 (closed)

      mentioned in merge request !447 (closed)

    • Maarten de Waard changed milestone to %0.8.0

      changed milestone to %0.8.0

    • Varac added 1 deleted label

      added 1 deleted label

    • Ana Aviles
      Ana Aviles @ana ·
      Owner

      Having those ports exposed can be a security risk, so we shoudl fix it. At least for now @simon wrote an easy fix using iptables-legacy together with nfs tables that rancher provides.

      We would like to apply this to all running instances

      Edited by Ana Aviles
    • Varac
      Varac @varac ·
      Author Contributor

      So there was some confusion about the state of the firewall in OAS.

      • oas.gh was left without a firewall, with open ports. Ops noticed and hot-fixed it in place (/usr/local/bin/simplefw)
      • !447 (closed) is about to remove the firewall, and /usr/local/bin/simplefw should be part of it
      • The related droplet to !447 (closed) should be fixed soon (or taken down, since it exposes ports)
      • No other hosts need to get fixed imo
      Edited by Varac
    • Varac removed 1 deleted label

      removed 1 deleted label

    • Varac mentioned in issue #923 (closed)

      mentioned in issue #923 (closed)

    • Varac marked this issue as related to #923 (closed)

      marked this issue as related to #923 (closed)

    • Varac
      Varac @varac ·
      Author Contributor

      I just realized that debian bullseye ships a newer version of iptables (1.8.7-1 instead of busters 1.8.2-4), which according to this post solves the issue. This is why I'd like to prioritize #923 (closed)

    • Varac added Blocked label

      added Blocked label

    • Varac
      Varac @varac ·
      Author Contributor

      Blocked by #923 (closed)

    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    • Loading
    Please register or sign in to reply