Skip to content

flux source-controller and helm-controller get oom killed frequently

On my cluster (12GB RAM) I have this:

root@oas:~# dmesg | grep -i oom
[45547.064109] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[45547.064144]  oom_kill_process.cold.30+0xb/0x1cf
[45547.064607] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[45547.078042] oom_reaper: reaped process 13045 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[45939.644458] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[45939.644577]  oom_kill_process.cold.30+0xb/0x1cf
[45939.645520] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[45939.666964] oom_reaper: reaped process 26828 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[46338.139759] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[46338.139778]  oom_kill_process.cold.30+0xb/0x1cf
[46338.140057] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[46338.145917] oom_reaper: reaped process 8122 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[50340.586842] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[50340.586873]  oom_kill_process.cold.30+0xb/0x1cf
[50340.587103] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[50340.602356] oom_reaper: reaped process 21289 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54033.497211] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[54033.497242]  oom_kill_process.cold.30+0xb/0x1cf
[54033.497424] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[54033.514797] oom_reaper: reaped process 21422 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54121.061268] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[54121.061331]  oom_kill_process.cold.30+0xb/0x1cf
[54121.062281] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[54121.102105] oom_reaper: reaped process 21136 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54226.451873] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[54226.451891]  oom_kill_process.cold.30+0xb/0x1cf
[54226.452234] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[54226.458414] oom_reaper: reaped process 24968 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54333.799042] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[54333.799059]  oom_kill_process.cold.30+0xb/0x1cf
[54333.799278] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[54333.806485] oom_reaper: reaped process 28920 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

root@oas:~# free -m
              total        used        free      shared  buff/cache   available
Mem:          12009        5097        1649          15        5263        6871
Swap:             0           0           0
Edited by Varac
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information