flux source-controller and helm-controller get oom killed frequently
On my cluster (12GB RAM) I have this:
root@oas:~# dmesg | grep -i oom
[45547.064109] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[45547.064144] oom_kill_process.cold.30+0xb/0x1cf
[45547.064607] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[45547.078042] oom_reaper: reaped process 13045 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[45939.644458] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[45939.644577] oom_kill_process.cold.30+0xb/0x1cf
[45939.645520] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[45939.666964] oom_reaper: reaped process 26828 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[46338.139759] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[46338.139778] oom_kill_process.cold.30+0xb/0x1cf
[46338.140057] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[46338.145917] oom_reaper: reaped process 8122 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[50340.586842] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[50340.586873] oom_kill_process.cold.30+0xb/0x1cf
[50340.587103] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[50340.602356] oom_reaper: reaped process 21289 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54033.497211] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[54033.497242] oom_kill_process.cold.30+0xb/0x1cf
[54033.497424] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[54033.514797] oom_reaper: reaped process 21422 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54121.061268] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[54121.061331] oom_kill_process.cold.30+0xb/0x1cf
[54121.062281] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[54121.102105] oom_reaper: reaped process 21136 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54226.451873] source-controll invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=995
[54226.451891] oom_kill_process.cold.30+0xb/0x1cf
[54226.452234] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[54226.458414] oom_reaper: reaped process 24968 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[54333.799042] source-controll invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
[54333.799059] oom_kill_process.cold.30+0xb/0x1cf
[54333.799278] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[54333.806485] oom_reaper: reaped process 28920 (source-controll), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
root@oas:~# free -m
total used free shared buff/cache available
Mem: 12009 5097 1649 15 5263 6871
Swap: 0 0 0
Edited by Varac