Nextcloud apache out of memory on oas.gh and oas-gh
root@oas:~# journalctl | grep -i 'out of memory'
Oct 21 12:47:03 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 894092 (apache2) total-vm:196676kB, anon-rss:33656kB, file-rss:13696kB, shmem-rss:64kB, UID:33 pgtables:228kB oom_score_adj:990
Oct 21 12:47:05 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254418 (apache2) total-vm:120440kB, anon-rss:31288kB, file-rss:12540kB, shmem-rss:64kB, UID:33 pgtables:216kB oom_score_adj:990
Oct 21 12:47:10 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254416 (apache2) total-vm:191288kB, anon-rss:28216kB, file-rss:16648kB, shmem-rss:64kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 22 16:26:56 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 890216 (apache2) total-vm:197248kB, anon-rss:34156kB, file-rss:15644kB, shmem-rss:64kB, UID:33 pgtables:232kB oom_score_adj:990
Oct 22 17:55:46 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254346 (apache2) total-vm:196660kB, anon-rss:33608kB, file-rss:13760kB, shmem-rss:64kB, UID:33 pgtables:232kB oom_score_adj:990
Oct 22 20:14:25 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254392 (apache2) total-vm:199164kB, anon-rss:35860kB, file-rss:15556kB, shmem-rss:64kB, UID:33 pgtables:236kB oom_score_adj:990
Oct 23 08:30:45 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254417 (apache2) total-vm:193700kB, anon-rss:30844kB, file-rss:15716kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:47 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254393 (apache2) total-vm:193556kB, anon-rss:30716kB, file-rss:15776kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:48 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254419 (apache2) total-vm:193504kB, anon-rss:30844kB, file-rss:15204kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:50 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254468 (apache2) total-vm:193108kB, anon-rss:30616kB, file-rss:15492kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:53 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1253543 (apache2) total-vm:193032kB, anon-rss:30528kB, file-rss:15556kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:54 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1254467 (apache2) total-vm:193220kB, anon-rss:30884kB, file-rss:14932kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:56 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3931107 (apache2) total-vm:193120kB, anon-rss:30572kB, file-rss:15036kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:57 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 890215 (apache2) total-vm:192792kB, anon-rss:29900kB, file-rss:13760kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 23 08:30:58 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 389446 (apache2) total-vm:192220kB, anon-rss:29712kB, file-rss:13624kB, shmem-rss:64kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 23 08:30:59 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320029 (apache2) total-vm:116076kB, anon-rss:25160kB, file-rss:12276kB, shmem-rss:64kB, UID:33 pgtables:208kB oom_score_adj:990
Oct 23 08:31:00 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320027 (apache2) total-vm:116076kB, anon-rss:24636kB, file-rss:12276kB, shmem-rss:64kB, UID:33 pgtables:208kB oom_score_adj:990
Oct 23 08:31:01 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320003 (apache2) total-vm:114136kB, anon-rss:23484kB, file-rss:12536kB, shmem-rss:64kB, UID:33 pgtables:204kB oom_score_adj:990
Oct 23 08:31:02 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320030 (apache2) total-vm:114028kB, anon-rss:24636kB, file-rss:12276kB, shmem-rss:64kB, UID:33 pgtables:204kB oom_score_adj:990
Oct 23 08:31:03 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320028 (apache2) total-vm:114028kB, anon-rss:24636kB, file-rss:12276kB, shmem-rss:64kB, UID:33 pgtables:204kB oom_score_adj:990
Oct 23 08:31:04 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3319945 (apache2) total-vm:114136kB, anon-rss:24800kB, file-rss:12536kB, shmem-rss:64kB, UID:33 pgtables:204kB oom_score_adj:990
Oct 23 08:31:05 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320327 (apache2) total-vm:116076kB, anon-rss:24636kB, file-rss:12276kB, shmem-rss:64kB, UID:33 pgtables:208kB oom_score_adj:990
Oct 23 08:31:06 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320625 (apache2) total-vm:116084kB, anon-rss:24676kB, file-rss:12284kB, shmem-rss:64kB, UID:33 pgtables:208kB oom_score_adj:990
Oct 23 08:31:07 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320623 (apache2) total-vm:116084kB, anon-rss:26500kB, file-rss:12284kB, shmem-rss:64kB, UID:33 pgtables:208kB oom_score_adj:990
Oct 23 08:31:08 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320707 (apache2) total-vm:118124kB, anon-rss:27540kB, file-rss:12348kB, shmem-rss:64kB, UID:33 pgtables:212kB oom_score_adj:990
Oct 23 08:31:09 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320624 (apache2) total-vm:118132kB, anon-rss:28340kB, file-rss:12412kB, shmem-rss:64kB, UID:33 pgtables:212kB oom_score_adj:990
Oct 24 06:19:43 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320754 (apache2) total-vm:193556kB, anon-rss:30772kB, file-rss:15704kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 24 06:19:48 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320915 (apache2) total-vm:195544kB, anon-rss:32260kB, file-rss:15696kB, shmem-rss:64kB, UID:33 pgtables:228kB oom_score_adj:990
Oct 24 06:20:09 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320913 (apache2) total-vm:193100kB, anon-rss:30584kB, file-rss:15492kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 24 06:20:19 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3320735 (apache2) total-vm:193028kB, anon-rss:30516kB, file-rss:15552kB, shmem-rss:64kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 25 09:21:38 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1841992 (apache2) total-vm:192848kB, anon-rss:30288kB, file-rss:15592kB, shmem-rss:68kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 25 09:21:39 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1838423 (apache2) total-vm:192548kB, anon-rss:29796kB, file-rss:13976kB, shmem-rss:68kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 25 09:21:41 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1836636 (apache2) total-vm:192548kB, anon-rss:29808kB, file-rss:13960kB, shmem-rss:68kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 25 09:21:42 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 1836635 (apache2) total-vm:192292kB, anon-rss:29788kB, file-rss:13976kB, shmem-rss:68kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 25 09:21:43 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 2493408 (apache2) total-vm:192788kB, anon-rss:29628kB, file-rss:15412kB, shmem-rss:68kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 25 09:21:45 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 2493407 (apache2) total-vm:196812kB, anon-rss:33360kB, file-rss:15328kB, shmem-rss:68kB, UID:33 pgtables:232kB oom_score_adj:990
Oct 25 09:55:39 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 2493406 (apache2) total-vm:197500kB, anon-rss:34128kB, file-rss:16404kB, shmem-rss:68kB, UID:33 pgtables:232kB oom_score_adj:990
Oct 27 10:52:45 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 2493404 (apache2) total-vm:193368kB, anon-rss:30844kB, file-rss:14864kB, shmem-rss:68kB, UID:33 pgtables:224kB oom_score_adj:990
Oct 27 18:07:38 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3015088 (apache2) total-vm:193484kB, anon-rss:30688kB, file-rss:7088kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 27 18:07:43 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3015085 (apache2) total-vm:192900kB, anon-rss:30076kB, file-rss:7116kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 27 18:07:47 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3021080 (apache2) total-vm:192296kB, anon-rss:29752kB, file-rss:7244kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 27 18:07:53 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3016900 (apache2) total-vm:192556kB, anon-rss:30008kB, file-rss:7072kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 27 19:01:37 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3612856 (apache2) total-vm:192972kB, anon-rss:30308kB, file-rss:7072kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 27 19:01:42 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3667406 (apache2) total-vm:192520kB, anon-rss:29816kB, file-rss:7224kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
Oct 27 19:01:51 oas.greenhost.net kernel: Memory cgroup out of memory: Killed process 3015084 (apache2) total-vm:192680kB, anon-rss:30072kB, file-rss:7060kB, shmem-rss:68kB, UID:33 pgtables:220kB oom_score_adj:990
The problem is that we have 2 pods that run apache: wordpress and nextcloud. I tried to find out which is what, and found interesting new methods how to find out which process belongs to which pod, but I don't know how to do this retrospectively.
From https://stackoverflow.com/a/48922355:
systemd-cgls will show all our cgroups and the processes running in them in a tree-view, like this:
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
├─docker
│ ├─070a034d27ed7a0ac0d336d72cc14671584cc05a4b6802b4c06d4051ce3213bd
│ │ └─14043 bash
│ ├─dd952fc28077af16a2a0a6a3231560f76f363359f061c797b5299ad8e2614245
│ │ └─3050 go-cron -s 0 0 * * * * -- automysqlbackup
Or:
root@oas:~# ps x | grep apa
3003087 ? Ss 1:09 apache2 -DFOREGROUND
root@oas:~# nsenter -t 3003087 -u hostname
nc-nextcloud-687577666b-rhwbh
Anyone has an idea ?
Edited by Varac