Promtail leaves too many files open
I've run into the following problem:
# kubectl logs -n oas-apps nc-nextcloud-7d4b68744b-vkcpc -f
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.42.0.72. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.42.0.72. Set the 'ServerName' directive globally to suppress this message
[Wed May 06 12:52:40.618714 2020] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.11 configured -- resuming normal operations
[Wed May 06 12:52:40.618734 2020] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
213.108.108.186 - - [06/May/2020:12:52:40 +0000] "GET /status.php HTTP/1.1" 400 1449 "-" "kube-probe/1.15"
failed to create fsnotify watcher: too many open filesroot
I think this is because of promtail, because when I run lsof, I get 213262 lines of this:
promtail 2134 2324 promtail root 125u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 126r FIFO 0,9 0t0 1945396 pipe
promtail 2134 2324 promtail root 127w FIFO 0,9 0t0 1945396 pipe
promtail 2134 2324 promtail root 128r REG 202,1 934 44400671 /var/lib/docker/containers/dc0aaf72f5b1f5741392c2944c3da97f00088694c3c215801b59c8b1f7a7e8b4/dc0aaf72f5b1f5741392c2944c3da97f00088694c3c215801b59c8b1f7a7e8b4-json.log
promtail 2134 2324 promtail root 129r a_inode 0,10 0 1027 inotify
promtail 2134 2324 promtail root 130u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 131r FIFO 0,9 0t0 1945398 pipe
promtail 2134 2324 promtail root 132w FIFO 0,9 0t0 1945398 pipe
promtail 2134 2324 promtail root 133r a_inode 0,10 0 1027 inotify
promtail 2134 2324 promtail root 134u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 135r FIFO 0,9 0t0 1945399 pipe
promtail 2134 2324 promtail root 136w FIFO 0,9 0t0 1945399 pipe
promtail 2134 2324 promtail root 137r a_inode 0,10 0 1027 inotify
promtail 2134 2324 promtail root 138u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 139r FIFO 0,9 0t0 1945400 pipe
promtail 2134 2324 promtail root 140w FIFO 0,9 0t0 1945400 pipe
promtail 2134 2324 promtail root 141r a_inode 0,10 0 1027 inotify
promtail 2134 2324 promtail root 142u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 143r FIFO 0,9 0t0 1945401 pipe
promtail 2134 2324 promtail root 144w FIFO 0,9 0t0 1945401 pipe
promtail 2134 2324 promtail root 145r a_inode 0,10 0 1027 inotify
promtail 2134 2324 promtail root 146r REG 202,1 39976 55287837 /var/lib/docker/containers/6ac90e37df3f3fff1818ceb2b2b0d22195e327832848dcf8068a65bc55a99481/6ac90e37df3f3fff1818ceb2b2b0d22195e327832848dcf8068a65bc55a99481-json.log
promtail 2134 2324 promtail root 147u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 148r FIFO 0,9 0t0 1945402 pipe
promtail 2134 2324 promtail root 149w FIFO 0,9 0t0 1945402 pipe
promtail 2134 2324 promtail root 150r REG 202,1 5793 34152535 /var/lib/docker/containers/7b9d9d5585e3f90e4a21353a515f3ec62bb7fddebe5dae5a0cfe5ad312984123/7b9d9d5585e3f90e4a21353a515f3ec62bb7fddebe5dae5a0cfe5ad312984123-json.log
promtail 2134 2324 promtail root 151r REG 202,1 11065 31015063 /var/lib/docker/containers/8580838d439c66f99c65d6086b6220e2e82bd3033bc002d1716b9e55fd63f7ad/8580838d439c66f99c65d6086b6220e2e82bd3033bc002d1716b9e55fd63f7ad-json.log
promtail 2134 2324 promtail root 152r REG 202,1 3364 38215733 /var/lib/docker/containers/9fcf5355d8b3fbda098117191070cb85bf389e79979ab8507eaa80d1494494c3/9fcf5355d8b3fbda098117191070cb85bf389e79979ab8507eaa80d1494494c3-json.log
promtail 2134 2324 promtail root 153r REG 202,1 3131 51249174 /var/lib/docker/containers/7080569737b66342a38e68983d450af4f00d22c2b43376876c1a04d9d1d97d23/7080569737b66342a38e68983d450af4f00d22c2b43376876c1a04d9d1d97d23-json.log
promtail 2134 2324 promtail root 154r a_inode 0,10 0 1027 inotify
promtail 2134 2324 promtail root 155u a_inode 0,10 0 1027 [eventpoll]
promtail 2134 2324 promtail root 156r FIFO 0,9 0t0 1945403 pipe
promtail 2134 2324 promtail root 157w FIFO 0,9 0t0 1945403 pipe
promtail 2134 2324 promtail root 158r REG 202,1 1039 34136072 /var/lib/docker/containers/8001c881f4ac438ab861a602a77680a36d6a68aee72a8e023903c4afb4f3a14b/8001c881f4ac438ab861a602a77680a36d6a68aee72a8e023903c4afb4f3a14b-json.log
I have a hunch this might worsen if you reinstall pods a few times. I also have a hunch that this can cause other processes than kubectl logs -f
to fail as well, because my cluster isn't working well at the moment...