Restarted pods sometimes fail to init

This happens fairly regularly when wordpress sites are restarted, in particular when a cluster node is rebooted.

Symptom

The wordpress pod goes into a Init:CrashLoopBackOff because the init script fails with

TASK [wordpress-init : Prepare wp-config permissions if it exists] *************
fatal: [localhost]: FAILED! => {"changed": false, "gid": 33, "group": "xfs", "mode": "0440", "msg": "chown failed: [Errno 1] Operation not permitted: b'/var/www/html/wp-config.php'", "owner": "xfs", "path": "/var/www/html/wp-config.php", "size": 3035, "state": "file", "uid": 33}
...ignoring

TASK [wordpress-init : Create wp-config from zero] *****************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "wp --path=/var/www/html config create --dbname=\"wordpress_db\" --dbuser=\"wordpress\" --dbpass=\"...\" --dbhost=\"test-database\" --dbprefix=\"wp_\" --dbcharset=\"utf8mb4\" --dbcollate=\"utf8mb4_unicode_ci\" --skip-check --force --extra-php", "delta": "0:00:00.739157", "end": "2020-12-09 14:42:40.085338", "msg": "non-zero return code", "rc": 1, "start": "2020-12-09 14:42:39.346181", "stderr": "[09-Dec-2020 14:42:40 UTC] PHP Warning:  file_put_contents(/var/www/html/wp-config.php): failed to open stream: Permission denied in phar:///usr/local/bin/wp/vendor/wp-cli/config-command/src/Config_Command.php on line 164\nError: Could not create new 'wp-config.php' file.", "stderr_lines": ["[09-Dec-2020 14:42:40 UTC] PHP Warning:  file_put_contents(/var/www/html/wp-config.php): failed to open stream: Permission denied in phar:///usr/local/bin/wp/vendor/wp-cli/config-command/src/Config_Command.php on line 164", "Error: Could not create new 'wp-config.php' file."], "stdout": "", "stdout_lines": []}

More information

• The path /var/www/html is an emptyDir volume called wordpress-wp-storage.
• The pertinent contents of that emptyDir during the crashloop are like this:

worker1 wordpress-wp-storage # ls -ln wp-config.php 
-r--r----- 1 33 33 3035 dec  9 14:13 wp-config.php
worker1 wordpress-wp-storage # ls -l wp-config.php 
-r--r----- 1 www-data www-data 3035 dec  9 14:13 wp-config.php

This is outside the docker container, on the worker node. In particular note that the user and group IDs 33 apparently belong to the xfs user/group inside the init container.

• The Prepare wp-config permissions if it exists task tries to set the owner/group to {{ wp_user }}/{{ wp_group }}, typically www-data/www-data, which have different IDs inside the container than outside.

added Bug label

A workaround is to delete the failing pod. It will be recreated by the statefulset automatically and in my experience typically the init process runs fine then.

assigned to @maarten

I know why this happens:

-r--r-----    1 xfs      xfs         2.9K Jan  4 12:49 wp-config.php

created merge request !28 (merged) to address this issue

mentioned in merge request !28 (merged)

closed with merge request !28 (merged)

mentioned in commit a38b49be