Flux upgrade breaks some apps
By upgrading flux from 0.25.3
to 0.28.5
(#976 (closed)), we ran into a flux issue: because the "managed fields" of (some of) our PVCs were not set properly, new supposedly better flux behaviour made it take over ownership of the .spec.volumeName
field of the PVCs, which results in breakage because flux then tries to update the value of that field, which is not allowed (PVCs are mostly immutable).
According to the issue comments, the only known workaround is to patch the resources involved, fixing the "managed field" so flux will no longer try to change the volumeName
. I tried this on the staging cluster, and it worked.
Steps:
kubectl patch pvc -n stackspin-apps wekan --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeName"}]'
flux reconcile ks wekan
kubectl patch pvc -n stackspin grafana --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeName"}]'
flux reconcile ks monitoring
kubectl patch pvc -n stackspin-apps nextcloud-files --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeName"}]'
kubectl patch pvc -n stackspin-apps nextcloud-mariadb --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeName"}]'
kubectl patch pvc -n stackspin-apps nextcloud-onlyoffice-data --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeName"}]'
flux reconcile ks nextcloud
The nextcloud reconciliation timed out for me, and retrying failed, like in wordpress-helm#119 (closed) . The usual solution of deleting the helmrelease and then retrying the kustomization reconciliation worked.
Depending on the age of the cluster and apps on it, maybe fewer or more PVCs could have this problem. From the issue comments it seemed that this would typically occur if the PVCs were created a relatively long time ago, but ours are not that old really (about half a year).
Maarten asked in the issue if there is a way to work around the problem without manually patching.