Disable re-use of CI machines
For some time now we re-use CI machines for the same branch. The big advantage is that we save time on subsequent CI runs. However, it's a messy hack because we have state on the cluster that sometimes alter the result of the run compared with a fresh installation. Some time we got bitten by that already.
I'm not sure but I think !304 (merged) could be because of this. The final pipeline for !304 (merged) was successful after a total of 9 pipelines. After !304 (merged) got merged into master, the master pipeline fails. This would be bad for normal MRs but for the 0.4 release MR this is really bad.
I don't want to blame only the CI caching, but we had some many issues with it that my trust in our CI setup is low. And I think the price we pay for speed is too high.
So I propose to disable re-using machines. We switched to k3s which speeds up the installation, and we could put more effort into speeding up the CI by using k3os in CI i.e., or other things.