Automate upgrade testing
Discussed on yesterday's meeting:
We want something that tests whether we can apply a commit to the current release branch. In other words, make sure we don't break anything that would require manual intervention on the cluster. If a commit does break stuff in such a way, we should either make changes to prevent it, or mark the commit for the next major release and include the manual intervention in an upgrade script or upgrade documentation.
A comment below describes my plan.