Automate upgrade testing

Discussed on yesterday's meeting:

We want something that tests whether we can apply a commit to the current release branch. In other words, make sure we don't break anything that would require manual intervention on the cluster. If a commit does break stuff in such a way, we should either make changes to prevent it, or mark the commit for the next major release and include the manual intervention in an upgrade script or upgrade documentation.

A comment below describes my plan.