7.1.1 Your live webapp is never in a state that can't be rolled back to in the future.

The basic idea here is that at any given point in time your site is probably in a working state, but it's only a matter of time before some hideous bug rears it's head. Frequently that hideous bug was introduced in a recent update to the site. Obviously you want to just fix the bug and update the site, but sometimes the bug can require far more time to fix than you are willing to allow your live site to be broken for. So, you have to roll back to the last known working version of your site until you can get a patch ready and tested. If you've got your site deployed on multiple boxes you need to be able to roll all of them back.

I've seen, and been responsible for, far too many sites where when a bug was found a fix would be made, but since we didn't have good branching habits, because we had tools that made it a pain, we would make that fix in the development branch, in the middle of half-finished new functionality. We obviously couldn't roll that out so we'd check in the patch and then upload just the affected files to the server(s). Ignoring the obvious problem that the dev branch might have been so altered that a fix there might not fix the bug on the server, we have the problem that, assuming our patch works well, when we go to roll out the new features in a week or so there is no way we'll be able to roll the server back to that patched, and working, state if we need to because it was an essentially random collection of files from various points in development.

The solution is simple, but depends on you having good branching habits. You have your production branch, you make your changes elsewhere and when they're done merge them into the production branch. Then, if the merging process didn't already do this, you tag this newly patched version of the production branch in such a way that you can easily remember/find it should you ever need to roll back to it. And, of course, you only push code from the production branch to the live server(s).

K. Rhodes 2007-05-18