Below is an extract of the continuous deployment strategy that I and my team used at Hulu over the last year and a half or so. It was extremely successful for us and allowed us to do continuous development, where as soon as we have something production ready, it hits production with little friction. The results of us doing this were that every developer was empowered to ship at their own pace, as often as possible. And because developers were in charge of the shipping process, it forced all of us to rely a lot more on automation & testing for pre-verification.
We’ve had days where we shipped to production more than 30 times in a 24 hour period, and we rarely had bad push incidents. Even when we did have bad releases, the fix was always just a push away :).
We are using a branching model inspired by ( http://nvie.com/posts/a-successful-git-branching-model/). We use development integration branches and release branches. The integration branches are shared development branches for sharing code, and release branches are branches which get deployed to our various environments. Code changes should originate only from integration branches, and then be propagated to the deployment branches. Both branch types are quickly outlined below.
The develop branch is a branch for integrating feature-complete code or fixes. It is considered unstable, but that does not mean broken . Run tests before committing to the develop branch. Most of us use develop for their day-to-day development. Most changes should originate from develop or if small production bugfixes are done off of master, they should quickly be integrated back into develop.
The master branch is a stable branch. All commits to master go through CI server, and tests are run after every push. The master branch is just an integration branch, which means no deployments are done from it. Changes should be pushed to master, generally only after being code reviewed and pushed to the test environment and tested there.
The hotfixes branch is a stable branch off of master for integrating small bugfixes that need to make it to production without tagging all of our development changes with them. All commits to the branch go through CI server, and tests are run after every push. The hotfixes branch is just an integration branch, which means no deployments are done from it. Changes should be pushed to hotfixes, generally only after being code reviewed and pushed to the test environment and tested there.
The cr branch is an auxiliary branch, used for pushing changes pending code reviews. This branch is useful in cases, for example, when remote teams would make changes during the night, and would then submit for code review, and which the next day, the core team can code review, rebase to head of develop and merge in, while remote team is sleeping. This is very useful, as during the day, there are number of changes that happen in the integration branches, and not integrating the code early, could cause merge conflicts. The use of this branch tries to reduce turnaround of code-changes merged into the integration branches.
The test branch is a release branch for our test environment. The test environment is generally for testing development changes in a real environment, and should not be used/referenced outside of the team. The environment should be deployed via CI, which will automatically run all available tests before deployment. Code changes should not originate from this branch. All changes in this branch should come from a merge of the origin/develop branch.
The stage branch is a release branch for our stage environment. The stage environment should be used for final verification of changes ready to be pushed to production. The branch should be kept as stable at all times, as its deployed environment is being used and is a dependency to other staging services and clients. The environment should be deployed via CI, which will automatically run all available tests before deployment. Code changes should not originate from this branch. All changes in this branch should come from a merge of the origin/master integration branch or from origin/develop for verifying more unstable changes.
The prod branch is a release branch for our production environment. The prod environment is obviously running production traffic. There are a variety of monitoring tools hooked up to the prod environment. In case you push broken changes to origin/prod, you’d likely hear from them. The prod branch should be kept stable at all times. The environment should be deployed via CI (or git push), which will automatically run all available tests before deployment. Code changes should not originate from this branch. All changes in this branch should come from a merge of the origin/master integration branch, after first pushing them to origin/stage and testing them in the staging environment.
A change workflow
Here is how to propagate a change from being ready ( in develop ) to production.
Integrate in develop:
git push origin develop
Send for code review to the core team:
rake codereview origin/develop
Merge and push to test. This will also deploy the change there , after running tests.
git checkout -b test origin/test
git merge develop
git push origin test
Test the change
Merge into master
git checkout -b master origin/master
git merge develop
git push origin master
Merge and deploy staging. git push will deploy you.
git checkout -b stage origin/stage
git merge master
git push origin stage
Test the change. Generally try and wait at least half an hour before deploying to prod to weed out any issues. There is a log-replay that takes production traffic and replays to stage, so if something is broken, you should be able to quickly spot it.
Merge and deploy to prod. git push will deploy you.
git checkout -b prod origin/prod
git merge master
git push origin prod