CI and Automated Deployments

Automating deployments with TeamCity has saved us hours of headaches! Finding the setup that works best for us has taken some time, but now we’re there, the great mist that was hanging over web deployments has lifted and the world feels like a clearer, simpler place.

A few years ago, we were in a bad place.

We didn’t really know it yet, but we were definitely not set up for reliable, repeatable and scalable deployments. In fairness, what we had worked. With a relatively small team and not many applications, keeping on top of things wasn’t too onerous and generally there weren’t many mistakes.

Fast forward a few months and the system was seeing some growing pains. The development/test environment we experiencing contention, people overwriting each other’s changes and no one really knowing what version they were looking at. All deployments were manual, publishing from developer machines and copying the files. In today’s world though, this is practically a criminal offence! (It’s at least morally indefensible…)

Over the coming months and years we embarked on a journey of discovery. For some of us, the journey felt familiar – we’d covered much of the ground before – but for others it was a mysterious (and scary) voyage into the unknown. Today we’re in a better place, and here’s why…

We’re using git for source control.

Previously we had a large and unmanageable SVN repository. Moving to git was challenging, but in the end the benefits are proving to be well worth it. As things stand:

  • The single large repo is now many smaller ones
  • Managing code between branches (dev, release, feature, hotfix, master) is simple
  • Core modules are distributed using internal NuGet, dependencies are nicely managed
  • All repositories are centrally hosted using Stash

Continuous Integration (CI), is the first step for anyone towards salvation. We have TeamCity working hard validating that all our code in all branches builds once it’s checked in, giving us a confidence in our code base that we just didn’t have before. Additional steps run unit tests and create deployment packages. We also use features in TeamCity such as assembly versioning, to allow us to track which version of the code a particular DLL was built from.

Speed up and take out the human

Automating deployments speed up getting code into a working environment, and takes away the human element of deploying code. We have taken advantage of publish profiles and web deploy in Visual Studio and IIS to control where applications are deployed and with what configuration. Web.config transforms make managing multiple environment configurations a breeze, and you get consistent and reliable results. No more searching for hours to find that a setting wasn’t added to a particular web.config file!

Automating this much of the process has allowed us to create more environments, with better controls and clearer defined purposes. A test and production environment are fine in most cases, until you’re doing formal release testing and want to QA features that are for a future release. So we added another environment specifically for this purpose. Want an environment that’s more like production so you can do final validation checks before deploying? Easy, we created a staging environment.

All in all we now have 5 environments, all built from a specific code base:

Environment Built from Purpose
CI “dev” branch For developers to validate their changes with everyone else’s in an integrated environment
Test “dev” branch Owned by QA. For testing features that are in the current development iteration.
Release Verification “release/x.x” branch Owned by QA. For validating a release candidate. Only bug fixes relating to this release get deployed to this environment.
Staging “master” branch Owned by Infrastructure. Pre-live validation environment to perform final checks on a release before it’s deployed to production.
Production “master” branch The real deal

At first glance that’s a lot of environments, but thanks to automation the overheads are minimal compared to the benefits. Deployments are one click and take just a few minutes.

Streamlined Processes

Even so, we streamlined the process of creating new builds and environments. First of all, I want to say that TeamCity and Web Deploy are fantastic tools. If you’re not using them, you should be!

Creating a new set of environments is fairly straightforward. You do have to create the sites in IIS and point them to a place on the file system. You also have to set up the security correctly, but then that’s about it. Web deploy does the copying of files and the configuration side of things for you.

TeamCity allows you to create build configuration templates, which means new builds can be set up in just a few minutes. By selecting the template, then overriding a few parameters, your build is ready to go!

So, the scary part is clearly the production deployment, and there are some things you can do here to make it less scary:

  • Create a short MSBuild script to run before the deploy step that backs up the current production code to a known location. This means you can easily rollback if something’s gone wrong just by copying the files back into place.
  • Secure access to the production build configurations. There’s nothing worse than an accidental or unexpected deployment, and you can minimise the risk of this in TeamCity by having all your production deploy configurations in a separate project. This also means you can restrict access to just your infrastructure team, that way it’s impossible for developers to accidentally push changes into production without authorisation.

And that’s it! It took us quite a while to get here, and we’re by no means in a perfect place, but a lot of the stress and pain is now gone from validating and deploying our web projects. That means we can spend more time creating great products and less time worrying about deployments.

Ralph Cooper, Head of Software Development

Leave a Reply