Enter unit testing. Unit testing gives you a way to build yourself a library of examples that are verifiable, executable, and self-documenting. This technique isn’t new. 4 years ago, Mike Clark wrote the best material I’ve found on the subject:

•    Ruby Learning Test #1: Are You There, World?
•    Ruby Learning Test #2: Active Reading

I only found these posts about 6 months ago, however, and I suspect many people haven’t seen this technique. I’ve found it to be a great way to practice in a safe environment rather than subjecting production code to my experiments. It’s the code equivalent of learning to play the daf in a sound studio: You can record yourself, use reference materials, and do as many takes as you want. And you won’t disturb the neighbors.

Andy Hunt of the Pragmatic Programmers has a blog post on test-driven learning in which he talks about mixing the technique with concepts like the spacing effect and spaced presentation. “Spaced repetition is a learning technique that exploits this psychological principle by incorporating increasing intervals of time between subsequent review of previously learned material.” I’m interested in exploring how this system could be used in mastering software API’s. In the meantime, I’ll keep working on test driving my way through a few more API’s and will blog any interesting surprises I find along the way.

Application code is one part of the system. So are the database, the configuration, the network, the documentation, and the host machine, to name a few others. One by one, these systemic concerns are being pulled into the CI process.

As the technical roadblocks to automation are removed, we’re also solving the political challenges: Operational silos are being knocked down in favor of collaboration. Product managers are thinking about how to structure features such that they can add value incrementally. Rapid delivery of high quality, valuable software is what agile methods is all about, and bringing automated deployment into the CI process is a natural extension of any agile process.

Just as Scrum introduced an “expanded definition of done”, the definition of “integrated” is expanding. Deployment is traditionally considered something that happens after the CI build. “Build, test, deploy”. I think we’re all coming around to the realization that we really need a “build, deploy, test” CI process in which we deploy the application to a staging environment before we run our functional tests, for example. Since this deployment is part of the CI build, this has to be automated. If we’ve solved deployment to a staging environment, why not deploy to production?

SmartFrog’s Patterns of Deployment includes the CI server right in the definition of continuous deployment. This theme resonated at CITCON NA 2009, with the discussions frequently turning to configuration and deployment. I mentioned I’d post the outcome of the discussion I facilitated on continuous deployment we had there. Here’s the first part, a summary of the top ten indications that continuous deployment makes sense for your team:

* To more quickly add value; get the code into the hands of users; remove inventory - inventory is waste.

* To elicit faster user feedback about defects, quality of the application, and value. Bugs should be fixed. Quality issues should be addressed immediately. Features that don’t add value should be dropped.

* For rehearsal; practice shipping to production using the same automation used to ship to a staging environment. Remove bottlenecks through early identification.

* The organization values happiness. How do people want to spend their time? Fighting fires and fixing production bugs isn’t fun. Why not spend the time up front to automate, rehearse, and avoid the “Last Mile Problem” that occurs when code piles up and before shipping? The organization must be willing and empowered to acknowledge and remove roadblocks.

* The cost of a failed deployment isn’t catastrophic - manual review gates in between automated steps might be deemed necessary as a safety valve in order to catch errors that can’t be automatically detected in the test scripts. Automation reduces risk, though, so even apps with a high cost of failure can benefit from automation as long as there is an element of control and auditing.

* Support costs of a new release are low. Companies don’t want to support hundreds of versions of a COTS product on store shelves, whereas a hosted web application effectively only has one live version at any given time.

*The licensing / revenue model is compatible with frequent releases. Subscription models, free upgrades within a minor revision, etc.

* The users cost of consumption is low, or users can opt to not upgrade. The previous point was about the cost of a deployment to the developing company. This point is about the cost, whether cognitive or financial, for consumers to make use of the new features as they roll out. An application that requires extensive training is obviously harder to continuously deploy than a chat avatar, for example.

* There aren’t complex legal issues with deployment. A new feature might have regulatory implications. Perhaps SOX compliance must be proven before a feature can be released. Situations like this aren’t the norm, but they do happen.

* The team is capable of excellence. One unreliable release will cost trust, user loyalty, and team morale. Too many unreliable releases will cost the product line and possibly the company. This is true regardless of whether or not automation used, obviously, but when we’re talking about continuous deployment with releases that can happen many times a day, we’re hopefully talking about feature releases, not just bug fixes. A solid development process that includes CI is a pre-requisite; I wouldn’t recommend continuous deployment to amateurs who haven’t yet written their first unit test.

With the justifications for deployment being a core part of continuous integration, I’m up for moving on to how it’s done. Now that distributed enterprise CI servers are becoming the norm, we can have a deployment agent on the machines in our QA and production environments that can grab our tested artifacts and run the same deployment scripts used in testing, for example. This is only one approach, though, and I’ll save a more in-depth discussion for another post.

The downtime and confusion that would ensue could hardly be considered agile.

Of course, this isn’t what’s intended by the terms ‘continuous integration’ or ‘continuous deployment’. Continual means ‘frequent, repeating at intervals’ and continuous means ‘going on without pause or interruption’. So shouldn’t we be talking about ‘continual integration’?

“Continuous Integration increases your opportunities for feedback. Through it, you learn the state of the project several times a day. CI can be used to reduce the time between when a defect is introduced and when it is fixed, thus improving overall software quality.” - Continuous Integration: Improving Software Quality and Reducing Risk

As agile developers, we choose to build and deploy as frequently as we can in order to maximize feedback and the opportunities for learning about the state of a system that is otherwise obfuscated. An interval as often as every check in seems to be a reasonable compromise between frequent feedback and absurdity.

The point of the term continuous isn’t to be linguistically precise: It’s to act as a challenge to improve our process by automating as many of our release management tasks as is practical. It’s about a commitment you make with your team to keep the build green and to fix broken windows as soon as they appear. Is “post check-in integration” going to change your team? Is “pretty frequent deployment” going to revolutionize your organization? Somehow those terms don’t quite have the same call to action to them, do they?

The benefits of continuous integration have been well-documented. Applying the same automation challenge to deployment is relatively new, but it makes sense. As continuous integration breaks out of it’s agile home ground and goes mainstream, more people are realizing that they’re already producing their release candidates as part of the build, so why not actually deploy the same way, using nothing more complicated than scripts and a CI server?

The why’s and why not’s of continuous deployment took up an hour when I facilitated a discussion with about 50 participants at CITCON NA 2009 this weekend; I’ll be posting the results of that discussion soon.

I’m happy to see this happening: It validates a lot of the theory we built into Cruise. The Cruise team and others at ThoughtWorks are as interested in deployment as they are in keeping the builds green. Getting code to production after it passed through successive stages of automated and manual testing fits with a build pipeline metaphor, so the pipeline became the core abstraction in Cruise.

The Cruise team uses Cruise as a CI and deployment server: There’s a UAT environment used by the Cruise development team, a pre-production environment used by about 100 ThoughtWorks developers, and the public release that’s sold to the world. Although the deployment was automated through Ant scripts, we left the gates between the deployment stages set to manual to make things a bit more predictable for the development team and, more importantly, our early users. A post on the XP mailing list I received at the beginning of my Cruise tenure presaged this outcome:

On November 27, 2007,  

The biggest barrier to continuous deployment
is gaining the trust that what you're going to
deploy isn't going to destabilize the environment.
This takes time and effort.

John Roth

Indeed. No developer wants the CI server to go down during the workday. We were using Cruise to build itself within a couple of iterations of starting the project, but of course the product took some time to stabilize. We rolled out our first production release two weeks after we started using Cruise in UAT. Once our user base increased, many felt that the risks of the production server being down due to a failed deployment outweighed the benefits of getting new features and bug fixes out as they became available: Users were perfectly content to wait until we’d fielded it in UAT, and even the dev team didn’t necessarily want to use every check in; it was enough to know that we could trigger a deployment on demand, as we frequently did throughout an iteration.

Should a CI server be concerned with deployment? I think so. Why do you use a CI server in the first place? Immmediate feedback. Visibility. Reporting. The peace of mind that follows. CI also implies automation, and that means executable documentation, repeatability, reduction in errors.

Don’t we also need all of these benefits in our deployment process? For many new to CI, the continuous integration process leaves off after testing, contributing to the “Last Mile” problem. I’ve seen some cases where the supposed CI build doesn’t have any tests! I hoped these were isolated incidents, but there are enough experience reports to the contrary that lead some to ask whether or not we need standards for CI. Assuming you’ve got a working CI build, why not go to the next level and deploy using the CI server?

Cruise, AnthillPro, and ControlTier have the concept of deployment baked right in. There are doubtless other servers offering these features. Either we’re all off on a tangent, or were on to something and Continuous Integration (CI) is becoming Continuous Integration & Deployment (CID). Maybe in a few years, leaving deployment out of your CI solution will seem gauche as leaving out tests.

There’s a lot more to say about how the challenge of continuous deployment affects the software lifecycle and how it fits with more established configuration management practices in CMMI and ITIL. Although I’ve since left ThoughtWorks, I’m working on a continuous deployment solution for my new employer, DRW Trading. I’ll continue to post more of my thoughts on CI and CD, and I’m looking forward to discussing more of these ideas with members of the community at CITCON North America in April.