Is Continuous Deployment Good for Users?

| 11.4.2009

The recent release of Windows 7 got me thinking about development cycles. For those of us who suffered through the last 2+ years of Vista, Windows 7 has been a welcome relief from the lagging, bugs, and constant hassle of a failed operating system. Overall, as a customer, I’m pretty happy with Windows 7. But, at least on my part, there is still some latent anger – if Windows 7 hadn’t been quite as good as it seems to be, they would have lost me to Apple. They still might.

A big part of my unhappiness is the fact that I had to wait for more than two years before they fixed my problems. That’s a lot of crashes and frustration to forget about.

One approach that many software companies have been adopting to combat the huge lag time built into traditional software releases is something called continuous deployment. This sort of deployment means that, instead of having large, planned releases that go through a strict process and may take months or years, engineers release new code directly to users constantly, sometimes multiple times a day. A “release” could include almost anything: a whole new feature, a bug fix, or a text change on the landing page.

I worked with a software development organization that practiced continuous deployment on a very large, complicated code base, and I can definitely say, the engineers loved it. From the point of view of the employees, continuous deployment was a giant win.

But how was it for the users? The fact is, some decisions that seem like they only affect engineering (or marketing, business, PR, etc.) can actually have a huge impact on end users. So, whenever organizations make decisions, they should always be asking, “how might this affect my customers, and how can I make it work best for them?”

Is Continuous Deployment Good For Users?

As with so many decisions, the answer is yes and no. Continuous deployment has some natural pros and cons for the customer experience, but knowing about them can help you fix the cons and benefit even more from the pros.

Big Customer Wins

Fast Bug Fixes

Perhaps the biggest win for users is that bugs can get addressed immediately. Currently, even Microsoft releases patches for some of its worst security holes, but there is certainly a class of non-critical, but still important bugs that have to wait until the next major release to get addressed. That means weeks, months, or even years of your users dealing with something broken, even if the fix is simple. In continuous deployment, a fix can be shipped as soon as it’s done.

Fast Things vs. Slow Things

Similar to the first point, continuous deployment lets you get everything, not just bug fixes, to users as soon as they’re ready to go. Small features, easy changes, and UI tweaks don’t have to wait for larger, unrelated features to be released to customers. After all, should a new design for the splash screen really have to wait on the implementation of a whole new payment system?

More Opportunities for Community Involvement

If you’re having a constant dialog with your customers (you are, right?), they’re probably making some pretty good suggestions about problems they’re having or ways to improve the product. A by-product of the first two benefits is that those users are going to feel even more involved in the development process when their suggestions or concerns are dealt with quickly, rather than if they have to wait months or even years for the next major release.

(Mostly) Avoidable Customer Problems

As with anything, continuous deployment can also cause some problems for users. Of course, some of these problems can exist in big, staged deployments as well, but these are a few things in particular to watch for.

Constant Change

Imagine if every day, the layout on your car changed, sometimes slightly, other times drastically. You want to drive to the store, but the steering wheel is on the other side and you’ve suddenly got an extra pedal. It would make it a lot harder to get where you were going, wouldn’t it?

Well, presumably your users are using your product to get something done, and they’ve got a certain way that they’ve learned to do it. Continuous deployment can mean that the interface for your product can change at any moment, even several times a day. If features constantly appear and disappear, it can be very disruptive to your users’ process.

There are a few things you can do to minimize the disruption. First, make sure that you’re testing your biggest changes on small cohorts of people. Iterate on a subset of your user base, rather than hitting every single user with every single change. This will limit the change that any individual sees while still giving you the benefit of constantly pushing code to customers.  In fact, use this as an opportunity to do the A/B testing you should be doing anyway.

Also, and this should be obvious, try to limit your truly disruptive user interface changes so that things don’t feel like they’re in constant flux. You can still change things, but be aware of how frequently you’re making major changes and stay in contact with your users to make sure they’re not feeling dizzy.

Inconsistent UIs

When a big new release is planned, often there is a comprehensive design phase where all the new changes are mapped out and discussed. This means that any inconsistencies in the UI can be found and addressed.

In continuous deployment, different pieces of the product are getting built and shipped to customers all the time, and there is rarely a time when the entire UI is reevaluated as a single entity. This means that sometimes UI standards can tend to…oh, let’s say evolve.

This problem can be controlled by having a UX team member embedded in the development team and constantly working with the engineers to enforce standards before things ship to customers. It can also be improved by providing wireframes, visual designs, and tools like templates to developers so that the look and feel of the product doesn’t shift too dramatically over time.

Avoiding QA

I’m certainly not claiming that every single thing in a traditional release process gets a full QA pass, but I do find that continuous deployment makes it easier for code to slip out to users without any human testing at all. Any time you give engineers the ability to ship code directly into production, you’re tempting fate. Somebody’s going to say, “oh, it’s just a tiny change,” and it’s going to get out without any testing. I’m not naming any names, but you only have to have a tiny change break the entire product once before you realize that there’s no such thing as a tiny change.

Also, continuous deployment can make certain types of testing much less likely to happen. While large release cycles tend to have a code freeze and weeks or months devoted to testing, hopefully including regression, unit tests, and end to end testing, continuous deployment doesn’t necessarily have that baked into the process.

You can always add periodic end to end testing of the product to your own process, of course, and it can be quite helpful in improving code quality, especially when your engineers occasionally slip through a “tiny change.”

Communication Issues

When you’re constantly releasing new features and bug fixes, communication with your users can be a challenge. You don’t have one big release with new help docs, a big roll out plan, and an advertising campaign. Instead, stuff is coming out all the time, and users can get overwhelmed by keeping up with the changes.

Context sensitive help and inline information for each feature can help users get quickly oriented.  Also, clearly marking new features as alpha or beta can let users know when a feature is still being developed so they can set their expectations accordingly.

Documentation can also easily get out of date when things are getting released constantly. Big, staged development cycles often have a built-in time for creating documents. Typically, manuals or help documentation and FAQs go through QA sometime after code freeze and before release. But since you’re not necessarily doing a big, monolithic release with continuous deployment, you can end up with this material never getting any sort of end to end editing to make sure they stay current. Make time for this. It’s good for both your customer experience and your customer support team.

Frequent Downloads and Updates

While continuous deployment is quite natural with web applications, even downloaded products can be constantly updated. However, you should always be aware of the burden you’re placing on your user base. If you’re forcing people to download a large file and go through an installation process too often, you’re going to annoy people.  As a personal example, iTunes appears to have a new version every week, and I’ve started to flinch every time I open it.

There are a few things you can do to make downloads easier on your users. First, you can ask the user to allow the update to download in the background and install automatically the next time the product opens. This means that the updating happens with very little user annoyance. Also, it’s best to keep the update quick by making the downloads incremental. For example, your virus protection software probably updates its virus information daily without asking you to reinstall the entire product every time.

So, Is It Good for Users or Not?

Continuous deployment can be done in a way that’s good for both engineering teams and users, but you do need to take some precautions. By taking care when you introduce changes that your users will really notice and making sure that you make time for important processes like QA, you can get features out faster and constantly improve your product. And that is very good for users.

Interested? You should follow me on Twitter.

For more information on the user experience, check out:

Share with

8 Comments

  1. Great article!

    I think one of the problems people have in thinking about continuous deployment is that they confuse pushing code with releasing features.

    There’s no particular reason releasing new code to your servers has to coincide with making new features visible to users. Indeed, if you’re already doing A/B testing, it’s a short technical step to create an interface for product managers to turn features on, off, or various states in between.

    One good example is YouTube. They strongly value a good user experience, but they also need to innovate rapidly. So every new feature goes out into fractional release, with only a small percentage of users seeing it at first. If there are big problems, the feature is switched off. More likely, the feature is tweaked based on use data and user testing until it’s ready for prime time, at which point they turn the dial up to full availability, with no code push needed. Eventually, as with finished A/B tests, engineers remove the extra code and the feature becomes just another part of the system.

    As far as testing goes, I think the right answer is heavy investment in automated testing, combined with the continuous on-team presence of exploratory testers and product managers. For anybody who needs to keep innovating, there’s no economic case for long-term manual QA.

    posted by William Pietri at 2:03 pm on 11.05.09
  2. Great post! One more personal example that comes to mind of a product with too many frequent download/update requests is Adobe. Their “updater” appears constantly on my computer– to the point that I’m always irritated when I see it and think it hurts their brand.

    posted by Mia Silverman at 2:49 pm on 11.05.09
  3. Thanks for the comments! William, you make a great point that pushing code doesn’t always have to have a visible impact on all users. Making an investment, like YouTube does, in tools that allow you to have good control over who sees what when will go a long way toward making sure that continuous deployment doesn’t have a negative impact on your customers.

    Thanks again!

    posted by Laura Klein at 7:21 pm on 11.09.09
  4. [...] about Agile « Zen Agile Bringing balance and harmony to your project 2 Likes Bread Board » Is Continuous Deployment Good for Users? 2 Likes Webcasts http://www.on24.com – Online events, with audio and video [...]

  5. Your comments about switching to the Mac are specious, most of your customers on the PC, so get used to it.

    posted by Jak at 12:06 pm on 11.19.09
  6. If anyone gets frustrated waiting for updates and fixes, there is another solution. Try Linux! After all, it’s free, the updates come automatically when they are ready for usage. OS error corrections can be made in sometimes the same day, or in just a couple of days. Quite a lot faster than MS. Linux doesn’t have to go through the bureaucratic layers that happens in MS to get things done. It is not subject to viruses, and malware. And the stability of the OS is great. A great pletora of various applications are available. And they are all free! In most cases with MS, you have to buy extra applications that you may need. Since I’ve been running Linux and FreeBSD, I’ve never had to cope with malicious attacks, and viruses, and trojans.

    Oh, and when an application is downloaded and installed, one doesn’t have to reboot to get it going. Simply install and keep on doing your thing.

    There are all kinds of flavors of Linux. Somewhere out there,there is one version that will fully suit your needs. If you need something for laptops, I’d recommend PCLinuxOS, or DesktopBSD. Those two OS’s have a great and easy GUI wireless application to use. I can set it up 5 minutes.

    Of course if you are not yet sure that you want to stick with Linux, and you still want to keep your Windows in the background (just in case), Linux has a great boot-loader manager, called GRUB. With that, you can dual boot to either of the OS’s, either in separate hard drives, or both in the same hard drive.

    I’m simply happy to announce that there are more choices than just being a slave to an expensive closed source OS. And Linux doesn’t weight you down with costly licenses. One copy can be installed in as many machines as your heart desires. Linux and BSD Unix are both open source systems. You get all the tools needed to make the changes to suit your requirements.

    Happy Computing!

    posted by Onederer at 11:12 pm on 11.19.09
  7. It is clear that you are not speaking of large complex industrial software such as factory automation software or in fact any production critical software or mention mission critical Software such as a CAT scanner or avionics. Imagine the chaos that would occur if software was dynamaically donloaded to a cisco core router that didnt take in the account the impact on other non upgraded routers

    posted by Joseph Enright at 2:24 pm on 12.09.09
  8. [...] firm slicedbread on when continuous feature/fix releasesare good and bad for [...]

Leave a comment