Well, for one thing, I believe that design should continue for as long as a product is being used by or sold to customers. It’s an iterative process, and there’s nothing that gives me quicker, more accurate insight into how a new product version or feature is performing than looking at user metrics.

But there’s something that I, as a user advocate, care about quite a lot that is really very hard to measure accurately. I care about User Happiness. Now, I don’t necessarily care about it for some vague, good karma reason. I care because I think that happy users are retained users and, often, paying users. I believe that happy users tell their friends about my product and reduce my acquisition costs. I truly believe that happy users can earn money for my product.

So, how can I tell whether my users are happy? You know, without talking to every single one of them?

Although I think that happy users can equal more registrations, more revenue, and more retention, I don’t actually believe that this implies the opposite. In other words, there are all sorts of things I can do to retain customers or get more money out of them that don’t actually make them happy. Here are a few of the important business metrics you might be tempted to use as shorthand for customer happiness – but it’s not always the case:

Retention

An increase in retention numbers seems like a good indication that your customers are happy. After all, happier customers stay longer, right?

But, do you mean retention or forced retention? For example, I can artificially increase my retention numbers by locking new users into a long contract, and that’s going to keep them with me for awhile. Once that contract’s up, they are free to move wherever they like, and then I need to acquire a customer to replace them. And, if my contract is longer than my competitors’, it can scare off new users.

Also, the retention metric is easy to affect with switching barriers, which may increase the number of months I have a customer while making them less happy. Of course, if those switching barriers are removed for any reason – for example, cell phone number portability – I can lose my hold over long time customers.

While retention can be an indicator of happy customers, increasing retention by any means necessary doesn’t necessarily make your customers happier.

Revenue

Revenue’s another metric that seems like it would point to happy customers. Increased revenue means people are spending more, which means they like your service!

There are all sorts of ways I can increase my revenue without making my customers happier. For example, I can rope them into paying for things they didn’t ask for or use deceptive strategies to get them to sign up for expensive subscriptions. This can work in the short term, but it’s likely to make some customers very unhappy, and maybe make them ex-customers in the long run.

Revenue is also tricky to judge for free or ad-supported products. Again, you can boost ad revenue on a site simply by piling more ads onto a page, but that doesn’t necessarily enhance your users’ experience or happiness.

While increased revenue may indicate that people are spending more because they find your product more appealing, it can also be caused by sacrificing long term revenue for short term gains.

NPS – Net Promoter Score

The net promoter score is a measure of how many of your users would recommend your product to a friend. It’s actually a pretty good measure of customer happiness, but the problem is that it can be tricky to gauge accurately. It generally needs to be obtained through surveys and customer contact rather than simple analytics, so it suffers from relying on self-reported data and small sample sizes. Also, it tends to be skewed in favor of the type of people who answer surveys and polls, which may or may not be representative of your customer base.

While NPS may be the best indicator of customer happiness, it can be difficult to collect accurately. Unless your sample size is quite large, the variability from week to week can make it tough to see smaller changes that may warn of a coming trend.

Conversion to Paying

For products using the freemium or browsing model, this can be a useful metric, since it lets you know that people like your free offering enough to pay for it. However, it can take awhile to collect the data after you make a change to your product because you have to wait for enough new users to convert to payers.

Also, it doesn’t work well on ad-supported products or products that require payment upfront.

Most importantly, it doesn’t let you know how happy your paying customers are, since they’ve already converted.

Conversion to Paying can be useful, but it is limited to freemium or browsing models, and it tends to skew toward measuring the free part of the product rather than the paid product.

Engagement

Engagement is an interesting metric to study, since it tells me how soon and often users are electing to come back to interact with my product and how long they’re spending. This can definitely be one of the indicators of customer happiness for ecommerce, social networking, or gaming products that want to maximize the amount of time spent by each user. However, increasing engagement for a utility product like processing payroll or managing personal information might actually be an indicator that users are being forced to do more work than they’d like.

Also, engagement is one of the easiest metrics to manipulate in the short run. One time efforts, like marketing campaigns, special offers, or prize giveaways can temporarily increase engagement, but unless they’re sustainable and cost effective, they’re not going to contribute to the long term happiness of your customers.

For example, one company I worked with tried inflating their engagement numbers by offering prizes for coming back repeatedly for the first few days. While this did get people to return after their first visit, it didn’t actually have any effect on long term user happiness or adoption rates.

Engagement can be one factor in determining customer happiness, but this may not apply if you don’t have an entertainment or shopping product. Also, make sure your engagement numbers are being driven by actual customer enjoyment of your product and not by artificial tricks.

Registration

While registration can be the fastest metric to see changes in, it’s basically worthless for figuring out how happy your users are, since they’re not interacting with the product until after they’ve registered. The obvious exception is products with delayed (i.e. lazy) registration, in which case it can act like a lower barrier-to-entry version of Conversion to Paying. When you allow users to use your product for awhile before committing, an increase in registration can mean that users are finding your product compelling enough to take the next step and register.

Registration is only an indicator of happy customers when it’s lazy, and even then it’s only a piece of the puzzle, albeit an important one.

Customer Service Contacts

You’d think that decreasing the number of calls and emails to your customer service team would give you a pretty good idea of how happy your customers are. Unfortunately, this one can be manipulated aggressively by nasty tactics like making it harder to get to a representative or find a phone number. A sudden decrease in the number of support calls might mean that people are having far fewer problems. Or, it might mean that people have given up trying to contact you and gone somewhere else.

Decreased Customer Service Contacts may be caused by happier customers, but that’s not always the case.

So which is it?

While all of these metrics can be extremely important to your business, no single one can tell you if you are making your customers happy. However, looking at trends in all of them can certainly help you determine whether a recent change to your product has made your customers happier.

For example, imagine that you introduce a new element to your social networking site that reminds users of their friends’ birthdays and then helps them choose and buy the perfect gifts. Before you release the feature, you decide that it is likely to positively affect:

• Engagement – every time you send a reminder of a birthday, it gives the user a reason to come back to the product and reengage.
• Revenue – assuming you are taking a cut of the gift revenue, you should see an increase when people find and buy presents.
• Conversion to Paying – you’re giving your users a new reason to spend money.
• (Lazy) Registration – if you only allow registered users to take advantage of the new feature, this can give people a reason to register.
• Retention – you’re giving users a reason to stay with you and keep coming back year after year, since people keep having birthdays.

Once the feature is released, you look at those numbers and see a statistically significant positive movement in all or most of those metrics. As long as the numbers aren’t being inflated by tricks or unsustainable methods (for example, you’re selling the gifts at a huge loss, or you’re giving people extra birthdays), you can assume that your customers are being made happy by your new feature and that the feature will have a positive impact on your business.

Of course, while you’re looking at all of your numbers and metrics and analysis, some good old fashioned customer outreach, where you actually get out and talk directly with users, can also do wonders for your understanding of WHY they’re feeling the way they’re feeling. But that’s another post.

Interested? You should follow me on Twitter.

For more information on the user experience, check out:

• 6 Reasons Users Hate Your New Feature
• Is Continuous Deployment Good For Users?
• A Faster Horse – When Not to Listen to Users
• Improving the ROI for Your Customer Feedback
• 5 Things People Get Wrong When Talking to Users

There is, sadly, no single reason why your new feature failed, but there are a number of possibilities. The failure of brand new products is its own complicated subject. To keep the scope narrow, I’m just going to concentrate on failed feature additions to current products with existing users.

Your Existing Product Needs Too Much Work

Ah, the allure of the shiny new feature! It’s so much more exciting to work on the next big thing than to fix bugs or improve the user experience of a boring old existing feature.

While working with one company, I spoke with and read forum posts written by thousands of users. I also used the product extensively myself. One of the recurring themes of the complaints I heard was that the main product was extremely buggy and slow. The problem was, fixing the bugs and the lagging was really, really hard. It involved a significant investment in infrastructure change and a serious rewrite of some very tricky code.

Instead of buckling down and making the necessary improvements, management spent a long time trying to build new features on top of the old, buggy product. Unfortunately, the response to each new, exciting feature tended to be, “Your product still crashes my computer.  Why didn’t you make it stop doing that instead of adding this worthless thing that I can’t use?”

Now, you obviously don’t need to fix every last bug in your existing offering before you move on and add something new. You do, however, need to be sensitive to the actual quality of your product and the current experience of your users before adding something new. You wouldn’t build a second story on a house with a shaky foundation. Don’t tack brand new features onto a product that has an unacceptably high crash rate, severe usability problems, or that runs too slowly for a significant percentage of your users.

Before you add a new feature to a product, ask yourself, “Have I fixed the major bugs, crashes, and UX issues that are currently preventing my users from taking advantage of core features?”

Your Product Interface Is a Giant Mess

Remember the old Saturday Night Live spoof commercial that advertised, “It’s a floor wax! It’s a dessert topping!”? It’s not as funny when it’s true. Products cannot do everything, and when they try, they end up with interfaces far too complicated for the average user to navigate.

I see this happen all the time, especially with startups looking for a way to make their product appeal to more people. Instead of improving their core product and adding features that enhance that experience, they add unrelated feature after unrelated feature, often stolen directly from more successful companies with larger user bases. Their goal is to find something that makes them blow up huge, but they just end up with an overly complicated product that tries to do too many things and doesn’t do any of them well.

It’s not just startups that suffer from this. Products that have been around for many years often get bogged down with feature after feature, all of which have to be supported because some fraction of the user base still uses it. These products then become vulnerable to new challengers with more focused, easy to use interfaces and smaller feature sets.

Of course, there are times when companies have to take their products in a new direction. For example, Flickr started as a set of tools for an MMORPG called Game Neverending. The game has ended, but Flickr lives on as an entirely different business.  PayPal began as a way to make PDA to PDA mobile payments, but that feature got killed years ago when they realized that web payments were a far better business model.

When you do find that killer feature that’s going to change your whole business model, commit to it and make it a serious focus rather than burying it under dozens of less popular features. Don’t try to be all things to all people, or you will end up with nothing but a giant, unusable mess.

Before adding a new feature, ask yourself, “Does this enhance my current product experience or just add to an already confusing and cluttered interface?” And, if it doesn’t fit with your current product offering but you still want to do it, ask, “Am I prepared to cut other features to make this part of my core offering and simplify my experience?”

You Didn’t Build What They Asked For

Let’s face it, sometimes your priorities are different from your users’. For whatever reason, be it a new business partnership, a need for a new revenue stream, or the desire to attract a different group of users, sometimes you’re going to build something that your current users don’t want and didn’t ask for.

This isn’t always a bad thing. For example, something that annoys your current non-paying users but attracts a whole slew of new, paying users is worth a few nasty emails to your customer service department. Just make sure that the new feature is really going to do what you think it will. It sucks to piss off your current, paying customers to build a feature that never really fulfills its initial promise. Trust me on this.

Before building a feature that potentially has more benefits to your company than your current users, ask yourself, “Am I prepared to deal with the fact that this is going to annoy some of my customers, and what is the real likelihood that I will get more out of this than I will lose?”

You Built EXACTLY What They Asked For

I know. It doesn’t seem fair. They’re angry if you don’t do what they want, and they’re angry if you do what they want.

The truth is that users will often ask you for a solution when it would really be more helpful to tell you that they have a problem. I’ve written more extensively about when you shouldn’t be listening to your users, but the upshot is that users aren’t great predictors of which brand new features will be big hits. Sometimes users will tell you that they want a toaster in their car, when what they really mean is that they don’t have time to make breakfast in the morning.

Before building a new feature that your customers are demanding, ask yourself, “What known user problems is this solving, and is this the best way to solve it for everybody?”

The Feature’s Not Finished

Now, I’m all for building the minimum viable product, getting it out in front of users, and then iterating on it to improve it, but some features just aren’t ready for prime time. By launching a half baked feature without key functionality, you’re running the risk of turning a lot of people off on the idea before they ever get a chance to really try it out.

Remember, your customers aren’t in the conference room with you when you come up with your grand vision. They don’t know where you’re going with this neat new idea. They’re judging the feature based on their first experience with it. Make sure that the first version is at least usable and hopefully that it’s far enough along that users can see the same promise that you do.

Also, good enough to ship doesn’t necessarily mean good enough to remain in your product long term. A big part of shipping early is continuing to improve the feature once it’s been out for awhile. One company I worked with had the tendency to ship early versions of features and then let them just sit there gathering dust, rather than iterating on them until they were truly high quality. What they ended up with was an enormous product that all seemed about half finished and a lot of unhappy customers who didn’t believe features would ever improve past version 1.0.

Before shipping a new feature, ask, “Is this good enough that users will get why they should care about it? And, if they do care about it, am I committed to improving it?”

The Feature’s Not Finishable

At many of the companies I’ve worked with, features have tended to evolve before they even get built. What generally happens is this: you have an idea based on something you’ve heard from users; that idea gets brainstormed and grows based on internal input; UX and visual designers spec out the whole idea, often expanding on the original idea; then engineering gets involved and gives a time estimate of how long the feature will take to build; finally the whole thing gets cut back by about 80% based on the estimates.

Unfortunately, the 20% you end up implementing may not solve the original problem. That means, when you finally announce your great new feature, users who originally asked to have that particular problem solved are justifiably upset.

Before drastically cutting your new feature back, ask yourself, “Does this still solve the original problem I was trying to solve?” If it doesn’t, ask, “Can this problem be solved with a reasonable level of investment?” There’s no shame in answering no to either or both of these questions, as long as you decide not to go forward with the new feature.

The Secret to Making Everybody Love Everything You Make

I’m joking. There’s no secret. The truth is that it’s almost certainly impossible. But by asking yourself the right questions during your feature development phase, you can dramatically cut back on time spent creating things your users hate.

And never forget, when you do build something they hate, acknowledge it, apologize for it, and fix it. It will go a long way toward making your users happy again, and it might even get them to like that neat new feature you just shipped.

Interested? You should follow me on Twitter.

For more information on the user experience, check out:

• Is Continuous Deployment Good For Users?
• A Faster Horse – When Not to Listen to Users
• Improving the ROI for Your Customer Feedback
• 5 Things People Get Wrong When Talking to Users
• A/B and Qualitative User Testing

A big part of my unhappiness is the fact that I had to wait for more than two years before they fixed my problems. That’s a lot of crashes and frustration to forget about.

One approach that many software companies have been adopting to combat the huge lag time built into traditional software releases is something called continuous deployment. This sort of deployment means that, instead of having large, planned releases that go through a strict process and may take months or years, engineers release new code directly to users constantly, sometimes multiple times a day. A “release” could include almost anything: a whole new feature, a bug fix, or a text change on the landing page.

I worked with a software development organization that practiced continuous deployment on a very large, complicated code base, and I can definitely say, the engineers loved it. From the point of view of the employees, continuous deployment was a giant win.

But how was it for the users? The fact is, some decisions that seem like they only affect engineering (or marketing, business, PR, etc.) can actually have a huge impact on end users. So, whenever organizations make decisions, they should always be asking, “how might this affect my customers, and how can I make it work best for them?”

Is Continuous Deployment Good For Users?

As with so many decisions, the answer is yes and no. Continuous deployment has some natural pros and cons for the customer experience, but knowing about them can help you fix the cons and benefit even more from the pros.

Big Customer Wins

Fast Bug Fixes

Perhaps the biggest win for users is that bugs can get addressed immediately. Currently, even Microsoft releases patches for some of its worst security holes, but there is certainly a class of non-critical, but still important bugs that have to wait until the next major release to get addressed. That means weeks, months, or even years of your users dealing with something broken, even if the fix is simple. In continuous deployment, a fix can be shipped as soon as it’s done.

Fast Things vs. Slow Things

Similar to the first point, continuous deployment lets you get everything, not just bug fixes, to users as soon as they’re ready to go. Small features, easy changes, and UI tweaks don’t have to wait for larger, unrelated features to be released to customers. After all, should a new design for the splash screen really have to wait on the implementation of a whole new payment system?

More Opportunities for Community Involvement

If you’re having a constant dialog with your customers (you are, right?), they’re probably making some pretty good suggestions about problems they’re having or ways to improve the product. A by-product of the first two benefits is that those users are going to feel even more involved in the development process when their suggestions or concerns are dealt with quickly, rather than if they have to wait months or even years for the next major release.

(Mostly) Avoidable Customer Problems

As with anything, continuous deployment can also cause some problems for users. Of course, some of these problems can exist in big, staged deployments as well, but these are a few things in particular to watch for.

Constant Change

Imagine if every day, the layout on your car changed, sometimes slightly, other times drastically. You want to drive to the store, but the steering wheel is on the other side and you’ve suddenly got an extra pedal. It would make it a lot harder to get where you were going, wouldn’t it?

Well, presumably your users are using your product to get something done, and they’ve got a certain way that they’ve learned to do it. Continuous deployment can mean that the interface for your product can change at any moment, even several times a day. If features constantly appear and disappear, it can be very disruptive to your users’ process.

There are a few things you can do to minimize the disruption. First, make sure that you’re testing your biggest changes on small cohorts of people. Iterate on a subset of your user base, rather than hitting every single user with every single change. This will limit the change that any individual sees while still giving you the benefit of constantly pushing code to customers.  In fact, use this as an opportunity to do the A/B testing you should be doing anyway.

Also, and this should be obvious, try to limit your truly disruptive user interface changes so that things don’t feel like they’re in constant flux. You can still change things, but be aware of how frequently you’re making major changes and stay in contact with your users to make sure they’re not feeling dizzy.

Inconsistent UIs

When a big new release is planned, often there is a comprehensive design phase where all the new changes are mapped out and discussed. This means that any inconsistencies in the UI can be found and addressed.

In continuous deployment, different pieces of the product are getting built and shipped to customers all the time, and there is rarely a time when the entire UI is reevaluated as a single entity. This means that sometimes UI standards can tend to…oh, let’s say evolve.

This problem can be controlled by having a UX team member embedded in the development team and constantly working with the engineers to enforce standards before things ship to customers. It can also be improved by providing wireframes, visual designs, and tools like templates to developers so that the look and feel of the product doesn’t shift too dramatically over time.

Avoiding QA

I’m certainly not claiming that every single thing in a traditional release process gets a full QA pass, but I do find that continuous deployment makes it easier for code to slip out to users without any human testing at all. Any time you give engineers the ability to ship code directly into production, you’re tempting fate. Somebody’s going to say, “oh, it’s just a tiny change,” and it’s going to get out without any testing. I’m not naming any names, but you only have to have a tiny change break the entire product once before you realize that there’s no such thing as a tiny change.

Also, continuous deployment can make certain types of testing much less likely to happen. While large release cycles tend to have a code freeze and weeks or months devoted to testing, hopefully including regression, unit tests, and end to end testing, continuous deployment doesn’t necessarily have that baked into the process.

You can always add periodic end to end testing of the product to your own process, of course, and it can be quite helpful in improving code quality, especially when your engineers occasionally slip through a “tiny change.”

Communication Issues

When you’re constantly releasing new features and bug fixes, communication with your users can be a challenge. You don’t have one big release with new help docs, a big roll out plan, and an advertising campaign. Instead, stuff is coming out all the time, and users can get overwhelmed by keeping up with the changes.

Context sensitive help and inline information for each feature can help users get quickly oriented.  Also, clearly marking new features as alpha or beta can let users know when a feature is still being developed so they can set their expectations accordingly.

Documentation can also easily get out of date when things are getting released constantly. Big, staged development cycles often have a built-in time for creating documents. Typically, manuals or help documentation and FAQs go through QA sometime after code freeze and before release. But since you’re not necessarily doing a big, monolithic release with continuous deployment, you can end up with this material never getting any sort of end to end editing to make sure they stay current. Make time for this. It’s good for both your customer experience and your customer support team.

Frequent Downloads and Updates

While continuous deployment is quite natural with web applications, even downloaded products can be constantly updated. However, you should always be aware of the burden you’re placing on your user base. If you’re forcing people to download a large file and go through an installation process too often, you’re going to annoy people.  As a personal example, iTunes appears to have a new version every week, and I’ve started to flinch every time I open it.

There are a few things you can do to make downloads easier on your users. First, you can ask the user to allow the update to download in the background and install automatically the next time the product opens. This means that the updating happens with very little user annoyance. Also, it’s best to keep the update quick by making the downloads incremental. For example, your virus protection software probably updates its virus information daily without asking you to reinstall the entire product every time.

So, Is It Good for Users or Not?

Continuous deployment can be done in a way that’s good for both engineering teams and users, but you do need to take some precautions. By taking care when you introduce changes that your users will really notice and making sure that you make time for important processes like QA, you can get features out faster and constantly improve your product. And that is very good for users.

Interested? You should follow me on Twitter.

For more information on the user experience, check out:

• A Faster Horse – When Not to Listen to Users
• Improving the ROI for Your Customer Feedback
• 5 Things People Get Wrong When Talking to Users
• A/B and Qualitative User Testing
• 6 Stupid Excuses for Not Getting Customer Feedback

You need to talk to users because, if you ask the right questions, they will help you build a better product. The key is figuring out the right questions.

For starters, users are great at telling you when there’s something wrong with your product. They can tell you exactly which parts of the product are particularly confusing for them or are keeping them from being happy, repeat customers. Figuring out what to do about those problems is your job.

In general, users are not going to be able to answer the following types of questions:

• What new technical innovation is going to revolutionize a particular industry?
• What’s the next cool gadget that you’d like to buy?
• Do you think that people like you would buy this new cool gadget that you’ve just learned about?
• What new features would make this product more interesting/compelling/fun/easy to use? (although, this question becomes more answerable when the user is presented with some options for which features they might prefer.)
• How exactly should we change the product to make it easier for you to use?

They are fantastic at answering questions like these:

• What do you most love or hate about this product?
• Do you find anything about this product hard to use or confusing?
• Does this product solve your problem better or worse than what you’re currently doing?
• How are you currently solving a particular problem that may or may not be addressed by this product?
• What don’t you like about your current solutions for a particular problem?
• Why did you choose this particular solution as opposed to another solution?

Obviously, there are innumerable other questions that you might want to ask your users, so how do you decide which ones they’ll be able to answer with any degree of accuracy?

Problems vs. Solutions

Users are much better at telling you about problems that they’re having than solutions that they want. In Ford’s example, when people asked for a faster horse, what they were really saying was that the horses they had were too slow. They didn’t specifically want a faster horse. They wanted a faster means of transportation that was no worse than a horse in other respects.

Frequently in user tests or customer feedback sessions, customers will tell you very clearly, “I want x!” Your job is to understand why they want x and then to determine whether or not x is really the right solution. It’s not that they never have good solutions, but users tend to only look at the product from their own perspective and usage patterns, while you should be talking to lots of different types of users with lots of different types of problems. They’re not thinking about the product as a whole or how to fix things for the other several million people who might have slightly different problems.

When users try to give you solutions, encourage them to talk about their problems instead. Then figure out what they’re really asking for, and give it to them.

Past vs. Future Events

It’s much easier for people to answer questions or give opinions about something specific that has already happened than about something that might happen in the future.

Consider the question, “What do you want to eat tonight?” vs. the question, “What did you think of the meal you just ate?” For the vast majority of us, the second one is much easier to answer. It simply asks you to formulate a concrete opinion about a single event that has recently happened. The first question asks you to imagine all the various available options for food and make a decision about what you might like in the future based on probably imperfect information.

This is true with products, as well. It will be much easier for your user to explain how performing a particular task went than to predict how he would like to perform that particular task in the future. That’s why, when you’re doing your preliminary research to determine product direction or early feature development, it’s very important to give users hands on tasks that they can perform for you and then give opinions on rather than to talk abstractly about the solution you’re considering providing for them with your product.

Users vs. Other People

Unless you’re really lucky, you’ve probably realized that people are terrible at figuring out what other people want. Perhaps you came to this realization on some birthday or other gift giving holiday. Users suffer from the same problems as gift givers. They’re almost always terrible at telling you how other people will react to a product.

And yet, talk to just a few customers or user test participants, and you’re guaranteed to hear one of them say, “Well, it’s not for me, but my mom/friend/boss/brother would be really into this…” Another one you hear a lot is, “My mom/friend/boss/brother would never be able to use this. It’s way too complicated.”

This information can be marginally useful if you’re trying to find the right customer segment, but take it with a grain of salt. Reassure the user that you’re also going to talk to people like their mom/friend/boss/brother, and what you’re really interested right now in is the user’s opinion. Then talk to the mom/friend/boss/brother to find out their real feelings. Chances are, the person you’re talking to doesn’t really know what anybody else wants as well as they think they do.

The Right Questions

So, what should Henry Ford have been asking his customers? Instead of, “What do you want?” he could have asked, “Is there anything you particularly like or don’t like about your horse and wagon?” If they chose not to buy a car, he could ask, “Why didn’t you buy that car?” Once they bought a car, he could have asked, “What made you decide to buy a car?” or “Was there anything you found particularly confusing or hard to use about your new car?” He could even have gone for a drive with some of them and observed the various problems that they encountered.

In fact, there were dozens of things he could have done that might have helped him improve the design and marketing of his product. He just couldn’t ask them, “What do you want?” because they almost certainly would have said, “a faster horse.”

For more information on our approach to getting customer feedback, check out:

• Improving the ROI for Your Customer Feedback
• 5 Things People Get Wrong When Talking to Users
• Fast Insight Testing
• A/B and Qualitative User Testing
• 6 Stupid Excuses for Not Getting Customer Feedback

If your answer is “talk about it all abstractly with everybody in the company or write a huge paper that nobody will read and then go on with business as usual,” you’re in good (bad?) company.

But you have to DO something with all that data. You have to analyze it and turn it into actionable items that your engineering department can use to fix your product. It’s not always easy, but I’m going to give you an approach that should make it a little easier. This isn’t the only way to do your test analysis, but it’s one of the quickest and easiest that I’ve found when you are concerned with key metrics.

When to use this method:

• You have an existing product with a way to measure key metrics, and you’re interested in improving in particular areas related to your bottom line
• You have a limited research and development budget and want to target your changes specifically to move key metrics
• You are looking for the “low hanging fruit” that is getting in the way of your users performing important tasks with your product
• You are working in an agile development environment that is constantly tweaking and improving your product  and then testing the changes

When not to use this method:

• You have an existing product that you are planning to completely overhaul, and you want to understand all of the major problems before you do your redesign
• You are trying to create an overall awesome, irresistible user experience that is not related to a specific metric
• You are designing a new product or feature and are observing people using other products to identify opportunities for innovation

If you fall into the first bucket, read on…

The Five Basic Steps:

• Identify key metrics you’d like to improve
• Identify the tasks on your site that correlate with improvement in those metrics
• Observe people performing the appropriate tasks
• Identify the barriers preventing people from completing or repeating the tasks
• Develop recommendations that address each specific barrier to task completion

Step 1: Identify Key Metrics

First, identify the metrics that you care about, and look at the distinct steps that your users are most likely to take to change those metrics. This way, you can plan your qualitative testing to show you exactly what is getting in the way of your users reaching each subsequent step.

As an example, let’s imagine that you have a social networking site that earns revenue when people give virtual gifts to each other to display on their profile pages. Because your revenue depends on lots of users purchasing lots of things for each others’ pages, you might care about the following metrics: how many people register, how many people customize their profile pages, how many people make friends, how many friends they make, and how many people purchase one or more gifts. Obviously, that last is the most important, since that’s where your revenue comes from, but the others are also important, since improving those numbers should increase everything downstream, if done in the correct way.

Your goal is to get people from point A to point D as quickly and pleasantly as possible. To do that, you want to look at all of the barriers you have erected in the way of your users accomplishing those tasks.

Here’s are the pieces of the user flow that change your key metrics:

Registration > Profile Customization > Finding a Friend > Purchasing a Gift

Step 2: Identify Individual Task Flows

Now that you know what your key metrics are, you need to make sure that you observe your test participants trying to accomplish the tasks that will eventually lead to changing those metrics. To plan your tasks and analyze your data, it helps to break down your individual tasks into smaller user flow steps based on your user interface.

For example, while the number of successful registrations may be a single metric, there may be several screens or discrete interactions involved in your entire registration process. Make a list of what they are now.

As an example, I ran a test recently for a product that had a 4 step registration process. The user flow for registration on the site looked like this:

Step 3: Observe People Performing Tasks

So let’s say that, based on the metrics you want to change in your hypothetical social networking site, you’ve determined that you need to gather data about registration, profile customization, finding friends, and purchasing a gift. You’ve scheduled 5 people in your target demographic to come in for user test sessions. I’m not going to teach you everything you need to know about running a test, but here are some basics.

First, let the user begin using your product however they want and note all of the things that they do that aren’t the tasks you’ve identified. Remember, any time spent not doing your target tasks is time that is not being spent improving your metrics, so you want to know what these other things are! If you give your participant a targeted task right away, you will never learn what other things they are doing instead. Finding these distractions allows you to eliminate them or change them so that they also improve your key metrics.

Watch everything your user does with the perspective of the ideal flow for your metric. Do your participants wander off track or getconfused during registration? Do they fail to find their own profile? Do they not have the information they need to find a friend? Do they fail to understand what gifts are or how to purchase them? Do they get distracted by a totally different part of your site that doesn’t contribute to improving your key metrics?

Once you’ve allowed the participant some free exploration time, you may need to move them to a particular task. For example, in our current example, it might be that you need to ask the participant to try to purchase a gift, even if that’s something they wouldn’t do in the context of a study environment. You can move them along by saying something casual like, “Did you notice that there is a way to purchase gifts? What do you think about that?”

Meanwhile, record what the participant is doing and encourage them to talk about their impressions of the product.

Step 4: Identify Barriers

Once you’ve gathered data by watching 4 or 5 people repeat the same tasks, you need to analyze your data to figure out where the barriers are and communicate them effectively to the team. Determining the barriers should be pretty easy. Just ask yourself the following questions:

• Where did participants seem confused or distracted and stop performing a task?
• Which things took longer than they should have?
• Why did participants fail to accomplish a task?
• What tasks did the participants perform that would not improve the key metrics?

In the product I tested with the four step registration process, the metrics showed significant drop off one each screen. On screen number 2, the user was asked to choose a unique user name and enter some information. It had a surprising amount of drop off considering how simple it was. After watching a few user tests, I noticed that many people were trying to select user names that were not available, but they were failing to notice the error message telling them to choose a different one. Even if they noticed the error, several people had a tough time finding a user name they liked and would spend minutes typing in word after word, getting frustrated, and finally giving up.

By watching people try to register for the product, we identified several significant barriers to successful registration in just one step of the process. In fact, we found similar problems – some large, some small – in each of the steps.

Once we’d identified the barriers, organizing and explaining them to the team was easy with a simple, annotated flow of the steps leading up to the goal along with the problems associated with each step. For example, the flow for the registration test might have looked like this:

Landing Page

• Didn’t understand the product
• Felt the product was aimed at teens
• Thought the network was very small

User name

• Didn’t know when a user name was taken
• Couldn’t come up with a decent name

Personal Info

• Were concerned about privacy
• When failed Captcha, all info had to be re-entered

Download

• Were uncomfortable downloading b/c of viruses
• Didn’t know what they were downloading

Overall

• Were bored during registration process or complained that it was taking a long time

Step 5: Develop Recommendations

Now that you know what barriers are keeping your customers from accomplishing the goals you’ve set for them, you need to generate recommendations for ways to remove those barriers. You can do this by looking at exactly what the barrier is and what the users’ reactions were to it, and then brainstorming ideas to help the user overcome that barrier. Yes, this is the tough, somewhat creative part. You don’t want to just take any recommendation that the user gives you. You need to understand what problem they’re having and come up with a way to fix it that doesn’t cause any other problems.

Going back to my example, once we found the problems on the user name page, we looked at several solutions. We definitely needed to make it more obvious to the user when they selected a name that was already taken. We came up with a few simple ways to solve this particular problem: we could suggest other user names when people tried one that was already taken; we could make the error more noticeable; we could make the Next button obviously disabled so that they couldn’t even try to move on; we could free up some names that had been taken but weren’t currently in use so that the selection of user names was better. We then selected a couple of these solutions based on expected ROI calculations.

Even the lowest effort of these suggestions, improving the visual design of the error, caused an immediate and statistically significant decrease in drop offs for that step in the registration flow, and eventually improved revenue by increasing the number of successful sign ups. The barrier was lowered, if not entirely removed.

You may have noticed that the last section is labeled “Overall.” This is where you look at the process as a whole, rather than a sum of its parts. For example, one comprehensive solution for drop off at each step might be to reduce the number of steps it takes to register (almost never a bad idea!). Doing this wouldn’t necessarily mean that you wouldn’t have to address the other problems individually, but you might get fewer drop offs simply because it would take less time for users to finish the process.

How Is This Different?

You might be wondering how this is different from more traditional ways of running user tests and analyzing data. In any user test, you’ll need to come up with tasks, observe users, figure out where they’re having problems, and come up with solutions for the problems.

There is a difference though. My experience has been that companies do not fix every single UX problem that they find in user research. I don’t love this, but I do understand the need to prioritize important changes.

So, if you’re only going to fix a few problems, you should make an effort to identify the most important ones. The counter intuitive part is that the most important problems might not be the worst problems from a user experience stand point, but they will definitely be the ones that are keeping your users from reaching the goals that improve your most critical metrics.

Using this framework will help you identify your key user flows, find and communicate the major barriers to success, and propose targeted solutions that will improve both your user experience and your business.

For more information on our approach to getting customer feedback, check out:

Before I get too far into this, let me define what I mean by a paper prototype, since I’ve heard people use the term to refer to everything from sketches on actual pieces of paper (or cocktail napkins in a couple of cases) to full color printed mockups with a polished visual design. In this instance, I’m referring to a totally non-interactive screen, mockup, or sketch of any sort of application that is meant to be shared with customers, test participants, or team members. It can be printed on actual paper or shown on a computer screen, but whatever the viewer does to it, a paper prototype is not interactive.

So, what don’t I like about them?

Screen vs. Paper

This first couple of peeves apply to screens that are actually printed out or drawn directly on paper. With a few exceptions that I’ve listed below, I’ve found this approach to be really counterproductive.

Iterating On a Design

One of the biggest problems with hand drawn sketches on paper has less to do with user interactions and more to do with my work flow as a designer. Sure, sketching something quickly on a piece of paper can be quick, but what happens when I realize that I want to swap two sections of the screen? I can draw arrows and lines all over it, but that gets messy pretty fast. Whenever I want to make any changes to my design, I need to create a whole new sketch. This can mean redrawing the entire screen quite a few times.

If I’m creating a design in HTML or any other prototyping tool, the very first version might take a little longer than a quick sketch, but the second through nth iterations are a whole lot faster. And, as a bonus, I can check them into source control, which means I’m a lot less likely to lose my work than if I have dozens of pieces of paper scattered all over my office.

Interacting With Paper

Whether they’re sketched out by hand or printed out on paper, people interact with paper screens differently than they do with computer screens. They view them at a different angle. They focus on different parts of the screen. They use their hands to interact with them rather than a mouse and keyboard. Any feedback that you get on a printed design will be colored by the fact that people are fundamentally interacting with it differently than they would if it were on a computer screen.

Given all of these drawbacks, there are a few situations when designs printed on paper can be used effectively:

• You are at the very beginning of the design process, and you want to explore a bunch of different possible directions with other team members very quickly before committing yourself to fleshing out one or two specific options.
• You’re designing material that is meant to be printed, like brochures, user manuals, books, etc. In this case, you want to know how people will interact with the printed media.
• Your product is an interface for some sort of embedded or small screen device that would be very difficult to replicate in a quick interactive prototype. For example, a screen for certain mobile devices or the heads-up display for a car dashboard might be hard to show interactively in the appropriate context.
• You have several different visual designs, and you’d like to show them all to users at the same time in order to see which one is the most attention-getting. You’ll still need to show the designs on screen, of course, since colors can vary so much between screen and print, but it can be helpful to lay out several pieces of paper so that the various options can easily be compared.
• You need to share screens with people in an environment with absolutely no access to a computer whatsoever. You know, maybe you’re in the middle of a meeting and need to sketch something quickly, or the rest of your design team is Amish, or you are designing in a post-apocalyptic wasteland where the computers are trying destroy humanity.

On the other hand, if you’re designing desktop or web applications for standard computers, at the very least, show your prototypes on a computer, even if they are not interactive!

A few reasons why it’s especially important to show interactive designs on a screen:

Animations and interactions

I am an interaction designer. A big part of my job is to determine what happens when a user interacts with a product. Sometimes that’s obvious. For example, if I click on a link with some text that reads, “Contact Us,” I expect to be able to communicate in some way with the people who have created the product.

But is it so obvious? Back in the day when links could only take you to another static web page, sure it was. But now, all sorts of things could happen. I might have different behavior on hover vs. on click. I could be given the option to have a live chat with a rep. I might be presented with an inline contact form so that I don’t have to leave the page I’m visiting. The contact form could have some information already prefilled for me based on my present location or account type. There could be animation involved in displaying the contact information. There could be some sort of contact wizard that would change later screens depending on choices the user makes initially.

All these interactions are much harder to convey with paper prototypes than with interactive wireframes. Sure, you can make a different screen showing each stage of the interaction, but with anything more complicated than a few steps, your observers can get lost pretty fast shuffling through papers. Having it all working in an interactive prototype allows users to click through and explore naturally. They can also discover things like hover interactions or animations on their own, which are particularly unsuited to paper.

Testing

What? You don’t need to test your designs with users? Your designs spring from your brain fully formed and perfect? Well, good for you. The rest of us mortals actually like to show our designs to test participants and get feedback from people who aren’t familiar with the product.

Now, I’ve run a lot of user tests. I’ve run them with working products, interactive prototypes, pictures of screens displayed on computers, pure paper prototypes, and physical mockups of products. I’ve used prototypes built with everything from HTML to Visio to cardboard. The one constant was that the closer the prototype mimicked the final product interaction, the fewer problems we found once the product was built. And since we recommend iterative testing during development rather than waiting  to test until the product is 100% built (you know, just in case your design wasn’t entirely perfect the first time around), an interactive wireframe is the best trade off of speed for functionality.

Communicating the design

You can have the best design in the world, but if you don’t communicate it effectively to the engineers who have to build the thing, it doesn’t matter one bit. Of course, once you’ve designed all of your static screens, you could paste them into a big spec with extensive documentation about what each button on each screen does and how many seconds an animation is supposed to take and blah blah blah. I’ve done it when it’s been specifically requested by a client. I’ve also never met an engineer who was particularly happy to read through one of those three hundred page documents. Besides, just the thought of making changes to every screen in that document every time there’s a change makes me tired.

An interactive prototype, when built intelligently, is a different story. It allows engineers to see exactly how every element of the design is supposed to work. What happens when you click on a link? Click on it to find out! Is the animation supposed to sweep horizontally or vertically? Take a look! Of course, you’re free to add notes to screens as necessary to fully communicate absolutely everything, but the difference between using this lightweight approach and a full design document for reference is night and day.

What’s the alternative?

So, what should you use other than paper prototypes? I’ve mentioned it throughout the post, but I’ll be perfectly clear. Your best bet is an interactive wireframe that mimics the behavior of the actual product as closely as possible. It doesn’t have to be hooked up to a database on the back end or use real data. It doesn’t even have to be fully functional if you’re only testing parts of it or if you’re annotating it with notes. But it should make the user, whether they’re a test participant, an engineer, or your design manager, feel like they’re actually performing tasks with the product.

Luckily, you’ll find plenty of products out there that will help you build interactive wireframes with very little technical expertise. You don’t need to be an engineer or a Flash expert to make your screens clickable. My favorite tool is Dreamweaver for creating quick HTML prototypes with basic animations and interactions, but other people have had good luck with applications like Axure and or even PowerPoint (although, that has some serious limitations).

There is, of course, one quick caveat. I know I’ve been harping on how closely your prototype should match your eventual product. However, you do not need, and probably shouldn’t have until quite late in the design process, a full visual design for your interactive prototypes. Why is that? Well, I’ve found that making the visuals look too polished can actually focus the user on the aesthetics rather than on whether they can accomplish a task. A lovely or flashy visual design can distract the user from interaction problems that you’d rather find early in the process.

For best results, focus on creating a simple, clean wireframe with as much interactivity as you can possibly give it. Your users will thank you for it.

I’m not trying to promote any specific methodology for testing your products or getting customer feedback. Whether you’re doing formal usability testing, Fast Insight Testing, contextual inquiries, surveys, a/b testing, or just calling up users to chat, you should be staying in contact with customers and potential customers throughout the entire design and development process.

To help get you to stop avoiding it, I’ve explored six of the most common stupid excuses for not testing your designs and getting feedback early.

Excuse 1: It’s a design standard

You can’t test every little change you make, right? Can’t you sometimes just rely on good design practices and standards? Maybe you moved a button or changed some text. But the problem is, sometimes design standards can get in the way of accomplishing your business goals.

For example, a few months ago at a talk given by Bill Scott, he talked about a developer who had a/b tested the text on a link. One option read, “I’m now on Twitter.” The second read, “Follow me on Twitter.” The third read, “Click here to follow me on Twitter.” Now, anybody familiar with “good design practices” will tell you that you should never, ever use the words “click here” to get somebody to click here. It’s SO Web 1.0. But guess which link converted best in the a/b test? That’s right. “Click here” generated significantly more Twitter followers than the other two. If that was the business goal, the bad design principle won hands down.

Does this mean that you have to do a full scale usability test every time you change link text? Of course not. Does it mean you have to use the dreaded words “click here” in all your links? Nope. What it does mean is that you should have some way to keep an eye on the metrics you care about for your site, and you should be testing how your design changes affect customer behavior, even when your changes adhere to all the best practices of good design. So, to put it simply:  prioritize what you care about and then make sure you test your top priorities.

Excuse 2: Company X does it this way

I can’t tell you how many times I’ve heard, “Oh, we know that will work. Google/Facebook/Apple does it that way.” This is the worst kind of cargo cult mentality. While it’s true that Google, Facebook, and Apple are all very successful companies, you aren’t solving exactly the same problem that those companies are, you don’t have exactly the same customers that they do, and you don’t know if they have tested their designs or even care about design in that particular area. You are, hopefully, building an entirely different product, even if it may have some of the same features or a similar set of users.

Is it ok to get design ideas from successful companies? Of course it is. But you still need to make sure your solutions work for your customers.

I previously worked with a company that had a social networking product. Before I joined them, the company decided that, since other companies had had good luck with showing friend updates, they would implement a similar feature, alerting users when their friends updated their profiles or bought products. Unfortunately, the company’s users weren’t very interested in the updates feature as it was implemented. When we finally asked them why they weren’t using the feature, the users told us that they would have been very interested in receiving an entirely different type of update. Of course, if the company had connected with users earlier in the process, they would have rolled the feature out with the right information and gotten a much more positive reaction on launch.

Another thing to remember is that just because a company is successful and has a particular feature doesn’t mean it’s that exact feature that makes them successful. Google has admitted that the “I’m Feeling Lucky” button loses them page views, but they keep it because they, and their customers, like the feature. That doesn’t mean it’s a good business plan for your budding search engine startup to adopt a strategy of only providing people with the equivalent of the “I’m Feeling Lucky” button. In fact, this is a great example of why you might need to employ multiple testing methods: qualitative (usability, contextual inquiry, surveys), to find out if users find the feature compelling and usable, and quantitative (a/b, analytics), to make sure that the feature doesn’t bankrupt you.

The bottom line is, it doesn’t matter if something works for another company. If it’s a core interaction that might impact your business or customer behavior, you need to test new features and designs with your customers to make sure that they work for you. Obviously, you also need to make sure that you’re not violating anybody’s IP, but that’s another blog post.

Excuse 3: We don’t have time or money

The fact is you don’t have time NOT to test. As your development cycle gets farther along, major changes get more and more expensive to implement. If you’re in an agile development environment, you can make updates based on user feedback quickly after a release, but in a more traditional environment, it can be a long time before you can correct a major mistake, and that spells slippage, higher costs, and angry development teams.

I know you have a deadline. I know it’s probably slipped already. It’s still a bad excuse for not getting customer feedback during the development process. You’re just costing yourself time later. Besides, it’s not as expensive as you think! You don’t always have to do some enormous user test in a formal lab with one way glass. We advocate cheap and dirty Fast Insight Testing, which  you can learn about  here.

Excuse 4: We’re new; We’ll fix it later

I hear this a lot from startups, especially agile ones, that are rushing to get something shipped, and it’s related to the previous excuse. Believe me, I do understand the pressures of startups. I know that, if you don’t ship SOMETHING you could be out of business in a few months. Besides, look at how terrible some really popular sites looked when they first started! You have to cut something, right?

Great. Cut something else. Cut features or visual polish. Trust me, people will forgive ugly faster than they’ll forgive completely unusable or confusing. Whatever you decide to cut, don’t cut getting customer feedback during your development process. As I mentioned in Excuse 3, you can do it faster than you think. But more importantly, if you ship something that customers can’t use, you can go out of business almost as fast as if you hadn’t shipped anything.

Potential users have a lot of options for products these days. If they don’t understand very quickly all the wonderful things your product can do for them, they’re going to move on.  Take a few hours to show your ideas to users informally, and you will save your future self many hours of rework.

Excuse 5: It’s my vision; users will just screw it up

This can also be called the “But Steve Jobs doesn’t listen to users…” excuse. Except, that’s not true. Recently, in an interview with the New York Times, when asked why the Nano got a camera while the Touch didn’t, Jobs responded, “Originally, we weren’t exactly sure how to market the Touch…What happened was, what customers told us was, they started to see it as a game machine. We started to market it that way, and it just took off. And now what we really see is it’s the lowest-cost way to the App Store, and that’s the big draw. So what we were focused on is just reducing the price to $199. We don’t need to add new stuff. We need to get the price down where everyone can afford it.” Apparently, this myth that Jobs doesn’t listen to customer feedback or make decisions about product features based on talking to users is overblown.

The fact is, understanding what your users like and don’t like about your product doesn’t mean giving up on your vision. You don’t have to make every single change suggested by your users . You don’t have to sacrifice a coherent design to the whims of a focus group.

What you should do is connect with your users or potential users in various different ways – user tests, contextual inquiry, metrics gathering, etc. – to understand whether your product is solving the problem you think it is for the people you think are your customers. And, if it’s not, it’s a good idea to try to understand why that is and develop some ideas for how to fix it.

Besides, even if Steve Jobs never listened to a single customer in his entire life, how many wanna-be Steves do you think there out there whose companies failed miserably because nobody wanted to use their products?

Excuse 6: It’s just a prototype to get funding

This is an interesting one, since I think it’s a fundamental misunderstanding of the entire concept of customer research. When you’re building a prototype or proof of concept, you still need to talk to your customers. The thing is, you may have an entirely different set of customers than you thought you did.

Maybe you think the target market for your new networked, Wi-Fi lunchbox is 11-13 year old girls, but they’re not going to pay you to build the first million units and get them into stores. Your first customers are the venture capitalists or the decision makers at your company or whoever is going to look at your product and decide whether or not to give you money.

Even if they’re not your eventual target market, it’s probably a good idea to spend some time talking with whomever you’re trying to get to fork over the cash. I’m not saying you should change your entire product concept based on this feedback. I mean, if you really want to start the company on your credit cards and a loan from your mom, don’t change a thing! The important take away here is that you may have different audiences at different points in your company’s life. And the best way to find what they all want is to talk to them!

Out of Excuses?

Those are the most common excuses I hear, but I’m sure you can think of some clever ones. Then again, your time is probably better spent connecting with your users, understanding their problems, and thinking of ways to address them.

For more information on our approach to getting customer feedback, check out:

• Improving the ROI for Your User Research
• 5 Things People Get Wrong When Talking to Users
• Fast Insight Testing
• A/B and Qualitative User Testing

• Get information about whether people thought the product was a good idea.
• Identify potential customer types, both for marketing and further research purposes.
• Talk to as many potential users as possible to get a broad range of feedback.
• Keep it as cheap as possible!

He had, unsurprisingly, a number of stories about mistakes they had made and lessons they’d learned during the process of talking to dozens of people. As he was sharing the stories with me, the thought that kept going through my head was, “OF COURSE that didn’t work! Why didn’t you [fill in the blank]?” Obviously, the reason he had to learn all this from scratch was because he hadn’t moderated and viewed hundreds of usability sessions or had any training in appropriate user interview techniques. Many of things that user researchers take for granted were brand new to him. Having spoken with many other people at small companies with almost non-existent research budgets, I can tell you that this is not an isolated incident. While it’s wonderful that more companies are taking user research seriously and understanding how valuable talking to users can be, it seems like people are relearning the same lessons over and over.

In order to help others who don’t have a user experience background not make those same mistakes, I’ve compiled a list of 5 things you’re almost certainly doing wrong if you’re trying to get customer feedback without much experience. Even if you’ve been talking to users for years, you might still be doing these things, since I’ve seen these mistakes made by people who really should know better. Of course, this list is not exhaustive. You could be making dozens of other mistakes, for all I know! But just fixing these few small problems will dramatically increase the quality of your user feedback, regardless of the type of research you’re doing.

Don’t give a guided tour

One of the most common problems I’ve seen in customer interviews is inexperienced moderators wanting to give way too much information about the product up front. Whether they’re trying to show off the product or trying to “help” the user not get lost, they start the test by launching into a long description of what the product is, who it’s for, what problems it’s trying to solve, and all the cool features it has. At the end of the tour, they wrap up with a question like, “So, do you think you would use this product to solve this exact problem that I told you about?” Is there any other possible answer than, “ummm…sure?”

Instead of the guided tour, start by letting the user explore a bit on his own. Then, give the user as little background information as possible to complete a task. For example, to test the cool new product we worked on for Superfish, I might give them a scenario they can relate to like, “You are shopping online for a new pair of pants to wear to work, and somebody tells you about this new product that might help. You install the product as a plug in to Firefox and start shopping. Show me what you’d do to find that pair of pants.” The only information I’ve given the user is stuff they probably would have figured out if they’d found the product on their own and installed it themselves. I leave it up to them to figure out what Superfish is, how it works, and whether or not it solves a problem that they have.

Shut up, already

Remember, while you may have been staring at this design for weeks or months, this may be the first time your participant has even heard of your product. When you first share a screen or present a task, you may want to immediately start quizzing the participant about it. Resist that impulse for a few minutes! Give people a chance to get their bearings and start to notice things on their own. There will be plenty of time to have a conversation with the person after they’ve become a little more comfortable with the product, and you’ll get more in depth comments than if you put them on the spot immediately.

Ask open ended questions

When you do start to ask questions, never give the participant a chance to simply answer yes or no. The idea here is to ask questions that start a discussion.

These questions are bad for starting a discussion:

• “Do you think this is cool?”
• “Was that easy to use?”

These questions are much better:

• “What do you think of this?”
• “How’d that go?”

The more broad and open ended you keep your questions, the less likely you are to lead the user and the more likely you are to get interesting answers to questions you didn’t even think to ask.

Follow up

This conversation happens at least a dozen times in every test:
Me: What did you think about that?
User: It was cool.
Me: WHAT WAS COOL ABOUT IT?
User: [something that's actually interesting and helpful.]

Study participants will often respond to questions with words that describe their feelings about the product but that don’t get at why they might feel that way. Words like “cool,” “intuitive,” “fun,” and “confusing” are helpful, but it’s more helpful to know what it was about the product that elicited that user reaction. Don’t assume you know what makes a product cool!

Let the user fail

This can be painful, I know. Especially if it’s your design or product that’s failing. I’ve had engineers observing study sessions grab the mouse and show the participant exactly what to do at the first sign of hesitation. But the problem is, you’re not testing to see if somebody can be SHOWN how to use the product. You’re testing to see if a person can FIGURE OUT how to use the product. And frequently, you learn the most from failures. When four out of four participants all fail to perform a task in exactly the same way, maybe that means that the product needs to change so that they can perform the task in that way.

Also, just because a participant fails to perform a task immediately doesn’t mean that they won’t discover the right answer with a little exploration. Watching where they explore first can be incredibly helpful in understanding the partipant’s mental model of the application. So let them for fail for awhile, and then give them a small hint to help them toward their goal. If they still don’t get it, you can keep giving them stronger hints until they’ve completed the task.

Are those all the tricks to a successful user study? Well, no. But they’re solutions to mistakes that get made over and over, especially by people without much experience or training in talking to users, and they’ll help you get much better information than you would otherwise. Now get out there and start talking to your users!

Qualitative user testing, on the other hand, involves showing a product or prototype to a small number of people while observing and interviewing them. It produces a different sort of information, but the goal is still to help you make better product decisions based on user feedback.

Now, a big part of my job involves talking to users about products in qualitative tests, so you might imagine that I would hate A/B testing. After all, wouldn’t something like that put somebody like me out of a job? Absolutely not! I love A/B testing. It’s a phenomenal tool for making decisions about products. It is not the only tool, however. In fact, qualitative user research combined with A/B testing creates the most powerful system for informing design that I have ever seen. If you’re not doing it yet, you probably should be.

A/B Testing

What It Does Well

A/B testing on its own is fantastic for certain things. It can help you:

• Get statistically significant data on whether a proposed new feature or change significantly increases metrics that matter – numbers like revenue, retention, and customer acquisition
• Understand more about what your customers are actually doing on your site
• Make decisions about which features to cut and which to improve
• Validate design decisions
• See which small changes have surprisingly large effects on metrics
• Get user feedback without actually interacting with users

For example, imagine that you are creating a new check out flow for your website. There is a request from your marketing department to include an extra screen that asks users for some demographic information. However, you feel that every additional step in a check out process represents a chance for users to drop out, which prevents purchases. By creating two flows in production, one with the extra screen and one without, and showing each flow to only half of your users, you can gather real data on how many purchases are completed by members of each group. This allows you to understand the exact impact on sales and helps you decide whether gathering the demographic information is really worth the cost.

Even more appealing, you can get all this user feedback without ever talking to a single user. A/B testing is, by its nature, an engineering solution to a product design problem, which makes it very popular with small, engineering-driven startups. Once the various versions of the feature are released to users, almost anybody can look at the results and understand which option is doing better, so it can all be done without having to recruit or interview test participants.

Of course, A/B testing in production works best on things like web or mobile applications where you can not only show different interfaces to different customers, but where you can also easily switch all of your users to the winning interface without having to ship them a new box full of software or a new physical device. I wouldn’t recommend trying it if you’re designing, for example, a car.

What It Does Poorly

Now imagine that, instead of adding a single screen to an already existing check out flow, you are tasked with designing an entirely new check out flow that should maximize revenue and minimize the number of people who abandon their shopping carts. In creating the new flow, there are hundreds of design decisions you need to make, both small and large. How many screens should it have? How much up-selling and cross-selling should you do? At what point in the flow do you ask users for payment information? What should the screens look like? Should they have the standard header and footer, or should those be removed to minimize potential distractions for users when purchasing? And on and on and on…

These are all just a series of small decisions, so, in an ideal world, you’d be able to A/B test each one separately, right? Of course, in the real world, this could mean creating an A/B test with hundreds of different variations, each of which has to be shown to enough users to achieve statistical significance. Since you want to roll out your new check out process sometime before the next century, this may not be a particularly appealing option.

A Bad Solution

Another option would be to fully implement several very different directions for the check out screens and test them all against one another. For example, let’s say you implemented four different check out processes with the following features to test against one another:

This might work in companies that have lots of bored engineers sitting around waiting to implement and test several different versions of the same code, most of which will eventually be thrown away. Frankly, I haven’t run across a lot of those companies. But even if you did decide to devote the resources to building four different check out flows, the big problem is that, if you get a clear winner, you really don’t have very clear idea of WHY users preferred a particular version of the check out flow over the others. Sure, you can make educated guesses. Perhaps it was the particularly soothing shade of blue. Or maybe it was the fact that there weren’t any marketing questions. Or maybe it was aggressive up-selling. Or maybe that version just had the fewest bugs.

But the fact is, unless you figure out exactly which parts users actually liked and which they didn’t like, it’s impossible to know that you’re really maximizing your revenue. It’s also impossible to use those data to improve other parts of your site. After all, what if people HATE the soothing shade of blue, but they like everything else about the new check out process? Think of all the money you’ll lose by not going with the yellow or orange or white. Think of all the time you’ll waste by making everything else on your site that particular shade of blue, since you think that you’ve statistically proven that people love it!

What Qualitative Testing Does Well

Despite the many wonderful things about A/B testing, there are a few things that qualitative testing just does better.

Find the Best of All Worlds

Qualitative testing allows you to test wildly different versions of a feature against one another and understand what works best about each of them, thereby helping you develop a solution that has the best parts from all the different options. This is especially useful when designing complicated features that require many individual decisions, any one of which might have a significant impact on metrics. By observing users interacting with the different versions, you can begin to understand the pros and cons of each small piece of the design without having to run each one individually in its own A/B test.

Find Out WHY Users Are Leaving

While a good A/B test (or plain old analytics) can tell you which page a user is on when they abandon a check out flow, it can’t tell you why they left. Did they get confused? Bored? Stuck? Distracted? Information like that helps you make better decisions about what exactly it is on the page that is causing people to leave, and watching people using your feature is the best way to to gather that information.

Save Engineering Time and Iterate Faster

Generally, qualitative tests are run with rich, interactive wireframes rather than fully designed and tested code. This means that, instead of having your engineers code and test four different versions of the flow, you can have a designer create four different HTML prototypes in a fraction of the time. HTML prototypes are significantly faster to produce since:

• They don’t have to run in multiple browsers, just the one you’re testing
• They don’t have any backend code that needs to be done
• They frequently don’t have a polished visual design (unless that’s part of what you’re testing)

And since making changes to a prototype doesn’t require any engineering or QA time, you can iterate on the design much faster, allowing you to refine the design in hours or days rather than weeks or months.

How Do They Work Together?

Qualitative Testing Narrows Down What You Need to A/B Test

Qualitative testing will let you eliminate the obviously confusing stuff, confirm the obviously good stuff, and narrow down the set of features you want to A/B test to a more manageable size. There will still be questions that are best answered by statistics, but there will be a lot fewer of them.

Qualitative Testing Generates New Ideas for Features and Designs

While A/B testing helps you eliminate features or designs that clearly aren’t working, it can’t give you new ideas. Users can. If every user you interview gets stuck in the same place, you’ve identified a new problem to solve. If users are unenthusiastic about a particular feature, you can explore what’s missing with them and let them suggest ways to make the product more engaging.

Talking to your users allows you to create a hypothesis that you can then validate with an A/B test. For example, maybe all of the users you interviewed about your check out flow got stuck selecting a shipment method. To address this, you might come up with ideas for a couple of new shipment flows that you can test in production once you’ve confirmed that they’re less confusing with another quick qualitative test.

A/B Testing Creates a Feedback Loop for Researchers

A/B tests can also improve your qualitative testing process by providing statistical feedback to your researchers. I, as a researcher, am going to observe participants during tests in order to see what they like and dislike. I’m then going to make some educated guesses about how to improve the product based on my observations. When I get feedback about which  recommendations are the most successful, it helps me learn more about what’s important to users so I make better recommendations in the future.

Any Final Words?

Separately, both A/B testing and qualitative testing are great ways to learn more about your users and how they interact with your product. Combined, they are more than the sum of their parts. They form an incredibly powerful tool that can help you make good, user-centered product decisions more quickly and with more confidence than you have ever imagined.

For more information on our approach to getting customer feedback, check out: