Latest Tweets

Improving the ROI for Your User Research

| 09.23.2009

So, you decided to do some user research in order to find out where you can make improvements. After a few hours of user interviews, you ended up with a notebook full of scribbled information that all seemed really critical. How in the world do you figure out what to do with all that information?

If your answer is “talk about it all abstractly with everybody in the company or write a huge paper that nobody will read and then go on with business as usual,” you’re in good (bad?) company.

But you have to DO something with all that data. You have to analyze it and turn it into actionable items that your engineering department can use to fix your product. It’s not always easy, but I’m going to give you an approach that should make it a little easier. This isn’t the only way to do your test analysis, but it’s one of the quickest and easiest that I’ve found when you are concerned with key metrics.

When to use this method:

  • You have an existing product with a way to measure key metrics, and you’re interested in improving in particular areas related to your bottom line
  • You have a limited research and development budget and want to target your changes specifically to move key metrics
  • You are looking for the “low hanging fruit” that is getting in the way of your users performing important tasks with your product
  • You are working in an agile development environment that is constantly tweaking and improving your product  and then testing the changes

When not to use this method:

  • You have an existing product that you are planning to completely overhaul, and you want to understand all of the major problems before you do your redesign
  • You are trying to create an overall awesome, irresistible user experience that is not related to a specific metric
  • You are designing a new product or feature and are observing people using other products to identify opportunities for innovation

If you fall into the first bucket, read on…

The Five Basic Steps:

  • Identify key metrics you’d like to improve
  • Identify the tasks on your site that correlate with improvement in those metrics
  • Observe people performing the appropriate tasks
  • Identify the barriers preventing people from completing or repeating the tasks
  • Develop recommendations that address each specific barrier to task completion

Step 1: Identify Key Metrics

First, identify the metrics that you care about, and look at the distinct steps that your users are most likely to take to change those metrics. This way, you can plan your qualitative testing to show you exactly what is getting in the way of your users reaching each subsequent step.

As an example, let’s imagine that you have a social networking site that earns revenue when people give virtual gifts to each other to display on their profile pages. Because your revenue depends on lots of users purchasing lots of things for each others’ pages, you might care about the following metrics: how many people register, how many people customize their profile pages, how many people make friends, how many friends they make, and how many people purchase one or more gifts. Obviously, that last is the most important, since that’s where your revenue comes from, but the others are also important, since improving those numbers should increase everything downstream, if done in the correct way.

Your goal is to get people from point A to point D as quickly and pleasantly as possible. To do that, you want to look at all of the barriers you have erected in the way of your users accomplishing those tasks.

Here’s are the pieces of the user flow that change your key metrics:

Registration > Profile Customization > Finding a Friend > Purchasing a Gift

Step 2: Identify Individual Task Flows

Now that you know what your key metrics are, you need to make sure that you observe your test participants trying to accomplish the tasks that will eventually lead to changing those metrics. To plan your tasks and analyze your data, it helps to break down your individual tasks into smaller user flow steps based on your user interface.

For example, while the number of successful registrations may be a single metric, there may be several screens or discrete interactions involved in your entire registration process. Make a list of what they are now.

As an example, I ran a test recently for a product that had a 4 step registration process. The user flow for registration on the site looked like this:

Landing Page > User name > Personal Info > Download

Step 3: Observe People Performing Tasks

So let’s say that, based on the metrics you want to change in your hypothetical social networking site, you’ve determined that you need to gather data about registration, profile customization, finding friends, and purchasing a gift. You’ve scheduled 5 people in your target demographic to come in for user test sessions. I’m not going to teach you everything you need to know about running a test, but here are some basics.

First, let the user begin using your product however they want and note all of the things that they do that aren’t the tasks you’ve identified. Remember, any time spent not doing your target tasks is time that is not being spent improving your metrics, so you want to know what these other things are! If you give your participant a targeted task right away, you will never learn what other things they are doing instead. Finding these distractions allows you to eliminate them or change them so that they also improve your key metrics.

Watch everything your user does with the perspective of the ideal flow for your metric. Do your participants wander off track or getconfused during registration? Do they fail to find their own profile? Do they not have the information they need to find a friend? Do they fail to understand what gifts are or how to purchase them? Do they get distracted by a totally different part of your site that doesn’t contribute to improving your key metrics?

Once you’ve allowed the participant some free exploration time, you may need to move them to a particular task. For example, in our current example, it might be that you need to ask the participant to try to purchase a gift, even if that’s something they wouldn’t do in the context of a study environment. You can move them along by saying something casual like, “Did you notice that there is a way to purchase gifts? What do you think about that?”

Meanwhile, record what the participant is doing and encourage them to talk about their impressions of the product.

Step 4: Identify Barriers

Once you’ve gathered data by watching 4 or 5 people repeat the same tasks, you need to analyze your data to figure out where the barriers are and communicate them effectively to the team. Determining the barriers should be pretty easy. Just ask yourself the following questions:

  • Where did participants seem confused or distracted and stop performing a task?
  • Which things took longer than they should have?
  • Why did participants fail to accomplish a task?
  • What tasks did the participants perform that would not improve the key metrics?

In the product I tested with the four step registration process, the metrics showed significant drop off one each screen. On screen number 2, the user was asked to choose a unique user name and enter some information. It had a surprising amount of drop off considering how simple it was. After watching a few user tests, I noticed that many people were trying to select user names that were not available, but they were failing to notice the error message telling them to choose a different one. Even if they noticed the error, several people had a tough time finding a user name they liked and would spend minutes typing in word after word, getting frustrated, and finally giving up.

By watching people try to register for the product, we identified several significant barriers to successful registration in just one step of the process. In fact, we found similar problems – some large, some small – in each of the steps.

Once we’d identified the barriers, organizing and explaining them to the team was easy with a simple, annotated flow of the steps leading up to the goal along with the problems associated with each step. For example, the flow for the registration test might have looked like this:

Landing Page

  • Didn’t understand the product
  • Felt the product was aimed at teens
  • Thought the network was very small

User name

  • Didn’t know when a user name was taken
  • Couldn’t come up with a decent name

Personal Info

  • Were concerned about privacy
  • When failed Captcha, all info had to be re-entered


  • Were uncomfortable downloading b/c of viruses
  • Didn’t know what they were downloading


  • Were bored during registration process or complained that it was taking a long time

Step 5: Develop Recommendations

Now that you know what barriers are keeping your customers from accomplishing the goals you’ve set for them, you need to generate recommendations for ways to remove those barriers. You can do this by looking at exactly what the barrier is and what the users’ reactions were to it, and then brainstorming ideas to help the user overcome that barrier. Yes, this is the tough, somewhat creative part. You don’t want to just take any recommendation that the user gives you. You need to understand what problem they’re having and come up with a way to fix it that doesn’t cause any other problems.

Going back to my example, once we found the problems on the user name page, we looked at several solutions. We definitely needed to make it more obvious to the user when they selected a name that was already taken. We came up with a few simple ways to solve this particular problem: we could suggest other user names when people tried one that was already taken; we could make the error more noticeable; we could make the Next button obviously disabled so that they couldn’t even try to move on; we could free up some names that had been taken but weren’t currently in use so that the selection of user names was better. We then selected a couple of these solutions based on expected ROI calculations.

Even the lowest effort of these suggestions, improving the visual design of the error, caused an immediate and statistically significant decrease in drop offs for that step in the registration flow, and eventually improved revenue by increasing the number of successful sign ups. The barrier was lowered, if not entirely removed.

You may have noticed that the last section is labeled “Overall.” This is where you look at the process as a whole, rather than a sum of its parts. For example, one comprehensive solution for drop off at each step might be to reduce the number of steps it takes to register (almost never a bad idea!). Doing this wouldn’t necessarily mean that you wouldn’t have to address the other problems individually, but you might get fewer drop offs simply because it would take less time for users to finish the process.

How Is This Different?

You might be wondering how this is different from more traditional ways of running user tests and analyzing data. In any user test, you’ll need to come up with tasks, observe users, figure out where they’re having problems, and come up with solutions for the problems.

There is a difference though. My experience has been that companies do not fix every single UX problem that they find in user research. I don’t love this, but I do understand the need to prioritize important changes.

So, if you’re only going to fix a few problems, you should make an effort to identify the most important ones. The counter intuitive part is that the most important problems might not be the worst problems from a user experience stand point, but they will definitely be the ones that are keeping your users from reaching the goals that improve your most critical metrics.

Using this framework will help you identify your key user flows, find and communicate the major barriers to success, and propose targeted solutions that will improve both your user experience and your business.

For more information on our approach to getting customer feedback, check out:

Why I Hate Paper Prototypes

| 09.16.2009

Ok, maybe hate is a little strong. Paper prototypes and sketches have their place in interaction design. For example, they’re great for helping to quickly brainstorm various different approaches to a problem at the beginning of a design process. They’re also a very fast and cheap way to illustrate a new idea, since most people can draw boxes faster than they can build interactive prototypes. But, in my opinion, they have several serious drawbacks.

Before I get too far into this, let me define what I mean by a paper prototype, since I’ve heard people use the term to refer to everything from sketches on actual pieces of paper (or cocktail napkins in a couple of cases) to full color printed mockups with a polished visual design. In this instance, I’m referring to a totally non-interactive screen, mockup, or sketch of any sort of application that is meant to be shared with customers, test participants, or team members. It can be printed on actual paper or shown on a computer screen, but whatever the viewer does to it, a paper prototype is not interactive.

So, what don’t I like about them?

Screen vs. Paper

This first couple of peeves apply to screens that are actually printed out or drawn directly on paper. With a few exceptions that I’ve listed below, I’ve found this approach to be really counterproductive.

Iterating On a Design

One of the biggest problems with hand drawn sketches on paper has less to do with user interactions and more to do with my work flow as a designer. Sure, sketching something quickly on a piece of paper can be quick, but what happens when I realize that I want to swap two sections of the screen? I can draw arrows and lines all over it, but that gets messy pretty fast. Whenever I want to make any changes to my design, I need to create a whole new sketch. This can mean redrawing the entire screen quite a few times.

If I’m creating a design in HTML or any other prototyping tool, the very first version might take a little longer than a quick sketch, but the second through nth iterations are a whole lot faster. And, as a bonus, I can check them into source control, which means I’m a lot less likely to lose my work than if I have dozens of pieces of paper scattered all over my office.

Interacting With Paper

Whether they’re sketched out by hand or printed out on paper, people interact with paper screens differently than they do with computer screens. They view them at a different angle. They focus on different parts of the screen. They use their hands to interact with them rather than a mouse and keyboard. Any feedback that you get on a printed design will be colored by the fact that people are fundamentally interacting with it differently than they would if it were on a computer screen.

Given all of these drawbacks, there are a few situations when designs printed on paper can be used effectively:

  • You are at the very beginning of the design process, and you want to explore a bunch of different possible directions with other team members very quickly before committing yourself to fleshing out one or two specific options.
  • You’re designing material that is meant to be printed, like brochures, user manuals, books, etc. In this case, you want to know how people will interact with the printed media.
  • Your product is an interface for some sort of embedded or small screen device that would be very difficult to replicate in a quick interactive prototype. For example, a screen for certain mobile devices or the heads-up display for a car dashboard might be hard to show interactively in the appropriate context.
  • You have several different visual designs, and you’d like to show them all to users at the same time in order to see which one is the most attention-getting. You’ll still need to show the designs on screen, of course, since colors can vary so much between screen and print, but it can be helpful to lay out several pieces of paper so that the various options can easily be compared.
  • You need to share screens with people in an environment with absolutely no access to a computer whatsoever. You know, maybe you’re in the middle of a meeting and need to sketch something quickly, or the rest of your design team is Amish, or you are designing in a post-apocalyptic wasteland where the computers are trying destroy humanity.

On the other hand, if you’re designing desktop or web applications for standard computers, at the very least, show your prototypes on a computer, even if they are not interactive!

A few reasons why it’s especially important to show interactive designs on a screen:

Animations and interactions

I am an interaction designer. A big part of my job is to determine what happens when a user interacts with a product. Sometimes that’s obvious. For example, if I click on a link with some text that reads, “Contact Us,” I expect to be able to communicate in some way with the people who have created the product.

But is it so obvious? Back in the day when links could only take you to another static web page, sure it was. But now, all sorts of things could happen. I might have different behavior on hover vs. on click. I could be given the option to have a live chat with a rep. I might be presented with an inline contact form so that I don’t have to leave the page I’m visiting. The contact form could have some information already prefilled for me based on my present location or account type. There could be animation involved in displaying the contact information. There could be some sort of contact wizard that would change later screens depending on choices the user makes initially.

All these interactions are much harder to convey with paper prototypes than with interactive wireframes. Sure, you can make a different screen showing each stage of the interaction, but with anything more complicated than a few steps, your observers can get lost pretty fast shuffling through papers. Having it all working in an interactive prototype allows users to click through and explore naturally. They can also discover things like hover interactions or animations on their own, which are particularly unsuited to paper.


What? You don’t need to test your designs with users? Your designs spring from your brain fully formed and perfect? Well, good for you. The rest of us mortals actually like to show our designs to test participants and get feedback from people who aren’t familiar with the product.

Now, I’ve run a lot of user tests. I’ve run them with working products, interactive prototypes, pictures of screens displayed on computers, pure paper prototypes, and physical mockups of products. I’ve used prototypes built with everything from HTML to Visio to cardboard. The one constant was that the closer the prototype mimicked the final product interaction, the fewer problems we found once the product was built. And since we recommend iterative testing during development rather than waiting  to test until the product is 100% built (you know, just in case your design wasn’t entirely perfect the first time around), an interactive wireframe is the best trade off of speed for functionality.

Communicating the design

You can have the best design in the world, but if you don’t communicate it effectively to the engineers who have to build the thing, it doesn’t matter one bit. Of course, once you’ve designed all of your static screens, you could paste them into a big spec with extensive documentation about what each button on each screen does and how many seconds an animation is supposed to take and blah blah blah. I’ve done it when it’s been specifically requested by a client. I’ve also never met an engineer who was particularly happy to read through one of those three hundred page documents. Besides, just the thought of making changes to every screen in that document every time there’s a change makes me tired.

An interactive prototype, when built intelligently, is a different story. It allows engineers to see exactly how every element of the design is supposed to work. What happens when you click on a link? Click on it to find out! Is the animation supposed to sweep horizontally or vertically? Take a look! Of course, you’re free to add notes to screens as necessary to fully communicate absolutely everything, but the difference between using this lightweight approach and a full design document for reference is night and day.

What’s the alternative?

So, what should you use other than paper prototypes? I’ve mentioned it throughout the post, but I’ll be perfectly clear. Your best bet is an interactive wireframe that mimics the behavior of the actual product as closely as possible. It doesn’t have to be hooked up to a database on the back end or use real data. It doesn’t even have to be fully functional if you’re only testing parts of it or if you’re annotating it with notes. But it should make the user, whether they’re a test participant, an engineer, or your design manager, feel like they’re actually performing tasks with the product.

Luckily, you’ll find plenty of products out there that will help you build interactive wireframes with very little technical expertise. You don’t need to be an engineer or a Flash expert to make your screens clickable. My favorite tool is Dreamweaver for creating quick HTML prototypes with basic animations and interactions, but other people have had good luck with applications like Axure and or even PowerPoint (although, that has some serious limitations).

There is, of course, one quick caveat. I know I’ve been harping on how closely your prototype should match your eventual product. However, you do not need, and probably shouldn’t have until quite late in the design process, a full visual design for your interactive prototypes. Why is that? Well, I’ve found that making the visuals look too polished can actually focus the user on the aesthetics rather than on whether they can accomplish a task. A lovely or flashy visual design can distract the user from interaction problems that you’d rather find early in the process.

For best results, focus on creating a simple, clean wireframe with as much interactivity as you can possibly give it. Your users will thank you for it.

6 Stupid Excuses for Not Getting Feedback

| 09.11.2009

Almost every company I talk to wants to test their products, get customer feedback, and iterate based on real user metrics, but all too often they have some excuse for why they just never get around to it. Despite people’s best intentions, products constantly get released with little to no customer feedback until it’s too late.

I’m not trying to promote any specific methodology for testing your products or getting customer feedback. Whether you’re doing formal usability testing, Fast Insight Testing, contextual inquiries, surveys, a/b testing, or just calling up users to chat, you should be staying in contact with customers and potential customers throughout the entire design and development process.

To help get you to stop avoiding it, I’ve explored six of the most common stupid excuses for not testing your designs and getting feedback early.

Excuse 1: It’s a design standard

You can’t test every little change you make, right? Can’t you sometimes just rely on good design practices and standards? Maybe you moved a button or changed some text. But the problem is, sometimes design standards can get in the way of accomplishing your business goals.

For example, a few months ago at a talk given by Bill Scott, he talked about a developer who had a/b tested the text on a link. One option read, “I’m now on Twitter.” The second read, “Follow me on Twitter.” The third read, “Click here to follow me on Twitter.” Now, anybody familiar with “good design practices” will tell you that you should never, ever use the words “click here” to get somebody to click here. It’s SO Web 1.0. But guess which link converted best in the a/b test? That’s right. “Click here” generated significantly more Twitter followers than the other two. If that was the business goal, the bad design principle won hands down.

Does this mean that you have to do a full scale usability test every time you change link text? Of course not. Does it mean you have to use the dreaded words “click here” in all your links? Nope. What it does mean is that you should have some way to keep an eye on the metrics you care about for your site, and you should be testing how your design changes affect customer behavior, even when your changes adhere to all the best practices of good design. So, to put it simply:  prioritize what you care about and then make sure you test your top priorities.

Excuse 2: Company X does it this way

I can’t tell you how many times I’ve heard, “Oh, we know that will work. Google/Facebook/Apple does it that way.” This is the worst kind of cargo cult mentality. While it’s true that Google, Facebook, and Apple are all very successful companies, you aren’t solving exactly the same problem that those companies are, you don’t have exactly the same customers that they do, and you don’t know if they have tested their designs or even care about design in that particular area. You are, hopefully, building an entirely different product, even if it may have some of the same features or a similar set of users.

Is it ok to get design ideas from successful companies? Of course it is. But you still need to make sure your solutions work for your customers.

I previously worked with a company that had a social networking product. Before I joined them, the company decided that, since other companies had had good luck with showing friend updates, they would implement a similar feature, alerting users when their friends updated their profiles or bought products. Unfortunately, the company’s users weren’t very interested in the updates feature as it was implemented. When we finally asked them why they weren’t using the feature, the users told us that they would have been very interested in receiving an entirely different type of update. Of course, if the company had connected with users earlier in the process, they would have rolled the feature out with the right information and gotten a much more positive reaction on launch.

Another thing to remember is that just because a company is successful and has a particular feature doesn’t mean it’s that exact feature that makes them successful. Google has admitted that the “I’m Feeling Lucky” button loses them page views, but they keep it because they, and their customers, like the feature. That doesn’t mean it’s a good business plan for your budding search engine startup to adopt a strategy of only providing people with the equivalent of the “I’m Feeling Lucky” button. In fact, this is a great example of why you might need to employ multiple testing methods: qualitative (usability, contextual inquiry, surveys), to find out if users find the feature compelling and usable, and quantitative (a/b, analytics), to make sure that the feature doesn’t bankrupt you.

The bottom line is, it doesn’t matter if something works for another company. If it’s a core interaction that might impact your business or customer behavior, you need to test new features and designs with your customers to make sure that they work for you. Obviously, you also need to make sure that you’re not violating anybody’s IP, but that’s another blog post.

Excuse 3: We don’t have time or money

The fact is you don’t have time NOT to test. As your development cycle gets farther along, major changes get more and more expensive to implement. If you’re in an agile development environment, you can make updates based on user feedback quickly after a release, but in a more traditional environment, it can be a long time before you can correct a major mistake, and that spells slippage, higher costs, and angry development teams.

I know you have a deadline. I know it’s probably slipped already. It’s still a bad excuse for not getting customer feedback during the development process. You’re just costing yourself time later. Besides, it’s not as expensive as you think! You don’t always have to do some enormous user test in a formal lab with one way glass. We advocate cheap and dirty Fast Insight Testing, which  you can learn about  here.

Excuse 4: We’re new; We’ll fix it later

I hear this a lot from startups, especially agile ones, that are rushing to get something shipped, and it’s related to the previous excuse. Believe me, I do understand the pressures of startups. I know that, if you don’t ship SOMETHING you could be out of business in a few months. Besides, look at how terrible some really popular sites looked when they first started! You have to cut something, right?

Great. Cut something else. Cut features or visual polish. Trust me, people will forgive ugly faster than they’ll forgive completely unusable or confusing. Whatever you decide to cut, don’t cut getting customer feedback during your development process. As I mentioned in Excuse 3, you can do it faster than you think. But more importantly, if you ship something that customers can’t use, you can go out of business almost as fast as if you hadn’t shipped anything.

Potential users have a lot of options for products these days. If they don’t understand very quickly all the wonderful things your product can do for them, they’re going to move on.  Take a few hours to show your ideas to users informally, and you will save your future self many hours of rework.

Excuse 5: It’s my vision; users will just screw it up

This can also be called the “But Steve Jobs doesn’t listen to users…” excuse. Except, that’s not true. Recently, in an interview with the New York Times, when asked why the Nano got a camera while the Touch didn’t, Jobs responded, “Originally, we weren’t exactly sure how to market the Touch…What happened was, what customers told us was, they started to see it as a game machine. We started to market it that way, and it just took off. And now what we really see is it’s the lowest-cost way to the App Store, and that’s the big draw. So what we were focused on is just reducing the price to $199. We don’t need to add new stuff. We need to get the price down where everyone can afford it.” Apparently, this myth that Jobs doesn’t listen to customer feedback or make decisions about product features based on talking to users is overblown.

The fact is, understanding what your users like and don’t like about your product doesn’t mean giving up on your vision. You don’t have to make every single change suggested by your users . You don’t have to sacrifice a coherent design to the whims of a focus group.

What you should do is connect with your users or potential users in various different ways – user tests, contextual inquiry, metrics gathering, etc. – to understand whether your product is solving the problem you think it is for the people you think are your customers. And, if it’s not, it’s a good idea to try to understand why that is and develop some ideas for how to fix it.

Besides, even if Steve Jobs never listened to a single customer in his entire life, how many wanna-be Steves do you think there out there whose companies failed miserably because nobody wanted to use their products?

Excuse 6: It’s just a prototype to get funding

This is an interesting one, since I think it’s a fundamental misunderstanding of the entire concept of customer research. When you’re building a prototype or proof of concept, you still need to talk to your customers. The thing is, you may have an entirely different set of customers than you thought you did.

Maybe you think the target market for your new networked, Wi-Fi lunchbox is 11-13 year old girls, but they’re not going to pay you to build the first million units and get them into stores. Your first customers are the venture capitalists or the decision makers at your company or whoever is going to look at your product and decide whether or not to give you money.

Even if they’re not your eventual target market, it’s probably a good idea to spend some time talking with whomever you’re trying to get to fork over the cash. I’m not saying you should change your entire product concept based on this feedback. I mean, if you really want to start the company on your credit cards and a loan from your mom, don’t change a thing! The important take away here is that you may have different audiences at different points in your company’s life. And the best way to find what they all want is to talk to them!

Out of Excuses?

Those are the most common excuses I hear, but I’m sure you can think of some clever ones. Then again, your time is probably better spent connecting with your users, understanding their problems, and thinking of ways to address them.

For more information on our approach to getting customer feedback, check out: