Category: Interaction Design
Mia Silverman | 02.25.10
We’re excited to share that our redesign of the Intuit Trends application has just launched!
Intuit Trends is a free online application that lets small businesses compare how they are doing financially (such as income, expenses, profits, etc.) with other businesses that are similar to them.
We talked with small business owners to find out what they most wanted to understand about how their businesses compared to their peers/competitors, and then introduced some big improvements to the previous design. This release delivers the first preliminary round of changes, and there are many more to come.
Key features of our redesign effort included in this preliminary release are:
- Introduced a new, personalized Scorecard
- Before the redesign, the Trends application only offered small businesses the ability to view general trends about how their peers and competitors were doing. Now, small business owners can also see how their own company compares to their peers along three key business metrics and receive individual and overall scores.
- Provided a simple bar chart and scoring solution
- Now that the site was going to show comparisons between a specific business and its peers, we needed a compelling way to present this information. To get started, we explored a variety of different design options for how best to show comparison data. Next, we usability tested our ideas and moved forward with the design that users liked best. Our final solution combined a simple bar chart with a visual quartile score, in a format that was easy for scanning.
- Created a clean, fresh, and consistent visual design
- We all know that it is important for information on a site to not only be useful, but to also be presented in a clear, visually appealing way. To help the Trends site appeal to its audience and convey the desired tone, we created a modern, clean look and feel for the site that also followed Intuit’s brand guidelines. We also introduced a consistent color scheme to use for “Me” and “Peers” across the site to help users differentiate between the two types of data.
You can check out our redesign for yourself at: https://workplace.intuit.com/db/bejqb2kpn.
We’d love to hear what you think!
Julie Stanford | 02.22.10
While consumer smart grid energy portals are an important area for user centered design, there is an often overlooked design challenge in helping utilities craft a demand response (DR) program that really works. For readers unfamiliar with the term, demand response is a program utilities are exploring which asks customers to reduce electricity use during peak times in exchange for financial incentives. Utilities have recently launched DR programs with the basic assumption that providing access to energy usage data and an economic incentive would motivate users to change their behavior. Turns out, encouraging behavior change is not so easy. With that challenge in mind, I decided to look at what’s been done in the past to motivate energy behavior change and see how learnings from past efforts can be applied to the design of demand response systems – from a consumer perspective.
Based on my literature review, the following are ten ideas to consider when crafting your demand response program to create an effective user experience:
1. Carefully craft and explain rate structures
Construct the rates and program carefully with consideration of more than the just the economics. A 2008 study of a time of use pricing pilot found that suggestions for behavior change were highly time sensitive to key family patterns such as mealtimes and did not work if they were disruptive to the household. To make sure you create a structure that is within the capabilities of your target audience, consider conducting a user study to understand how household behaviors align with specific time periods. Then you can craft a program with realistic expectations for consumption management and provide users with actionable advice that they can follow without changing their family patterns.
2. Create a goal – get commitment – provide feedback
Consider structuring the DR program so that participants get a specific difficult goal for participation, commit to the goal, and then get feedback on their goal. This type of structure has proven repeatedly to be one of the strongest approaches for motivating energy behavior change. In one study, researchers gave households a difficult goal (20% energy reduction), easy goal (2% reduction), or no goal for energy use. All groups (including the no goal control) were then given information on which appliances used the most energy. The goal was also combined with feedback or not. Households who received a difficult goal + feedback conserved the most (15.1%) and were the only group to significantly differ from the control. Participants with the easy goal did not differ in behavior from the control at all. To make an even stronger program, consider an extra reward if the goal is reached.
3. Provide frequent feedback
The more continuous the feedback, the more effective the intervention. In a seminal study conducted over 30 years ago in 1979, households were given continuous feedback over a period of 11 months about monetary costs of electricity use by means of a monitor displaying electricity use cents per hour. On average, households that had a monitor installed reduced electricity by 12%. Although hourly, daily, weekly, and monthly feedback all create savings effects, the more frequent the feedback, the more effective it is. Consider creative ways to deliver that feedback via web portals, in home devices, smart phones or on SMS (intermittently).
4. Emphasize choice and control
One study considered people’s resistance to installing automatic day/night thermostats. Once the thermostat was redesigned to allow residents to override the system temporarily, the thermostat was much more attractive to residents – even though in actual use most people never overrode them. Similarly, a DR program should emphasize choice and control – people can opt into the program and still have full control over their consumption.
5. Tap into the power of the group
One energy conservation program that had a lot of success enrolled people in groups where they discussed and compared conservation behavior with their social group over a long term basis. Similarly, virtual networks of known groups can be set up to motivate participation in DR programs – for example by tapping into existing social networks of friends on Facebook to encourage participation.
In addition, consider a structure that offers additional savings if everyone in a group or neighborhood participates and reaches a set goal (see point 1 above). One study indicated that an incentive that offered on an individual and group level – in this case for all residents of one apartment building — was more effective than solely an individual incentive.
6. Frame program benefits as avoiding loss rather than emphasizing gain
The amount of joy that someone experiences when winning $100 is not equal to the consternation suffered when losing the same amount. Most people are more willing to take risks to avoid or minimize a loss than to increase their fortune. So, focus on showing residents how much money they are losing every month by not enrolling in demand response. Once the loss is obvious, people will take action.
7. Integrate complex information
When calculating energy savings, people usually can’t take into account all the elements such as rising fuel costs, the real long terms benefit, etc… So, do the math for them! Give price information that shows the full savings, presented as avoidance of a negative consequence of non-action (see point above). Use the actual data you already know about the consumer’s energy usage to make the information actionable and real.
8. Present information using vivid personal stories and videos
Statistical data summaries and impersonal information are less effective than case studies and colorful stories for motivating participation. For example, imagine that you are considering a new car and are choosing between a Volvo and a Saab. Consumer Reports informs you that the consensus of its studies is that the Volvo has a better repair record. That evening, you go to a party and run into an acquaintance who tells you a horrific story about a Volvo. Although the Consumer Reports article is based on hundred of repair records and your friend’s story is just one additional data point, most people will be swayed by their friend not to buy that car.
When communicating the benefits of a direct response program, demonstrate benefits with concrete stories about real people who save more energy than average but are “just like you. To be even more effective, present the content in videos. Numerous studies have also shown that videos of people modeling the desired actions are more effective in getting people to change their behavior than written information or lectures.
9. Use a foot in the door strategy
Individuals who agree to a small initial task are much more likely to agree to a larger request. So, instead of asking people to enroll in the full DR program immediately, first ask people to participate in a small act, such as filling out a survey, and then later ask them to consider signing up for demand response as a follow up to the first request. For example, one representative study showed that the percentage of people agreeing to an unattractive sign being put on their front lawn encouraging people to drive carefully increased dramatically (from 17% to 55%) if they had first been given the opportunity to sign a petition favoring safe driving.
10. Communicate trust
One key differentiator for successful energy programs is successful marketing to get people to even consider trying it out. We’ve found in our research that people inherently don’t trust their utility so partner with a local organization people do trust to market your program. In a marketing experiment conducted in Minnesota, a county government contracted with a private company to install energy saving equipment in homes in exchange for payment of a percentage of the value of the energy saved. To market the program, households received one of three types of letters: one letter was sent on company letterhead with no mention of cooperation with the county, one letter went out on company letterhead and mentioned the county’s role, and the third went out on county letterhead and was signed by the County Board of Commissioners. The source of information had a profound effect on consumer response – request for energy audits came from 6%, 11% and 26% respectively of households receiving the three types of letters.
Next Steps
We’re continuing to do more research in this area and will publish more insight expanding into some areas mentioned above. In the meantime, here’s a partial list of references…happy reading!
- Abrahamsen, W., Steg, L., Vlek, C., & Rothengatter, T. (2005). A review of intervention studies aimed at household energy conservation. Journal of Environmental Psychology, 25, 273-291.
- Krantz, D. H., & Kunreuther, H. C. (2007). Goals and plans in decision making. Judgment and Decision Making, 2(3), 137-168.
- Geller, E. S., (1992) It takes more than information to save energy. American Psychologist, 814-815.
- Geller, E. S., Winett, R. A., & Everett, P. B. (1982). Preserving the environment: New strategies for behavior change. Elmsford, NY: Pergamon Press.
- Lutzenhiser, S. et al (2009) Beyond the Price Effect in Time-of-Use Programs: Results from a Municipal Utility Pilot, 2007-2008. Presented at the International Energy Program Evaluation Conference, Portland, OR, August 12-14, 2009. http://drrc.lbl.gov/pubs/lbnl-2750e.pdf
- McKenzie-Mohr, D. and Smith, W. (1999) Fostering Sustainable Behavior. Gabriola Island, B.C., Canada: New Society Publishers.
- Swim, Janet et al. (2009) Psychology and global climate change: addressing a multi-faceted phenomenon and set of challenges, A report by the American Psychological Association Task Force on the Interface between psychology and global climate change. http://www.apa.org/science/about/publications/climate-change.aspx
- Winett, R. A., Hatcher, J. W., Fort, T. R., Leckliter, I. N., Love, S. Q., Riley, A. W., et al. (1982). The effects of videotape modeling and daily feedback on residential electricity conservation, home temperature and humidity, perceived comfort, and clothing worn: Winter and summer. Journal of Applied Behavior Analysis, 15(3), 381-402.
- Winett, R.A. and Geller, E.S., (1981) Comment on “Psychological research and energy policy”. American Psychologist, (425-426).
- Yates, S. and Aronson, E., (1983) A social psychological perspective on energy conservation in residential buildings. American Psychologist, (435-444).
- Stern, P. C., Aronson, E., Darley, J. M., Hill, D. H., Hirst, E., & Kempton, W. et al. (1986). The effectiveness of incentives for residential energy conservation. Evaluation Review, 10(2),
147-176.
For related posts about designing for the smart grid, check out:
Mia Silverman | 12.09.09
A recent SmartGridNews.com article praised Greenbox Technology for “deliver[ing] understanding to utility customers.” While Greenbox does provide useful functionality that differentiates it from its competitors, key improvements to its interaction design would go a long way to provide a better overall user experience.
The Good
To be fair, Greenbox does deserve a gold star for displaying energy data specifically in dollars ($). As I mentioned in an earlier post, consumers don’t understand energy units, such as kWh, and are motivated to change their behavior by saving money.
The Not So Good
But, what about the rest of the Greenbox design? Greenbox gets caught up in the same usability pitfalls I’ve seen in other consumer energy portals as well –too much information and not enough direct reference to the things that matter most to users. Here are my top three suggestions to help Greenbox, or any consumer energy portal, deliver an excellent user experience:
1) Simplify, simplify, simplify!
My foremost suggestion is simplicity. At first glance, the Greenbox interface looks very busy. There are many controls and buttons, and it tries to accomplish too much on a single page. Do users really need to be able to adjust their thermostat controls right on the main landing page, and also view their overview and data details at the same time? Instead, the home page could show only the most important information at first glance, and use color to further highlight and differentiate content on the page. The user can always drill-in to inside pages to access the less urgent information or access info on mouseover.
2) Engage users immediately with compelling content and proactive recommendations
My first response to the Greenbox dashboard is, “So what?” I’m missing context to understand the significance of my data. For example, is my overall usage good or bad? Am I using more or less energy than I did last month? Am I on track so far this month? Also, the portal doesn’t offer any suggestions for changing my behavior so it’s unclear what my next step should be.
To engage its users, Greenbox should focus on delivering insightful content that immediately provides context and motivation. Delight users by anticipating their questions and displaying answers at-a-glance. Provide simple charts and visuals on the dashboard to compare energy usage to different time periods and to others in the community so that people understand why this information matters.
3) Adhere to basic design principles
In addition to overall experience enhancements, here are some basic user interface guidelines to help GreenBox or any other site be more usable:
- Use clear titles Is the bar chart below showing data for all energy resource usage, just the electricity usage, or only heating and cooling (because that’s what is selected in green)? It’s hard to know for sure without a clear title for the chart.
- Use visual cues to help users appropriately group related itemsThe content relationships on GreenBox are a little hard to parse. For example, in the screenshot below, the “Heating/Cooling” button selected on the left side of the dashboard, actually updates the content shown on the right side, which is not immediately obvious.
Instead, GreenBox could make these two items more visually related to connect them. For example, put the same green border around the box on the right and give both elements the same background color (see update below). Then, I would be much more likely to quickly understand their relationship.
Updated design with green box:
- Visually identify hyperlinks & interactive featuresTo help users understand what’s clickable, it’s important to differentiate hyperlinks and interactive features from other static elements on the page. For example, on the GreenBox site buttons and links currently use the same dark text color as other static elements, which makes them difficult to identify.
A Minimal Redesign Idea
If we take the recommendations above and do a very minimal redesign of Greenbox limiting ourselves to only minor changes to content and layout, we already begin to see large improvements in terms of usability.
For example, in this minimal redesign, the relationships between content is clearer due to stronger visual cues and use of titles, the interactive elements more readily “pop” from the page, and inclusion of arrow icons begin to tell an interesting story about the user’s energy consumption.
But we’re not done yet…
Of course, it would take GreenBox more than just a simple redesign with no change in functionality to address all of our recommendations. As a next step, I think Greenbox would benefit to take a more critical look at its overall structure and content to ensure they lend themselves to a good user experience – in particular thinking about the best items to put on the home page and trying to answer the question “So what?”
Will we see a new and improved Greenbox site in the near future? We hope so. In the meantime, let’s learn both from the things that Greenbox does well and from the areas that Greenbox falls short so that we can continue to focus our energy on creating intuitive, meaningful, and persuasive user experiences.
Related post: Watts all the buzz about smart grid energy?
Laura Klein | 12.03.09
One of the greatest tools available to me as an interaction designer is the ability to see real metrics. I’m guessing that’s surprising to some people. After all, many people still think that design all happens before a product ever gets into the hands of users, so how could I possibly benefit from finding out what users are actually doing with my products?
Well, for one thing, I believe that design should continue for as long as a product is being used by or sold to customers. It’s an iterative process, and there’s nothing that gives me quicker, more accurate insight into how a new product version or feature is performing than looking at user metrics.
But there’s something that I, as a user advocate, care about quite a lot that is really very hard to measure accurately. I care about User Happiness. Now, I don’t necessarily care about it for some vague, good karma reason. I care because I think that happy users are retained users and, often, paying users. I believe that happy users tell their friends about my product and reduce my acquisition costs. I truly believe that happy users can earn money for my product.
So, how can I tell whether my users are happy? You know, without talking to every single one of them?
Although I think that happy users can equal more registrations, more revenue, and more retention, I don’t actually believe that this implies the opposite. In other words, there are all sorts of things I can do to retain customers or get more money out of them that don’t actually make them happy. Here are a few of the important business metrics you might be tempted to use as shorthand for customer happiness – but it’s not always the case:
Retention
An increase in retention numbers seems like a good indication that your customers are happy. After all, happier customers stay longer, right?
But, do you mean retention or forced retention? For example, I can artificially increase my retention numbers by locking new users into a long contract, and that’s going to keep them with me for awhile. Once that contract’s up, they are free to move wherever they like, and then I need to acquire a customer to replace them. And, if my contract is longer than my competitors’, it can scare off new users.
Also, the retention metric is easy to affect with switching barriers, which may increase the number of months I have a customer while making them less happy. Of course, if those switching barriers are removed for any reason – for example, cell phone number portability – I can lose my hold over long time customers.
While retention can be an indicator of happy customers, increasing retention by any means necessary doesn’t necessarily make your customers happier.
Revenue
Revenue’s another metric that seems like it would point to happy customers. Increased revenue means people are spending more, which means they like your service!
There are all sorts of ways I can increase my revenue without making my customers happier. For example, I can rope them into paying for things they didn’t ask for or use deceptive strategies to get them to sign up for expensive subscriptions. This can work in the short term, but it’s likely to make some customers very unhappy, and maybe make them ex-customers in the long run.
Revenue is also tricky to judge for free or ad-supported products. Again, you can boost ad revenue on a site simply by piling more ads onto a page, but that doesn’t necessarily enhance your users’ experience or happiness.
While increased revenue may indicate that people are spending more because they find your product more appealing, it can also be caused by sacrificing long term revenue for short term gains.
NPS – Net Promoter Score
The net promoter score is a measure of how many of your users would recommend your product to a friend. It’s actually a pretty good measure of customer happiness, but the problem is that it can be tricky to gauge accurately. It generally needs to be obtained through surveys and customer contact rather than simple analytics, so it suffers from relying on self-reported data and small sample sizes. Also, it tends to be skewed in favor of the type of people who answer surveys and polls, which may or may not be representative of your customer base.
While NPS may be the best indicator of customer happiness, it can be difficult to collect accurately. Unless your sample size is quite large, the variability from week to week can make it tough to see smaller changes that may warn of a coming trend.
Conversion to Paying
For products using the freemium or browsing model, this can be a useful metric, since it lets you know that people like your free offering enough to pay for it. However, it can take awhile to collect the data after you make a change to your product because you have to wait for enough new users to convert to payers.
Also, it doesn’t work well on ad-supported products or products that require payment upfront.
Most importantly, it doesn’t let you know how happy your paying customers are, since they’ve already converted.
Conversion to Paying can be useful, but it is limited to freemium or browsing models, and it tends to skew toward measuring the free part of the product rather than the paid product.
Engagement
Engagement is an interesting metric to study, since it tells me how soon and often users are electing to come back to interact with my product and how long they’re spending. This can definitely be one of the indicators of customer happiness for ecommerce, social networking, or gaming products that want to maximize the amount of time spent by each user. However, increasing engagement for a utility product like processing payroll or managing personal information might actually be an indicator that users are being forced to do more work than they’d like.
Also, engagement is one of the easiest metrics to manipulate in the short run. One time efforts, like marketing campaigns, special offers, or prize giveaways can temporarily increase engagement, but unless they’re sustainable and cost effective, they’re not going to contribute to the long term happiness of your customers.
For example, one company I worked with tried inflating their engagement numbers by offering prizes for coming back repeatedly for the first few days. While this did get people to return after their first visit, it didn’t actually have any effect on long term user happiness or adoption rates.
Engagement can be one factor in determining customer happiness, but this may not apply if you don’t have an entertainment or shopping product. Also, make sure your engagement numbers are being driven by actual customer enjoyment of your product and not by artificial tricks.
Registration
While registration can be the fastest metric to see changes in, it’s basically worthless for figuring out how happy your users are, since they’re not interacting with the product until after they’ve registered. The obvious exception is products with delayed (i.e. lazy) registration, in which case it can act like a lower barrier-to-entry version of Conversion to Paying. When you allow users to use your product for awhile before committing, an increase in registration can mean that users are finding your product compelling enough to take the next step and register.
Registration is only an indicator of happy customers when it’s lazy, and even then it’s only a piece of the puzzle, albeit an important one.
Customer Service Contacts
You’d think that decreasing the number of calls and emails to your customer service team would give you a pretty good idea of how happy your customers are. Unfortunately, this one can be manipulated aggressively by nasty tactics like making it harder to get to a representative or find a phone number. A sudden decrease in the number of support calls might mean that people are having far fewer problems. Or, it might mean that people have given up trying to contact you and gone somewhere else.
Decreased Customer Service Contacts may be caused by happier customers, but that’s not always the case.
So which is it?
While all of these metrics can be extremely important to your business, no single one can tell you if you are making your customers happy. However, looking at trends in all of them can certainly help you determine whether a recent change to your product has made your customers happier.
For example, imagine that you introduce a new element to your social networking site that reminds users of their friends’ birthdays and then helps them choose and buy the perfect gifts. Before you release the feature, you decide that it is likely to positively affect:
- Engagement – every time you send a reminder of a birthday, it gives the user a reason to come back to the product and reengage.
- Revenue – assuming you are taking a cut of the gift revenue, you should see an increase when people find and buy presents.
- Conversion to Paying – you’re giving your users a new reason to spend money.
- (Lazy) Registration – if you only allow registered users to take advantage of the new feature, this can give people a reason to register.
- Retention – you’re giving users a reason to stay with you and keep coming back year after year, since people keep having birthdays.
Once the feature is released, you look at those numbers and see a statistically significant positive movement in all or most of those metrics. As long as the numbers aren’t being inflated by tricks or unsustainable methods (for example, you’re selling the gifts at a huge loss, or you’re giving people extra birthdays), you can assume that your customers are being made happy by your new feature and that the feature will have a positive impact on your business.
Of course, while you’re looking at all of your numbers and metrics and analysis, some good old fashioned customer outreach, where you actually get out and talk directly with users, can also do wonders for your understanding of WHY they’re feeling the way they’re feeling. But that’s another post.
Interested? You should follow me on Twitter.
For more information on the user experience, check out:
Laura Klein | 11.13.09
You spend months on a new feature for your existing product: researching it, designing and building it, launching it. Finally, it’s out in the world, and you sit back and wait for all those glowing comments to come in about how happy your users are that you’ve finally solved their biggest problems. Except, when the emails, forum posts, and adoption data actually come in, you realize that they hate it.
There is, sadly, no single reason why your new feature failed, but there are a number of possibilities. The failure of brand new products is its own complicated subject. To keep the scope narrow, I’m just going to concentrate on failed feature additions to current products with existing users.
Your Existing Product Needs Too Much Work
Ah, the allure of the shiny new feature! It’s so much more exciting to work on the next big thing than to fix bugs or improve the user experience of a boring old existing feature.
While working with one company, I spoke with and read forum posts written by thousands of users. I also used the product extensively myself. One of the recurring themes of the complaints I heard was that the main product was extremely buggy and slow. The problem was, fixing the bugs and the lagging was really, really hard. It involved a significant investment in infrastructure change and a serious rewrite of some very tricky code.
Instead of buckling down and making the necessary improvements, management spent a long time trying to build new features on top of the old, buggy product. Unfortunately, the response to each new, exciting feature tended to be, “Your product still crashes my computer. Why didn’t you make it stop doing that instead of adding this worthless thing that I can’t use?”
Now, you obviously don’t need to fix every last bug in your existing offering before you move on and add something new. You do, however, need to be sensitive to the actual quality of your product and the current experience of your users before adding something new. You wouldn’t build a second story on a house with a shaky foundation. Don’t tack brand new features onto a product that has an unacceptably high crash rate, severe usability problems, or that runs too slowly for a significant percentage of your users.
Before you add a new feature to a product, ask yourself, “Have I fixed the major bugs, crashes, and UX issues that are currently preventing my users from taking advantage of core features?”
Your Product Interface Is a Giant Mess
Remember the old Saturday Night Live spoof commercial that advertised, “It’s a floor wax! It’s a dessert topping!”? It’s not as funny when it’s true. Products cannot do everything, and when they try, they end up with interfaces far too complicated for the average user to navigate.
I see this happen all the time, especially with startups looking for a way to make their product appeal to more people. Instead of improving their core product and adding features that enhance that experience, they add unrelated feature after unrelated feature, often stolen directly from more successful companies with larger user bases. Their goal is to find something that makes them blow up huge, but they just end up with an overly complicated product that tries to do too many things and doesn’t do any of them well.
It’s not just startups that suffer from this. Products that have been around for many years often get bogged down with feature after feature, all of which have to be supported because some fraction of the user base still uses it. These products then become vulnerable to new challengers with more focused, easy to use interfaces and smaller feature sets.
Of course, there are times when companies have to take their products in a new direction. For example, Flickr started as a set of tools for an MMORPG called Game Neverending. The game has ended, but Flickr lives on as an entirely different business. PayPal began as a way to make PDA to PDA mobile payments, but that feature got killed years ago when they realized that web payments were a far better business model.
When you do find that killer feature that’s going to change your whole business model, commit to it and make it a serious focus rather than burying it under dozens of less popular features. Don’t try to be all things to all people, or you will end up with nothing but a giant, unusable mess.
Before adding a new feature, ask yourself, “Does this enhance my current product experience or just add to an already confusing and cluttered interface?” And, if it doesn’t fit with your current product offering but you still want to do it, ask, “Am I prepared to cut other features to make this part of my core offering and simplify my experience?”
You Didn’t Build What They Asked For
Let’s face it, sometimes your priorities are different from your users’. For whatever reason, be it a new business partnership, a need for a new revenue stream, or the desire to attract a different group of users, sometimes you’re going to build something that your current users don’t want and didn’t ask for.
This isn’t always a bad thing. For example, something that annoys your current non-paying users but attracts a whole slew of new, paying users is worth a few nasty emails to your customer service department. Just make sure that the new feature is really going to do what you think it will. It sucks to piss off your current, paying customers to build a feature that never really fulfills its initial promise. Trust me on this.
Before building a feature that potentially has more benefits to your company than your current users, ask yourself, “Am I prepared to deal with the fact that this is going to annoy some of my customers, and what is the real likelihood that I will get more out of this than I will lose?”
You Built EXACTLY What They Asked For
I know. It doesn’t seem fair. They’re angry if you don’t do what they want, and they’re angry if you do what they want.
The truth is that users will often ask you for a solution when it would really be more helpful to tell you that they have a problem. I’ve written more extensively about when you shouldn’t be listening to your users, but the upshot is that users aren’t great predictors of which brand new features will be big hits. Sometimes users will tell you that they want a toaster in their car, when what they really mean is that they don’t have time to make breakfast in the morning.
Before building a new feature that your customers are demanding, ask yourself, “What known user problems is this solving, and is this the best way to solve it for everybody?”
The Feature’s Not Finished
Now, I’m all for building the minimum viable product, getting it out in front of users, and then iterating on it to improve it, but some features just aren’t ready for prime time. By launching a half baked feature without key functionality, you’re running the risk of turning a lot of people off on the idea before they ever get a chance to really try it out.
Remember, your customers aren’t in the conference room with you when you come up with your grand vision. They don’t know where you’re going with this neat new idea. They’re judging the feature based on their first experience with it. Make sure that the first version is at least usable and hopefully that it’s far enough along that users can see the same promise that you do.
Also, good enough to ship doesn’t necessarily mean good enough to remain in your product long term. A big part of shipping early is continuing to improve the feature once it’s been out for awhile. One company I worked with had the tendency to ship early versions of features and then let them just sit there gathering dust, rather than iterating on them until they were truly high quality. What they ended up with was an enormous product that all seemed about half finished and a lot of unhappy customers who didn’t believe features would ever improve past version 1.0.
Before shipping a new feature, ask, “Is this good enough that users will get why they should care about it? And, if they do care about it, am I committed to improving it?”
The Feature’s Not Finishable
At many of the companies I’ve worked with, features have tended to evolve before they even get built. What generally happens is this: you have an idea based on something you’ve heard from users; that idea gets brainstormed and grows based on internal input; UX and visual designers spec out the whole idea, often expanding on the original idea; then engineering gets involved and gives a time estimate of how long the feature will take to build; finally the whole thing gets cut back by about 80% based on the estimates.
Unfortunately, the 20% you end up implementing may not solve the original problem. That means, when you finally announce your great new feature, users who originally asked to have that particular problem solved are justifiably upset.
Before drastically cutting your new feature back, ask yourself, “Does this still solve the original problem I was trying to solve?” If it doesn’t, ask, “Can this problem be solved with a reasonable level of investment?” There’s no shame in answering no to either or both of these questions, as long as you decide not to go forward with the new feature.
The Secret to Making Everybody Love Everything You Make
I’m joking. There’s no secret. The truth is that it’s almost certainly impossible. But by asking yourself the right questions during your feature development phase, you can dramatically cut back on time spent creating things your users hate.
And never forget, when you do build something they hate, acknowledge it, apologize for it, and fix it. It will go a long way toward making your users happy again, and it might even get them to like that neat new feature you just shipped.
Interested? You should follow me on Twitter.
For more information on the user experience, check out:
Mia Silverman | 11.11.09
We recently worked on a new energy tracking site to help consumers monitor their energy use and find ways to save money. With President Obama’s recent announcement awarding a few billion dollars in smart grid grants, we expect to see an even larger effort devoted to creating new energy tracking systems and devices. So, let’s save all of us some energy by sharing our top tips for creating a consumer energy portal.
1) Simplicity is key
We’re noticing that far too many of the new energy portals on the market are delivering complicated interfaces and busy dashboard-style pages with dense data charts and lots of buttons. Although heavy data, analysis tools, and controls might be interesting to data geeks, most consumers will find this information overwhelming or just plain boring. Consumers don’t want it to be rocket science just to learn to set their thermostat, and they don’t want to spend hours reviewing their usage details just to determine how they can save money.
A few examples of interfaces with too much data for consumers:
Greenbox
Fat Spaniel
Tendril
So, we encourage you to simplify, simplify, simplify! Anticipate user’s most common questions, then make it easy to find these answers. Highlight key information in an easily scannable format, and engage consumers with friendly language, like “How much energy am I using?” and “Is my electric bill on track this month?” If you have more data, you can always offer it on drill down for people who want to learn more.
2) Present energy data in meaningful unit equivalents, specifically dollars
Which in-home energy device would you want to use in your home?
Option 1
Option 2
I bet you chose Option 2.
To engage consumers, phrase information in a way that makes sense to them. What we say has to be both measurable and meaningful. Consumers do not understand the electricity unit of ”kwH”, unless they’ve had considerable experience with it. And the words “tons of carbon” are just as meaningless even to those who are in the industry. Instead, all energy data should be presented in terms of dollars ($), with kwH and other meaningful equivalents shown as alternative views that can be visualized. For example, “You’ve saved $53.44, or enough energy to watch 362 hours of TV.” Check out Chevron’s “Energy Generator” as another great example of how to present meaningful unit equivalents that consumers understand.
3) Consider a “new user” experience
Most consumers have not had a lot of experience seeing detailed analysis of their energy data, so there is going to be a 3-6 month period of active learning for new users. During this time, users are going to be interested in identifying some basics about their energy usage. For example,
- How much energy do I typically consume during a single day?
- What is the impact of different items and behaviors in my home?
After this initial learning period, consumers will have a good sense of the basics that will remain fairly static over time, and will start shifting their focus to a different type of monitoring. For example:
- Is my energy usage on track?
- How does my usage now compare to my usage in the past?
We suggest that you recognize this consumer learning curve by considering a “new user” experience for your consumer portal. The purpose is to educate users on their energy basics and to appropriately highlight information that is most relevant while they’re still learning, but might remain static overtime or become less interesting after initial use. Then, after this initial ramp-up period, keep users engaged by presenting an ongoing use experience that highlights the dynamic information they want to continue monitoring overtime.
4) Deliver proactive recommendations with bottom-line savings
Consumers want to know what concrete steps they can take to reduce their energy consumption, and they want to know what impact those steps will have on their bottom-line savings. Our research has shown that people are highly concerned and knowledgeable about environmental issues, but their primary motivation is still saving money. We recommend creating a predictive savings calculator based on actual energy usage that would allow users to see how various changes would affect their consumption and bill. Users could even use this calculator to help convince other household members to make the behavioral changes that matter most for their bottom dollar. Will the calculator show you which trade-off is right for you? Probably not, but at the very least, it will provide you with your top options for having the biggest impact on your bottom dollar. You can take it from there.
5) Offer a highly-visible, integrated in-home solution
In addition to creating a web portal for access to consumer’s historical data, we suggest also providing a highly visible point-in-time meter for integrated placement within the home. Consumers are looking for visible, real-time meters that can become an effortless part of their daily routines – much like their thermostats – because they know that everyone in their house has to be able to stay on track with one simple glance. Otherwise, it will be “one day up then one day down” instead of a forward-moving effort by everyone involved.
Also, remember what we’ve learned– keep the device simple, and present energy data in dollars and other meaningful equivalents, such the following example from Energy Aware.
We hope you found these tips in creating consumer energy portals helpful. Think about it… talk about it… try it… and get out there and create your own green power designs so that others can give it a try, too.
Laura Klein | 11.04.09
The recent release of Windows 7 got me thinking about development cycles. For those of us who suffered through the last 2+ years of Vista, Windows 7 has been a welcome relief from the lagging, bugs, and constant hassle of a failed operating system. Overall, as a customer, I’m pretty happy with Windows 7. But, at least on my part, there is still some latent anger – if Windows 7 hadn’t been quite as good as it seems to be, they would have lost me to Apple. They still might.
A big part of my unhappiness is the fact that I had to wait for more than two years before they fixed my problems. That’s a lot of crashes and frustration to forget about.
One approach that many software companies have been adopting to combat the huge lag time built into traditional software releases is something called continuous deployment. This sort of deployment means that, instead of having large, planned releases that go through a strict process and may take months or years, engineers release new code directly to users constantly, sometimes multiple times a day. A “release” could include almost anything: a whole new feature, a bug fix, or a text change on the landing page.
I worked with a software development organization that practiced continuous deployment on a very large, complicated code base, and I can definitely say, the engineers loved it. From the point of view of the employees, continuous deployment was a giant win.
But how was it for the users? The fact is, some decisions that seem like they only affect engineering (or marketing, business, PR, etc.) can actually have a huge impact on end users. So, whenever organizations make decisions, they should always be asking, “how might this affect my customers, and how can I make it work best for them?”
Is Continuous Deployment Good For Users?
As with so many decisions, the answer is yes and no. Continuous deployment has some natural pros and cons for the customer experience, but knowing about them can help you fix the cons and benefit even more from the pros.
Big Customer Wins
Fast Bug Fixes
Perhaps the biggest win for users is that bugs can get addressed immediately. Currently, even Microsoft releases patches for some of its worst security holes, but there is certainly a class of non-critical, but still important bugs that have to wait until the next major release to get addressed. That means weeks, months, or even years of your users dealing with something broken, even if the fix is simple. In continuous deployment, a fix can be shipped as soon as it’s done.
Fast Things vs. Slow Things
Similar to the first point, continuous deployment lets you get everything, not just bug fixes, to users as soon as they’re ready to go. Small features, easy changes, and UI tweaks don’t have to wait for larger, unrelated features to be released to customers. After all, should a new design for the splash screen really have to wait on the implementation of a whole new payment system?
More Opportunities for Community Involvement
If you’re having a constant dialog with your customers (you are, right?), they’re probably making some pretty good suggestions about problems they’re having or ways to improve the product. A by-product of the first two benefits is that those users are going to feel even more involved in the development process when their suggestions or concerns are dealt with quickly, rather than if they have to wait months or even years for the next major release.
(Mostly) Avoidable Customer Problems
As with anything, continuous deployment can also cause some problems for users. Of course, some of these problems can exist in big, staged deployments as well, but these are a few things in particular to watch for.
Constant Change
Imagine if every day, the layout on your car changed, sometimes slightly, other times drastically. You want to drive to the store, but the steering wheel is on the other side and you’ve suddenly got an extra pedal. It would make it a lot harder to get where you were going, wouldn’t it?
Well, presumably your users are using your product to get something done, and they’ve got a certain way that they’ve learned to do it. Continuous deployment can mean that the interface for your product can change at any moment, even several times a day. If features constantly appear and disappear, it can be very disruptive to your users’ process.
There are a few things you can do to minimize the disruption. First, make sure that you’re testing your biggest changes on small cohorts of people. Iterate on a subset of your user base, rather than hitting every single user with every single change. This will limit the change that any individual sees while still giving you the benefit of constantly pushing code to customers. In fact, use this as an opportunity to do the A/B testing you should be doing anyway.
Also, and this should be obvious, try to limit your truly disruptive user interface changes so that things don’t feel like they’re in constant flux. You can still change things, but be aware of how frequently you’re making major changes and stay in contact with your users to make sure they’re not feeling dizzy.
Inconsistent UIs
When a big new release is planned, often there is a comprehensive design phase where all the new changes are mapped out and discussed. This means that any inconsistencies in the UI can be found and addressed.
In continuous deployment, different pieces of the product are getting built and shipped to customers all the time, and there is rarely a time when the entire UI is reevaluated as a single entity. This means that sometimes UI standards can tend to…oh, let’s say evolve.
This problem can be controlled by having a UX team member embedded in the development team and constantly working with the engineers to enforce standards before things ship to customers. It can also be improved by providing wireframes, visual designs, and tools like templates to developers so that the look and feel of the product doesn’t shift too dramatically over time.
Avoiding QA
I’m certainly not claiming that every single thing in a traditional release process gets a full QA pass, but I do find that continuous deployment makes it easier for code to slip out to users without any human testing at all. Any time you give engineers the ability to ship code directly into production, you’re tempting fate. Somebody’s going to say, “oh, it’s just a tiny change,” and it’s going to get out without any testing. I’m not naming any names, but you only have to have a tiny change break the entire product once before you realize that there’s no such thing as a tiny change.
Also, continuous deployment can make certain types of testing much less likely to happen. While large release cycles tend to have a code freeze and weeks or months devoted to testing, hopefully including regression, unit tests, and end to end testing, continuous deployment doesn’t necessarily have that baked into the process.
You can always add periodic end to end testing of the product to your own process, of course, and it can be quite helpful in improving code quality, especially when your engineers occasionally slip through a “tiny change.”
Communication Issues
When you’re constantly releasing new features and bug fixes, communication with your users can be a challenge. You don’t have one big release with new help docs, a big roll out plan, and an advertising campaign. Instead, stuff is coming out all the time, and users can get overwhelmed by keeping up with the changes.
Context sensitive help and inline information for each feature can help users get quickly oriented. Also, clearly marking new features as alpha or beta can let users know when a feature is still being developed so they can set their expectations accordingly.
Documentation can also easily get out of date when things are getting released constantly. Big, staged development cycles often have a built-in time for creating documents. Typically, manuals or help documentation and FAQs go through QA sometime after code freeze and before release. But since you’re not necessarily doing a big, monolithic release with continuous deployment, you can end up with this material never getting any sort of end to end editing to make sure they stay current. Make time for this. It’s good for both your customer experience and your customer support team.
Frequent Downloads and Updates
While continuous deployment is quite natural with web applications, even downloaded products can be constantly updated. However, you should always be aware of the burden you’re placing on your user base. If you’re forcing people to download a large file and go through an installation process too often, you’re going to annoy people. As a personal example, iTunes appears to have a new version every week, and I’ve started to flinch every time I open it.
There are a few things you can do to make downloads easier on your users. First, you can ask the user to allow the update to download in the background and install automatically the next time the product opens. This means that the updating happens with very little user annoyance. Also, it’s best to keep the update quick by making the downloads incremental. For example, your virus protection software probably updates its virus information daily without asking you to reinstall the entire product every time.
So, Is It Good for Users or Not?
Continuous deployment can be done in a way that’s good for both engineering teams and users, but you do need to take some precautions. By taking care when you introduce changes that your users will really notice and making sure that you make time for important processes like QA, you can get features out faster and constantly improve your product. And that is very good for users.
Interested? You should follow me on Twitter.
For more information on the user experience, check out:
Mia Silverman | 10.17.09
We’re excited to share the infographic we designed for the Intuit Health Benefits Center project to answer this question.
In July 2009, Intuit conducted a survey with over one thousand small business owners in California asking about the health insurance options they offered their employees and how much they were paying for health insurance plans. From prior discussions with business owners, we knew employers were very interested in finding out the results of this survey, as they wanted to remain competitive with other businesses in their industry. We decided that an infographic would be the perfect way to transform our huge collection of data into a single, compelling visual. Our goal was to create a comprehensive infographic that would be instantly meaningful and easily understandable to the many employers looking to get a handle on the best way to offer competitive insurance packages to their employees.
Sound easy? Not exactly. Like most aspects of design, it turns out that creating infographics is a tricky business – on the one hand, we had to be careful not to present too much information so as to be overwhelming, but on the other hand, our infographic had to be able to provide enough detail to be useful to a wide range of employers looking for very specific information. And, of course, we had to do all of this without getting bogged down in the details or jargons of health insurance plans that often don’t make sense to someone at first glance who isn’t in the healthcare industry.
Here are our top design principles for creating an infographic:
1) Focus the infographic around a single, key question
We centered our infographic around the fundamental question new small business owners cared about most: “How do my health insurance costs compare?” Then, to answer this question, we created an infographic design to breakdown some of the important specifics – like employer and employee costs by region, and the average employer contribution. We placed secondary information, such as the various plan types offered by employers, on the side of the infographic and used subtler visuals so it would not to compete with our key message.
2) Identify a visual metaphor to help expose differences quickly
We wanted viewers to be able to notice variations in data quickly, as this is the fundamental power of a good infographic. Consequently, we came up with helpful visual metaphors that allowed us to visually represent plan costs by region in two ways. First, we used a heat map of California to indicate employer costs so viewers could quickly see regional differences at-a-glance. Second, we displayed horizontal bars that corresponded to the relative costs by employers and employees per region so viewers could visually compare differences in amount.
3) Use minimal text, and rely on visuals to tell the story
We wanted to keep our infographic clean and simple by requiring minimal reading, so we included a visual key for our chart at the bottom of the infographic to help keep the text and fine print to a minimum.
Overall, our quest to create an effective infographic helped us to double-deliver to our client — we not only revealed interesting findings, but also presented the findings in a ready-to-go format that met the needs of the employers Intuit wanted to reach. We’re excited to share our newest piece of healthcare artwork with you, so take a look and let us know what you think!
https://benefits.intuit.com/content/infographic.pdf
Laura Klein | 10.02.09
Henry Ford once said that, if he’d asked his customers what they wanted, they’d have asked for a faster horse. In the high tech industry, this quote is often used to justify not talking to users. After all, if customers don’t know what they want, why bother talking to them?
You need to talk to users because, if you ask the right questions, they will help you build a better product. The key is figuring out the right questions.
For starters, users are great at telling you when there’s something wrong with your product. They can tell you exactly which parts of the product are particularly confusing for them or are keeping them from being happy, repeat customers. Figuring out what to do about those problems is your job.
In general, users are not going to be able to answer the following types of questions:
- What new technical innovation is going to revolutionize a particular industry?
- What’s the next cool gadget that you’d like to buy?
- Do you think that people like you would buy this new cool gadget that you’ve just learned about?
- What new features would make this product more interesting/compelling/fun/easy to use? (although, this question becomes more answerable when the user is presented with some options for which features they might prefer.)
- How exactly should we change the product to make it easier for you to use?
They are fantastic at answering questions like these:
- What do you most love or hate about this product?
- Do you find anything about this product hard to use or confusing?
- Does this product solve your problem better or worse than what you’re currently doing?
- How are you currently solving a particular problem that may or may not be addressed by this product?
- What don’t you like about your current solutions for a particular problem?
- Why did you choose this particular solution as opposed to another solution?
Obviously, there are innumerable other questions that you might want to ask your users, so how do you decide which ones they’ll be able to answer with any degree of accuracy?
Problems vs. Solutions
Users are much better at telling you about problems that they’re having than solutions that they want. In Ford’s example, when people asked for a faster horse, what they were really saying was that the horses they had were too slow. They didn’t specifically want a faster horse. They wanted a faster means of transportation that was no worse than a horse in other respects.
Frequently in user tests or customer feedback sessions, customers will tell you very clearly, “I want x!” Your job is to understand why they want x and then to determine whether or not x is really the right solution. It’s not that they never have good solutions, but users tend to only look at the product from their own perspective and usage patterns, while you should be talking to lots of different types of users with lots of different types of problems. They’re not thinking about the product as a whole or how to fix things for the other several million people who might have slightly different problems.
When users try to give you solutions, encourage them to talk about their problems instead. Then figure out what they’re really asking for, and give it to them.
Past vs. Future Events
It’s much easier for people to answer questions or give opinions about something specific that has already happened than about something that might happen in the future.
Consider the question, “What do you want to eat tonight?” vs. the question, “What did you think of the meal you just ate?” For the vast majority of us, the second one is much easier to answer. It simply asks you to formulate a concrete opinion about a single event that has recently happened. The first question asks you to imagine all the various available options for food and make a decision about what you might like in the future based on probably imperfect information.
This is true with products, as well. It will be much easier for your user to explain how performing a particular task went than to predict how he would like to perform that particular task in the future. That’s why, when you’re doing your preliminary research to determine product direction or early feature development, it’s very important to give users hands on tasks that they can perform for you and then give opinions on rather than to talk abstractly about the solution you’re considering providing for them with your product.
Users vs. Other People
Unless you’re really lucky, you’ve probably realized that people are terrible at figuring out what other people want. Perhaps you came to this realization on some birthday or other gift giving holiday. Users suffer from the same problems as gift givers. They’re almost always terrible at telling you how other people will react to a product.
And yet, talk to just a few customers or user test participants, and you’re guaranteed to hear one of them say, “Well, it’s not for me, but my mom/friend/boss/brother would be really into this…” Another one you hear a lot is, “My mom/friend/boss/brother would never be able to use this. It’s way too complicated.”
This information can be marginally useful if you’re trying to find the right customer segment, but take it with a grain of salt. Reassure the user that you’re also going to talk to people like their mom/friend/boss/brother, and what you’re really interested right now in is the user’s opinion. Then talk to the mom/friend/boss/brother to find out their real feelings. Chances are, the person you’re talking to doesn’t really know what anybody else wants as well as they think they do.
The Right Questions
So, what should Henry Ford have been asking his customers? Instead of, “What do you want?” he could have asked, “Is there anything you particularly like or don’t like about your horse and wagon?” If they chose not to buy a car, he could ask, “Why didn’t you buy that car?” Once they bought a car, he could have asked, “What made you decide to buy a car?” or “Was there anything you found particularly confusing or hard to use about your new car?” He could even have gone for a drive with some of them and observed the various problems that they encountered.
In fact, there were dozens of things he could have done that might have helped him improve the design and marketing of his product. He just couldn’t ask them, “What do you want?” because they almost certainly would have said, “a faster horse.”
For more information on our approach to getting customer feedback, check out:
Laura Klein | 09.23.09
So, you decided to do some user research in order to find out where you can make improvements. After a few hours of user interviews, you ended up with a notebook full of scribbled information that all seemed really critical. How in the world do you figure out what to do with all that information?
If your answer is “talk about it all abstractly with everybody in the company or write a huge paper that nobody will read and then go on with business as usual,” you’re in good (bad?) company.
But you have to DO something with all that data. You have to analyze it and turn it into actionable items that your engineering department can use to fix your product. It’s not always easy, but I’m going to give you an approach that should make it a little easier. This isn’t the only way to do your test analysis, but it’s one of the quickest and easiest that I’ve found when you are concerned with key metrics.
When to use this method:
- You have an existing product with a way to measure key metrics, and you’re interested in improving in particular areas related to your bottom line
- You have a limited research and development budget and want to target your changes specifically to move key metrics
- You are looking for the “low hanging fruit” that is getting in the way of your users performing important tasks with your product
- You are working in an agile development environment that is constantly tweaking and improving your product and then testing the changes
When not to use this method:
- You have an existing product that you are planning to completely overhaul, and you want to understand all of the major problems before you do your redesign
- You are trying to create an overall awesome, irresistible user experience that is not related to a specific metric
- You are designing a new product or feature and are observing people using other products to identify opportunities for innovation
If you fall into the first bucket, read on…
The Five Basic Steps:
- Identify key metrics you’d like to improve
- Identify the tasks on your site that correlate with improvement in those metrics
- Observe people performing the appropriate tasks
- Identify the barriers preventing people from completing or repeating the tasks
- Develop recommendations that address each specific barrier to task completion
Step 1: Identify Key Metrics
First, identify the metrics that you care about, and look at the distinct steps that your users are most likely to take to change those metrics. This way, you can plan your qualitative testing to show you exactly what is getting in the way of your users reaching each subsequent step.
As an example, let’s imagine that you have a social networking site that earns revenue when people give virtual gifts to each other to display on their profile pages. Because your revenue depends on lots of users purchasing lots of things for each others’ pages, you might care about the following metrics: how many people register, how many people customize their profile pages, how many people make friends, how many friends they make, and how many people purchase one or more gifts. Obviously, that last is the most important, since that’s where your revenue comes from, but the others are also important, since improving those numbers should increase everything downstream, if done in the correct way.
Your goal is to get people from point A to point D as quickly and pleasantly as possible. To do that, you want to look at all of the barriers you have erected in the way of your users accomplishing those tasks.
Here’s are the pieces of the user flow that change your key metrics:
Registration > Profile Customization > Finding a Friend > Purchasing a Gift
Step 2: Identify Individual Task Flows
Now that you know what your key metrics are, you need to make sure that you observe your test participants trying to accomplish the tasks that will eventually lead to changing those metrics. To plan your tasks and analyze your data, it helps to break down your individual tasks into smaller user flow steps based on your user interface.
For example, while the number of successful registrations may be a single metric, there may be several screens or discrete interactions involved in your entire registration process. Make a list of what they are now.
As an example, I ran a test recently for a product that had a 4 step registration process. The user flow for registration on the site looked like this:
| Landing Page |
> |
User name |
> |
Personal Info |
> |
Download |
Step 3: Observe People Performing Tasks
So let’s say that, based on the metrics you want to change in your hypothetical social networking site, you’ve determined that you need to gather data about registration, profile customization, finding friends, and purchasing a gift. You’ve scheduled 5 people in your target demographic to come in for user test sessions. I’m not going to teach you everything you need to know about running a test, but here are some basics.
First, let the user begin using your product however they want and note all of the things that they do that aren’t the tasks you’ve identified. Remember, any time spent not doing your target tasks is time that is not being spent improving your metrics, so you want to know what these other things are! If you give your participant a targeted task right away, you will never learn what other things they are doing instead. Finding these distractions allows you to eliminate them or change them so that they also improve your key metrics.
Watch everything your user does with the perspective of the ideal flow for your metric. Do your participants wander off track or getconfused during registration? Do they fail to find their own profile? Do they not have the information they need to find a friend? Do they fail to understand what gifts are or how to purchase them? Do they get distracted by a totally different part of your site that doesn’t contribute to improving your key metrics?
Once you’ve allowed the participant some free exploration time, you may need to move them to a particular task. For example, in our current example, it might be that you need to ask the participant to try to purchase a gift, even if that’s something they wouldn’t do in the context of a study environment. You can move them along by saying something casual like, “Did you notice that there is a way to purchase gifts? What do you think about that?”
Meanwhile, record what the participant is doing and encourage them to talk about their impressions of the product.
Step 4: Identify Barriers
Once you’ve gathered data by watching 4 or 5 people repeat the same tasks, you need to analyze your data to figure out where the barriers are and communicate them effectively to the team. Determining the barriers should be pretty easy. Just ask yourself the following questions:
- Where did participants seem confused or distracted and stop performing a task?
- Which things took longer than they should have?
- Why did participants fail to accomplish a task?
- What tasks did the participants perform that would not improve the key metrics?
In the product I tested with the four step registration process, the metrics showed significant drop off one each screen. On screen number 2, the user was asked to choose a unique user name and enter some information. It had a surprising amount of drop off considering how simple it was. After watching a few user tests, I noticed that many people were trying to select user names that were not available, but they were failing to notice the error message telling them to choose a different one. Even if they noticed the error, several people had a tough time finding a user name they liked and would spend minutes typing in word after word, getting frustrated, and finally giving up.
By watching people try to register for the product, we identified several significant barriers to successful registration in just one step of the process. In fact, we found similar problems – some large, some small – in each of the steps.
Once we’d identified the barriers, organizing and explaining them to the team was easy with a simple, annotated flow of the steps leading up to the goal along with the problems associated with each step. For example, the flow for the registration test might have looked like this:
Landing Page
- Didn’t understand the product
- Felt the product was aimed at teens
- Thought the network was very small
User name
- Didn’t know when a user name was taken
- Couldn’t come up with a decent name
Personal Info
- Were concerned about privacy
- When failed Captcha, all info had to be re-entered
Download
- Were uncomfortable downloading b/c of viruses
- Didn’t know what they were downloading
Overall
- Were bored during registration process or complained that it was taking a long time
Step 5: Develop Recommendations
Now that you know what barriers are keeping your customers from accomplishing the goals you’ve set for them, you need to generate recommendations for ways to remove those barriers. You can do this by looking at exactly what the barrier is and what the users’ reactions were to it, and then brainstorming ideas to help the user overcome that barrier. Yes, this is the tough, somewhat creative part. You don’t want to just take any recommendation that the user gives you. You need to understand what problem they’re having and come up with a way to fix it that doesn’t cause any other problems.
Going back to my example, once we found the problems on the user name page, we looked at several solutions. We definitely needed to make it more obvious to the user when they selected a name that was already taken. We came up with a few simple ways to solve this particular problem: we could suggest other user names when people tried one that was already taken; we could make the error more noticeable; we could make the Next button obviously disabled so that they couldn’t even try to move on; we could free up some names that had been taken but weren’t currently in use so that the selection of user names was better. We then selected a couple of these solutions based on expected ROI calculations.
Even the lowest effort of these suggestions, improving the visual design of the error, caused an immediate and statistically significant decrease in drop offs for that step in the registration flow, and eventually improved revenue by increasing the number of successful sign ups. The barrier was lowered, if not entirely removed.
You may have noticed that the last section is labeled “Overall.” This is where you look at the process as a whole, rather than a sum of its parts. For example, one comprehensive solution for drop off at each step might be to reduce the number of steps it takes to register (almost never a bad idea!). Doing this wouldn’t necessarily mean that you wouldn’t have to address the other problems individually, but you might get fewer drop offs simply because it would take less time for users to finish the process.
How Is This Different?
You might be wondering how this is different from more traditional ways of running user tests and analyzing data. In any user test, you’ll need to come up with tasks, observe users, figure out where they’re having problems, and come up with solutions for the problems.
There is a difference though. My experience has been that companies do not fix every single UX problem that they find in user research. I don’t love this, but I do understand the need to prioritize important changes.
So, if you’re only going to fix a few problems, you should make an effort to identify the most important ones. The counter intuitive part is that the most important problems might not be the worst problems from a user experience stand point, but they will definitely be the ones that are keeping your users from reaching the goals that improve your most critical metrics.
Using this framework will help you identify your key user flows, find and communicate the major barriers to success, and propose targeted solutions that will improve both your user experience and your business.
For more information on our approach to getting customer feedback, check out:
