Comments on: A/B and Qualitative User Testing

By: What hackers have to say about A/B testing « I love split testing – Visual Website Optimizer Blog

Wed, 28 Apr 2010 14:22:54 +0000

[...] A/B and Qualitative User Testing (HN discussion) [...]

By: Giff

Giff — Fri, 04 Dec 2009 01:04:38 +0000

enjoyed the post — thanks for taking the time to write out your thoughts

By: Sarah DeAtley

Sarah DeAtley — Thu, 03 Dec 2009 21:49:39 +0000

Really good article, Laura! I try to argue for combining these things as well. Something else to consider is using multivariate testing, which does allow you to test each combination of different design elements together, but it is harder to get statistical significance with it.

Sarah

By: uberVU - social comments

uberVU - social comments — Thu, 03 Dec 2009 21:37:42 +0000

Social comments and analytics for this post…

This post was mentioned on Twitter by ericries: Learn about hypothesis generation vs validation: “A/B and Qualitative User Testing” http://bit.ly/5Em5UO…

By: Laura Klein

Laura Klein — Wed, 09 Sep 2009 19:58:54 +0000

Thanks all for the comments!

Dan, you asked what surprised me when I combined qualitative with A/B testing. It was honestly surprising how well they went together. I had heard from so many engineers “oh, we don’t need to talk to users. We have metrics!” that I started to believe them.

But of course I talked to users anyway, because that’s what I do, and I started to approach the engineers saying, “Hey, we’re hearing that this thing is a problem. Let’s change it in this particular way and use the metrics to find out if we were right.” It pretty quickly got to the point where we could predict which metrics (registration? revenue? retention?) were going to move significantly when we made a change based on user feedback.

I think there is still resistance on both sides of the qual vs quant (or sometimes designers vs engineers) debate, but I have have been absolutely convinced that an integrated approach works brilliantly for producing a better product.

Laura

By: Rob Edwards

Rob Edwards — Wed, 09 Sep 2009 10:04:02 +0000

Really good post.

There’s a culture in web that testing can be left out – it’s an addition to a project if there is enough time and money.

This is really the wrong approach – you need to be testing right from the project outset. By testing early and testing often you’ll save money in the long run.

Rob — http://testled.com

By: Dan Hodgins

Dan Hodgins — Wed, 09 Sep 2009 03:54:53 +0000

Laura-

I am always on the prowl for more practical tips on A/B and multivariate testing for my web properties. Kudos to you for teaching me something new tonight by weaving qualitative testing in there. Thanks!

FYI- I have emailed the article URL to myself for future reference in my testing_metrics email folder.

Can you tell me what, if anything, has suprised you when you combined qualitative testing with A/B testing?

By: Isaac Weinhausen

Isaac Weinhausen — Tue, 08 Sep 2009 18:22:40 +0000

We’re in the process of starting our test bed and plan on utilizing it in a very similar way to what you’ve described. For our qualitative tests, we’ve been using UserTesting.com. What’s nice about their service, is that their users respond very quick, which pairs nicely with an iterative/agile environment.

Thx for sharing your wisdom.

-isaacw