Why A/B Testing Needs Usability Testing, aka Why Opposites Attract
Why does A/B testing need usability testing? Because opposites attract, and the benefits received by coupling A/B and Usability testing can go far in improving the UX of a website or app. In fact, it’s better than any other method. Here’s why.
Way back in the dark ages of 2009 I wrote a three part series on A/B Alpha/Beta and Usability testing, asking the question of which one is better.
To summarize the pros and cons of A/B testing and usability testing from the three part series:
A/B Testing positives:
- Fast (testing can be set up and run in a day)
- Tests reality, not theory
- Quantifiable with statistical significance
- Accurate (winner almost always performs at that level)
A/B Testing negatives:
- Can hurt results (B version failure can cause reduced conversion)
- Missing critical “why” data (no data to explain why a version won)
- Not predictive (data not available to determine what to test nex)
- Needs traffic (doesn’t work on prototypes or apps with no users)
Usability Testing positives:
- Doesn’t hurt results (testing does not impact conversion)
- Provides the “why” data (qualitative data answers why users do what they do)
- Predictive (results can determine what next to test)
- Doesn’t need traffic or even a live website or app (testing available from concept through production stages)
Usability Testing negatives:
- Trained professionals required for unbiased results
- Won’t reveal all issues
- Results can vary
- On-going testing is difficult
How A/B and Usability Testing Compliment Each Other
A/B and usability testing complement each other in several ways.
First. Did you happen to notice how the four negatives for A/B testing are actually the four positives for usability testing? That’s right, the issues inherent in A/B testing are actually strengths of usability testing, and vice versa.
Chart of A/B Testing Cons and Usability Testing Pros:
|A/B Testing Cons||Usability Testing Pros|
Second. A/B testing provides the quantitative side of UX data, and usability testing provides the qualitative side. The quantitative ‘what’ of user experience choices your website and app visitors make is the A/B data is augmented with the ‘why’ for those choices coming from qualitative usability testing data.
Third. A/B Testing coupled with usability testing provides an end-to-end view into the user experience of your website or app. By coupling the data, you have a much clearer picture of the engagement happening in your site or app. You also will have the data to know how to impact and improve on that engagement. Finally, you’ll have the data to evaluate your results and move forward with your next set of testing and optimization.
Why A/B Testing Needs Usability Testing
The benefits of combining A/B testing with usability testing are many and include:
- Complete quantitative and qualitative data for more informed decision making
- Comprehensive view into the user experience of your app or website
- Ability to use data to predict what additional tests can increase engagement
- A 360 degree view of data to use for continuously optimizing your app or website
By combining A/B testing with usability testing, you’ll find your testing and optimization efforts produce far better results, which is a win for you, your firm and your app or website users.
A/B Testing and Usability Testing Resources:
Three Part Series: Usability, A/B or Alpha/Beta Testing, What’s Better?