Usability, A/B or Alpha/Beta Testing, What’s Better? Part I A/B Testing


Web site managers have many choices when testing their sites, including usability, A/B or Alpha/Beta testing, but which one is best, and why?

A recent Tweet from Eric Harrigan asked an intriguing question:

“If you could implement only one website testing program: A/B, Alpha/Beta or Usability, which one would you pick and why?”

I found myself thinking about this question well, to be honest I actually started thinking about this question right after I was done thinking about whether I was going to eat a Bagel or piece of Cinnamon toast for breakfast (hey, first things first and food’s first in my mind, sorry). I think (and by the way I need to place a brief disclaimer here and tell you; I’m a bit biased what with being a Certified Usability Analyst and all, so you have to consider the source) that the answer is…

“It depends”

Cop-out answer? Hardly! You see, there are positives and negatives with each web site testing method. When applied in certain circumstances, one choice may actually be a better pick than another (assuming you have to pick one).

At first, I was going to blog about the pros and cons of each of the three types. But, my post kept getting longer and longer and longer, and I began to realize that if Jakob Nielsen is right and you’re only going to read about 20% of the words on this page, I better cut this post back. Therefore I’m chopping this into 3 parts:

Part I  Pros and Cons of A/B testing (you’re reading it!)
Part II  Pros and Cons of Alpha/Beta testing
Part III  Pros and Cons of Usability testing with a summary comparison of all three.

I’d like to stop you here and mention that a smart web site manager will use all three testing methods, plus additional sources of testing, when optimizing a site, but that spoils the fun of being forced to mentally pick one, so let’s proceed.

Part I  The Pros and Cons of A/B Testing

First, let’s not assume that everyone knows what A/B testing means. I like to think of the definition of web site A/B testing as:

“Simultaneously testing an existing item against a modified version (or versions) of the item by splitting traffic evenly between the items, and measuring results.”

Wikipedia likes to think of A/B testing as:

“A/B testing, or split testing, is a method of advertising testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates.”

Click here to view latest job postings

I actually like my definition better (but again, disclaimer, I’m biased). So, let’s examine the pros and cons of A/B testing on a web site.

Pros of Web Site A/B Testing:

1. Fast Of all the test types, A/B is way, way fast. That’s because it takes very little time to create a modified version of an existing web page that includes a modified item (like a new picture, new copy or other new element) and throw it up on your site. Then, it’s just a matter of splitting traffic to the two pages.

Results can be gained quickly too, by splitting traffic 50-50 between the existing and the test version of the pages you can measure the results in short order (assuming you have traffic to your web site that is). Scaredy-cat Cautious web site managers may wish to keep most of their traffic on the existing page, and only split 10, 20 or some other low percentage of traffic to the test page, just in case the test item doesn’t work to expectations. The result is the test might take longer. More on that in the Cons below.

2. Tests reality, not theory The good news about A/B testing on a live web site is you’re obtaining real results from real users doing real things. That means you’re not using theory, estimates, forecasts, predictions, your Horoscope or Fairy cards to base decisions on.

3. Quantifiable A/B web site testing provides actual numbers that can be compared, sliced and diced to evaluate results. Interaction, conversion, number of abandonments – all those numbers are accessible during and after testing. No guessing required!

4. Accurate Unlike other forms of web site testing, A/B testing is 100% accurate ASSUMING you have statistically significant data. Understanding error rate and statistical significance and all those other statistics terms you were supposed to be learning in Statistics class is very important. You were paying attention in Statistics class, right? If not, find someone who was and have them examine your results before assuming you’ve got accurate data. More on this in the Cons.

Cons of Web Site A/B Testing:

1. Can Hurt Web Site Results Unless you always win at everything you do (in which case I’m instantly suspicious of you, or want to go to Las Vegas with you – either way) you’re going to make some bad decisions from time to time. Remember that horrible hair style you just HAD to have that one time? Ugh!

In A/B testing that bad decision, meaning what you thought was an excellent B test item in an A/B test, may go terribly wrong. When that happens (and it will, don’t forget your whole “hair style” incident) you’re going to end up hurting your overall web site results. Be afraid, be very afraid.

True, splitting a smaller portion of traffic will reduce any potential huge catastrophe, but just remember that reduced traffic flow to a test page also means it takes longer to get enough data to make an accurate evaluation. And that takes away from the whole “A/B testing is way, way fast” thing. Dang! This stuff is harder than it looks!

2. Missing the “Why” Have you ever noticed a dog or cat staring blankly into space, at apparently absolutely nothing. WHY are they doing that?! What could possibly be in their minds?! Well, that’s the same feeling you’ll get when you use A/B testing. A/B web site testing does not explore the rationale behavioral decisions that are being made by the web site visitors.

Oh sure, you’ll see the numbers and results of the test, but you won’t know for sure WHY all those web site visitors picked the item they picked. You may have theories, but you won’t know for sure. Worse, you won’t know why they DIDN’T pick the that new and shiny and (what you thought was) totally great B item. You’ll be saying,

“Darn those users! What the heck could they possibly be thinking?! Why didn’t they pick my great B item?”

Just like trying to understand why your cat stares into space, you won’t know why A or B was picked, A/B testing can’t tell you.

3. Not Predictive A/B testing is great and all, but it can’t be used to predict future design change impacts. To a certain extent this means that you’re always stuck doing A/B testing. At some point it would be handy to be able to predict if a whole new web page, web site or application will (or won’t) work, based on fairly accurate predictions of use, without the hassle of actually having to create the web page, web site or application then test it.

Remember, A/B testing at it’s best and most accurate is “live” testing, The golden rule of A/B web site testing?

“Thou shalt not A/B test a live item vs a non-live or non-functioning item, because thou art now testing Apples and Oranges.”

4. Needs Traffic In order to provide quick, consistent and reliable results, you’re going to need a pretty good amount of traffic to your web page to run an A/B test. Remember that for the time the test is running, that traffic split will be siphoning off a percentage of your visitors from the existing page to the test page.

If you are testing on low-volume pages, then that whole “fast” and “accurate” thing just won’t be true, because you’ll need enough traffic (data) to have valid analysis, which means more time. Put another way, A/B testing works great in high-traffic areas, but becomes more problematic in lower-traffic areas.

A/B Web Site Testing Overview:

A/B testing summarySo here’s a summary of A/B web site testing you can print out and carry with you in your purse or wallet. Collect them, trade them with your friends! Well, maybe not. Just memorize these as there will be a test later.

Next: Part II  Pros and Cons of Alpha/Beta testing


Comments are closed.