Usability, A/B or Alpha/Beta Testing, What’s Better? Part III Usability Testing


Web site managers have many choices when testing their sites, including usability, A/B or Alpha/Beta testing, but which one is best, and why?

Part III, The Pros and Cons of Usability Testing

This is Part III, the Pros and Cons of usability testing, of a three part series on web site testing methods. Part I, the pros and cons of A/B testing started the series, and Part II, on the pros and cons of alpha/beta testing, well, was in the middle.

A recent Tweet from Eric Harrigan asked an intriguing question:

“If you could implement only one website testing program: A/B, Alpha/Beta or Usability, which one would you pick and why?”

I found myself thinking about this question and I think (and by the way I need to place a brief disclaimer here and tell you; I’m a bit biased what with being a Certified Usability Analyst and all, so you have to consider the source) that the answer is…

“It depends”

Lame answer you say? Hardly!

You see, there are positives and negatives with each web site testing method. When applied in certain circumstances, one choice may actually be a better pick than another (assuming you have to pick one).

At first, I was going to blog about the pros and cons of each of the three types. But, my post kept getting longer and longer and longer, so I thought I better cut this post back. Therefore I’m chopping this into 3 parts:

Part I  Pros and Cons of A/B testing
Part II  Pros and Cons of Alpha/Beta testing
Part III  Pros and Cons of Usability testing with a summary comparison of all three.

I’d like to stop you here and mention that a smart web site manager will use all three testing methods, plus additional sources of testing, when optimizing a site, but that spoils the fun of being forced to mentally pick one, so let’s proceed.

Pros of Usability Testing a Web Site or Application:

Click here to view latest job postings

1. Tests Real Users – A core attribute and strength of proper usability testing is – real users are the test participants. If usability could flex its biceps at you, this would be the reason why it would do it. Usability is a user testing stud!

Now this assumes that Personas were developed prior to testing, and that users who match the Persons were invited to participate in the testing. What’s a Persona? It’s a fictional representation of a typical user. Personas include behavioral and needs-based information about the typical user, enough such that any design or testing decisions can be made based on the information that comprises the Persona.

Testing the actual people that the web site or application was originally designed for in 1-on-1 testing sessions provides unparalleled opportunity to observe actual user behavior. You’ll discover where the road-blocks are in a task flow for those users almost immediately.

2. Tests Reality, Not Opinions – Usability testing, specifically 1-on-1 usability testing, is typically used to test a task-flow, to determine where in the process there are issues that either slow down, or completely stop a user from completing a task.

Unlike other forms of testing this means you’re getting measurements based on actual behavior and actual events, not opinions or assumed intentions based on secondary data.

This also means there is no need for large numbers of usability testing participants, because usability testing is not opinion-based survey testing, and doesn’t rely on statistically-significant numbers to produce valid results. If 6 out of 7 usability test participants fail to complete a task, you don’t need to test 300 additional participants to see if the results are valid. The task has failed reliably and usability testing documents this.

3. Can test throughout design & development – If you fly from London to New York, or any long distance, your pilots are making minor course corrections as you journey along the way. Consider what might happen if they didn’t! By the time you are nearing the end of your trip your airplane could be hundreds, perhaps thousands of miles off course! You might have to land in Canada, or worse, New Jersey (just kidding all yous guys from Joisy, I heart ya!).

The point is minor course corrections while traveling are easy, major course corrections near the end of a trip are hard.

Your web site or application development process is also a trip. At the end of this trip, you’re supposed to have this amazing web site or application that your users find so easy to use, they just love it and use it and you make lots of money have lots of satisfaction!

Consider usability testing as your opportunity to conduct little course corrections along the development process journey.

If you conduct usability tests during key points in the design and development process, it allows minor (aka low cost) corrections to be made to improve the user experience. This ensures your web site or application will not have to make any major (expensive!) corrections at the end of your development process, or worse, have your application end up in Joisy (just kidding again)!

No other form of testing enables you to have specific user-based data that is so clear, and so actionable, for the entire length of your development process.

4. Provides the Why of user behavior – All those mysteries around the numbers in log files, like; “why does everyone click on A and not B?” can now be answered.

The usability testers will (if prompted correctly) explain their thoughts and actions to you. This provides the behavioral insights as to why certain choices are being made, or not being made.

Of all the testing methods out there, including by the way tracking software, this is the only way to learn why users do the crazy things they do interact with your web site the way they do.

Cons of Usability Testing for Web Sites and Applications

1. Trained Usability Professionals required – Unlike Doctors and Lawyers, you don’t need a license, degree or ice-cold waiting room to practice the art and science of usability testing. This means anyone who wants to claim they are a usability practitioner can do so. Without extensive knowledge of usability testing theories, methods and best-practices, you might end up with a usability test that provides poor, or worse, bad advice.

The best way to ensure you’re dealing with a reliable trained usability professional is to do your homework before you hire them. Ask for referrals and call those referrals. Research the usability practitioner and their company. Have they been in business a long time? Are they active in Associations like the Usability Professionals Association, SIGCHI (Special Interest Group on Computer-Human Interaction) or related groups? Do they have training and/or a degree in the Human-Computer Interaction field? Do they blog about usability? Do others in the usability field blog about them?

As with plastic surgery and iron-clad prenuptial agreements, your usability fate lies with the expertise of the practitioner, make sure you hire a good one.

2. Testing won’t reveal all issues – Usability testing is not going to find every problem with a web site or application. There are great debates that rage from time to time in the usability field about how many 1-on-1 usability test participants it takes to do a proper test.

But that’s not the correct question. The correct question is, “How many problems do I think I need to find?” Is it 50%, 75%, 80%? Based on that estimate, the proper number of test participants can be gathered.

Even with lots and lots of test participants, usability testing simply won’t find all the problems with a web site or application.

The important thing to remember about applying usability testing to find problems is that you’ll probably find most of the more significant issues, and fixing those probably will get you much closer to an optimal user experience.

Just realize going in that it’s unlikely you’ll find all the problems, at least in one testing session.

3. Usability Testing results can vary – A while back some studies of usability testing methods and results were conducted. A single web site was tested by groups of usability professionals. You may be surprised to learn that there were just as many test methods, and test results, as there were groups of testers.

This is because usability is part art (dealing with humans and design) and part science (using standard methods of conducting tests that generate repeatable results). Meld the two together and you have an inexact science.

Remember that whole “make sure you do your homework about your potential usability professional” message above? Ditto here! Your usability testing results can and will vary based on the expertise of the usability professional.

4. Testing & fixing are different things – A common misconception among usability clients is that the act of conducting usability testing will fix their web site or web application. In reality, as with other forms of testing usability testing only finds the usability issues and makes recommendations for improvements. It takes a separate effort to actually do the work to make the recommended changes.

Problems arise when the recommendations for usability improvements cannot be made, either due to business or technology issues, or due to the costs in terms of resources or time necessary to make the recommended changes.

When this happens, the dance that is called “trade-offs” happens, meaning sometimes important usability optimizations are not done. Sadly, it’s a rare usability engagement that has all of the suggestions for improvements done. Sometimes some easy changes (the low hanging fruit so to speak) are completed, but the more difficult but important changes are left undone.

This might leave a bad-taste about usability with some business owners. After all, wasn’t the usability testing supposed to fix our problems? Um, no. Find the problems and suggest solutions? Yes! Actually fix them? No.

Be sure to include time and resources at the conclusion of usability testing to make the recommended optimizations. This ensures your usability testing money is not wasted.

Conclusion – Usability Testing Web Sites and Applications:

usability testing

So here’s a summary of usability testing for web sites or application you can print out and carry with you in your purse or wallet. Collect them, trade them with your friends! Well, maybe not. Just memorize these as there will be a test later.

Summary of A/B, Alpha/Beta and Usability Testing Methods:

As promised, here’s your handy-dandy summary of testing methods that you can print out and share with your boss, friends or the guy that you see jogging every day and wonder how come he doesn’t have shin-splints already. I hope you’ve enjoyed the series, please let me know if you did, or even if you didn’t!

And don’t worry about the test, I was just kidding about that. The real test is will you actually USE these testing methods to improve your web site or application!

A/B testing summary

usability testing

Prior testing articles in this series:
Part I  Pros and Cons of A/B testing
Part II  Pros and Cons of Alpha/Beta testing