Usability, A/B or Alpha/Beta Testing, What’s Better? Part II Alpha/Beta Testing


Web site managers have many choices when testing their sites, including usability, A/B or Alpha/Beta testing, but which one is best, and why?

Part II  The Pros and Cons of Alpha or Beta Testing

This is Part II, the Pros and Cons of Alpha or Beta testing, of a three part series on web site testing methods. Part I, the pros and cons of A/B testing started the series, and Part III, on the pros and cons of usability testing will conclude the series.

A recent Tweet from Eric Harrigan asked an intriguing question:

“If you could implement only one website testing program: A/B, Alpha/Beta or Usability, which one would you pick and why?”

I found myself thinking about this question and I think (and by the way I need to place a brief disclaimer here and tell you; I’m a bit biased what with being a Certified Usability Analyst and all, so you have to consider the source) that the answer is…

“It depends”

Lame answer you say? Hardly!

You see, there are positives and negatives with each web site testing method. When applied in certain circumstances, one choice may actually be a better pick than another (assuming you have to pick one).

At first, I was going to blog about the pros and cons of each of the three types. But, my post kept getting longer and longer and longer, so I thought I better cut this post back. Therefore I’m chopping this into 3 parts:

Part I  Pros and Cons of A/B testing
Part II  Pros and Cons of Alpha/Beta testing
(You’re reading it!)
Part III  Pros and Cons of Usability testing with a summary comparison of all three.

I’d like to stop you here and mention that a smart web site manager will use all three testing methods, plus additional sources of testing, when optimizing a site, but that spoils the fun of being forced to mentally pick one, so let’s proceed.

Click here to view latest job postings

According that that compendium of all knowledge, Wikipedia, Alpha and Beta testing is defined as…

“Alpha testing is simulated or actual operational testing by potential users/customers or an independent test team at the developers’ site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.

Beta testing comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the programming team. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.”

For both Alpha and Beta testing, the web site (or application) is typically already built, finished and pretty much ready for use.

There is an assumption that testing in multiple forms has already occurred prior to Alpha and Beta testing, but sometimes that testing is not comprehensive to the user experience.

No matter how much prior testing however, it is assumed that there will be potential “bugs” (aka issues, problems, snafus or other “oops, we didn’t realize XYZ was broke” items) and that Alpha and Beta testing will uncover these issues prior to the web site or application going to the public at large.

The practice of Alpha and Beta testing assumes changes will need to be made, and will be made, prior to official “go-live.”

I like to think of Alpha testing as an Internal live test of a new web site, where people in a company become the end users and are subjected to the bug hunt final testing before the site goes public.

I like to call Beta testing the “Google Way” as it seems to me that most, if not all, of Google’s web applications at one time or another are put forth to the public for testing, but in a controlled manner (meaning you have to sign up to Beta test, so Google knows who’s testing what).

Considering that Gmail is still in Beta, and has been for the past 5 years or so, I’m guessing Google takes this whole Beta testing thing pretty seriously. There are other business reasons why it’s sometimes advantageous to use Beta testing, but more on that below in the Pros and Cons section.

Warn Beta Testers in Advance

Usually Alpha and Beta testing is conducted with users who are informed in advance that things may or may not work as expected. I like to call this an “eyes wide open” policy, meaning any potential users of the test web site (or application) are warned in advance that what they are about to use may or may not work correctly, and should hold the company harmless for any losses that might occur.

Often, users are asked to provide feedback about the issues they uncover either on an on-going basis, or through surveys or other tools for obtaining feedback.

Focusing on the more widely available Beta testing format of Alpha/Beta testing, let’s examine some of the pros to using the public to do your testing for you:

Pros of Alpha or Beta Testing of Web Sites or Applications:

1. Large Volume of Testers – Beta testing offers a company the ability to get a large number of testers interacting with the web site (or application) quickly. This provides plenty of opportunities to obtain feedback from real-world usage. Smart companies will usually post a Bug log or list of known issues, so that they will (hopefully) not receive a continuous stream of reports of the same issue, over and over and over again.

2. Live Testing – The great thing about Alpha and Beta testing is it’s live testing, using the actual environment that the web site or application was originally intended for. And because it’s live testing, results are obtained in real-time, meaning it won’t take long before users report items are not working as desired.

3. Real User Testing – The beauty of Beta testing is it is being done by real users, who have no pre-built assumptions about why things work, or how they are supposed to work. This means a wide variety of items are tested and reported on by real users, including usability, function, content (including help or instructions content!), error messages and other more nebulous items. All of these and more will be scrutinized and reported.

Perhaps some of these items were forgotten lightly covered by the design and development teams. Real users will test all systems and their feedback can greatly help add additional information and optimization of these critical areas.

4. Beta Means Continual Tweaking – Unlike boyfriends or husbands, you can change a web site or application, especially one that is being Alpha or Beta tested. With Beta testing, new versions with adjustments and optimizations can and will be deployed on an on-going basis. Because users have been warned, they should accept this and even help by testing the updates, to ensure they achieved the desired goal of fixing issues.

This by the way is an advantage over a final version of software, in that Beta versions allow for design and development improvements that do not have to be provided to OEMs or business partners as a software release. This is a distinct advantage for a company that’s not quite ready to support a mass-produced (and consumed) application post launch.

Cons of Alpha or Beta Testing of Web Sites or Applications:

1. The Web site or Application Must be BuiltPerhaps one of the major drawbacks of Alpha and Beta testing is the fact that the web site or application has to be built, meaning significant time and resources (meaning cost) were applied to construct it.

The cost of going back and making drastic changes to fix issues uncovered by Alpha or Beta testing can cause concern, and sometimes resistance, especially if a company has not allocated enough resources and time for post-Beta optimization.

This often results in what I like to call “Beta Momentum,” which is the resistance to change applied by the business due to the costs of making substantial tweaks found during Beta testing. I can think of several examples of web sites and applications that were rolled out to the public after Beta without all problems being fixed, perhaps you’ve seen a few too.

This trade-off of live but flawed executions might even defeat the purpose of using Beta testing as an optimization strategy.

2. Beta Testing is ChaoticThe thing that makes Beta testing so great, multiple users with little or no knowledge of the expected outcomes testing various random items, also makes it weak.

Since there is often no organized structure to how and where users test, there is no organized structure to receiving feedback. Some issues may be reported over and over again, while other issues are virtually ignored due to lack of use, or lack of awareness that there’s even a problem.

Throwing an unlimited number of Monkeys at typewriters may end up in a Shakespearean play being written, but throwing an unlimited number of testers in a beta test does not mean a perfect test of the application will be developed (even though some testers doth protest too much).

3. Beta Testers May Not Match Expected End-UsersOpening a web site or application to any and all testers could be opening a Pandora’s Box, in that the feedback received may be more harmful than helpful. Typically web sites and applications are targeted for specific end-user types (in usability they are referred to as Personas). If testers don’t match the profile of the expected Persona’s, then feedback collected from testers may not be appropriate, or worse, may be counter-productive.

Consider a public beta test of a web site targeted for females over the age of 55 that are interested in health information. Unless the developers have created a beta registration process that screens to match this profile of user, they may not know who is using the site, and whether the feedback they are receiving from their beta testers is accurate for that target.

4. Beta Testing Exposes Your Secret SauceMost web sites and applications have the ultimate goal of generating income for a company. Most of the time, there’s a competitive advantage of a new gee-whiz feature or function that makes using this new web site or application compelling, and makes people want to pay for it and use it.

Exposing your gee-whiz feature or function to the public for free in Beta form means others now know about it too, and can potentially reverse engineer it, or go it one better and start working on an even more exciting gee-whiz item. Your secret sauce will be out there for others, including your competitors, to see well before you may be in a position to start selling the final production version.

Alpha / Beta Web Site Testing Overview:

So here’s a summary of Alpha / Beta web site or application testing you can print out and carry with you in your purse or wallet. Collect them, trade them with your friends! Well, maybe not. Just memorize these as there will be a test later.

Next: Part III  Pros and Cons of Usability testing with a summary comparison of all three.


  1. I’ve enjoyed this series so far. One thing that’s really begun to bug me about alpha/beta is the number of companies that put software in beta just to rush it out early, with no intent to really test it or plan for how that testing will occur. Even reputable companies are putting products (*cough* Gmail) in beta for months and years.

    Like any testing program, I think beta testing should have objectives and timelines. Companies need to know what they’re trying to achieve, have a structure in place for cataloging and handling bugs that pop up, and have a target for what they consider “bug-free”.

  2. A well-used website with a member community relaunched it’s website recently – without testing. The site didn’t work, information and functionality was inaccessible. Things that worked on the mobile version, now didn’t work. It undermined trust in the company, they lost a lot of users / paid members in this time. They left the badly designed version up – and decided to fix it while it was live – which took a good 4 months. It damaged the company’s reputation quite badly. I was a user / paying member, and to be honest, I haven’t wanted to visit the site since that time. I couldn’t believe how unprofessionally they had managed the whole process. They seemed to have no clue. And didn’t care about all the users/members they lost – they had the approach, “we’ll find new ones once the site is working” – using SEO, paid search etc…it was a very poor attitude, and has damaged their reputation irreparably. BETA testing is a must, companies must be prepared to accept feedback – whether it is positive or negative – in the end, it is all positive feedback if it leads to a better user experience and more engaging experience for the greater audience once launched.

  3. Hi Beth,

    Good story. Yes, it’s sad how some organizations blindly push website redesign projects forward based more on achieving a date for launch, and less on doing it right.

    It is said, “not taking the time to do it right the first time means taking the time to get it right the next time.”

    A/B and Beta testing could have helped the company you mentioned immeasurably.

    Thanks for sharing your story.

Comments are closed.