Tags Posts tagged with "usability"


1 591

Why A/B Testing Needs Usability Testing, aka Why Opposites Attract

Why A/B Testing Needs Usability Testing by UsefulUsabilityWhy does A/B testing need usability testing? Because opposites attract, and the benefits received by coupling A/B and Usability testing can go far in improving the UX of a website or app. In fact, it’s better than any other method. Here’s why.

Way back in the dark ages of 2009 I wrote a three part series on A/B Alpha/Beta and Usability testing, asking the question of which one is better.

To summarize the pros and cons of A/B testing and usability testing from the three part series:

A/B Testing positives:

  • Fast (testing can be set up and run in a day)
  • Tests reality, not theory
  • Quantifiable with statistical significance
  • Accurate (winner almost always performs at that level)


A/B Testing negatives:

  • Can hurt results (B version failure can cause reduced conversion)
  • Missing critical “why” data (no data to explain why a version won)
  • Not predictive (data not available to determine what to test nex)
  • Needs traffic (doesn’t work on prototypes or apps with no users)


Usability Testing positives:

  • Doesn’t hurt results (testing does not impact conversion)
  • Provides the “why” data (qualitative data answers why users do what they do)
  • Predictive (results can determine what next to test)
  • Doesn’t need traffic or even a live website or app (testing available from concept through production stages)


Usability Testing negatives:

  • Trained professionals required for unbiased results
  • Won’t reveal all issues
  • Results can vary
  • On-going testing is difficult

How A/B and Usability Testing Compliment Each Other

A/B and usability testing complement each other in several ways.

First. Did you happen to notice how the four negatives for A/B testing are actually the four positives for usability testing? That’s right, the issues inherent in A/B testing are actually strengths of usability testing, and vice versa.

Chart of A/B Testing Cons and Usability Testing Pros:

A/B Testing Cons Usability Testing Pros
  • Can hurt results
  • Doesn’t hurt results
  • Missing critical “why” data
  • Provides the “why” data
  • Not predictive
  • Predictive
  • Needs traffic
  • Doesn’t need traffic

Second. A/B testing provides the quantitative side of UX data, and usability testing provides the qualitative side. The quantitative ‘what’ of user experience choices your website and app visitors make is augmented with the ‘why’ for those choices, which comes from qualitative usability testing data.

Third. A/B Testing coupled with usability testing provides an end-to-end view into the user experience of your website or app. By coupling the data, you have a much clearer picture of the engagement happening in your site or app. You also will have the data to know how to impact and improve on that engagement. Finally, you’ll have the data to evaluate your results and move forward with your next set of testing and optimization.

Why A/B Testing Needs Usability Testing

The benefits of combining A/B testing with usability testing are many and include:

  • Complete quantitative and qualitative data for more informed decision making
  • Comprehensive view into the user experience of your app or website
  • Ability to use data to predict what additional tests can increase engagement
  • A 360 degree view of data to use for continuously optimizing your app or website

By combining A/B testing with usability testing, you’ll find your testing and optimization efforts produce far better results, which is a win for you, your firm and your app or website users.

A/B Testing and Usability Testing Resources:

Three Part Series: Usability, A/B or Alpha/Beta Testing, What’s Better?

Part 1 – Usability, A/B or Alpha/Beta Testing, What’s Better? Part I A/B Testing

Part 2 –  Usability, A/B or Alpha/Beta Testing, What’s Better? Part 2 Alpha/Beta Testing

Part 3 – Usability, A/B or Alpha/Beta Testing, What’s Better? Part 3 Usability Testing

Usability.gov Usability Testing Section

WikiPedia Usability Testing

14 Usability Testing Tools


0 245

How to Make a UX Research Persona video, Part 2 of 2

Watch the second part of the two part video series to learn how to make a UX Research Persona for user experience research or usability testing. Part 2 of a 2 part series.

UX Research Personas, we have the contextual inquiry data, now what?

Consolidate data and look for commonalities and patterns

Now that you have all your observations and data we discussed in the How to Create a UX Research Persona part 1 video, how do you synthesize it all into a Persona? Look for common patterns, specifically in terms of how the end-user goes about accomplishing their goal. There are several steps to looking for patterns, which include (PS, more detail in the video):

  • What are consistent work-arounds to existing problems?
  • What does everyone say repeatedly about a goal or desire?
  • What consistent task-flow successes, or failures, are shared among your end-users?

Put your draft Persona together (hint, work backwards)

Now that you’ve identified your common patterns, it’s time to start putting your Persona together. Many practitioners start by rushing off to find a great picture and/or name for their persona. However, there’s a better way to do this, and it’s by working backwards:

1. Start with end goal – What ultimately is the end-user wanting, what’s the desired end state?

2. Identify critical tasks – What are the top 1-3 critical tasks necessary for the end-user to be successful? This identification is necessary for usability testing, but not for general design or product Personas (although why WOULDN’T you find it helpful to know the critical tasks necessary for the end-users success)?

3. Document environment of use – Are there common places, devices or 3rd party tools that consistently are used or needed? If so (and they are important to the completion of the end user’s desired end-state) document them.

4. Define domain expertise – Is there a common domain expertise, meaning familiarity with the systems, terminology or processes? If yes, document them. An example is a claims processor for a large insurance company who has to be trained on the terminology and processes before utilizing an internal claims-entry system.

5. Create a name – Be sure to be culturally sensitive, focus on common names that are easy to remember and that can easily be used by your team. Names are important, don’t scrimp on spending time to find just the right name for your Persona!

6. Find a picture – As humans, we are visual creatures, so a face and name are important to humanize our Persona. I recommend real pictures versus cartoons or clip art, and if possible use pictures showing the end-user in context of use of the system. For example, if creating an app to find a lost dog or cat, a picture of a happy pet owner hugging their pet would be a good choice.

Things that all UX Personas have in common for UX research or usability testing:

There are thousands and thousands of variations of Personas out there, just do a search for “UX Persona” in Google images to see what I mean. However, for UX research and usability testing purposes most Personas should share the same things in common, including:

  • Picture – Important to personalize and humanize our persona, MUST be an accurate visual representation of our Persona. Don’t just use any random picture that sort of looks like a Persona.
  • Critical Tasks – Typically no more than 3
  • Scenario – Specific to the critical task or tasks, what is this Persona trying to accomplish?
  • Background – The background for the scenario, why is this Persona trying to accomplish a critical task?
  • Devices – What device or devices does our Persona typically use, or what 3rd party tools are required? Less important now as most people can and do use multiple devices, however still important especially if we are discussing B2B software or solutions that require specific devices.
  • Domain Expertise – How educated or familiar is the Persona with the subject matter, terminology and existing process flow? Do they have a good understanding of terminology and a solid mental map of how the process and task-flow should work, or not?
  • Environment – Again, less meaningful now that the internet is everywhere via mobile, still, it’s important to consider the context in which the user is engaged with your website or app.

You will see a huge variety in types of Personas, from the very detailed to the very basic. But for UX design and research purposes you can’t go wrong making sure you have the above data clearly defined for your Personas. Watch the video to learn why.

UX Research Persona Resources:

How to Create a UX Research Persona Part 1 – The first video in this two part video series on how to create a UX research Persona for user experience research and usability testing

How to Create a UX Research Persona Part 2 – The second video in this two part video series on how to create a UX research Persona for user experience research and usability testing

Personas – Research and techniques as documented by Forrester

Personas Introduction Video – A brief introduction into Personas, what they are, types of personas, and why Personas are important for you.

7 Signs You May Have a Problem Persona – Good article on how to spot potential issues with your Personas, and how to fix them.


1 394

Should I usability test my own site is a question I am often asked.

Should-I-Usability-Test-My-Own-Website-UsefulUsability-The answer to the question “should I usability test my own site” is not as clear nor as obvious as you might think.

There are multiple reasons why a quick “yes” or “no” doesn’t necessarily apply.

In this article I will explain the issues of testing your own website so you can be more informed when it’s time for you to conduct usability testing.

Usability Testing Your Own Site Definition:

So what exactly is usability testing a website? My definition is:

What I mean by that definition is “usability testing” is conducting performance-based tests on a website’s critical tasks using people who match the website’s Personas. What’s a Persona? A Persona is a representation of the most common website visitors who all share a set of critical tasks.

It is very important to remember that it is the critical tasks that are being tested, not the tester’s opinions about those tasks.

Likewise, usability testing is not about capturing survey data, or voice of the customer information (although both of those can be helpful additions to the performance-based testing).

So when people ask me,

“Should I usability test my own website myself?”

I reply with,

“If you mean you being the test participant? No. Because you can’t usability test your own site yourself conducting performance based tasks. You probably don’t match the Persona, and you are already too biased and know the correct task flow.”

Running But Not Participating In Tests:

Some of you may be wondering a similar question, which is,

“Should I set up and run a usability test on my website only (ie. not participate in the testing)?”

Here the answer is a bit more cloudy.

In general, I would caution against usability testing your own website, there are 5 reasons why:

1. Usability Training Matters:

Unless you are trained in usability testing, it’s dangerous to assume you have the knowledge and expertise necessary to very carefully write a non-biased testing protocol.

It is amazingly easy to find tools that anyone with or without training can use to conduct usability testing. But it is also amazingly easy for those without usability training to accidentally create a biased test that provides bad data.

When I was in the process of becoming certified in usability analysis I attend many classes prior to taking and passing my certification test. I would say about half the classes were focused on helping students understand biases; where they can creep in, how to uncover them, and how to write protocols that carefully eliminate them.

Without that advanced training, conducting your own testing can introduce hidden biases in the test protocol, which could make results invalid and cause resulting website usability changes to make things worse, not better.

2. Internal Usability Teams Can Be Biased:

Although less common, I’ve come across situations in which I was surprised to find that even internal company usability teams had hidden bias’ in protocols. Not always of course, but often enough over the past 20 years of my experience that I’ve decided the longer you spend time with a website or company, the more difficult it becomes for you to remove your own biases from the process.

That’s not to say that internal company usability teams can’t write non-biased protocols. It’s just that it becomes more and more difficult as domain expertise rises AND testing patterns become standardized.

Let’s face it, as humans, we are all creatures of habit. And for internal teams this can sometimes unfortunately mean picking up biased habits and sticking to those habits when writing protocols and creating tests.

3. Usability Testing Protocols Are Mandatory:

Sadly there’s a common misconception out there that anyone can usability test simply by walking down a hallway or using software to ask people to try to do something on a website and observe them.

I’m here to tell you that’s a dangerous oversimplification of how to actually conduct non-biased tests.

To ensure that bias is not in the usability test, it’s critical that a usability testing protocol be developed AND be reviewed in advance. Without the protocol, random “guerrilla usability testing” or “agile testing” can accidentally introduce bias into the process.

Some usability practitioners, entrepreneurs and usability testing software services may disagree with me.

But I’m here to tell you the primary mission of a protocol is to ensure you’re reducing bias and that you’re testing across a standard set of criteria for each test you conduct. Without it, test results may vary due to the unintended introduction of biases.

4. Moderating Usability Testing is Harder Than It Seems:

Closely associated with needing a protocol, moderating the actual usability tests without bias is not easy.

The test participant is looking for clues (after all, it is a test). They will instinctively read a moderator’s body language and other verbal or non-verbal queues. Being on guard to not generate those clues while at the same time gathering think aloud feedback plus keeping the tester focused takes skill.

The common misconception that anyone can usability test without training or bias is false.

It is best to let trained usability professionals handle moderating usability testing, else risk bad data entering the process.

Again, entrepreneurs and do-it-yourselfers may scoff, but I’ve seen plenty of bad websites made worse by erroneous data coming from badly moderated testing sessions.

5. Analyzing and Prioritizing Usability Testing Results Requires Expertise:

Analyzing usability testing results and prioritizing them also takes skill that I caution should not be handled by untrained usability practitioners.

Here’s a common scenario:

You’ve conducted usability testing and found 8 places in a task-flow with errors causing poor performance.

What’s the severity, and thus priority, of those errors?

Analyzing issues and identifying possible solutions requires a fair amount of user experience and usability best practices knowledge.

Going with gut feelings about the best way to solve an issue can sometimes work, and sometimes not. It’s better to analyze the results and make recommendations based on extensive experience in understanding UX and usability best practices.

Associated with this, I have seen plenty of bias in prioritizing results even by internal usability teams. That’s because being so familiar with their organization, they knew what could and could not easily be worked on by developers. They also knew what would and wouldn’t be easily supported in terms of workload by business owners.

How many times did the “hard to do stuff” take a back seat to the “easy to do stuff?” And, did the “hard to do stuff” have a much greater impact on usability than the “easy to do stuff?” Often it does, but often it’s not prioritized that way due to internal bias.

Conclusion: Should I Usability Test My Own Site?

So should I test my own site?

Now that you have a clearer picture of the issues around testing your own website, I think you’ll agree a quick answer of ‘yes’ is not always appropriate.

This is NOT to say that usability testing your own site cannot or should not be done!

Instead, it’s a warning that when testing your own site it is very easy to let bias into the results, thus causing bad data and potentially bad results from subsequent optimization.

Being aware of these issues, my recommendation is to consider your resources, and if it makes sense contact a professional usability consultant like me or others in our industry anytime you’re considering usability testing on your own website.

Further Usability Testing Resources:

Usability.gov – Usability Testing Methods

UsefulUsability.com – 14 Usability Testing Tools

Human Factors International – Certified Usability Analyst (CUA) Training

Nielsen Norman Group – Full Day Usability Testing Training

0 334

So How Did My Earlier 7 Controversial Usability Predictions Turn Out from 5 Years Ago?

Results-7-Controversial-Usability-Predictions-2010-326x235-UsefulUsabilityFive years ago, I published a post of 7 controversial usability predictions for 2010. So what were the results? Did my prognostications come true? Did I get them right?

So with 100 percent transparency, and a small amount of trepidation on my part, let us review how my predictions turned out.

For those of you who need may need a refresher on my 2010 controversial usability predictions, here they are with a brief update on how I did:

  1. The cost of conducting usability testing will decrease by a factor of 10.

I think I nailed this one (means I got it correct for those of you not up on U.S. slang).

Costs for conducting usability testing are a fraction of what they were five to six years ago. Why? Primarily because more and more firms and consultants are using remote moderated and un-moderated usability testing in place of expensive in-person testing at remote locations.

Remote moderated and un-moderated usability testing tools enable research to be conducted for a fraction of what it costs to fly a team to a remote location, rent a facility or hotel room, schedule and conduct the tests, pay for food, travel and lodging for the team, fly them back, and wait for their results.  The savings easily beat the factor of 10 prediction I made.

Add to that that there are now a plethora of low cost remote moderated and un-moderated usability testing tools available, and the savings are even greater.

  1. There will be a dramatic increase in the use of low cost web-based usability testing tools.

Again, nailed it!

Looking out at the tools available today, (you can see a recent list in my 14 usability testing tools article), it’s clear that web and mobile based solutions are plentiful, and seem to be added to daily.

  1. True usability ROI will continue to elude usability practitioners

I think I mostly got this one right, and that this statement is still mostly true, as I have witnessed some firms who have become able to accurately predict their Return On Investment (ROI) for usability. But, sadly, I’ve seen plenty of other firms that are still clueless about usability and ROI.

It helps to have people like Jeff Sauro of Measuring Usability providing helpful information on how to measure and estimate ROI for usability. But I am going to go out on a limb here and state that there are still plenty of organizations and consultants who have no idea how improved usability adds to their bottom line.

  1. Use of remote moderated usability testing will increase by a factor of 10

This one I think I missed. Firms that provide remote un-moderated usability testing like UserTesting, UsabilityTools, Loop11 and plenty more have experienced tremendous growth in utilization. But to a certain extent, that growth has I believe come at the expense of conducting usability testing sessions using remote moderated methods.

I have to admit that even for myself, it is sometimes easier, faster and quicker to conduct un-moderated remote usability testing versus moderated remote sessions. The allure of obtaining results in 10 minutes sometimes pulls decisions to use remote un-moderated, when in fact remote moderated would have been equal to or potentially better for a particular test.

Remote moderated will never go away, but because it takes more leg work to set up and administer it will probably never see utilization increase anywhere near remote un-moderated utilization.

  1. The UK will become a major source of usability expertise

Nailed it! Have you seen the huge number of UX and Usability conferences in the UK? Here’s a list of the UK’s past 337 UX events from Lanyrd. Yes! 337 events!

There are also scores of UX, Design and IA shops in the British Isles. The UK is no slouch when it comes to usability expertise. Our friends over the pond have embraced all things usability and UX and have used it to great extent. Is there room to do more? Of course, but considering the number of full time usability and UX shops that were there five years ago versus today, there has been tremendous growth in this area. So raise a hefty pint of ale, and three cheers in celebration of usability expertise in the UK!

  1. The phrase User Experience Design will become overused and almost meaningless

I’ll give myself a partial correct on this one. True, UX is now a far more common term than usability and in some ways UX has killed Usability. And true, most business folks or non-techies may not know one from the other, but still, there has been some consolidation and standardization of the term user experience design that most in our circle understand and use. It’s far from meaningless, so although there are still multiple ways to define ‘UX,’ the common theme of the experience a user has with a product, website, application or whatever seems to be fairly well understood.

  1. Without professional certification being required, more and more charlatans will be attracted to usability

I’ll give myself a partial correct on this one. A Certification course, test and certified practitioner list still eludes our ranks. I would have hoped the User Experience Professional’s Association could have made some progress on this in the past five years, but sadly that is not the case.

I have seen plenty of suspect ‘UX Audits’ and ‘Usability Reports’ floating around that seem to be very sub-standard in terms of actual UX and Usability expertise. Still, for the most part the players continue to be the players, and new consultants that pop-up for the most part seem to be interested in doing the right thing by their clients and providing real value. I may have been a bit negative in my attitude on this one. But still, until there is an official Certification and evaluation of practitioners, it really is a ‘buyer beware’ world for our prospective customers.

Conclusion: How I did On My 7 Controversial 2010 Usability Predictions

So overall I scored myself with:

  • 3 Correct
  • 3 Partially Correct
  • 1 Incorrect

How would you score my predictions? Do you agree with my scoring? Be sure to leave your comments below!

A Website Audit can improve your ROI (and respect) at least 7 different ways, here’s how:

Website audit funny quote

 Website Audit Definition

The definition of a website audit for UX purposes is to evaluate both behavioral data (visitor actions) and user experience data (task-flow other UX vs. best practices) to identify issues and recommend opportunities to improve conversion.

Take My Website – PLEASE!

Sometimes a website audit can make you feel like you get no respect, as if you are the Rodney Dangerfield of the web.

Side note: Odd how many companies do not respect their own websites. I’m not sure why that is, but a separate follow-up study may be needed.

In any event, I digress…

Remember his famous quote?

“My psychiatrist told me I’m going crazy. I told him, “If you don’t mind, I’d like a second opinion.” He said, “All right. You’re ugly too!””

– Rodney Dangerfield

But I’m here to assure you that a website audit, specifically an audit of your website’s UX, can and will make your website ROI better, and get you the respect from your website visitors (and bosses) that you deserve. Here’s how…

Website Audits Can Improve ROI

Improving the ROI of a website is best accomplished by ‘fixing the leaky bucket.’ What I mean by that is finding and fixing the issues that are causing website abandonment, task flow failure, poor visitor engagement and disappointing conversion.

There are many variations of website audits, and all have their unique value. These include:

  • Accessibility Audit
  • Analytics Audit
  • Conversion Optimization Audit
  • Page Speed Audit
  • SEO Audit
  • Usability Audit
  • Website Competitor Audit

However, I like to combine the best elements of several of the above audits into a comprehensive website audit that evaluates the larger user experience. This is a great way to ensure all the data available is utilized to analyze the website and make recommendations for optimization. ROI improvements based on recommendations from website audits are typically significant and quick.

Website Audit Elements

The two categories of information that a website audit evaluates are the critical elements of a well done audit:

Behavioral Data – Behavioral data will define what actions visitors are doing (or not doing) on the site. This data typically comes from website analytics tools such as Google Analytics, Adobe Analytics, Coremetrics, WebTrends, etc. Typically this data includes information such as:

  • Overall website performance conversion data
  • Paid search conversion data
  • Organic conversion data
  • High-level PPC keyword data
  • Website bounce rate
  • Visits by browser
  • Screen resolution
  • Top content
  • Content paths
  • Length of visits
  • Location (by geography)
  • Page fold
  • Devices
  • Operating Systems
  • And more…

User Experience Data – UX data includes information about how the website compares to usability and UX best practices. This data comes from tools such as the 5 second test, eye tracking, usability testing, and a comparison of the critical website interaction elements versus best practices (sometimes called a heuristic or website review).  This type of data typically includes information such as:

  • Elements that are or are not attracting attention
  • Page fold ramifications on CTAs or critical copy
  • Form field best practices vs. existing forms
  • Navigation flow and labeling
  • Product page elements vs. best practices
  • Contact Us page elements vs. best practices
  • And more…

Website Audit Findings and Analysis

The website audit includes a document with detailed findings and analysis of the behavioral and UX data that precisely defines where the website is performing well, and where there are opportunities for improvements. The subsequent recommendations are then tested, typically with A/B testing, to verify that the optimizations are having the desired benefit.

I always include screen shots of each of the items being audited, with callouts that explain what the issue and opportunity for testing could be. These can sometimes be fairly large documents, upwards of 70 to 90 pages. But because the information is presented one item at a time getting through the analysis document is easy, and fairly quick.

Advantages to Website Audits

There are at least 7 primary advantages to website audits, they include:

1. Ability to use website data to prioritize A/B testing: Nothing beats using your actual website and UX data to define and prioritize where your conversion is not optimized. This removes the guessing game that too often occurs as part of A/B or Multivariate testing.

2. Benchmarks existing versus potential conversion: By using the data from a website audit as a benchmark, the down-stream changes to traffic, navigation flow and conversion can be quantified. Benchmarking removes much of the guess work out of determining if conversion improvements based on testing are temporary, or permanent.

3. Pinpoints issues: The analysis of the website audit provides pinpoint clarity on which elements of a page, form or flow are potentially hindering performance. Specific recommendations can be extremely detailed, which helps focus where to spend testing resources. If you’ve ever heard the principle of “don’t throw out the baby with the bath water” you’ll understand and appreciate the detailed approach of a website audit

4. Clearly defines success: Many of my clients appreciate the incremental value that comes from the analysis of a website audit, in that they appreciate knowing what ‘success’ looks like. When A/B testing, how do you know you’ve truly optimized the ‘B’ version to achieve maximum success? Unless you have data that defines what that success looks like, you don’t really know how successful the test could be. A website audit provides that ability to better define success.

5. Doesn’t ‘fix’ what’s not broken: Too often companies optimize using an ad-hoc approach to testing. Sometimes this causes something that was working just fine before, to work poorly, or not at all. A website audit helps define what’s working from what’s not, so that items that are working well are left alone, and items that are not are tested and optimized.

6. Maximizes resources: Unless you have a full time optimization team, the odds are that testing is but a small part of your overall work activities. Maximizing your time and other resources is crucial to optimizing ROI. Spending time optimizing only that which needs to be optimized makes your resources that much more productive.

7. Proves your value (and earns you respect): Facing your bosses and answering to them for how you specifically are helping the company can sometimes be a challenge. Having the demonstrated results from the testing that comes from a usability audit provides you with a plethora of actual data that precisely defines how you are contributing to ROI. Nothing speaks better to your bosses than data that proves you (and thus they) are adding value to the company. You want respect, give them optimization numbers and you’ll get it!

Conclusion:  Website Audit Improves ROI and Respect

A website audit and the resulting optimization of the conversion of the site can greatly improve the ROI and performance of a site. It can also provide you with the respect you need. If you have any questions or would like more information on how you can use a website audit to improve your website ROI just contact me. By using the results of the website audit, you will have a more informed, prioritized and clearly defined road to improving the success of your website.

35 38181

A Comprehensive Review and Matrix of 14 Usability Testing Tools

Comprehensive review and matrix of 14 usability testing toolsIn 2009 I wrote a popular article on 24 usability testing tools. Since that time, there has been tremendous growth in the quantity and capability of usability testing tools. An update to the article is due, but because of the broad array of tools now available a more formalized definition and a matrix of usability tools is needed.  To that end, the following matrix and comprehensive review of 14 usability testing tools will help practitioners who are looking for tools for specific testing tasks.

Usability Testing Tools Matrix

For purposes of classifying tools to evaluate in this matrix, the tools must have three critical functions necessary to conduct a classically defined usability test:

  1. Audio and video recordings of testers (ala Think Out Loud method)
  2. Ability to edit recordings of tester sessions into highlight reels
  3. Capability to test using mobile devices (Smartphones and tablets)

The classical definition of usability testing is a systematic observation under controlled conditions to evaluate how well testers can complete critical tasks.

Thus, having the ability to record testers, to see their interactions and hear their thoughts (ala the Think Out Loud method) is critical to the observation process. It is also important to be able to edit those recordings, so that key insights can be shared with others in the form of highlight reels. Finally, the increasing adoption and usage of mobile devices to interact with websites and applications makes it necessary to conduct usability tests of the mobile user experience. Thus, tools should be able to test the mobile as well as PC based experience.

Not included in this review are Feedback or Click Tracking tools. This is because they only provide feedback or opinions about user interfaces, and lack the ability to observe testers conducting tasks. I will conduct the same comprehensive review of Feedback and Click Tracking tools in a future article.

Usability testing tools matrix from UsefulUsability.com

Four Quadrants of the Usability Testing Tools Matrix

Usability testing tools are divided into four quadrants on the matrix, based on where they fall when evaluated against two critical attributes:

  • Depth of Technology: What functionality and features are available with the tool, and how deep does the array of features run? Tools that score higher going up the vertical Depth of Technology axis have more functions and features.
  • Tool Impact: This is a measure of the ability of the tool to impact overall testing, and thus the observations and eventual recommendations for usability improvements. Tools that score higher (going to the right) on the Tool Impact axis have the ability to have a greater impact on overall testing results.

Based on these definitions, tools can fall into one of four quadrants on the matrix: Authority, Contender, Innovator and Niche.

  • Authority tools are those that provide a greater amount of usability testing functionality as well as a higher potential impact on discovering, diagnosing and ultimately optimizing the user experience.
  • Contender tools have the ability to provide a greater impact in terms of their depth of technology, but may have more limited potential impact in discovering, diagnosing and optimizing the user experience
  • Innovator tools have a more limited depth of technology, but have the potential to have a greater impact in usability testing and recommendations
  • Niche tools have a relatively limited depth of technology as well as a lower potential impact on usability testing and recommendations results, but can still be very useful for the specific tasks they were created for

All of these tools have specific features that make them unique.  Depending on your testing needs, any of these tools may be good choices and can compliment your toolkit.

Mobile Usability Testing Still Rudimentary

The upper right corner (Authority) of the matrix is blank for a very good reason. It represents the current lack of a fully integrated mobile usability testing solution. In my opinion we still lack a comprehensive, fully integrated mobile device usability testing tool, something that records both mobile device screen interaction and the audio/video of testers as they try to accomplish tasks. Because they are mobile devices, testing in this manner should be possible anywhere the tester goes, at any time, and using any smartphone or tablet (as long as it has a camera). Real time streaming of the screen and audio/video of the tester should be available for remote teams.

Mobile usability testing is currently in a neophyte stage. Yes, there are cludgy and cumbersome ways to test usability with a mobile device, typically requiring testers to place their phone on a desk with a camera hovering over it (thus not being at all ‘mobile’). Most methods today lack the sophistication of being a truly integrated and mobile experience.

Consider how just a few years ago new tools like Camtasia, Morae, UserTesting and others made usability testing websites an anytime, anywhere, quick and cost effective solution versus using expensive and cludgy labs or home grown equipment. We need that same fully integrated experience for mobile, but we do not have it, yet.

Authority Usability Testing Tools:

Usability testing tools matrix from UsefulUsability.com

Ovo Logger

Ovo Logger image from UsefulUsability.comOvo Logger by Ovo Studios can be considered to be the Rolls Royce of usability testing solutions. This is the tool for large Fortune 1000 enterprises or usability testing studios that conduct a high volume of testing and have teams spread around the globe. Typically these situations call for the need to have a robust tool to capture, edit and share results. It should be noted that Ovo Studios creates custom usability testing labs, and the Ovo Logger fits in well with those labs. The tool scores at the top of comparable tools for depth of technology, but is slightly limited in terms of impact due to requiring a dedicated PC, and being somewhat expensive to deploy.

Pros: Enterprise level usability testing software and hardware that records multiple inputs from PCs or Macs (cameras, microphones, screen interaction) and streams the output to observers. Editing tools enable real-time analysis of tests with the ability of observers to document important points. Can capture un-tethered iPads, non-jail broken iPhones and Android devices if using the Logger mobile app.

Cons: Designed specifically for larger firms with multiple locations and or usability teams. Requires a dedicated PC to run the software. Pricing is more expensive versus other usability testing tools and is based on number of sources, associated lab requirements, etc.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $3,000 and up depending on scope of associated usability lab

Camtasia plus a Screen Sharing Tool

Camtasia image from UsefulUsability.comA very powerful Do It Yourself approach for remote usability testing is to use a screen capture tool such as Camtasia (from TechSmith) coupled with a screen sharing tool such as GoToMeeting, WebEx or similar applications. The major benefit of this approach is the ability to conduct moderated remote tests, because the practitioner has a direct connection to the participant. In addition, the recording of the screen interaction with the voice and or video of the tester is immediately available for editing on the practitioners computer. The downside to this approach is lack of a simple mobile solution.

Pros: Enables remote moderated usability testing, because the practitioner has a direct connection to the participant. Sessions can be shared with other remote observers using the screen sharing tool. Audio and video recordings of the screen interaction with the voice of the tester are immediately available for editing on the practitioners computer.

Cons: For part-time or ad-hoc uses, this approach may be more expensive than using some of the other remote unmoderated tools available from vendors. Screen sharing tools are notorious for occasionally having technical problems that cause poor connection performance, or worst case dropping the connection entirely. This is not an optimal solution for conducting mobile usability testing, although cludgy methods of using a webcam pointed at the testers mobile device can be tried.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $299 for Camtasia, $19/month for GoToMeeting or $24/month for Webex


UserTesting image from UsefulUsability.comUserTesting provides a network of pre-screened testers who can meet several specified demographics. Or, practitioners can invite their own testers using the tool. Practitioners create a set of tasks for the testers to complete, either on your website or even competitor sites. Testers screen interaction and voices are recorded. UserTesting includes powerful and easy to use video editing tools to create highlight reels. UserTesting also provides mobile usability testing, which is a major feature considering the increasing utilization of mobile devices. Written responses to questions are included. UserTesting is one of the most robust tools to use for remote unmoderated usability testing and thus scores higher  for depth of technology and testing impact on the matrix.

Pros: Relatively low cost way to conduct remote unmoderated usability testing. Audio and video of the tester are recorded, and editing tools enable highlight reels to be created easily and quickly. Mobile device usability testing is available. The large network of testers typically means testing session results are available in an hour or two. Demographic filtering tools enable finding the appropriate testers for most Persona situations.

Cons: Because UserTesting pays small fees to the testers for each completed test, it is critical to carefully create non-biased tasks, else you risk the tester trying to cut corners to accomplish the test as quickly as possible. As a practitioner, you must watch for this kind of activity and request a re-do of that test if you are concerned about the quality of the tester. As an alternative, using your own testers can alleviate much of this. Mobile testing is available, but requires the tester to be at a desk so a webcam can be pointed down to the screen, which is not typically how most people use their mobile devices.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $49 per test with bulk pricing discounts available


Morae image from UsefulUsability.comMorae from Techsmith can be considered the Granddaddy of usability testing tools. This is because it was the first tool specifically created for usability testing. Even today it is still one of the best. It provides a comprehensive set of features available to record, analyze, edit and share usability testing sessions. Morae is an application that resides on the practitioners computer (PC only, sorry Macs). Morae records all screen interaction, including the testers voice and or face if using a webcam. Testers can be on any Mac or PC if testing websites or applications. Morae includes very powerful documentation, editing and sharing tools. Morae scores fairly high for depth of technology, and also fairly high for testing impact, the limiters being lack of a Mac practitioner solution and the issue of cludgy mobile testing.

Pros: Powerful screen capture and editing tools make this the choice for larger scale usability testing. Morae includes additional tools that can track a variety of screen and click interactions, plus has the ability to incorporate survey/opinion based data. Sharing test sessions with other practitioners and testing results with teams is relatively easy and quick.

Cons: Desktop & Laptop based system with no ability to record Mobile device screen interaction directly. Relatively expensive and PC-only (not Mac compatible) for the Manager and Recorder tools.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $1,995

Contender Usability Testing Tools:

Usability testing tools matrix from UsefulUsability.com


Userlytics image from UsefulUsability.comUserlytics provides both PC and Mobile based remote unmoderated usability testing. For mobile testing, the more expensive Gold or Enterprise plans are required. Testers can come from the Userlytics panel, or from testers you provide. Screen interaction, tester voices and webcam video images of testers can be recorded. Written answers to post test questions are also included.  The depth of technology is good, but because of the requirement to use the more expensive Gold plans for downloading and editing of videos, capturing of sessions longer than 15 minutes and testing mobile, this tools has somewhat limited testing impact versus some competitors. Practitioners that conduct larger volumes of tests should consider Userlytics.

Pros: Userlytics is relatively low in cost, and is definitely easy to set up and use. The more expensive plans allow download of recordings for editing and sharing. In addition, images, videos and other assets can be uploaded to the Userlytics secure servers for early prototype testing. White label testing results sharing is possible for Gold and above plans.

Cons: Userlytics has the same issues as any other remote unmoderated usability testing tool, with the burden being on the test creator to ensure non-biased questions. Userlytics has a rather confusing set of test functionality based on plans (Basic, Silver, Gold, Enterprise). For example, tests recordings are less than 15 minutes for sub-Gold plans, and mobile testing is only available on Gold and above plans. Depending on the plan (Silver or above), each monthly subscription price includes one or more ‘free’ tests, with each additional test costing $33 or less per test. There are also limits in terms of number of post test questions that can be asked based on plans.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $39 per month or more with bulk discount pricing available


UserFeel image from UsefulUsability.comUserFeel.com provides an assortment of testers, many of whom speak languages other than English, for remote unmoderated usability testing. Having the ability to choose a language other than English (many are offered) is a great benefit for testers of websites that are targeted for other countries. Written responses to post test questions are also available. Videos are available of the test on the website (but not available for download for editing purposes), and a white label playback option of the video is free. The depth of technology is good, but due to the lack of mobile testing and inability to download test recordings the tool impact is lower than some competitors. For those conducting usability testing in languages other than English, this could be a good solution.

Pros: A good tool for non-English language remote unmoderated usability testing. Demographic and language filtering tools are available. Pricing is relatively low and post test questionnaire functionality is available.  The free white label playback of the test recordings is a plus.

Cons: Like other remote unmoderated usability tools, practitioners must be careful to create non-biased tasks or else risk having testers try to quickly finish the test, causing poor test results. The advanced video editing capabilities of some competitor tools are not available. Videos cannot be downloaded and edited. Mobile testing is not available.

Tester Audio and Video Recordings? Yes

Recordings can be edited? No

Mobile testing capable? No

Price: $39 per month with bulk discount pricing available

Innovator Usability Testing Tools


OpenHallway image from UsefulUsability.comOpenHallway is a remote unmoderated usability testing tool for PC or Mac-based research. Testers come from the practitioner (no panel is offered) via a link that is generated by OpenHallway when the scenario and tasks are set up. OpenHallway requires no code to use, which makes it a great tool for testing applications, websites and even competitor websites. Screen interaction is recorded, along with voice and or the testers face, if the tester has a webcam. Plans (Basic, Plus and Premium) determine the amount of video storage available. Unlimited number of test sessions are available, however the video storage amount will act as the limiter to how many sessions are actually conducted.  Because of video storage limits, and videos being MP4 only, and no mobile solution, OpenHallway has somewhat more limited depth of technology than some competitors, however the tool impact is good in that it can be used on almost any website with no code required.

Pros: OpenHallway is simple to use, easy to set up and therefore is a handy tool for recording unmoderated remote usability tests. As of the writing of this article there is a free trial.

Cons: OpenHallway is PC or Mac compatible, but does not support mobile.  Although no code is required, Java must be present on the tester computer for the tool to work. Videos are MP4 only, and only the Premium plan ($199 per month) allows downloading (and thus editing) of the video.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $49 per month or higher, depending on the plan


UsabilityTools image from UsefulUsability.comUsability provides a variety of usability, feedback and click tracking tools, with the primary advantage being they all use a single practitioner interface. This makes creating tests that require multiple tools much easier. There are two versions, a UX Suite and Conversion Suite. The UX Suite contains click testing, web testing, survey, card sorting and a Persona Creator. Practitioners can set up web tests that provide scenarios to website visitors, who then try to accomplish the tasks and click success or abandon buttons based on whether they thought they could accomplish the task. Results are provided in data and charts. No code is required unless you choose to utilize their website intercept request on your site. The lack of tester audio and video recordings for PC and mobile is a limiter for depth of technology.  However, having the ability to utilize the tools efficiently and quickly provides a good tool impact.

Pros: A toolkit of the most common tools needed to fully understand the user experience. Having all tools in a single practitioner interface streamlines the process of creating and running multiple tool tests. In larger quantity testing scenarios, this set of tools can be more economical than using separate usability tools.

Cons: UsabilityTools does not provide audio and or video recordings of tester sessions. Mobile interaction using audio and video recordings is also not available. Pricing is not available on their website, although as of Sept. 2013 they had one time fees ($300) and monthly fees starting at $69 for the Professional version. Free panels of testers are not offered, although paid sources are available.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? No

Price: Pricing information is not available on their website.


Loop11 image from UsefulUsability.comLoop11 provides the ability to conduct unmoderated remote usability testing on PCs, Macs or mobile devices. The tool is easy to set up and fairly easy to administer (no code required). A key advantage is it can provide real-time data. Loop11 provides useful information for evaluating task flow on websites, including competitor websites. No audio and video recordings of unique test sessions are available, and because of this the depth of technology, and tool impact are lower than some competitors.

Pros: Loop11 requires no code, thus is useful for conducting comparison tests of user experience on competitor or other websites. Setting up and running tests is relatively quick and easy. Real time information can be advantageous for situations where quick results are needed.

Cons: As with other unmoderated tools Loop11 does not allow interaction with the testers to probe or ask follow-up questions. The test creator has to find his or her own test participants. No ability to record the tester via ‘think out loud’ methodology, something that exists with competitor tools. Somewhat expensive versus competitors.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? Yes

Price: $350 per project or $158 per month


UserZoom image from UsefulUsability.comUserZoom includes multiple tools including remote unmoderated testing on PCs and mobile devices (via UZ Mobile), along with other tools such as card sorts, tree testing, screenshot click testing and more. For the remote unmoderated usability testing tool UserZoom enables tasks to be created and deployed to either 3rd party panels or the practitioner’s own panel. Intercepts are also available, a handy feature when trying to reach actual visitors to a website. UserZoom does include video recordings of screen interaction, but only at the pricey $9,000 per year or higher plans, and no voice capture of testers are available, thus Think Out Loud recordings of users explaining their actions are not captured. This limits the depth of technology and tool impact when compared to some competitors.

Pros: UserZoom includes powerful reporting and analytics tools that can greatly help researchers synthesize and analyze post test data, including heatmaps, time on task reports and much more. Plans allow practitioners to pick the best price point and functionality for their needs. Plans include a free Pilot, UZ Proof of Concept at $1,000, Basic plan at $9,000, Pro plan at $29,000 and an Enterprise plan that requires a custom quote. As of the writing of this article UserZoom offers a free pilot study.

Cons: UserZoom does not enable voice and or video capture of the tester as they try to accomplish a task, a major differentiator compared to competitor services. UserZoom does not have its own panel of testers available for free, as other competitors do. Practitioners can view and edit recordings of testers while they were conducting tasks to make highlight reels, but there is no audio nor video capture of the actual tester. UserZoom is rather expensive versus its competitors, and lacking the ability to hear and see testers via session recordings while they were conducting the task is another major differentiator between UserZoom and other remote unmoderated usability testing tools.

Tester Audio and Video Recordings? No

Recordings can be edited? Yes, but only recordings of screen interaction (no tester voice is available)

Mobile testing capable? Yes

Price: $9,000 for the Basic plan and up

Niche Usability Testing Tools

Usability testing tools matrix from UsefulUsability.com


TryMyUI image from UsefulUsability.comTrymyUI.com is another of the many SaaS-based remote unmoderated usability testing tools. Testers create tasks, and videos are provided that include screen interaction and the testers voice. Written answers to post questions are also included. As of the writing of this article a free trial is included. Practitioners can use their own testers, but at a license cost of $99 for a single use test. Because video editing and mobile testing are not provided, the depth of technology and testing impact are lower than some competitors.

Pros: TryMyUI provides an alternative to other remote unmoderated tools that is slightly cheaper than many competitors. As a tool that provides video and audio recording of testers as they try to complete tasks it fits the classical definition of a usability testing tool.

Cons: All the same issues with remote unmoderated usability testing apply, with the burden being on the practitioner to create non-biased questions, and to carefully analyze the results to ensure the tester was not trying to use short cuts to complete the test. Video editing is not available, nor is mobile testing.

Tester Audio and Video Recordings? Yes

Recordings can be edited? No

Mobile testing capable? No

Price: $35 per test with bulk discount pricing available


WebEffective image from UsefulUsability.comWebEffective by Keynote Systems provides remote, unmoderated usability testing via tasks that are assigned and recorded. A panel is available, or practitioners can use their own testers. Data including click stream information is captured and displayed in a variety of reports, which can be downloaded. Individual sessions showing screen interaction are captured, however there is no audio nor video commentary captured of the tester as they go through the tasks.  Mobile testing is available. The lack of audio and video of testers as they try to complete tasks means WebEffective has a lower depth of technology and testing impact than some of its competitors.

Pros: A powerful tool to capture user interaction on websites or mobile devices, with a wide variety of reports available.

Cons: WebEffective does not provide audio and or video recordings of tester sessions. In this regard, WebEffective operates more as a click stream tool than a true usability testing tool.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? Yes

Price: Pricing is not available on the Keynote WebEffective website

Ovo Solo

OvoSolo image by UsefulUsability.comThe Ovo Solo tool by Ovo Studios provides a small business version of Ovo Logger that enables screen interaction video capture along with the testers face and voice.  Simple logging controls enable analysis of the resulting test videos. However the tool is PC only. The relatively low price point makes this a good candidate for practitioners on a tight budget. Because of the PC only nature of the tool, the basic editing functions and the inability to record mobile sessions, this tool has lower depth of technology and testing impact than some of its competitors.

Pros: The relatively low price point makes this a very affordable way to capture, edit and display usability testing sessions.

Cons: This tool is PC only and is not able to record screen interaction on mobile devices. Editing tools are basic and may not meet more advanced usability tester needs.

Tester Audio and Video Recordings? Yes (but tool is PC only)

Recordings can be edited? Yes

Mobile testing capable? No

Price: $99


Silverback image by UsefulUsability.comFor Mac users, Silverback from Clearleft enables remote and in-person moderated usability testing. Silverback is not available for PC based practitioners. Session screen interaction and audio and or video of the tester are recorded, and simple editing features like task tagging are available. Recordings are saved in QuickTime, which makes editing of highlight videos relatively easy for Mac users. The lack of PC functionality and video editing capability means this tool has lower depth of technology and testing impact than some other competitors. Mac practitioners will find Silverback a very easy to use tool at a very affordable price.

Pros: A relatively low cost and easy to use tool for Mac users that provides the basics in terms of classical usability testing functionality.

Cons: Mac desktop and laptop only, does not work with PCs. No ability to record mobile device screen interaction directly. As of the writing of this article currently does not support FaceTime camera with newer Macbooks, but the app is being re-written to fix this. Lacks powerful editing tools available in other applications.

Tester Audio and Video Recordings? Yes (but tool is Mac only)

Recordings can be edited? No

Mobile testing capable? Yes

Price: $69.95

14 Usability Testing Tools Conclusion

Usability testing tools matrix from UsefulUsability.com

Any of the 14 usability testing tools reviewed here can provide excellent results if applied in a manner which best fits their unique functionality and features.

If Your Favorite Tool is Missing

Not seeing your tool here? It could be because I’m classifying it as a Feedback or Click Tracking analytics type of tool. My original 24 usability testing tools article included many of these tools, but also included some tools that can better be defined as Feedback or Click Tracking analytics tools. In future articles I will provide a matrix and reviews separately for those types of tools. And if your tool is not a feedback or click tracking tool, let me know by leaving a comment.

Note: Companies mentioned in this article may or may not be advertisers on this site. However, in no way does their sponsorship or lack thereof impact the results of this or any other editorial content on the site. In all cases, the same rubric for evaluation is used to compare tools or services and all results reflect the outcome of the comparison without regard to whether a company is advertising on the site or not.