UX and Usability Testing Analysis


UX and Usability Testing Analysis for Website Optimization

ux-and-usability-testing-analysis-usefulusabilityLearn how to use UX research and usability testing analysis for website optimization. UX and usability testing analysis are one of the big 4 UX Steps to website optimization. Learn how UX usability testing analysis and qualitative data helps improve websites.

UX research and usability testing analysis provide qualitative WHY data for website interaction. By coupling this with the WHAT IS HAPPENING quantitative Behavioral UX data analysis discussed in the prior article, we are able to make far more informed optimization recommendations for websites.  This article will provide an overview of; what UX research and usability testing analysis is, the types of data available, how to analyze this data and examples of how it is used.

UX research and usability testing analysis is Step 3 in the Four Big UX Optimization Steps, which are:

UX and Usability Testing Analysis Goal:

The goal for UX research and usability testing analysis is to obtain qualitative WHY data for the already observed WHAT data coming from the Behavioral UX Data analysis. Our method for conducting UX research and usability testing is to gather qualitative data by observing actual people (who match your Personas) as they conduct critical tasks on your website.

Questions we want to answer by conducting this qualitative analysis include:

  • What is working for website visitors?
  • What is not working for them?
  • What confuses them or causes them concerns?
  • Are their expectations met for the experience? Why or why not?

We will use this data to evaluate and identify patterns of task-flow errors. In addition, the qualitative component can help in identifying the potential WHY for any other task-flow issues that may be uncovered.

Types of UX research and usability testing methods:

There are many UX research and usability testing methods we can use to capture the qualitative data. They include:

Moderated Usability Test – Richest source of qualitative data, but resource intensive

Unmoderated Usability Test – Good source of qualitative data, less resource intensive

5 Second Test – Provides data for how well a page communicates with visitors

Click here to view latest job postings

Click Test – Identifies navigation and taxonomy issues

Card Sort – Excellent for evaluating information architecture questions

Question Test – Useful for identifying task-flow and page communication problems

Preference Test – Can add the WHY to preferences among several choices

Eye Tracking – Identifies good and bad elements that may be helping or hurting attention

Etc. Etc. Etc. – There are other types of UX research and usability testing qualitative methods and tools available. Which ones you choose depends on a number of factors including; the type of analysis needed, the type of Personas being tested, the type of website (B2B, B2C, eCommerce, etc.) and the detail needed for analysis.

Examples of UX Research and Usability Testing:

Moderated Usability Testing

moderated-usability-testing-usefulusabilityModerated in-person usability testing provides the richest and most detailed form of usability data.

However, it also requires the most significant investment in resources and time.

Recruiting, scheduling and compensating participants, arranging for the test facility or remote live testing, conducting the tests and evaluating results can take a relatively larger amount of time, effort and cost versus other methods such as unmoderated testing.

If conducted properly, moderated in-person usability testing delivers the best and most detailed WHY qualitative data. This WHY data can be a very helpful source for answering the questions raised by the quantitative data analysis. This is because moderated in-person testing allows probing of the participant by the moderator using the think-aloud or retrospective methods of testing.

Now don’t get me wrong. The kind of probing we’re talking about here is not the kind of probing that aliens do on unsuspecting earthlings.

Rather, this probing is the good kind of probing where the moderator asks the participant what they are thinking when the moderator observes non-verbal clues.

What sort of non-verbal clues?

Clues of behavioral nuances that may indicate the participant is having difficulty (ie, cognitive load), such as;

  • Raised eyebrows
  • Big sighs
  • Perplexed looks
  • Sitting back
  • Staring at the screen
  • And many other non-verbal clues

There are many tools for moderated usability testing. For more information please read my 14 Usability Testing Tools article.

Unmoderated Usability Testing

As with moderated usability tests, unmoderated usability testing is a powerful way to capture the all-important WHY qualitative data. The key difference is that in unmoderated usability testing the moderator is not present with the participant when he or she conducts the test.

Unmoderated testing can thus be conducted much faster than moderated testing, requiring lower effort, time and cost. It can also be conducted at scale, and because it can be remote, unmoderated usability testing can be conducted anywhere, anytime, and with any participants who have the technology to conduct the test.

So how then do you obtain that WHY information if you’re not there as a moderator?

The answer lies in how carefully the unmoderated test is set-up. By focusing on specific task-based questions and reminding the participant to think aloud as they conduct the task the WHY information can be captured. Recording the session allows the UX researcher to be able to observe the session and analyze it for the WHY of the performance issues.

If set up correctly, unmoderated can be almost as effective as moderated testing. Almost.

There are a variety of vendors such as UserTesting, TryMyUI, UserZoom, Userlytics and others that provide good unmoderated usability testing tools, you can learn more about those in my 14 usability testing tools article.

Five Second Test

Five Second Tests are extremely helpful for evaluating how well a website communicates three critical pieces of information to visitors within 5 seconds, else risking the visitor leaving (often permanently).

These three critical elements are:

  • Who are you?
  • What do you offer?
  • Why should they care (what’s in it for the visitor?)

Why five seconds?

Because according to research conducted by Lindgaard et. al.* and other research studies, most visitors actually make their minds up about a website’s quality within only 50 MILLISECONDS.


And as demonstrated in our earlier discussion of quantitative data, the Session Duration report for websites will typically show that the vast majority of visits are actually LESS THAN 10 SECONDS.


So five seconds has become something of an industry standard for evaluating how well a website communicates those all important pieces of information to visitors. For the details on how it works you can read my Five Second Test article.

But in general, here’s how to conduct a five second test. Testers view an image for 5 seconds, the image is then removed and they are asked several questions.

As with unmoderated testing, the devil is in the details in setting up an un-biased test. It is VERY important that questions are carefully worded to reduce bias, and are focused on answering the three critical communication elements of; who you are, what you do, and why they should care (how it helps them).

5 second test results usefulusabilityIn the 5 second test results example above, 63 percent of the participants were unable to determine what the product or service was of this website. That’s not good because this was a test of the website’s home page!


The good news is a five second test is a very powerful tool for identifying not only WHAT is not working, but WHY it is not working. That’s because you can include probing questions at the end of the test to help uncover the WHY qualitative data.

Eye Tracking

Eye tracking is effective for obtaining data on where people look, and sometimes just as importantly where people do NOT look. Eye tracking studies use technology to record where a participant is looking on the website. This is done by tracking the fixations (the series of short stops our eyes automatically do) when a person is reading or scanning a website.

We use this data to help evaluate what elements may or may not be attracting several types of attention:

  • “Good” attention points
  • “Neutral” attention points
  • “Bad” attention points

Graphics and visual design on a website can be powerful tools for capturing a visitor’s attention. This is extremely helpful when we want our visitors to know important information, or direct their attention to a Call To Action.

However, graphics and visual design can also accidently pull a visitor’s attention AWAY from the very information or Calls To Action we were hoping our visitors would see.

eye-tracking-results-usefulusabilityAs the example heat map of eye tracking data results above demonstrates, areas where there are more fixations are colored green to yellow to red to indicate higher amounts of fixations. Areas of no fixations have no color.

This data can help inform us about what elements on a page are attracting or not attracting attention. It can also inform us about objects that should NOT be attracting attention, but are. These can potentially be labeled as ‘bad’ attention sources and may be harming the user experience by pulling visitors away from focusing on the elements we intended.

So in the above example, the hot spot areas of the search bar, shopping cart and product images are all good attention areas. But perhaps some of the text in the middle, which is not clickable, may be pulling attention away from the all important products and could be considered neutral or potentially bad attention areas.

Other UX and Usability Testing Analysis Tools:

There are plenty of other UX and Usability testing analysis tools that we can use to help us obtain the WHY qualitative data. But hopefully the examples above give you a sense of what qualitative data is possible to capture and how that data is used.

So what’s next?

Once you’ve obtained your behavioral UX data analysis, and your UX and Usability testing data analysis, it is time to combine them for analysis and recommendations.

Combine Behavioral Data Analysis with UX and Usability Testing Analysis

In the next article I will explain how to combine the quantitative behavioral UX data with the qualitative UX and usability testing data to provide a ‘360 degree view’ of what’s happening on your website, and why it’s happening. Using that comprehensive view enables us to make far more informed analysis and recommendations on how to optimize the site.

Conclusion: UX and Usability Testing Analysis

In conclusion, UX research and usability testing analysis is Step 3 in the Four Big UX Optimization Steps, which are:

We use a variety of UX and usability testing tools to gather qualitative WHY information about website activity. These tools can include; moderated usability testing, unmoderated usability testing, five second tests, eye tracking, click tests, card sorts, question tests and many more.

The goal of using these tests is to obtain the all-important WHY qualitative data coming from your website activity. By analyzing this data, we will have a much better picture of what’s happening on your website and why it’s happening.

By combining the quantitative WHAT behavioral data we captured earlier with this qualitative WHY data we get a 360 degree view of what’s happening on the website and why it is happening. This gives us a much better set of data with which to make website optimization recommendations.


14 Usability Testing Tools

Remote Moderated Usability Testing Tools

Five Second Test

Four Big UX Optimization Steps

Define Personas

Behavioral UX Data Analysis

*Lindgaard, G., Fernandes, G., Dudek, C., and Brown, J. “Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!” Behaviour & Information Technology 25, 2 (2006), 115–126.

Photo courtesy Kevin Case via Flickr and a Creative Commons License

Comments are closed.