Tags Posts tagged with "usability testing"

usability testing

1 592

Why A/B Testing Needs Usability Testing, aka Why Opposites Attract

Why A/B Testing Needs Usability Testing by UsefulUsabilityWhy does A/B testing need usability testing? Because opposites attract, and the benefits received by coupling A/B and Usability testing can go far in improving the UX of a website or app. In fact, it’s better than any other method. Here’s why.

Way back in the dark ages of 2009 I wrote a three part series on A/B Alpha/Beta and Usability testing, asking the question of which one is better.

To summarize the pros and cons of A/B testing and usability testing from the three part series:

A/B Testing positives:

  • Fast (testing can be set up and run in a day)
  • Tests reality, not theory
  • Quantifiable with statistical significance
  • Accurate (winner almost always performs at that level)


A/B Testing negatives:

  • Can hurt results (B version failure can cause reduced conversion)
  • Missing critical “why” data (no data to explain why a version won)
  • Not predictive (data not available to determine what to test nex)
  • Needs traffic (doesn’t work on prototypes or apps with no users)


Usability Testing positives:

  • Doesn’t hurt results (testing does not impact conversion)
  • Provides the “why” data (qualitative data answers why users do what they do)
  • Predictive (results can determine what next to test)
  • Doesn’t need traffic or even a live website or app (testing available from concept through production stages)


Usability Testing negatives:

  • Trained professionals required for unbiased results
  • Won’t reveal all issues
  • Results can vary
  • On-going testing is difficult

How A/B and Usability Testing Compliment Each Other

A/B and usability testing complement each other in several ways.

First. Did you happen to notice how the four negatives for A/B testing are actually the four positives for usability testing? That’s right, the issues inherent in A/B testing are actually strengths of usability testing, and vice versa.

Chart of A/B Testing Cons and Usability Testing Pros:

A/B Testing Cons Usability Testing Pros
  • Can hurt results
  • Doesn’t hurt results
  • Missing critical “why” data
  • Provides the “why” data
  • Not predictive
  • Predictive
  • Needs traffic
  • Doesn’t need traffic

Second. A/B testing provides the quantitative side of UX data, and usability testing provides the qualitative side. The quantitative ‘what’ of user experience choices your website and app visitors make is augmented with the ‘why’ for those choices, which comes from qualitative usability testing data.

Third. A/B Testing coupled with usability testing provides an end-to-end view into the user experience of your website or app. By coupling the data, you have a much clearer picture of the engagement happening in your site or app. You also will have the data to know how to impact and improve on that engagement. Finally, you’ll have the data to evaluate your results and move forward with your next set of testing and optimization.

Why A/B Testing Needs Usability Testing

The benefits of combining A/B testing with usability testing are many and include:

  • Complete quantitative and qualitative data for more informed decision making
  • Comprehensive view into the user experience of your app or website
  • Ability to use data to predict what additional tests can increase engagement
  • A 360 degree view of data to use for continuously optimizing your app or website

By combining A/B testing with usability testing, you’ll find your testing and optimization efforts produce far better results, which is a win for you, your firm and your app or website users.

A/B Testing and Usability Testing Resources:

Three Part Series: Usability, A/B or Alpha/Beta Testing, What’s Better?

Part 1 – Usability, A/B or Alpha/Beta Testing, What’s Better? Part I A/B Testing

Part 2 –  Usability, A/B or Alpha/Beta Testing, What’s Better? Part 2 Alpha/Beta Testing

Part 3 – Usability, A/B or Alpha/Beta Testing, What’s Better? Part 3 Usability Testing

Usability.gov Usability Testing Section

WikiPedia Usability Testing

14 Usability Testing Tools


0 2217

Remote Moderated Usability Tools Description, Benefits and Listings

Remote-Moderated-Usability-Tools-UsefulUsabilityRemote moderated usability tools offer a better way to identify user experience task flow issues. Learn the description, benefits, method and tools for conducting remote moderated usability testing.

Remote moderated usability testing tools are powerful, they are almost equal to the ultimate best-case usability testing method, which is in-person moderated usability testing.

There are several online remote moderated tools that enable remote moderated testing. These include some older tried-and-true do-it-yourself tools, along with several all-in-one services that provide additional functionality and features.

I’ll briefly touch on how remote moderated usability testing works, the benefits of remote moderated testing, and then focus on several remote moderated usability tools that enable you to use this powerful technique to optimize your website or app.

How Remote Moderated Usability Tools Work

Remote moderated usability testing tools work by enabling a moderator (researcher) to be able to connect through real-time two-way communications with a test subject who is participating in the test. Because it is a two-way connection, the moderator can follow-up or probe the tester as the tester goes through the usability test.

Why is follow-up or probing during testing so important?

Because many times clues that testers are having difficulty completing tasks are non-verbal. Things like lifted eyebrows, head movements, stopping to pause or other non-verbal clues let a moderator know the user is having difficulty. Being able to follow-up or probe is critical to enabling the moderator to more deeply understand exactly what that usability issue is.

In addition, being able to follow-up and probe during testing can elicit a wealth of information from the tester, typically around what they were expecting to happen or find, versus what they actually had happen or found. This is crucial information for understanding the WHY of usability task flow errors. And without the WHY, it is difficult to determine what to do to improve the usability of that task.

Following-up and probing during testing ensures that the tester’s memory is fresh, enabling the tester to fully share their expectations, thoughts and feelings as they are being tested.

Remote unmoderated usability testing tools are very popular today, but they simply do not allow this essential data to be captured during testing. That is why remote or in-person moderated testing is better, it enables the moderator to capture all the information happening during testing.

Remote Moderated Usability Testing Method

Remote moderated usability testing methods are centered on creating a two-way communication between the moderator and the test participant. In addition, most methods use a recording and screen-capture tool to document both the tester’s screen interaction as well as the verbal “think aloud” information the tester is sharing during the test.

Benefits of Remote Moderated Usability Testing:

  1. Enables the moderator to follow-up and probe the tester while testing is occurring
  2. Adds critical WHY data to the usability test, by eliciting expectations versus reality information directly from the user
  3. Allows moderated testing of remote testers who may be in another city, State or even Country which expands the tester pool
  4. Saves large amounts of money and time versus in-person moderated testing
  5. Scaleable in enabling multiple tests in a minimal amount of time
  6. Can leverage website intercepts to enable testing of actual website visitors who match Personas

Remote Moderated Usability Tools:

There are two types of remote moderated usability testing tools:

  • Do-it-yourself tools
  • All-in-one remote moderated service
Do-It-Yourself Remote Moderated Tools

The do-it-yourself remote moderated usability testing tools are typically an assortment of three types of online tools:

  1. Two-way real-time online connection tool. These tools are widely available, think Skype, Webex, GoToMeeting, JoinMe etc. The purpose of this tool is connecting the moderator in real-time with the tester, enabling both audio and visual communication between the moderator and tester. Some of these tools enable still others to join in to watch the session, which is useful for allowing design team members to also observe the test.
  2. Screen capture and recording tool. These tools are also widely available, think Camtasia, SnagIt, Adobe Captivate and a host of others. The purpose of this tool is to record the session including the audio and the video of the tester’s screen as he or she conducts the test, along with the voice of the moderator. Although it’s not mandatory to have a recording of the session, most researchers like to do so to go back and analyze specific times in the test when usability issues were uncovered. It is also helpful to have these sessions recorded for sharing the tests with others (such as design team members) after the test is complete.
  3. Video editing tool. These are also widely available tools, consider everything from Camtasia, Movie Maker, Adobe Premiere and more to Mac based tools like iMovie. The purpose of this tool is to edit the testing sessions into highlight reels that specifically identify common usability problems. This enables the moderator to be able to concisely point out what’s causing the issue without the observers having to sit through watching all of the usability tests from beginning to end.

There are other tools that make life easier for a usability practitioner, including website intercept tools like Ethnio or survey pop-ups to find actual website visitors to use as test participants.

A mixture of the three types of tools listed above enables the researcher to be able to conduct their own do-it-yourself remote moderated usability testing in a cost-effective and scalable manner.

All-In-One Remote Moderated Usability Services

There are several all-in-one remote moderated usability services now available for the usability practitioner that eliminate the hassle of having to have all the tools above, while also offering valuable additional features designed specifically for remote moderated testing.

List of remote moderated usability services include:

  1. Validately – A relative newcomer with an easy to use interface that makes remote moderated usability testing a snap. Includes an intercept and screener with unlimited recording. Includes a scheduler to make scheduling sessions easy, as well as a built-in payment system for incentives to testers after their tests are complete. Highlight reel creation is not currently available, but is coming soon. Recordings cannot be downloaded. Finally, Validately is a very affordable solution when compared to the competition.
  2. UserTesting Pro Version – An all-in-one solution that includes the ability to have moderated remote usability testing with either the UserTesting.com panel or your own participants.

I spoke with Steven Cohn, the CEO and a co-founder of Validately, along with Jessica Ogilvie, the CMO. Here’s what they have to say about Validately and why they chose to focus on a remote moderated usability testing service:

“We created Validately because we couldn’t find a remote moderated testing service that we really liked for developing and refining our own prototypes. With Validately, we simplify every aspect of remote moderated testing; finding testers who match Personas, scheduling sessions, paying incentives, storing and sharing large testing video files and lots more. And we worked very hard to make it all happen with NO downloads or plugins, which simplifies the user experience for the moderator AND the tester.”

To summarize the benefits of using a remote moderated usability testing service versus do-it-yourself: A service saves you from having to juggle multiple software tools yourself, it also provides the ability to utilize large panels (or better yet your own website visitors via their built-in intercepts), the ability to handle payment of incentives, scheduling sessions, and scaleability in the event you need to conduct continuous testing or testing at larger volume.

Conclusion: Remote Moderated Usability Tools

Remote moderated usability tools are almost equal to the best method of usability testing, which is in-person moderated testing.

The method for conducting remote moderated usability testing consists of connecting the moderator and the test participant in real-time using a two-way online communication tool, using a recording software tool to capture the audio and video of the tester session, and an editing tool to create highlight reels of the recorded tests to share the results with design teams.

The benefits of remote moderated usability testing include; enabling the moderator to follow-up or probe the tester as the test is occurring, the relatively low cost and time required for testing versus conducting in-person moderated tests, and the ability to intercept actual visitors of a site to use as testers.

Remote moderated usability testing tools come in two varieties, the do-it-yourself tools that are widely available, or an all-in-one remote moderated usability testing service that provides all the tools necessary plus additional functionality and features such as access to large panels.


For information on remote unmoderated usability testing tools read 14 Usability Testing Tools which includes comprehensive reviews of each usability tool and a comparison matrix.

6 868

UsefulUsability-Poll-What-Usability-Testing-Tool-Do-You-UseIt is time for a quick poll. This poll asks you to check any and all usability tools you are currently using. It is fine to check more than one tool.

And if your tool is not listed here, just check ‘Other’ and then in the comments please list your tool.

Which usability tools do you currently use?

View Results

Loading ... Loading ...

Remember, if your usability testing tool is not listed above just check ‘Other’ and enter your tool or tools in the comments.

5 Second Test:  An important conversion optimization tool

Five Second Test for Conversion Optimization image from UsefulUsability.comA 5 second test can help increase website conversion and improve online ROI, here’s how.

Five seconds may seem like a short time, but in fact it is more than enough time for a website visitor to determine if there is enough quality in your website to stay, or to leave potentially never to return. Using a 5 second test to optimize conversion is a powerful way to improve the ROI of a website. This is because in hundreds of website audits I have conducted over the past years I have found that a critical driver of website success is the ability of the home page, or any page for that matter, to deliver three pieces of critical information in five seconds or less:

  • Who are you?
  • What product or service do you provide?
  • Why should I care (what’s in it for ME)?

Websites that are able to quickly and efficiently communicate these three critical elements within 5 seconds typically have much better conversion, and thus ROI than websites that don’t.

Why Five Seconds

But why five seconds? The reason five seconds is so important is because of research studies which demonstrate that visitors to websites take a very short amount of time (in some cases a  fraction of a second, as little as 50 milliseconds) to judge the quality of a website.  As stated in an important study of timing of website visual quality judgments by Lindgaard et. al.…

“Our ambition was to determine how quickly people decide whether they like or dislike what they see, and whether such judgments may constitute a mere exposure effect. The above data suggest that a reliable decision can be made in 50 ms, which supports the contention that judgments of visual appeal could represent a mere exposure effect. The level of agreement between participants and between experiments was impressive and highly correlated even for the 50-ms condition.”*

In addition, hundreds of conversion optimization testing studies I have conducted over the years corroborate this, with test participants quickly scanning a page for just a few seconds before either staying, or moving on in their hunt for information.

5 Second Test Definition

I define a 5 second test for websites as…

“A five second test is a usability testing method in which the participant is exposed to an image of a webpage for five seconds. The image is then removed and the participant is asked questions about what they remember seeing on the page. The test is used for evaluating how well the page communicates the purpose and content within.”

5 Second Test Provides Quantitative and Qualitative Data

A major benefit of a 5 second test is the data that can be obtained, which is both quantitative and qualitative. Because the test is so fast and easy to distribute, it can be deployed to dozens, hundreds or even thousands of testers in a matter of hours, or at the most days. It is relatively quick and easy to obtain statistically significant results that can be organized into charts and graphs for analysis. This takes almost all of the guesswork out of validating if a page is communicating effectively or not, and makes it easy to develop useful conversion optimization recommendations.

As the example below shows, this kind of quantitative data is useful for analyzing exactly how well the webpage is working in terms of communicating with the intended audience. In this example we can see that the vast majority of participants had partial or no idea as to what service or product the company provides.

 5 Second Test results image from UsefulUsability.com

Homepage and the Five Second Test

In fact, one of the better uses of a 5 second test is to conduct tests of the home page. That is because the home page, of all the pages of a website, is most critical for communicating who the firm is, what products or services the firm offers and why the visitor should care (what’s in it for them). Conversion optimizations of the home page based on analysis of 5 second test results can greatly improve page flow, number of pages visited and bounce rate.

To demonstrate how effectively this test can work, examine one at a time for 5 seconds the two images below. Both are for websites that are providing a particular product. Can you derive in those 5 seconds:

  • What product or service is provided?
  • Who is the company?
  • What’s the benefit to you, the visitor?
5 Second Test page example image from UsefulUsability.com
What product or service does this firm provide?


5 Second Test example test image from UsefulUsability.com
What product or service does this firm provide?

Both sites are in the retail business, and both sell women’s shoes. Both appeared in search results for “women’s shoes.” But did one clearly communicate women’s shoes better than the other?

Most likely you found it easier to identify the product, company and benefit from the second website, which happens to be the Nordstrom shoe page. Even if you were not aware of the Brands prior to seeing the image, you probably would have found it easier to understand the product, company and benefit to you of the Nordstrom page. The first image is also a shoe retailer, but was that as obvious as the Nordstrom page?

Conversion optimization is most effective when quantitative data gathered from the 5 second test exposes issues with the page content, leading to optimization recommendations.

5 Second Test Methodology

To conduct a 5 second test, and use the results for conversion optimization, apply the following methodology…

  1. Evaluate which page you would like to test. Typically I like to start with the home page, as most often it receives the highest amount of traffic, has the highest bounce rate, and also is the most important page to communicate the three critical elements to the visitor. However, there are other opportunities to optimize page conversion with a 5 second test including; landing pages, category pages, product pages, information pages, customer service pages, contact us pages and more.
  2. Capture an image of the page. It’s typically best to control the test by providing an image of the page, versus sending the test participant to the page. This is primarily because if you are using the actual website, within the 5 seconds some testers may click away to other pages to help them identify the purpose of the site. Likewise, displaying an image instead of the actual webpage reduces the chances that the test will be flawed by slow load times or other technology glitches that cause the full website to be displayed less than the full 5 seconds. By only displaying an image, you the tester can control how long the participant views the page and curtail any desire to escape the page to find the missing information.
  3. Identify test participants. If the website has a specific target audience, say for example educators in Universities, then it can be helpful to find testers who match that Persona. Likewise, if the website is oriented more to the general public then you can find testers that match the general public at large. One word of caution here, ALL website home pages, whether targeting specific audiences or not, should be able to effectively and efficiently communicate the three core elements no matter WHO is viewing the page. This is why it is less essential to be overly focused on the exact Persona for those pages.
  4. Conduct the test. There are multiple ways to conduct the test. A low-tech way is to find someone in the hallway that matches your test participant profile, show them a print out of the page for 5 seconds, then remove the image and ask them your questions. There are online tools you can use as well, including the 5 Second Test website that I mentioned in the 24 Usability Testing Tools review. To record the results, a simple Excel spreadsheet can be used to document each participant’s comments, or if using the 5 Second Test tool a download of the results is available.
  5. Analyze the results. Depending on the question, I typically divide the results into three categories; incorrect answers, correct answers, and partially correct answers. For questions about what captured the most attention I typically list the elements that are important for communicating the purpose of the page including: Company logo, heading or explanation copy, any strong image. Interestingly, often the strong image is what captures the most attention, but is often guilty of not communicating what the product or service is. In all cases I total the results for each group of answers and provide that data, including a chart, in my report.
  6. Make recommendations for conversion optimization. The analysis of the results will quickly reveal where there are opportunities for conversion optimization. Typically these fall into several areas including; Brand name or logo too difficult to see, a strong non-product image or images capturing all the attention of the participant, the value proposition is unclear or totally missing. The recommendations for optimizations will become clear, based on what elements of the page are not working.

5 Second Test of Carousel Sliders

Many websites today use a carousel with a set of sliders that briefly display a set of horizontally sliding images at the top of the home page. Testing this type of page is important as the number of sliders, and their duration on the screen are impacting the communicative ability of that page. But how do you test animated sliders? Conversion optimization of multiple sliders may seem difficult, but in fact it is not if the following approach is used.

If the initial slider stays on the home page for 5 seconds or longer, then the problem is solved and that image can be used. If however the time is shorter between slider changes, then try categorizing all of the sliders into common groups, and use an example image from each category. For an eCommerce website as an example, there may be several sliders of a product that include the product image, and there may be several that are information only with no product image, or an image of something other than a product. I typically try to test each category of images. Ultimately you could test all the slider images, but that will necessitate having a much larger pool of testers to draw from.

Conclusion: 5 Second Test and Conversion Optimization

The 5 second test is a powerful tool to test the communicative ability of a webpage and provide data for conversion optimization. Remember that the test purpose is to evaluate how well the page communicates the following three critical elements;

  • Who is the company?
  • What product or service is provided?
  • What’s the benefit to your visitor?

A 5 second test can provide quantitative data as well as qualitative data, and because of the speed and efficiency can produce very quick results. Using a five second test on critical pages of a website is one of the best ways to identify opportunities to optimize the UX and thus improve conversion and ROI.

*Lindgaard, G., Fernandes, G., Dudek, C., and Brown, J. “Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression!” Behaviour & Information Technology 25, 2 (2006), 115–126.

How not to throw out 125 Million dollars when creating applications

No-Usability-Equals-125-Million-Dollars-In-The-Trash-from-UsefulUsabilityDo NOT throw out 125 million dollars like Avon did after their application was deemed unusable. Instead, learn from their mistakes and protect your investment (and job). Here is how.

Avon will throw out 125 Million Dollars because it has cancelled an SAP implementation that took four years to develop, as recently reported by the Wall Street Journal.

According to the article in the Wall Street Journal (Avon’s Failed SAP Implementation Reflects Rise of Usability). Avon did so after field testing in Canada revealed that users found the software unusable, and started leaving Avon in droves.

According to the article…

“As the WSJ’s Drew Fitzgerald reported earlier today, Avon is pulling the plug on a $125 million software system rollout that has been in the works for four years after a test of the system in Canada drove away representatives the door-to-door beauty product company relies on to drive sales.

Avon began testing the new order management software system in Canada in the second quarter. While the new system based on software supplied by SAP AG worked as planned, it was so burdensome and disruptive to the representatives’ daily routine that they left in meaningful numbers. Avon relies on a direct sales model where its representatives aren’t employees, which makes it difficult to add new tasks associated with the software system.”


If you are a CIO and you like your job I have a simple tip for you…

“Conduct usability testing often and extensively when developing new applications.”

If you are a CIO you can stop reading now. Thanks for stopping by!

But for the rest of us, let us examine why Avon ended up throwing out $125,000,000.00 and four long years of work by what was probably a large team. It is called usability, or user experience. Without an easy to use and satisfying user experience your application will NOT be used by your users and you will have wasted your money.

As the article in the Wall Street Journal states…

“At a time when people are accustomed to using well-designed applications from companies such as Google Inc. and Apple Inc. in their personal lives, they have little patience for workplace applications that leave them confused. Functionality is no longer the definition of success. Usability is key.”

Just to repeat a very important phrase


Here are five tips you can use to make sure YOU do not throw out one hundred and twenty five million dollars:

1. Identify Your Personas

A Persona is a fictional representation of your typical users, and includes both behavioral and goal oriented information that is critical for design decisions. In a previous article I wrote about tips you can use to create Personas and how to avoid bad ones, but suffice it to say it is critical that you have Personas BEFORE starting any application development project.

2. Design for Persona Critical Tasks

Personas include information on the top 3 or so critical tasks they MUST do to be successful. Your application should include bold statements that failure to make those tasks brain-dead simple means failure of the application. Never forget about those critical tasks, and make sure your application design is focused on continually seeking ways to make those tasks simple, fast and super-easy.

3. Conduct Early Prototype Usability Testing

Conducting usability testing early and often is not just a catch-phrase. Early prototype testing includes testing wireframes and even conducting Card Sorts very early in the process. The data gathered from this testing will ensure your application design is focused from the users perspective. There are lots of free or pay card sorting tools that make conducting card sorting and creating information architectures easy. There is no excuse for not gathering this data. Testing paper wireframes is also a really easy but extremely helpful data point.

4. Test Often During Development

Testing often during your application development is another key to creating usable (and thus successful) applications. Using remote usability testing methods means testing can be done almost instantly, and data can be captured in hours, not days or weeks.  There are excellent books on how to conduct remote testing that make it fast and easy for even newbies to create and run usability tests. Using an agile method of application development? No problem, remote testing during sprints means never having to say you are sorry (to your spouse after you come home with a box of your stuff in hand because you just got fired for a bad application).

5. Include users in your team

An excellent idea that few organizations seem to use is including actual users as part of the application design team. Having a group of actual users that you reach out to for feedback and input will clear up disagreements and clarify your purpose as you move through your process. Panels of users are worth their weight in gold. Use the input and commentary you receive from actual users as you go through your sprints or waterfall process. Their input will ensure you are keeping your application on track from a usability perspective.

Conclusion on How to Not Throw Out 125 Million Dollars

By incorporating usability testing and conducting user-centered design as part of your application development process you will ensure your design is user-friendly and successful. Failure to do so risks the potential of your application not being used, which can waste 125 Million Dollars and four years of work.  As has been said before, Failure is Not an Option!

For more tips on how to include testing as part of an application development process read the article on 24 usability testing tools.

Amazon versus Walmart and the usability testing results

Comparing Amazon and Walmart with simple but critical usability testing tasks: finding and buying an iPad, who won?

Useful Usability article Amazon-Vs-Walmart-Usability-TestingAmazon and Walmart are kings of eCommerce. But how do they compare in usability? To answer this, I created a simple but useful usability test: something thousands of users were trying to do this holiday season, finding and buying an iPad.

The usability testing protocol I created was simple, but not meant to be exhaustive in terms of comparing the user experience of both sites. Rather, the test was a quick evaluation of how easy or difficult it was for users to find an iPad with the best possible features for the price (the value of which had to be less than $550) and then buy it.

Here’s the usability testing protocol I set up for the test. It’s simple, quick, but importantly meant to be directional only. I used usertesting.com as my tool for this test.

usertesting.comAmazon versus Walmart Usability Testing Protocol

Introduction: You are buying an iPad as a gift for a family member. You only have a total of $550. You want to buy the best one you can for the price in terms of functionality and features.

Task 1: Please show me how you would find an iPad or iPads that are equal to or less than your price range?

Task 2: Let’s assume you’ve decided to purchase one of the iPads, please show me what you would do to buy it. Please go through all the steps without actually purchasing it.

Tester Age: 18 to 65+

Tester Household Income: $40k to $150k+

Gender: Any

Web Expertise: Any

Country: United States

Number of Testers: Six total (3 for Amazon and a different 3 for Walmart)

Testing Dates: December 6-17, 2012

Usability Testing Results of Amazon versus Walmart

The results of the usability tests are revealing and point to several areas where both Walmart and Amazon may need to explore further usability optimization. And even though this test was simple, quick and used a rather small amount of testers (3 for Walmart and a separate 3 for Amazon), it clearly shows how even minor amounts of usability testing can reveal important places where the user experience can be potentially improved. For eCommerce, this also means improving revenue!

Let’s look first at the results for Amazon and Walmart in terms of how they performed for several key tasks, including:

  1. Finding an iPad
  2. Filtering product search results to find the price range that fits our tester’s budget
  3. Being offered the opportunity to purchase a protection plan (something that no doubt is high on the Walmart and Amazon team’s radar as it is a good source of incremental revenue per shopping cart)
  4. Purchasing the item

We also look at several other errors that seem to be obvious things that can be fixed, or at least evaluated.

And now, on with the results!

1. Amazon versus Walmart Usability Test Task, Finding an iPad

Winner Walmart

Amazon and Walmart take different approaches to displaying and filtering product search results. So a true apples to apples comparison is not possible. However, we can compare the overall ease of use of each system based on the task of asking a tester to “find an iPad in the $550 or less range,” a real world scenario.

Based on the results of this test, the advantage goes to Walmart. This is primarily due to the displayed list of results after the user enters iPad into the search tool. All our testers were able to easily navigate the results, and take the next step promptly, which was to use filtering to find the product in the right price range.

Amazon did not do as well in this test as it could have. The search results are critical to helping our testers to sift through the hundreds of thousands of products Amazon sells to find an iPad in the $550 or less range. Even here at the very start of searching there were potential usability issues.

One of our testers almost immediately became confused when he noticed that the top result for the search term “iPad” was an iPad 2 Second Generation, which caused him to spin off in a different direction spending larger amounts of time trying to find the newer models (as of the writing of this article the iPad Fourth Generation is the newest iPad). Interestingly, all testers mentioned that “typically the best product is at the top” even though this clearly was not the case, and all of our testers had to do a fair amount of searching by scrolling up and down, or clicking on various links, to find the newer iPad models that fit their $550 price limit.

I am guessing Amazon has a usability team so I’m hoping they can evaluate this test result, to determine if there’s a need to find a better way to put the newer (aka “hotter”) products at the top of their search results display. I’m thinking perhaps some Search algorithm testing is in order.

For Walmart, things went well for testers who used the search bar, but the one tester who did not use the test bar had a much harder time of finding iPads. Lesson for Walmart? Consider making your search bar bigger, to attract more attention and cause fewer users to try to navigate through a more difficult process.

Amazon versus Walmart usability test task 1Click to Play the Amazon Highlight Reel

Amazon versus Walmart usability testing results and WalmartClick to Play the Walmart Highlight Video

2. Amazon versus Walmart Usability Test Task, Finding an iPad Using Filters

Winner Walmart

A critical element of eCommerce is using filter tools to narrow search results, which both Amazon and Walmart do, but using vastly different methods. For Amazon, there’s not a specific filter tool that’s readily apparent such as with Walmart, however users do have the ability to filter results, IF they know where to look.

Screen shot of Amazon filter controlsWalmart filter controlsInterestingly, the testers using Amazon had a more difficult time finding the iPad that fit our parameters, in this case a model that gave the most performance and features at a cost of $550 or less, than did the Walmart testers. This was specifically because the Walmart filter tool enables users to easily filter based on price. Not that our testers found the Walmart tool without problems (which they did).

Still, Amazon’s filtering (or lack thereof) of product results based on pricing parameters was something that all our testers struggled with. All testers resorted to scrolling through pages of results, some gave up early and selected a product because it was listed near the top and seemed to fit the test parameters. In the real world, I’m betting this behavior happens more often than may be realized, I’m not sure always to the benefit of Amazon or Amazon users.

Several times, testers became lost in their search due to scrolling through so many results and had to “reset” themselves by going back to the starting results page. The inclusion of peripherals spread in what seems random fashion in the results did not help matters, as it made hunting in the results for the latest model iPad even more difficult.

Because of the extra cognitive load Amazon puts on users, we give the nod to Walmart for this part of the test.

Amazon versus Walmart usability test and Amazon test results of filteringClick to Play the Amazon Highlight Reel

It would be interesting to see what the usability test results for Amazon would be if they were to offer their users a filtering set of tools along the lines of the Walmart tool, versus what Amazon users currently have available.

Walmart Filtering Tools are Good, but not Great

Walmart has one advantage over Amazon in terms of our test of finding an iPad in our price range, and that is the filter tools on the left side. Interestingly, all of the testers used this tool, and all of them were able to reduce what was a much larger list of products down to those they felt met their parameters by using the tool. That’s not to say the tool didn’t cause issues. Several found the refresh that happens without warning rather disconcerting, and one mentioned that slides were preferred, as that way the exact pricing parameters they wanted could be entered.

Screen shot of Walmart usability testing highlight reelClick to Play Walmart Filtering Highlight Reel

3. Amazon versus Walmart Usability Test Task, Product Protection Plan Offering

Winner Amazon

A critical element of eCommerce success is adding in additional SKUs to a shopping cart, in this case a protection plan. Typically this is good for the company, as it is an incremental source of revenue. But it can be a good idea for the shopper too. Reminding them to buy additional items or a protection plan they (if they are anything like my family) will end up using when something bad happens to their product is not a bad idea.

In terms of the offers, both Amazon and Walmart pop-up the protection plan, but that is where the similarities end. Notice the critical difference, Amazon has the “Add Coverage” button the bright, yellow, some would almost say Default button. Because of this, people evaluating adding the extra coverage may have more of a tendency to click the highlighted button, all other things being equal. In essence, the default is YES.

But with WalMart, note the choice is “I prefer not to add coverage.” Ouch. The default here is NO. Also, note that with Amazon you only have to click one button to make your selection, Walmart requires two clicks, one on the radio choice button, and then one way down at the bottom of the pop-up for “Continue.” My guess is WalMart is losing hundreds of thousands, maybe millions, of incremental dollars with their current protection plan offering user experience. Perhaps the WalMart usability or metrics team may disagree with me, but I would test a much more Amazon like user experience here, just to see if there’s a difference (I am betting lunch with the entire WalMart usability team that there is, if you know any of them forward them that message from me).

Amazon pop up with the protection plan offerThe Amazon pop up with the Protection Plan offer has a single button to buy the product


Walmart protection plan offering pop up

The Walmart pop up with the Protection Plan offer requires two clicks, and does not highlight the YES choice

And just to provide an additional data point, it’s interesting that the only tester to choose the protection plan was an Amazon tester, although there was a Walmart tester that was tempted.

Amazon usability testing results of protection plan offeringClick to Play Amazon Protection Plan Highlight Reel

Walmart usability test results protection plan offeringClick to Play Walmart Protection Plan Highlight Reel

4. Amazon versus Walmart Usability Test Task, Purchasing

Winner Amazon

In fact, both Amazon and Walmart are about equal in terms of the ease of moving through the buy-flow. Both have what can be described as best in class user experiences in terms of the shopping cart to purchase task flow. That said, Amazon has a slight edge with their ability to move users through the process with a bit less cognitive load, as witnessed by the several errors that occurred for our Walmart testers that did not happen for our Amazon testers.

Since so much went right for both purchase flows, let us focus on the errors we picked up, both in the buy-flow as well as in other places. Amazon more than once tripped our testers up with offers to buy a product at a price that seemed to disappear when they actually went to the results pages to find the product at that price. Walmart had several avoidable user errors in their buy-flow, mostly caused by simple things like not labeling required fields or hiding critical choices in the middle of a rather busy purchase page. Simple usability and A/B testing could easily improve all these easy to fix errors.

Avoidable errors uncovered in the Amazon versus Walmart usability testClick to Play the Highlight Reel

Summary of Amazon versus Walmart in Usability, Who Won?

So in summary, based on this simple usability test we performed, it would appear that Amazon and Walmart are about equal in terms of the usability of finding and purchasing an iPad, with Amazon winning two categories and Walmart winning two.

However, I actually believe that based on this test Walmart has the edge in usability. The primary reason? I believe Walmart provides an overall easier and faster user experience in the searching, filtering and vetting process associated with seeking out and purchasing a product.

The primary advantage Walmart has over Amazon is the availability of filters on the left side of the products search results pages. This filter set enables users to very easily target products that meet their parameters, to find the best product possible for the given budget range.