Amazon versus Walmart Usability Testing Results


Amazon versus Walmart and the usability testing results

Comparing Amazon and Walmart with simple but critical usability testing tasks: finding and buying an iPad, who won?

Useful Usability article Amazon-Vs-Walmart-Usability-TestingAmazon and Walmart are kings of eCommerce. But how do they compare in usability? To answer this, I created a simple but useful usability test: something thousands of users were trying to do this holiday season, finding and buying an iPad.

The usability testing protocol I created was simple, but not meant to be exhaustive in terms of comparing the user experience of both sites. Rather, the test was a quick evaluation of how easy or difficult it was for users to find an iPad with the best possible features for the price (the value of which had to be less than $550) and then buy it.

Here’s the usability testing protocol I set up for the test. It’s simple, quick, but importantly meant to be directional only. I used as my tool for this test.

usertesting.comAmazon versus Walmart Usability Testing Protocol

Introduction: You are buying an iPad as a gift for a family member. You only have a total of $550. You want to buy the best one you can for the price in terms of functionality and features.

Task 1: Please show me how you would find an iPad or iPads that are equal to or less than your price range?

Task 2: Let’s assume you’ve decided to purchase one of the iPads, please show me what you would do to buy it. Please go through all the steps without actually purchasing it.

Tester Age: 18 to 65+

Tester Household Income: $40k to $150k+

Gender: Any

Web Expertise: Any

Click here to view latest job postings

Country: United States

Number of Testers: Six total (3 for Amazon and a different 3 for Walmart)

Testing Dates: December 6-17, 2012

Usability Testing Results of Amazon versus Walmart

The results of the usability tests are revealing and point to several areas where both Walmart and Amazon may need to explore further usability optimization. And even though this test was simple, quick and used a rather small amount of testers (3 for Walmart and a separate 3 for Amazon), it clearly shows how even minor amounts of usability testing can reveal important places where the user experience can be potentially improved. For eCommerce, this also means improving revenue!

Let’s look first at the results for Amazon and Walmart in terms of how they performed for several key tasks, including:

  1. Finding an iPad
  2. Filtering product search results to find the price range that fits our tester’s budget
  3. Being offered the opportunity to purchase a protection plan (something that no doubt is high on the Walmart and Amazon team’s radar as it is a good source of incremental revenue per shopping cart)
  4. Purchasing the item

We also look at several other errors that seem to be obvious things that can be fixed, or at least evaluated.

And now, on with the results!

1. Amazon versus Walmart Usability Test Task, Finding an iPad

Winner Walmart

Amazon and Walmart take different approaches to displaying and filtering product search results. So a true apples to apples comparison is not possible. However, we can compare the overall ease of use of each system based on the task of asking a tester to “find an iPad in the $550 or less range,” a real world scenario.

Based on the results of this test, the advantage goes to Walmart. This is primarily due to the displayed list of results after the user enters iPad into the search tool. All our testers were able to easily navigate the results, and take the next step promptly, which was to use filtering to find the product in the right price range.

Amazon did not do as well in this test as it could have. The search results are critical to helping our testers to sift through the hundreds of thousands of products Amazon sells to find an iPad in the $550 or less range. Even here at the very start of searching there were potential usability issues.

One of our testers almost immediately became confused when he noticed that the top result for the search term “iPad” was an iPad 2 Second Generation, which caused him to spin off in a different direction spending larger amounts of time trying to find the newer models (as of the writing of this article the iPad Fourth Generation is the newest iPad). Interestingly, all testers mentioned that “typically the best product is at the top” even though this clearly was not the case, and all of our testers had to do a fair amount of searching by scrolling up and down, or clicking on various links, to find the newer iPad models that fit their $550 price limit.

I am guessing Amazon has a usability team so I’m hoping they can evaluate this test result, to determine if there’s a need to find a better way to put the newer (aka “hotter”) products at the top of their search results display. I’m thinking perhaps some Search algorithm testing is in order.

For Walmart, things went well for testers who used the search bar, but the one tester who did not use the test bar had a much harder time of finding iPads. Lesson for Walmart? Consider making your search bar bigger, to attract more attention and cause fewer users to try to navigate through a more difficult process.

Amazon versus Walmart usability test task 1Click to Play the Amazon Highlight Reel

Amazon versus Walmart usability testing results and WalmartClick to Play the Walmart Highlight Video

2. Amazon versus Walmart Usability Test Task, Finding an iPad Using Filters

Winner Walmart

A critical element of eCommerce is using filter tools to narrow search results, which both Amazon and Walmart do, but using vastly different methods. For Amazon, there’s not a specific filter tool that’s readily apparent such as with Walmart, however users do have the ability to filter results, IF they know where to look.

Screen shot of Amazon filter controlsWalmart filter controlsInterestingly, the testers using Amazon had a more difficult time finding the iPad that fit our parameters, in this case a model that gave the most performance and features at a cost of $550 or less, than did the Walmart testers. This was specifically because the Walmart filter tool enables users to easily filter based on price. Not that our testers found the Walmart tool without problems (which they did).

Still, Amazon’s filtering (or lack thereof) of product results based on pricing parameters was something that all our testers struggled with. All testers resorted to scrolling through pages of results, some gave up early and selected a product because it was listed near the top and seemed to fit the test parameters. In the real world, I’m betting this behavior happens more often than may be realized, I’m not sure always to the benefit of Amazon or Amazon users.

Several times, testers became lost in their search due to scrolling through so many results and had to “reset” themselves by going back to the starting results page. The inclusion of peripherals spread in what seems random fashion in the results did not help matters, as it made hunting in the results for the latest model iPad even more difficult.

Because of the extra cognitive load Amazon puts on users, we give the nod to Walmart for this part of the test.

Amazon versus Walmart usability test and Amazon test results of filteringClick to Play the Amazon Highlight Reel

It would be interesting to see what the usability test results for Amazon would be if they were to offer their users a filtering set of tools along the lines of the Walmart tool, versus what Amazon users currently have available.

Walmart Filtering Tools are Good, but not Great

Walmart has one advantage over Amazon in terms of our test of finding an iPad in our price range, and that is the filter tools on the left side. Interestingly, all of the testers used this tool, and all of them were able to reduce what was a much larger list of products down to those they felt met their parameters by using the tool. That’s not to say the tool didn’t cause issues. Several found the refresh that happens without warning rather disconcerting, and one mentioned that slides were preferred, as that way the exact pricing parameters they wanted could be entered.

Screen shot of Walmart usability testing highlight reelClick to Play Walmart Filtering Highlight Reel

3. Amazon versus Walmart Usability Test Task, Product Protection Plan Offering

Winner Amazon

A critical element of eCommerce success is adding in additional SKUs to a shopping cart, in this case a protection plan. Typically this is good for the company, as it is an incremental source of revenue. But it can be a good idea for the shopper too. Reminding them to buy additional items or a protection plan they (if they are anything like my family) will end up using when something bad happens to their product is not a bad idea.

In terms of the offers, both Amazon and Walmart pop-up the protection plan, but that is where the similarities end. Notice the critical difference, Amazon has the “Add Coverage” button the bright, yellow, some would almost say Default button. Because of this, people evaluating adding the extra coverage may have more of a tendency to click the highlighted button, all other things being equal. In essence, the default is YES.

But with WalMart, note the choice is “I prefer not to add coverage.” Ouch. The default here is NO. Also, note that with Amazon you only have to click one button to make your selection, Walmart requires two clicks, one on the radio choice button, and then one way down at the bottom of the pop-up for “Continue.” My guess is WalMart is losing hundreds of thousands, maybe millions, of incremental dollars with their current protection plan offering user experience. Perhaps the WalMart usability or metrics team may disagree with me, but I would test a much more Amazon like user experience here, just to see if there’s a difference (I am betting lunch with the entire WalMart usability team that there is, if you know any of them forward them that message from me).

Amazon pop up with the protection plan offerThe Amazon pop up with the Protection Plan offer has a single button to buy the product


Walmart protection plan offering pop up

The Walmart pop up with the Protection Plan offer requires two clicks, and does not highlight the YES choice

And just to provide an additional data point, it’s interesting that the only tester to choose the protection plan was an Amazon tester, although there was a Walmart tester that was tempted.

Amazon usability testing results of protection plan offeringClick to Play Amazon Protection Plan Highlight Reel

Walmart usability test results protection plan offeringClick to Play Walmart Protection Plan Highlight Reel

4. Amazon versus Walmart Usability Test Task, Purchasing

Winner Amazon

In fact, both Amazon and Walmart are about equal in terms of the ease of moving through the buy-flow. Both have what can be described as best in class user experiences in terms of the shopping cart to purchase task flow. That said, Amazon has a slight edge with their ability to move users through the process with a bit less cognitive load, as witnessed by the several errors that occurred for our Walmart testers that did not happen for our Amazon testers.

Since so much went right for both purchase flows, let us focus on the errors we picked up, both in the buy-flow as well as in other places. Amazon more than once tripped our testers up with offers to buy a product at a price that seemed to disappear when they actually went to the results pages to find the product at that price. Walmart had several avoidable user errors in their buy-flow, mostly caused by simple things like not labeling required fields or hiding critical choices in the middle of a rather busy purchase page. Simple usability and A/B testing could easily improve all these easy to fix errors.

Avoidable errors uncovered in the Amazon versus Walmart usability testClick to Play the Highlight Reel

Summary of Amazon versus Walmart in Usability, Who Won?

So in summary, based on this simple usability test we performed, it would appear that Amazon and Walmart are about equal in terms of the usability of finding and purchasing an iPad, with Amazon winning two categories and Walmart winning two.

However, I actually believe that based on this test Walmart has the edge in usability. The primary reason? I believe Walmart provides an overall easier and faster user experience in the searching, filtering and vetting process associated with seeking out and purchasing a product.

The primary advantage Walmart has over Amazon is the availability of filters on the left side of the products search results pages. This filter set enables users to very easily target products that meet their parameters, to find the best product possible for the given budget range.

Comments are closed.