Tags Posts tagged with "usability"


0 243

So How Did My Earlier 7 Controversial Usability Predictions Turn Out from 5 Years Ago?

Results-7-Controversial-Usability-Predictions-2010-326x235-UsefulUsabilityFive years ago, I published a post of 7 controversial usability predictions for 2010. So what were the results? Did my prognostications come true? Did I get them right?

So with 100 percent transparency, and a small amount of trepidation on my part, let us review how my predictions turned out.

For those of you who need may need a refresher on my 2010 controversial usability predictions, here they are with a brief update on how I did:

  1. The cost of conducting usability testing will decrease by a factor of 10.

I think I nailed this one (means I got it correct for those of you not up on U.S. slang).

Costs for conducting usability testing are a fraction of what they were five to six years ago. Why? Primarily because more and more firms and consultants are using remote moderated and un-moderated usability testing in place of expensive in-person testing at remote locations.

Remote moderated and un-moderated usability testing tools enable research to be conducted for a fraction of what it costs to fly a team to a remote location, rent a facility or hotel room, schedule and conduct the tests, pay for food, travel and lodging for the team, fly them back, and wait for their results.  The savings easily beat the factor of 10 prediction I made.

Add to that that there are now a plethora of low cost remote moderated and un-moderated usability testing tools available, and the savings are even greater.

  1. There will be a dramatic increase in the use of low cost web-based usability testing tools.

Again, nailed it!

Looking out at the tools available today, (you can see a recent list in my 14 usability testing tools article), it’s clear that web and mobile based solutions are plentiful, and seem to be added to daily.

  1. True usability ROI will continue to elude usability practitioners

I think I mostly got this one right, and that this statement is still mostly true, as I have witnessed some firms who have become able to accurately predict their Return On Investment (ROI) for usability. But, sadly, I’ve seen plenty of other firms that are still clueless about usability and ROI.

It helps to have people like Jeff Sauro of Measuring Usability providing helpful information on how to measure and estimate ROI for usability. But I am going to go out on a limb here and state that there are still plenty of organizations and consultants who have no idea how improved usability adds to their bottom line.

  1. Use of remote moderated usability testing will increase by a factor of 10

This one I think I missed. Firms that provide remote un-moderated usability testing like UserTesting, UsabilityTools, Loop11 and plenty more have experienced tremendous growth in utilization. But to a certain extent, that growth has I believe come at the expense of conducting usability testing sessions using remote moderated methods.

I have to admit that even for myself, it is sometimes easier, faster and quicker to conduct un-moderated remote usability testing versus moderated remote sessions. The allure of obtaining results in 10 minutes sometimes pulls decisions to use remote un-moderated, when in fact remote moderated would have been equal to or potentially better for a particular test.

Remote moderated will never go away, but because it takes more leg work to set up and administer it will probably never see utilization increase anywhere near remote un-moderated utilization.

  1. The UK will become a major source of usability expertise

Nailed it! Have you seen the huge number of UX and Usability conferences in the UK? Here’s a list of the UK’s past 337 UX events from Lanyrd. Yes! 337 events!

There are also scores of UX, Design and IA shops in the British Isles. The UK is no slouch when it comes to usability expertise. Our friends over the pond have embraced all things usability and UX and have used it to great extent. Is there room to do more? Of course, but considering the number of full time usability and UX shops that were there five years ago versus today, there has been tremendous growth in this area. So raise a hefty pint of ale, and three cheers in celebration of usability expertise in the UK!

  1. The phrase User Experience Design will become overused and almost meaningless

I’ll give myself a partial correct on this one. True, UX is now a far more common term than usability and in some ways UX has killed Usability. And true, most business folks or non-techies may not know one from the other, but still, there has been some consolidation and standardization of the term user experience design that most in our circle understand and use. It’s far from meaningless, so although there are still multiple ways to define ‘UX,’ the common theme of the experience a user has with a product, website, application or whatever seems to be fairly well understood.

  1. Without professional certification being required, more and more charlatans will be attracted to usability

I’ll give myself a partial correct on this one. A Certification course, test and certified practitioner list still eludes our ranks. I would have hoped the User Experience Professional’s Association could have made some progress on this in the past five years, but sadly that is not the case.

I have seen plenty of suspect ‘UX Audits’ and ‘Usability Reports’ floating around that seem to be very sub-standard in terms of actual UX and Usability expertise. Still, for the most part the players continue to be the players, and new consultants that pop-up for the most part seem to be interested in doing the right thing by their clients and providing real value. I may have been a bit negative in my attitude on this one. But still, until there is an official Certification and evaluation of practitioners, it really is a ‘buyer beware’ world for our prospective customers.

Conclusion: How I did On My 7 Controversial 2010 Usability Predictions

So overall I scored myself with:

  • 3 Correct
  • 3 Partially Correct
  • 1 Incorrect

How would you score my predictions? Do you agree with my scoring? Be sure to leave your comments below!

A Website Audit can improve your ROI (and respect) at least 7 different ways, here’s how:

Website audit funny quote

 Website Audit Definition

The definition of a website audit for UX purposes is to evaluate both behavioral data (visitor actions) and user experience data (task-flow other UX vs. best practices) to identify issues and recommend opportunities to improve conversion.

Take My Website – PLEASE!

Sometimes a website audit can make you feel like you get no respect, as if you are the Rodney Dangerfield of the web.

Side note: Odd how many companies do not respect their own websites. I’m not sure why that is, but a separate follow-up study may be needed.

In any event, I digress…

Remember his famous quote?

“My psychiatrist told me I’m going crazy. I told him, “If you don’t mind, I’d like a second opinion.” He said, “All right. You’re ugly too!””

- Rodney Dangerfield

But I’m here to assure you that a website audit, specifically an audit of your website’s UX, can and will make your website ROI better, and get you the respect from your website visitors (and bosses) that you deserve. Here’s how…

Website Audits Can Improve ROI

Improving the ROI of a website is best accomplished by ‘fixing the leaky bucket.’ What I mean by that is finding and fixing the issues that are causing website abandonment, task flow failure, poor visitor engagement and disappointing conversion.

There are many variations of website audits, and all have their unique value. These include:

  • Accessibility Audit
  • Analytics Audit
  • Conversion Optimization Audit
  • Page Speed Audit
  • SEO Audit
  • Usability Audit
  • Website Competitor Audit

However, I like to combine the best elements of several of the above audits into a comprehensive website audit that evaluates the larger user experience. This is a great way to ensure all the data available is utilized to analyze the website and make recommendations for optimization. ROI improvements based on recommendations from website audits are typically significant and quick.

Website Audit Elements

The two categories of information that a website audit evaluates are the critical elements of a well done audit:

Behavioral Data – Behavioral data will define what actions visitors are doing (or not doing) on the site. This data typically comes from website analytics tools such as Google Analytics, Adobe Analytics, Coremetrics, WebTrends, etc. Typically this data includes information such as:

  • Overall website performance conversion data
  • Paid search conversion data
  • Organic conversion data
  • High-level PPC keyword data
  • Website bounce rate
  • Visits by browser
  • Screen resolution
  • Top content
  • Content paths
  • Length of visits
  • Location (by geography)
  • Page fold
  • Devices
  • Operating Systems
  • And more…

User Experience Data – UX data includes information about how the website compares to usability and UX best practices. This data comes from tools such as the 5 second test, eye tracking, usability testing, and a comparison of the critical website interaction elements versus best practices (sometimes called a heuristic or website review).  This type of data typically includes information such as:

  • Elements that are or are not attracting attention
  • Page fold ramifications on CTAs or critical copy
  • Form field best practices vs. existing forms
  • Navigation flow and labeling
  • Product page elements vs. best practices
  • Contact Us page elements vs. best practices
  • And more…

Website Audit Findings and Analysis

The website audit includes a document with detailed findings and analysis of the behavioral and UX data that precisely defines where the website is performing well, and where there are opportunities for improvements. The subsequent recommendations are then tested, typically with A/B testing, to verify that the optimizations are having the desired benefit.

I always include screen shots of each of the items being audited, with callouts that explain what the issue and opportunity for testing could be. These can sometimes be fairly large documents, upwards of 70 to 90 pages. But because the information is presented one item at a time getting through the analysis document is easy, and fairly quick.

Advantages to Website Audits

There are at least 7 primary advantages to website audits, they include:

1. Ability to use website data to prioritize A/B testing: Nothing beats using your actual website and UX data to define and prioritize where your conversion is not optimized. This removes the guessing game that too often occurs as part of A/B or Multivariate testing.

2. Benchmarks existing versus potential conversion: By using the data from a website audit as a benchmark, the down-stream changes to traffic, navigation flow and conversion can be quantified. Benchmarking removes much of the guess work out of determining if conversion improvements based on testing are temporary, or permanent.

3. Pinpoints issues: The analysis of the website audit provides pinpoint clarity on which elements of a page, form or flow are potentially hindering performance. Specific recommendations can be extremely detailed, which helps focus where to spend testing resources. If you’ve ever heard the principle of “don’t throw out the baby with the bath water” you’ll understand and appreciate the detailed approach of a website audit

4. Clearly defines success: Many of my clients appreciate the incremental value that comes from the analysis of a website audit, in that they appreciate knowing what ‘success’ looks like. When A/B testing, how do you know you’ve truly optimized the ‘B’ version to achieve maximum success? Unless you have data that defines what that success looks like, you don’t really know how successful the test could be. A website audit provides that ability to better define success.

5. Doesn’t ‘fix’ what’s not broken: Too often companies optimize using an ad-hoc approach to testing. Sometimes this causes something that was working just fine before, to work poorly, or not at all. A website audit helps define what’s working from what’s not, so that items that are working well are left alone, and items that are not are tested and optimized.

6. Maximizes resources: Unless you have a full time optimization team, the odds are that testing is but a small part of your overall work activities. Maximizing your time and other resources is crucial to optimizing ROI. Spending time optimizing only that which needs to be optimized makes your resources that much more productive.

7. Proves your value (and earns you respect): Facing your bosses and answering to them for how you specifically are helping the company can sometimes be a challenge. Having the demonstrated results from the testing that comes from a usability audit provides you with a plethora of actual data that precisely defines how you are contributing to ROI. Nothing speaks better to your bosses than data that proves you (and thus they) are adding value to the company. You want respect, give them optimization numbers and you’ll get it!

Conclusion:  Website Audit Improves ROI and Respect

A website audit and the resulting optimization of the conversion of the site can greatly improve the ROI and performance of a site. It can also provide you with the respect you need. If you have any questions or would like more information on how you can use a website audit to improve your website ROI just contact me. By using the results of the website audit, you will have a more informed, prioritized and clearly defined road to improving the success of your website.

21 7667

A Comprehensive Review and Matrix of 14 Usability Testing Tools

Comprehensive review and matrix of 14 usability testing toolsIn 2009 I wrote a popular article on 24 usability testing tools. Since that time, there has been tremendous growth in the quantity and capability of usability testing tools. An update to the article is due, but because of the broad array of tools now available a more formalized definition and a matrix of usability tools is needed.  To that end, the following matrix and comprehensive review of 14 usability testing tools will help practitioners who are looking for tools for specific testing tasks.

Usability Testing Tools Matrix

For purposes of classifying tools to evaluate in this matrix, the tools must have three critical functions necessary to conduct a classically defined usability test:

  1. Audio and video recordings of testers (ala Think Out Loud method)
  2. Ability to edit recordings of tester sessions into highlight reels
  3. Capability to test using mobile devices (Smartphones and tablets)

The classical definition of usability testing is a systematic observation under controlled conditions to evaluate how well testers can complete critical tasks.

Thus, having the ability to record testers, to see their interactions and hear their thoughts (ala the Think Out Loud method) is critical to the observation process. It is also important to be able to edit those recordings, so that key insights can be shared with others in the form of highlight reels. Finally, the increasing adoption and usage of mobile devices to interact with websites and applications makes it necessary to conduct usability tests of the mobile user experience. Thus, tools should be able to test the mobile as well as PC based experience.

Not included in this review are Feedback or Click Tracking tools. This is because they only provide feedback or opinions about user interfaces, and lack the ability to observe testers conducting tasks. I will conduct the same comprehensive review of Feedback and Click Tracking tools in a future article.

Usability testing tools matrix from UsefulUsability.com

Four Quadrants of the Usability Testing Tools Matrix

Usability testing tools are divided into four quadrants on the matrix, based on where they fall when evaluated against two critical attributes:

  • Depth of Technology: What functionality and features are available with the tool, and how deep does the array of features run? Tools that score higher going up the vertical Depth of Technology axis have more functions and features.
  • Tool Impact: This is a measure of the ability of the tool to impact overall testing, and thus the observations and eventual recommendations for usability improvements. Tools that score higher (going to the right) on the Tool Impact axis have the ability to have a greater impact on overall testing results.

Based on these definitions, tools can fall into one of four quadrants on the matrix: Authority, Contender, Innovator and Niche.

  • Authority tools are those that provide a greater amount of usability testing functionality as well as a higher potential impact on discovering, diagnosing and ultimately optimizing the user experience.
  • Contender tools have the ability to provide a greater impact in terms of their depth of technology, but may have more limited potential impact in discovering, diagnosing and optimizing the user experience
  • Innovator tools have a more limited depth of technology, but have the potential to have a greater impact in usability testing and recommendations
  • Niche tools have a relatively limited depth of technology as well as a lower potential impact on usability testing and recommendations results, but can still be very useful for the specific tasks they were created for

All of these tools have specific features that make them unique.  Depending on your testing needs, any of these tools may be good choices and can compliment your toolkit.

Mobile Usability Testing Still Rudimentary

The upper right corner (Authority) of the matrix is blank for a very good reason. It represents the current lack of a fully integrated mobile usability testing solution. In my opinion we still lack a comprehensive, fully integrated mobile device usability testing tool, something that records both mobile device screen interaction and the audio/video of testers as they try to accomplish tasks. Because they are mobile devices, testing in this manner should be possible anywhere the tester goes, at any time, and using any smartphone or tablet (as long as it has a camera). Real time streaming of the screen and audio/video of the tester should be available for remote teams.

Mobile usability testing is currently in a neophyte stage. Yes, there are cludgy and cumbersome ways to test usability with a mobile device, typically requiring testers to place their phone on a desk with a camera hovering over it (thus not being at all ‘mobile’). Most methods today lack the sophistication of being a truly integrated and mobile experience.

Consider how just a few years ago new tools like Camtasia, Morae, UserTesting and others made usability testing websites an anytime, anywhere, quick and cost effective solution versus using expensive and cludgy labs or home grown equipment. We need that same fully integrated experience for mobile, but we do not have it, yet.

Authority Usability Testing Tools:

Usability testing tools matrix from UsefulUsability.com

Ovo Logger

Ovo Logger image from UsefulUsability.comOvo Logger by Ovo Studios can be considered to be the Rolls Royce of usability testing solutions. This is the tool for large Fortune 1000 enterprises or usability testing studios that conduct a high volume of testing and have teams spread around the globe. Typically these situations call for the need to have a robust tool to capture, edit and share results. It should be noted that Ovo Studios creates custom usability testing labs, and the Ovo Logger fits in well with those labs. The tool scores at the top of comparable tools for depth of technology, but is slightly limited in terms of impact due to requiring a dedicated PC, and being somewhat expensive to deploy.

Pros: Enterprise level usability testing software and hardware that records multiple inputs from PCs or Macs (cameras, microphones, screen interaction) and streams the output to observers. Editing tools enable real-time analysis of tests with the ability of observers to document important points. Can capture un-tethered iPads, non-jail broken iPhones and Android devices if using the Logger mobile app.

Cons: Designed specifically for larger firms with multiple locations and or usability teams. Requires a dedicated PC to run the software. Pricing is more expensive versus other usability testing tools and is based on number of sources, associated lab requirements, etc.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $3,000 and up depending on scope of associated usability lab

Camtasia plus a Screen Sharing Tool

Camtasia image from UsefulUsability.comA very powerful Do It Yourself approach for remote usability testing is to use a screen capture tool such as Camtasia (from TechSmith) coupled with a screen sharing tool such as GoToMeeting, WebEx or similar applications. The major benefit of this approach is the ability to conduct moderated remote tests, because the practitioner has a direct connection to the participant. In addition, the recording of the screen interaction with the voice and or video of the tester is immediately available for editing on the practitioners computer. The downside to this approach is lack of a simple mobile solution.

Pros: Enables remote moderated usability testing, because the practitioner has a direct connection to the participant. Sessions can be shared with other remote observers using the screen sharing tool. Audio and video recordings of the screen interaction with the voice of the tester are immediately available for editing on the practitioners computer.

Cons: For part-time or ad-hoc uses, this approach may be more expensive than using some of the other remote unmoderated tools available from vendors. Screen sharing tools are notorious for occasionally having technical problems that cause poor connection performance, or worst case dropping the connection entirely. This is not an optimal solution for conducting mobile usability testing, although cludgy methods of using a webcam pointed at the testers mobile device can be tried.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $299 for Camtasia, $19/month for GoToMeeting or $24/month for Webex


UserTesting image from UsefulUsability.comUserTesting provides a network of pre-screened testers who can meet several specified demographics. Or, practitioners can invite their own testers using the tool. Practitioners create a set of tasks for the testers to complete, either on your website or even competitor sites. Testers screen interaction and voices are recorded. UserTesting includes powerful and easy to use video editing tools to create highlight reels. UserTesting also provides mobile usability testing, which is a major feature considering the increasing utilization of mobile devices. Written responses to questions are included. UserTesting is one of the most robust tools to use for remote unmoderated usability testing and thus scores higher  for depth of technology and testing impact on the matrix.

Pros: Relatively low cost way to conduct remote unmoderated usability testing. Audio and video of the tester are recorded, and editing tools enable highlight reels to be created easily and quickly. Mobile device usability testing is available. The large network of testers typically means testing session results are available in an hour or two. Demographic filtering tools enable finding the appropriate testers for most Persona situations.

Cons: Because UserTesting pays small fees to the testers for each completed test, it is critical to carefully create non-biased tasks, else you risk the tester trying to cut corners to accomplish the test as quickly as possible. As a practitioner, you must watch for this kind of activity and request a re-do of that test if you are concerned about the quality of the tester. As an alternative, using your own testers can alleviate much of this. Mobile testing is available, but requires the tester to be at a desk so a webcam can be pointed down to the screen, which is not typically how most people use their mobile devices.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $49 per test with bulk pricing discounts available


Morae image from UsefulUsability.comMorae from Techsmith can be considered the Granddaddy of usability testing tools. This is because it was the first tool specifically created for usability testing. Even today it is still one of the best. It provides a comprehensive set of features available to record, analyze, edit and share usability testing sessions. Morae is an application that resides on the practitioners computer (PC only, sorry Macs). Morae records all screen interaction, including the testers voice and or face if using a webcam. Testers can be on any Mac or PC if testing websites or applications. Morae includes very powerful documentation, editing and sharing tools. Morae scores fairly high for depth of technology, and also fairly high for testing impact, the limiters being lack of a Mac practitioner solution and the issue of cludgy mobile testing.

Pros: Powerful screen capture and editing tools make this the choice for larger scale usability testing. Morae includes additional tools that can track a variety of screen and click interactions, plus has the ability to incorporate survey/opinion based data. Sharing test sessions with other practitioners and testing results with teams is relatively easy and quick.

Cons: Desktop & Laptop based system with no ability to record Mobile device screen interaction directly. Relatively expensive and PC-only (not Mac compatible) for the Manager and Recorder tools.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $1,995

Contender Usability Testing Tools:

Usability testing tools matrix from UsefulUsability.com


Userlytics image from UsefulUsability.comUserlytics provides both PC and Mobile based remote unmoderated usability testing. For mobile testing, the more expensive Gold or Enterprise plans are required. Testers can come from the Userlytics panel, or from testers you provide. Screen interaction, tester voices and webcam video images of testers can be recorded. Written answers to post test questions are also included.  The depth of technology is good, but because of the requirement to use the more expensive Gold plans for downloading and editing of videos, capturing of sessions longer than 15 minutes and testing mobile, this tools has somewhat limited testing impact versus some competitors. Practitioners that conduct larger volumes of tests should consider Userlytics.

Pros: Userlytics is relatively low in cost, and is definitely easy to set up and use. The more expensive plans allow download of recordings for editing and sharing. In addition, images, videos and other assets can be uploaded to the Userlytics secure servers for early prototype testing. White label testing results sharing is possible for Gold and above plans.

Cons: Userlytics has the same issues as any other remote unmoderated usability testing tool, with the burden being on the test creator to ensure non-biased questions. Userlytics has a rather confusing set of test functionality based on plans (Basic, Silver, Gold, Enterprise). For example, tests recordings are less than 15 minutes for sub-Gold plans, and mobile testing is only available on Gold and above plans. Depending on the plan (Silver or above), each monthly subscription price includes one or more ‘free’ tests, with each additional test costing $33 or less per test. There are also limits in terms of number of post test questions that can be asked based on plans.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $39 per month or more with bulk discount pricing available


UserFeel image from UsefulUsability.comUserFeel.com provides an assortment of testers, many of whom speak languages other than English, for remote unmoderated usability testing. Having the ability to choose a language other than English (many are offered) is a great benefit for testers of websites that are targeted for other countries. Written responses to post test questions are also available. Videos are available of the test on the website (but not available for download for editing purposes), and a white label playback option of the video is free. The depth of technology is good, but due to the lack of mobile testing and inability to download test recordings the tool impact is lower than some competitors. For those conducting usability testing in languages other than English, this could be a good solution.

Pros: A good tool for non-English language remote unmoderated usability testing. Demographic and language filtering tools are available. Pricing is relatively low and post test questionnaire functionality is available.  The free white label playback of the test recordings is a plus.

Cons: Like other remote unmoderated usability tools, practitioners must be careful to create non-biased tasks or else risk having testers try to quickly finish the test, causing poor test results. The advanced video editing capabilities of some competitor tools are not available. Videos cannot be downloaded and edited. Mobile testing is not available.

Tester Audio and Video Recordings? Yes

Recordings can be edited? No

Mobile testing capable? No

Price: $39 per month with bulk discount pricing available

Innovator Usability Testing Tools


OpenHallway image from UsefulUsability.comOpenHallway is a remote unmoderated usability testing tool for PC or Mac-based research. Testers come from the practitioner (no panel is offered) via a link that is generated by OpenHallway when the scenario and tasks are set up. OpenHallway requires no code to use, which makes it a great tool for testing applications, websites and even competitor websites. Screen interaction is recorded, along with voice and or the testers face, if the tester has a webcam. Plans (Basic, Plus and Premium) determine the amount of video storage available. Unlimited number of test sessions are available, however the video storage amount will act as the limiter to how many sessions are actually conducted.  Because of video storage limits, and videos being MP4 only, and no mobile solution, OpenHallway has somewhat more limited depth of technology than some competitors, however the tool impact is good in that it can be used on almost any website with no code required.

Pros: OpenHallway is simple to use, easy to set up and therefore is a handy tool for recording unmoderated remote usability tests. As of the writing of this article there is a free trial.

Cons: OpenHallway is PC or Mac compatible, but does not support mobile.  Although no code is required, Java must be present on the tester computer for the tool to work. Videos are MP4 only, and only the Premium plan ($199 per month) allows downloading (and thus editing) of the video.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $49 per month or higher, depending on the plan


UsabilityTools image from UsefulUsability.comUsability provides a variety of usability, feedback and click tracking tools, with the primary advantage being they all use a single practitioner interface. This makes creating tests that require multiple tools much easier. There are two versions, a UX Suite and Conversion Suite. The UX Suite contains click testing, web testing, survey, card sorting and a Persona Creator. Practitioners can set up web tests that provide scenarios to website visitors, who then try to accomplish the tasks and click success or abandon buttons based on whether they thought they could accomplish the task. Results are provided in data and charts. No code is required unless you choose to utilize their website intercept request on your site. The lack of tester audio and video recordings for PC and mobile is a limiter for depth of technology.  However, having the ability to utilize the tools efficiently and quickly provides a good tool impact.

Pros: A toolkit of the most common tools needed to fully understand the user experience. Having all tools in a single practitioner interface streamlines the process of creating and running multiple tool tests. In larger quantity testing scenarios, this set of tools can be more economical than using separate usability tools.

Cons: UsabilityTools does not provide audio and or video recordings of tester sessions. Mobile interaction using audio and video recordings is also not available. Pricing is not available on their website, although as of Sept. 2013 they had one time fees ($300) and monthly fees starting at $69 for the Professional version. Free panels of testers are not offered, although paid sources are available.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? No

Price: Pricing information is not available on their website.


Loop11 image from UsefulUsability.comLoop11 provides the ability to conduct unmoderated remote usability testing on PCs, Macs or mobile devices. The tool is easy to set up and fairly easy to administer (no code required). A key advantage is it can provide real-time data. Loop11 provides useful information for evaluating task flow on websites, including competitor websites. No audio and video recordings of unique test sessions are available, and because of this the depth of technology, and tool impact are lower than some competitors.

Pros: Loop11 requires no code, thus is useful for conducting comparison tests of user experience on competitor or other websites. Setting up and running tests is relatively quick and easy. Real time information can be advantageous for situations where quick results are needed.

Cons: As with other unmoderated tools Loop11 does not allow interaction with the testers to probe or ask follow-up questions. The test creator has to find his or her own test participants. No ability to record the tester via ‘think out loud’ methodology, something that exists with competitor tools. Somewhat expensive versus competitors.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? Yes

Price: $350 per project or $158 per month


UserZoom image from UsefulUsability.comUserZoom includes multiple tools including remote unmoderated testing on PCs and mobile devices (via UZ Mobile), along with other tools such as card sorts, tree testing, screenshot click testing and more. For the remote unmoderated usability testing tool UserZoom enables tasks to be created and deployed to either 3rd party panels or the practitioner’s own panel. Intercepts are also available, a handy feature when trying to reach actual visitors to a website. UserZoom does include video recordings of screen interaction, but only at the pricey $9,000 per year or higher plans, and no voice capture of testers are available, thus Think Out Loud recordings of users explaining their actions are not captured. This limits the depth of technology and tool impact when compared to some competitors.

Pros: UserZoom includes powerful reporting and analytics tools that can greatly help researchers synthesize and analyze post test data, including heatmaps, time on task reports and much more. Plans allow practitioners to pick the best price point and functionality for their needs. Plans include a free Pilot, UZ Proof of Concept at $1,000, Basic plan at $9,000, Pro plan at $29,000 and an Enterprise plan that requires a custom quote. As of the writing of this article UserZoom offers a free pilot study.

Cons: UserZoom does not enable voice and or video capture of the tester as they try to accomplish a task, a major differentiator compared to competitor services. UserZoom does not have its own panel of testers available for free, as other competitors do. Practitioners can view and edit recordings of testers while they were conducting tasks to make highlight reels, but there is no audio nor video capture of the actual tester. UserZoom is rather expensive versus its competitors, and lacking the ability to hear and see testers via session recordings while they were conducting the task is another major differentiator between UserZoom and other remote unmoderated usability testing tools.

Tester Audio and Video Recordings? No

Recordings can be edited? Yes, but only recordings of screen interaction (no tester voice is available)

Mobile testing capable? Yes

Price: $9,000 for the Basic plan and up

Niche Usability Testing Tools

Usability testing tools matrix from UsefulUsability.com


TryMyUI image from UsefulUsability.comTrymyUI.com is another of the many SaaS-based remote unmoderated usability testing tools. Testers create tasks, and videos are provided that include screen interaction and the testers voice. Written answers to post questions are also included. As of the writing of this article a free trial is included. Practitioners can use their own testers, but at a license cost of $99 for a single use test. Because video editing and mobile testing are not provided, the depth of technology and testing impact are lower than some competitors.

Pros: TryMyUI provides an alternative to other remote unmoderated tools that is slightly cheaper than many competitors. As a tool that provides video and audio recording of testers as they try to complete tasks it fits the classical definition of a usability testing tool.

Cons: All the same issues with remote unmoderated usability testing apply, with the burden being on the practitioner to create non-biased questions, and to carefully analyze the results to ensure the tester was not trying to use short cuts to complete the test. Video editing is not available, nor is mobile testing.

Tester Audio and Video Recordings? Yes

Recordings can be edited? No

Mobile testing capable? No

Price: $35 per test with bulk discount pricing available


WebEffective image from UsefulUsability.comWebEffective by Keynote Systems provides remote, unmoderated usability testing via tasks that are assigned and recorded. A panel is available, or practitioners can use their own testers. Data including click stream information is captured and displayed in a variety of reports, which can be downloaded. Individual sessions showing screen interaction are captured, however there is no audio nor video commentary captured of the tester as they go through the tasks.  Mobile testing is available. The lack of audio and video of testers as they try to complete tasks means WebEffective has a lower depth of technology and testing impact than some of its competitors.

Pros: A powerful tool to capture user interaction on websites or mobile devices, with a wide variety of reports available.

Cons: WebEffective does not provide audio and or video recordings of tester sessions. In this regard, WebEffective operates more as a click stream tool than a true usability testing tool.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? Yes

Price: Pricing is not available on the Keynote WebEffective website

Ovo Solo

OvoSolo image by UsefulUsability.comThe Ovo Solo tool by Ovo Studios provides a small business version of Ovo Logger that enables screen interaction video capture along with the testers face and voice.  Simple logging controls enable analysis of the resulting test videos. However the tool is PC only. The relatively low price point makes this a good candidate for practitioners on a tight budget. Because of the PC only nature of the tool, the basic editing functions and the inability to record mobile sessions, this tool has lower depth of technology and testing impact than some of its competitors.

Pros: The relatively low price point makes this a very affordable way to capture, edit and display usability testing sessions.

Cons: This tool is PC only and is not able to record screen interaction on mobile devices. Editing tools are basic and may not meet more advanced usability tester needs.

Tester Audio and Video Recordings? Yes (but tool is PC only)

Recordings can be edited? Yes

Mobile testing capable? No

Price: $99


Silverback image by UsefulUsability.comFor Mac users, Silverback from Clearleft enables remote and in-person moderated usability testing. Silverback is not available for PC based practitioners. Session screen interaction and audio and or video of the tester are recorded, and simple editing features like task tagging are available. Recordings are saved in QuickTime, which makes editing of highlight videos relatively easy for Mac users. The lack of PC functionality and video editing capability means this tool has lower depth of technology and testing impact than some other competitors. Mac practitioners will find Silverback a very easy to use tool at a very affordable price.

Pros: A relatively low cost and easy to use tool for Mac users that provides the basics in terms of classical usability testing functionality.

Cons: Mac desktop and laptop only, does not work with PCs. No ability to record mobile device screen interaction directly. As of the writing of this article currently does not support FaceTime camera with newer Macbooks, but the app is being re-written to fix this. Lacks powerful editing tools available in other applications.

Tester Audio and Video Recordings? Yes (but tool is Mac only)

Recordings can be edited? No

Mobile testing capable? Yes

Price: $69.95

14 Usability Testing Tools Conclusion

Usability testing tools matrix from UsefulUsability.com

Any of the 14 usability testing tools reviewed here can provide excellent results if applied in a manner which best fits their unique functionality and features.

If Your Favorite Tool is Missing

Not seeing your tool here? It could be because I’m classifying it as a Feedback or Click Tracking analytics type of tool. My original 24 usability testing tools article included many of these tools, but also included some tools that can better be defined as Feedback or Click Tracking analytics tools. In future articles I will provide a matrix and reviews separately for those types of tools. And if your tool is not a feedback or click tracking tool, let me know by leaving a comment.

Note: Companies mentioned in this article may or may not be advertisers on this site. However, in no way does their sponsorship or lack thereof impact the results of this or any other editorial content on the site. In all cases, the same rubric for evaluation is used to compare tools or services and all results reflect the outcome of the comparison without regard to whether a company is advertising on the site or not.

12 Critical Conversion Science Tools You Should Use to Improve your Website Testing and Optimization

Conversion Science and Data

12-Conversion-Science-Tools from Useful UsabilityConversion science is all about using actual behavioral and task-flow data from your website or applications to make informed decisions about what to test and optimize. But where do you get the data needed? This listing of the 12 conversion science tools will provide you with many of the top tools the experts use to obtain behavioral and usability data. It is this data that is needed, and used, to create informed prioritization of conversion optimization tests. Here then in random order are the 12 critical conversion science tools…

1. Google Analytics

Google Analytics from Useful UsabilityGoogle Analytics (or GA as it is commonly referred to among frequent users) is a free data capture and analysis tool from Google that enables tracking of user behavior across websites. Having and analyzing behavioral data is critical to conversion science. Google Analytics enables conversion data tracking that is essential for making informed decisions about optimizations. Google Analytics or paid versions of similar tools such as Adobe Analytics, CoreMetrics, WebTrends and the like are essential tools for data-based conversion optimization.

Google Analytics also provides a tool to conduct simple A/B testing (call Experiments) which operates well enough for the price (free), although there are more sophisticated paid A/B testing tools available like Optimizely or Maxymiser.

Price: Free

2. Crazy Egg

CrazyEgg image from useful usabilityCrazyEgg provides click tracking and scroll maps of your website, extremely important information for conducting conversion science testing and optimization. Knowing where your visitors are clicking, or just as important where they are NOT clicking, is essential for evaluating behavior on and among pages. Additional tools like the Confetti report enable tracking of keyword data based on clicks, to help you define which audiences are interested in what topics and where in your site they go. Other tools similar to CrazyEgg are available such as Mouseflow, ClickTale and more.

Price: $9 per month and up depending on the plan, first 30 days free

3. 5 Second Test

5 Second Test image from Useful UsabilityThe Five Second Test tool offered by UsabilityHub is a great way to test your home page, or any page of your website, to learn if it is communicating effectively with your visitors. Studies have shown that most visitors make up their minds about a website well within 5 seconds. So this tool enables testers to see an image of your website page for 5 seconds, and then answer questions about it. What questions? Who you are, what you provide and why they should care are three important questions I like to test. Use this tool to learn if your website is communicating your value proposition effectively. The data captured is extremely helpful for optimization and related conversion science purposes.

Price: $20 per month and up depending on the plan, free if you earn Karma points by participating in 5 second tests other users have set up.

4. UserTesting

UserTesting image from Useful UsabilityUserTesting.com provides a powerful way to quickly capture unmoderated usability testing data. Nothing beats observing real users interacting with your website while they try to complete tasks. Seeing your actual users stumble over certain tasks will quickly help you define areas on your site that need optimization. This kind of conversion science data is often overlooked by many firms that conduct A/B testing, but in fact is extremely helpful for optimized A/B testing success. Rather than guess what to A/B test, use UserTesting.com data and analysis to find and prioritize task flow failure points. Use this information to then set up A/B tests. UserTesting enables testing of any website or mobile experience, including testing your competitors! Other tools akin to UserTesting include Usabilla and UserZoom.

Price: $49 per tester with discounts available for pre-paying larger numbers of tests.

5. DeviceAnywhere

DeviceAnywhere image from Useful UsabilityDeviceAnywhere is a very helpful tool for evaluating websites on real mobile devices. More and more websites and applications are viewed and interacted with using mobile devices. Evaluating the user interface using this mobile tool will provide extremely helpful data for optimizing the mobile experience.  This is not an emulator. Unlike other free services, these are real mobile devices, so you can quickly see how your mobile website will render across a variety of popular smartphones.

Price: The Spot-checking tool is free, paid versions for developers and Enterprise-level usage are available starting at $260 per month and up, depending on the plan.

6. iPhoneTester

iPhone Tester image from Useful UsabilityThe iPhoneTester is an emulator that enables quick checks of your website content on an iPhone interface (3G and 4G) as well as on the iPod Touch. It is quick, easy and best of all free. Just enter your URL into the emulator and what you will see is a close approximation of what your iPhone visitors will see. It is a great way to quickly validate what your mobile user experience will be like. And considering the plethora of iPhones currently being used to access website content it is a very good idea to check your site from an iPhone usage perspective. I fully expect this year to be the transition year from mobile testing being a nice to have to mobile testing being a requirement of any optimization plan.

Price: Free

7. GoToMeeting

GoToMeeting image from Useful UsabilityGoToMeeting is a very helpful tool for conducting remote moderated usability testing of websites to capture essential task flow data. It is actually part of the tool set, the other tool being a screen video capturing utility like my favorite, Camtasia Studio (explained below). By using GoToMeeting with a usability test participant, you can enable the tester to screen share his or her screen while they go through the exercises of trying to accomplish critical tasks on your website. Recording the tester screen interaction and voice as they go through the test will provide data on where tasks are failing (much like UserTesting). However, being present during testing enables you to go one better in that you can stop the tester or ask follow-up questions as you go through the testing exercise. Data from this type of testing is extremely helpful in finding and prioritizing task flow errors, which ultimately can be optimized through A/B testing of improved interfaces.

Price: $19 per month or more depending on the plan, free 30 day trial

8. Camtasia Studio

Camtasia Studio image from Useful UsabilityCamtasia Studio by TechSmith is the second half of the remote moderated usability testing toolkit. Coupled with GoToMeeting, it enables real-time video screen capturing of the tester as he or she conducts usability testing on your website. All screen interactions (and voice) can be captured to gather usability data on critical tasks. And in addition, Camtasia Studio provides powerful but easy to use video editing tools that help create professional videos quickly and efficiently. Use Camtasia Studio to record on-screen activity, customize and edit content, add interactive elements or import media, and share your videos with anyone, on nearly any device. Conversion science is all about using data, and Camtasia enables capturing of interaction data in real time. If a picture is worth a thousand words then video of actual users on your site is worth a million.

Price: $299 per license or less if multiple copies are purchased.

9. Ethnio

Ethnio image from Useful UsabilityEthnio is a website intercept tool that enables you to invite your actual website visitors to participate in a survey or website usability test. It is an excellent way to get data from those all important actual users on your website. The tool includes screeners and the ability to manage your website intercept by a number of parameters including time of day and more. Conversion science is all about real data from real users, and this tool enables finding those users. The tool is simple to use, easy to set up, and makes finding testers easy and quick.

Price: $49 per month or more depending on the plan.

10. SnagIt

SnagIt image from Useful UsabilitySnagIt is another very helpful tool in the conversion science toolkit. By capturing screen shots of an interface and editing them with the built-in SnagIt photo editing tools you can quickly highlight areas of interest, move elements around on the page to create B version mock-ups, and add helpful callouts that specify where on the page the issue is. By demonstrating issues and opportunities using actual screen shots of your website, you will be able to communicate very effectively with your teams as you create new B versions of web pages or elements on a page to test. Visualizations of optimizations are a conversion science best practice.

Price: $49.95 per license or cheaper for multi-license purchases.

11. SurveyMonkey:

SurveyMonkey image from Useful UsabilitySurveyMonkey provides the ability to capture qualitative and quantitative data quickly and easily. The science part of conversion science is based on having actionable and valid data. The best way to capture large amounts of statistically significant data is with surveys of prospects and or customers. This data is extremely useful when evaluating qualitative aspects of your web user experience, and can help guide your prioritization of conversion optimizations. SurveyMonkey enables almost anyone to be able to quickly and easily set up surveys for capturing that all important data. A note of caution about surveys, asking non-biased questions that provide actionable data is harder than it seems. It is a good idea to get second opinions from researchers or other expert survey developers on your questions, answers and scales prior to launching a survey live.

Price: Free

12. Feng-GUI

Feng-GUI image from Useful UsabilityLast but certainly not least is Feng-GUI, an automated eye tracking tool that simulates based on algorithms where a typical human might fixate as they view a page. Eye tracking provides data on what objects on the page are capturing the attention of your website visitors. Fixations on an object are noted and the data is provided in multiple useful ways including heatmaps, gaze plots, opacity maps and more. For more precise (but more costly) actual human-based eye tracking data consider paid tools such as the Tobii eye tracker.

Price: $25 and up depending on the plan.

Conclusion: 12 Critical Conversion Science Tools You Should Use

Conversion science is all about the data. And the above 12 critical conversion science tools will enable you to capture the data you need to make more informed and more productive analysis of what elements to test and optimize on your site. Using several or all of these tools is almost guaranteed to provide you with a rich assortment of behavioral data that will help you test and optimize your site on a continual basis. A word of caution, you may enjoy using these tools so much that you become a conversion science junky! Enjoy!

For more information on how to conduct conversion optimization using some or all of the above tools, read the Top 10 Conversion Optimization Tips the Experts Use.

Note: Companies mentioned in this article may or may not be advertisers on this site. However, in no way does their sponsorship or lack thereof impact the results of this or any other editorial content on the site. In all cases, the same rubric for evaluation is used to compare tools or services and all results reflect the outcome of the comparison without regard to whether a company is advertising on the site or not.

How not to throw out 125 Million dollars when creating applications

No-Usability-Equals-125-Million-Dollars-In-The-Trash-from-UsefulUsabilityDo NOT throw out 125 million dollars like Avon did after their application was deemed unusable. Instead, learn from their mistakes and protect your investment (and job). Here is how.

Avon will throw out 125 Million Dollars because it has cancelled an SAP implementation that took four years to develop, as recently reported by the Wall Street Journal.

According to the article in the Wall Street Journal (Avon’s Failed SAP Implementation Reflects Rise of Usability). Avon did so after field testing in Canada revealed that users found the software unusable, and started leaving Avon in droves.

According to the article…

“As the WSJ’s Drew Fitzgerald reported earlier today, Avon is pulling the plug on a $125 million software system rollout that has been in the works for four years after a test of the system in Canada drove away representatives the door-to-door beauty product company relies on to drive sales.

Avon began testing the new order management software system in Canada in the second quarter. While the new system based on software supplied by SAP AG worked as planned, it was so burdensome and disruptive to the representatives’ daily routine that they left in meaningful numbers. Avon relies on a direct sales model where its representatives aren’t employees, which makes it difficult to add new tasks associated with the software system.”


If you are a CIO and you like your job I have a simple tip for you…

“Conduct usability testing often and extensively when developing new applications.”

If you are a CIO you can stop reading now. Thanks for stopping by!

But for the rest of us, let us examine why Avon ended up throwing out $125,000,000.00 and four long years of work by what was probably a large team. It is called usability, or user experience. Without an easy to use and satisfying user experience your application will NOT be used by your users and you will have wasted your money.

As the article in the Wall Street Journal states…

“At a time when people are accustomed to using well-designed applications from companies such as Google Inc. and Apple Inc. in their personal lives, they have little patience for workplace applications that leave them confused. Functionality is no longer the definition of success. Usability is key.”

Just to repeat a very important phrase


Here are five tips you can use to make sure YOU do not throw out one hundred and twenty five million dollars:

1. Identify Your Personas

A Persona is a fictional representation of your typical users, and includes both behavioral and goal oriented information that is critical for design decisions. In a previous article I wrote about tips you can use to create Personas and how to avoid bad ones, but suffice it to say it is critical that you have Personas BEFORE starting any application development project.

2. Design for Persona Critical Tasks

Personas include information on the top 3 or so critical tasks they MUST do to be successful. Your application should include bold statements that failure to make those tasks brain-dead simple means failure of the application. Never forget about those critical tasks, and make sure your application design is focused on continually seeking ways to make those tasks simple, fast and super-easy.

3. Conduct Early Prototype Usability Testing

Conducting usability testing early and often is not just a catch-phrase. Early prototype testing includes testing wireframes and even conducting Card Sorts very early in the process. The data gathered from this testing will ensure your application design is focused from the users perspective. There are lots of free or pay card sorting tools that make conducting card sorting and creating information architectures easy. There is no excuse for not gathering this data. Testing paper wireframes is also a really easy but extremely helpful data point.

4. Test Often During Development

Testing often during your application development is another key to creating usable (and thus successful) applications. Using remote usability testing methods means testing can be done almost instantly, and data can be captured in hours, not days or weeks.  There are excellent books on how to conduct remote testing that make it fast and easy for even newbies to create and run usability tests. Using an agile method of application development? No problem, remote testing during sprints means never having to say you are sorry (to your spouse after you come home with a box of your stuff in hand because you just got fired for a bad application).

5. Include users in your team

An excellent idea that few organizations seem to use is including actual users as part of the application design team. Having a group of actual users that you reach out to for feedback and input will clear up disagreements and clarify your purpose as you move through your process. Panels of users are worth their weight in gold. Use the input and commentary you receive from actual users as you go through your sprints or waterfall process. Their input will ensure you are keeping your application on track from a usability perspective.

Conclusion on How to Not Throw Out 125 Million Dollars

By incorporating usability testing and conducting user-centered design as part of your application development process you will ensure your design is user-friendly and successful. Failure to do so risks the potential of your application not being used, which can waste 125 Million Dollars and four years of work.  As has been said before, Failure is Not an Option!

For more tips on how to include testing as part of an application development process read the article on 24 usability testing tools.

2 256

The Top 7 HealthCare.gov UX failures caused by poor forecasting of usage

Executive Summary: HealthCare.gov is experiencing a UX Failure due to inaccurate forecasting of usage, but this has deeper impact on attitudes about government health care in general.

Healthcare.gov UX FailureA recent Healthcare.gov UX study highlights the importance of forecasting simultaneous user sessions to ensure applications are accessible.

Failure in forecasting hurts user adoption and in this case increases already uneasy attitudes about the site specifically, and government-provided health care in general.

HealthCare.gov User Experience:

HealthCare.gov is the new government health care plan website that is now offered to U.S. citizens seeking health care insurance in the U.S. The site opened to the public on October 1st, 2013. Unfortunately, as many news stories about overloaded servers have already testified, there are serious and significant UX failures with the site.

The most grievous UX failure currently is the inability for users to access the site due to overloaded servers. As the following UX test reveals, this is potentially causing great harm in opinions about the government health care website specifically and the government provided health care plan in general.

Better simultaneous user session forecasting coupled with simple usability fixes prior to launch could have made for a much better user experience and resulting attitude about the site specifically, and the government health care plan in general.

The Healthcare.gov UX Test:

I was specifically interested in testing the Federal site Healthcare.gov, and not State specific sites such as the ones available in California, New York or other States that have their own websites.

To test the user experience of Healthcare.gov I created a simple usability test with three tasks that a typical person interested in the health care plans would need to perform. The questions the testers were presented with are as follows:

  1. Use this website to find the health care plan you think is best for you. Which plan is it?
  2. Use this website to determine how much the plan you think is best for you will cost. Can you clearly identify the differences in price, and the differences in what is covered?
  3. Use this website to start the application process, it is NOT necessary to complete the application process. Is this process easy or difficult?

I recruited testers from usertesting.com with the following demographics:

  • Testers should be located in one of the following states and should NOT have already signed up for a Healthcare.gov plan: Florida, Georgia, Illinois, Indiana, Texas, New Jersey
  • Age: 18 – 65+
  • Country: U.S. only
  • Gender: Any
  • Income: $0k – $150k+
  • Operating Systems: Any
  • Web Browsers: Any
  • Web Expertise: Any

Testing occurred across multiple days, States and times, from October 2nd through October 4th of 2013.

UX Test Results:

The results of the UX test demonstrated that even the simple processes of trying to find information about plans and costs were virtually impossible. This was due to the website being overloaded and thus incapable of providing users with the ability to perform the sign up process, the first step prior to finding plan and pricing information.

As this video below demonstrates, the ‘waiting page’ that testers landed on did not provide any information about how long they had to wait. Because of this lack of information, most testers abandoned the process after several minutes.

Click to play Healthcare.gov UX testing highlight video #1

Inadequate planning of usage by the team at HealthCare.gov is the prime culprit.

According to WikiPedia as of January 1, 2013, the United States had a total resident population of 316,783,000.

If just 1% of those people wanted to access this application after the October 1st launch of the program, that would be about 3,167,830 trying to use the site at more or less the same time.

Even if just one tenth of that 1% tried to access the site simultaneously, that would still be about 300,000 simultaneous site visitors.

Yet according to the USA Today news story, the HealthCare.gov team planned for peak sessions not even close to that number:

“U.S. Chief Technology Officer Todd Park said the government expected HealthCare.gov to draw 50,000 to 60,000 simultaneous users, but instead it has drawn as many as 250,000 at a time since it launched Oct. 1.

More than 8.1 million consumers visited the site from Tuesday through Friday, according to the White House.”

Website and Chat UX Failure Causes Consumer Confidence Issues:

Because of the website and chat system failure, most of the testers had concerns about the government’s abilities to administer health care insurance at all. This is a good example of the old saying that ‘you only have one chance for a good first impression.’

As this highlight video demonstrates, testers assumed that if the website and chat system were not working, the phone lines would be even worse:

Click to play Healthcare.gov ux testing highlight video #2

HealthCare.gov Receives a 2 out of Possible 10 Score:

Testers were overwhelmingly in agreement that the bad user experience of not being able to find basic plan information and pricing due to the non-working site earned HealthCare.gov a very low satisfaction rating. As the video below highlights, the user experience was rated extremely difficult:

Click to play Healthcare.gov UX testing highlight video #3

Other HealthCare.gov UX Issues:

HealthCare.gov has other usability issues that cause an overall bad UX score. Listed below are just a few of the other findings of this study:

1. Slow response time with no information about how long the queue is causes confusion, which leads to very high abandonment rates and re-tries, and thus adds additional server load.

 Healthcare.gov does not provide information on wait times which causes user confusion

2. Chat also experiences slow or no response time with no information about how long it will be before the person can receive an answer to a question.

3. All testers stated they were confident the phone line will be even worse, and do not attempt to use it.

4. Login instructions are confusing:

“The username is case sensitive. Choose a username that is 6-74
characters long and must contain a lowercase or capital letter, a
number, or one of these symbols _.@/-“

Several users stumbled on these instructions, and received multiple errors due to their User Name not complying with the instructions. Some wondered if one of the symbols could replace a number, because of the use of “or” for lowercase or capital letter, and a number “or” a symbol.

 Healthcare.gov does not follow typical user name UX practices causing user confusion

5. One tester who needed a User Name and Password reminder from the system received an email that had her user name in all caps, but she had entered initial cap and lower case. The system and thus the user’s password is case sensitive, the incorrect all caps User Name in the email caused the user to enter her User Name incorrectly as all caps, this caused the user to be unable to login to the system.

6. There is no detailed information letting the user know the steps in the process to create a login, or the specific set of tasks associated with creating an account and choosing a plan, or where the user is in that process.

7. All testers assumed they would be able to find plan information and pricing without having to apply. Several referred to how other health insurance companies handle plan and pricing quotes, using basic demographic data to provide coverage information prior to applying for coverage.

eHealth provides and example of what users expect when visiting HealthCare.gov
eHealth provides and example of what users expect when visiting HealthCare.gov

From the sampling of usability issues the site is already experiencing, we can only wonder about the other user experience issues awaiting those who successfully complete the application process.

This is a reminder that really smart people (and I’m sure the designers and developers of the Healthcare.gov user experience are some of the best out there) can still stumble when creating applications for large sets of users.

The best way to avoid this is with constant testing and evaluation throughout the process, and even then UX issues will still surface. Hopefully the UX team responsible for HealthCare.gov is addressing many of these issues as we read this.

Conclusion: HealthCare.gov UX Failures

Considering the impact and far reaching consequences HealthCare.gov will have on millions of Americans, it is disturbing that so many severe UX failures are present in the system at this late stage.

Unfortunately, the bad user experience the HealthCare.gov site is currently providing is impacting attitudes and perceptions about the site specifically, and about the government’s ability to adequately administer health care insurance in general.

This should be a strong reminder to everyone in the UX community that the user experience is the Brand. Without adequate planning and testing prior to launch, a bad first impression of a system will result in bad perceptions about the Brand. And sadly in this case, that Brand is the United States Government.

Let's Connect