Tags Posts tagged with "usability"


A Website Audit can improve your ROI (and respect) at least 7 different ways, here’s how:

Website audit funny quote

 Website Audit Definition

The definition of a website audit for UX purposes is to evaluate both behavioral data (visitor actions) and user experience data (task-flow other UX vs. best practices) to identify issues and recommend opportunities to improve conversion.

Take My Website – PLEASE!

Sometimes a website audit can make you feel like you get no respect, as if you are the Rodney Dangerfield of the web.

Side note: Odd how many companies do not respect their own websites. I’m not sure why that is, but a separate follow-up study may be needed.

In any event, I digress…

Remember his famous quote?

“My psychiatrist told me I’m going crazy. I told him, “If you don’t mind, I’d like a second opinion.” He said, “All right. You’re ugly too!””

- Rodney Dangerfield

But I’m here to assure you that a website audit, specifically an audit of your website’s UX, can and will make your website ROI better, and get you the respect from your website visitors (and bosses) that you deserve. Here’s how…

Website Audits Can Improve ROI

Improving the ROI of a website is best accomplished by ‘fixing the leaky bucket.’ What I mean by that is finding and fixing the issues that are causing website abandonment, task flow failure, poor visitor engagement and disappointing conversion.

There are many variations of website audits, and all have their unique value. These include:

  • Accessibility Audit
  • Analytics Audit
  • Conversion Optimization Audit
  • Page Speed Audit
  • SEO Audit
  • Usability Audit
  • Website Competitor Audit

However, I like to combine the best elements of several of the above audits into a comprehensive website audit that evaluates the larger user experience. This is a great way to ensure all the data available is utilized to analyze the website and make recommendations for optimization. ROI improvements based on recommendations from website audits are typically significant and quick.

Website Audit Elements

The two categories of information that a website audit evaluates are the critical elements of a well done audit:

Behavioral Data – Behavioral data will define what actions visitors are doing (or not doing) on the site. This data typically comes from website analytics tools such as Google Analytics, Adobe Analytics, Coremetrics, WebTrends, etc. Typically this data includes information such as:

  • Overall website performance conversion data
  • Paid search conversion data
  • Organic conversion data
  • High-level PPC keyword data
  • Website bounce rate
  • Visits by browser
  • Screen resolution
  • Top content
  • Content paths
  • Length of visits
  • Location (by geography)
  • Page fold
  • Devices
  • Operating Systems
  • And more…

User Experience Data – UX data includes information about how the website compares to usability and UX best practices. This data comes from tools such as the 5 second test, eye tracking, usability testing, and a comparison of the critical website interaction elements versus best practices (sometimes called a heuristic or website review).  This type of data typically includes information such as:

  • Elements that are or are not attracting attention
  • Page fold ramifications on CTAs or critical copy
  • Form field best practices vs. existing forms
  • Navigation flow and labeling
  • Product page elements vs. best practices
  • Contact Us page elements vs. best practices
  • And more…

Website Audit Findings and Analysis

The website audit includes a document with detailed findings and analysis of the behavioral and UX data that precisely defines where the website is performing well, and where there are opportunities for improvements. The subsequent recommendations are then tested, typically with A/B testing, to verify that the optimizations are having the desired benefit.

I always include screen shots of each of the items being audited, with callouts that explain what the issue and opportunity for testing could be. These can sometimes be fairly large documents, upwards of 70 to 90 pages. But because the information is presented one item at a time getting through the analysis document is easy, and fairly quick.

Advantages to Website Audits

There are at least 7 primary advantages to website audits, they include:

1. Ability to use website data to prioritize A/B testing: Nothing beats using your actual website and UX data to define and prioritize where your conversion is not optimized. This removes the guessing game that too often occurs as part of A/B or Multivariate testing.

2. Benchmarks existing versus potential conversion: By using the data from a website audit as a benchmark, the down-stream changes to traffic, navigation flow and conversion can be quantified. Benchmarking removes much of the guess work out of determining if conversion improvements based on testing are temporary, or permanent.

3. Pinpoints issues: The analysis of the website audit provides pinpoint clarity on which elements of a page, form or flow are potentially hindering performance. Specific recommendations can be extremely detailed, which helps focus where to spend testing resources. If you’ve ever heard the principle of “don’t throw out the baby with the bath water” you’ll understand and appreciate the detailed approach of a website audit

4. Clearly defines success: Many of my clients appreciate the incremental value that comes from the analysis of a website audit, in that they appreciate knowing what ‘success’ looks like. When A/B testing, how do you know you’ve truly optimized the ‘B’ version to achieve maximum success? Unless you have data that defines what that success looks like, you don’t really know how successful the test could be. A website audit provides that ability to better define success.

5. Doesn’t ‘fix’ what’s not broken: Too often companies optimize using an ad-hoc approach to testing. Sometimes this causes something that was working just fine before, to work poorly, or not at all. A website audit helps define what’s working from what’s not, so that items that are working well are left alone, and items that are not are tested and optimized.

6. Maximizes resources: Unless you have a full time optimization team, the odds are that testing is but a small part of your overall work activities. Maximizing your time and other resources is crucial to optimizing ROI. Spending time optimizing only that which needs to be optimized makes your resources that much more productive.

7. Proves your value (and earns you respect): Facing your bosses and answering to them for how you specifically are helping the company can sometimes be a challenge. Having the demonstrated results from the testing that comes from a usability audit provides you with a plethora of actual data that precisely defines how you are contributing to ROI. Nothing speaks better to your bosses than data that proves you (and thus they) are adding value to the company. You want respect, give them optimization numbers and you’ll get it!

Conclusion:  Website Audit Improves ROI and Respect

A website audit and the resulting optimization of the conversion of the site can greatly improve the ROI and performance of a site. It can also provide you with the respect you need. If you have any questions or would like more information on how you can use a website audit to improve your website ROI just contact me. By using the results of the website audit, you will have a more informed, prioritized and clearly defined road to improving the success of your website.

17 2480

A Comprehensive Review and Matrix of 14 Usability Testing Tools

Comprehensive review and matrix of 14 usability testing toolsIn 2009 I wrote a popular article on 24 usability testing tools. Since that time, there has been tremendous growth in the quantity and capability of usability testing tools. An update to the article is due, but because of the broad array of tools now available a more formalized definition and a matrix of usability tools is needed.  To that end, the following matrix and comprehensive review of 14 usability testing tools will help practitioners who are looking for tools for specific testing tasks.

Usability Testing Tools Matrix

For purposes of classifying tools to evaluate in this matrix, the tools must have three critical functions necessary to conduct a classically defined usability test:

  1. Audio and video recordings of testers (ala Think Out Loud method)
  2. Ability to edit recordings of tester sessions into highlight reels
  3. Capability to test using mobile devices (Smartphones and tablets)

The classical definition of usability testing is a systematic observation under controlled conditions to evaluate how well testers can complete critical tasks.

Thus, having the ability to record testers, to see their interactions and hear their thoughts (ala the Think Out Loud method) is critical to the observation process. It is also important to be able to edit those recordings, so that key insights can be shared with others in the form of highlight reels. Finally, the increasing adoption and usage of mobile devices to interact with websites and applications makes it necessary to conduct usability tests of the mobile user experience. Thus, tools should be able to test the mobile as well as PC based experience.

Not included in this review are Feedback or Click Tracking tools. This is because they only provide feedback or opinions about user interfaces, and lack the ability to observe testers conducting tasks. I will conduct the same comprehensive review of Feedback and Click Tracking tools in a future article.

Usability testing tools matrix from UsefulUsability.com

Four Quadrants of the Usability Testing Tools Matrix

Usability testing tools are divided into four quadrants on the matrix, based on where they fall when evaluated against two critical attributes:

  • Depth of Technology: What functionality and features are available with the tool, and how deep does the array of features run? Tools that score higher going up the vertical Depth of Technology axis have more functions and features.
  • Tool Impact: This is a measure of the ability of the tool to impact overall testing, and thus the observations and eventual recommendations for usability improvements. Tools that score higher (going to the right) on the Tool Impact axis have the ability to have a greater impact on overall testing results.

Based on these definitions, tools can fall into one of four quadrants on the matrix: Authority, Contender, Innovator and Niche.

  • Authority tools are those that provide a greater amount of usability testing functionality as well as a higher potential impact on discovering, diagnosing and ultimately optimizing the user experience.
  • Contender tools have the ability to provide a greater impact in terms of their depth of technology, but may have more limited potential impact in discovering, diagnosing and optimizing the user experience
  • Innovator tools have a more limited depth of technology, but have the potential to have a greater impact in usability testing and recommendations
  • Niche tools have a relatively limited depth of technology as well as a lower potential impact on usability testing and recommendations results, but can still be very useful for the specific tasks they were created for

All of these tools have specific features that make them unique.  Depending on your testing needs, any of these tools may be good choices and can compliment your toolkit.

Mobile Usability Testing Still Rudimentary

The upper right corner (Authority) of the matrix is blank for a very good reason. It represents the current lack of a fully integrated mobile usability testing solution. In my opinion we still lack a comprehensive, fully integrated mobile device usability testing tool, something that records both mobile device screen interaction and the audio/video of testers as they try to accomplish tasks. Because they are mobile devices, testing in this manner should be possible anywhere the tester goes, at any time, and using any smartphone or tablet (as long as it has a camera). Real time streaming of the screen and audio/video of the tester should be available for remote teams.

Mobile usability testing is currently in a neophyte stage. Yes, there are cludgy and cumbersome ways to test usability with a mobile device, typically requiring testers to place their phone on a desk with a camera hovering over it (thus not being at all ‘mobile’). Most methods today lack the sophistication of being a truly integrated and mobile experience.

Consider how just a few years ago new tools like Camtasia, Morae, UserTesting and others made usability testing websites an anytime, anywhere, quick and cost effective solution versus using expensive and cludgy labs or home grown equipment. We need that same fully integrated experience for mobile, but we do not have it, yet.

Authority Usability Testing Tools:

Usability testing tools matrix from UsefulUsability.com

Ovo Logger

Ovo Logger image from UsefulUsability.comOvo Logger by Ovo Studios can be considered to be the Rolls Royce of usability testing solutions. This is the tool for large Fortune 1000 enterprises or usability testing studios that conduct a high volume of testing and have teams spread around the globe. Typically these situations call for the need to have a robust tool to capture, edit and share results. It should be noted that Ovo Studios creates custom usability testing labs, and the Ovo Logger fits in well with those labs. The tool scores at the top of comparable tools for depth of technology, but is slightly limited in terms of impact due to requiring a dedicated PC, and being somewhat expensive to deploy.

Pros: Enterprise level usability testing software and hardware that records multiple inputs from PCs or Macs (cameras, microphones, screen interaction) and streams the output to observers. Editing tools enable real-time analysis of tests with the ability of observers to document important points. Can capture un-tethered iPads, non-jail broken iPhones and Android devices if using the Logger mobile app.

Cons: Designed specifically for larger firms with multiple locations and or usability teams. Requires a dedicated PC to run the software. Pricing is more expensive versus other usability testing tools and is based on number of sources, associated lab requirements, etc.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $3,000 and up depending on scope of associated usability lab

Camtasia plus a Screen Sharing Tool

Camtasia image from UsefulUsability.comA very powerful Do It Yourself approach for remote usability testing is to use a screen capture tool such as Camtasia (from TechSmith) coupled with a screen sharing tool such as GoToMeeting, WebEx or similar applications. The major benefit of this approach is the ability to conduct moderated remote tests, because the practitioner has a direct connection to the participant. In addition, the recording of the screen interaction with the voice and or video of the tester is immediately available for editing on the practitioners computer. The downside to this approach is lack of a simple mobile solution.

Pros: Enables remote moderated usability testing, because the practitioner has a direct connection to the participant. Sessions can be shared with other remote observers using the screen sharing tool. Audio and video recordings of the screen interaction with the voice of the tester are immediately available for editing on the practitioners computer.

Cons: For part-time or ad-hoc uses, this approach may be more expensive than using some of the other remote unmoderated tools available from vendors. Screen sharing tools are notorious for occasionally having technical problems that cause poor connection performance, or worst case dropping the connection entirely. This is not an optimal solution for conducting mobile usability testing, although cludgy methods of using a webcam pointed at the testers mobile device can be tried.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $299 for Camtasia, $19/month for GoToMeeting or $24/month for Webex


UserTesting image from UsefulUsability.comUserTesting provides a network of pre-screened testers who can meet several specified demographics. Or, practitioners can invite their own testers using the tool. Practitioners create a set of tasks for the testers to complete, either on your website or even competitor sites. Testers screen interaction and voices are recorded. UserTesting includes powerful and easy to use video editing tools to create highlight reels. UserTesting also provides mobile usability testing, which is a major feature considering the increasing utilization of mobile devices. Written responses to questions are included. UserTesting is one of the most robust tools to use for remote unmoderated usability testing and thus scores higher  for depth of technology and testing impact on the matrix.

Pros: Relatively low cost way to conduct remote unmoderated usability testing. Audio and video of the tester are recorded, and editing tools enable highlight reels to be created easily and quickly. Mobile device usability testing is available. The large network of testers typically means testing session results are available in an hour or two. Demographic filtering tools enable finding the appropriate testers for most Persona situations.

Cons: Because UserTesting pays small fees to the testers for each completed test, it is critical to carefully create non-biased tasks, else you risk the tester trying to cut corners to accomplish the test as quickly as possible. As a practitioner, you must watch for this kind of activity and request a re-do of that test if you are concerned about the quality of the tester. As an alternative, using your own testers can alleviate much of this. Mobile testing is available, but requires the tester to be at a desk so a webcam can be pointed down to the screen, which is not typically how most people use their mobile devices.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $49 per test with bulk pricing discounts available


Morae image from UsefulUsability.comMorae from Techsmith can be considered the Granddaddy of usability testing tools. This is because it was the first tool specifically created for usability testing. Even today it is still one of the best. It provides a comprehensive set of features available to record, analyze, edit and share usability testing sessions. Morae is an application that resides on the practitioners computer (PC only, sorry Macs). Morae records all screen interaction, including the testers voice and or face if using a webcam. Testers can be on any Mac or PC if testing websites or applications. Morae includes very powerful documentation, editing and sharing tools. Morae scores fairly high for depth of technology, and also fairly high for testing impact, the limiters being lack of a Mac practitioner solution and the issue of cludgy mobile testing.

Pros: Powerful screen capture and editing tools make this the choice for larger scale usability testing. Morae includes additional tools that can track a variety of screen and click interactions, plus has the ability to incorporate survey/opinion based data. Sharing test sessions with other practitioners and testing results with teams is relatively easy and quick.

Cons: Desktop & Laptop based system with no ability to record Mobile device screen interaction directly. Relatively expensive and PC-only (not Mac compatible) for the Manager and Recorder tools.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $1,995

Contender Usability Testing Tools:

Usability testing tools matrix from UsefulUsability.com


Userlytics image from UsefulUsability.comUserlytics provides both PC and Mobile based remote unmoderated usability testing. For mobile testing, the more expensive Gold or Enterprise plans are required. Testers can come from the Userlytics panel, or from testers you provide. Screen interaction, tester voices and webcam video images of testers can be recorded. Written answers to post test questions are also included.  The depth of technology is good, but because of the requirement to use the more expensive Gold plans for downloading and editing of videos, capturing of sessions longer than 15 minutes and testing mobile, this tools has somewhat limited testing impact versus some competitors. Practitioners that conduct larger volumes of tests should consider Userlytics.

Pros: Userlytics is relatively low in cost, and is definitely easy to set up and use. The more expensive plans allow download of recordings for editing and sharing. In addition, images, videos and other assets can be uploaded to the Userlytics secure servers for early prototype testing. White label testing results sharing is possible for Gold and above plans.

Cons: Userlytics has the same issues as any other remote unmoderated usability testing tool, with the burden being on the test creator to ensure non-biased questions. Userlytics has a rather confusing set of test functionality based on plans (Basic, Silver, Gold, Enterprise). For example, tests recordings are less than 15 minutes for sub-Gold plans, and mobile testing is only available on Gold and above plans. Depending on the plan (Silver or above), each monthly subscription price includes one or more ‘free’ tests, with each additional test costing $33 or less per test. There are also limits in terms of number of post test questions that can be asked based on plans.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? Yes

Price: $39 per month or more with bulk discount pricing available


UserFeel image from UsefulUsability.comUserFeel.com provides an assortment of testers, many of whom speak languages other than English, for remote unmoderated usability testing. Having the ability to choose a language other than English (many are offered) is a great benefit for testers of websites that are targeted for other countries. Written responses to post test questions are also available. Videos are available of the test on the website (but not available for download for editing purposes), and a white label playback option of the video is free. The depth of technology is good, but due to the lack of mobile testing and inability to download test recordings the tool impact is lower than some competitors. For those conducting usability testing in languages other than English, this could be a good solution.

Pros: A good tool for non-English language remote unmoderated usability testing. Demographic and language filtering tools are available. Pricing is relatively low and post test questionnaire functionality is available.  The free white label playback of the test recordings is a plus.

Cons: Like other remote unmoderated usability tools, practitioners must be careful to create non-biased tasks or else risk having testers try to quickly finish the test, causing poor test results. The advanced video editing capabilities of some competitor tools are not available. Videos cannot be downloaded and edited. Mobile testing is not available.

Tester Audio and Video Recordings? Yes

Recordings can be edited? No

Mobile testing capable? No

Price: $39 per month with bulk discount pricing available

Innovator Usability Testing Tools


OpenHallway image from UsefulUsability.comOpenHallway is a remote unmoderated usability testing tool for PC or Mac-based research. Testers come from the practitioner (no panel is offered) via a link that is generated by OpenHallway when the scenario and tasks are set up. OpenHallway requires no code to use, which makes it a great tool for testing applications, websites and even competitor websites. Screen interaction is recorded, along with voice and or the testers face, if the tester has a webcam. Plans (Basic, Plus and Premium) determine the amount of video storage available. Unlimited number of test sessions are available, however the video storage amount will act as the limiter to how many sessions are actually conducted.  Because of video storage limits, and videos being MP4 only, and no mobile solution, OpenHallway has somewhat more limited depth of technology than some competitors, however the tool impact is good in that it can be used on almost any website with no code required.

Pros: OpenHallway is simple to use, easy to set up and therefore is a handy tool for recording unmoderated remote usability tests. As of the writing of this article there is a free trial.

Cons: OpenHallway is PC or Mac compatible, but does not support mobile.  Although no code is required, Java must be present on the tester computer for the tool to work. Videos are MP4 only, and only the Premium plan ($199 per month) allows downloading (and thus editing) of the video.

Tester Audio and Video Recordings? Yes

Recordings can be edited? Yes

Mobile testing capable? No

Price: $49 per month or higher, depending on the plan


UsabilityTools image from UsefulUsability.comUsability provides a variety of usability, feedback and click tracking tools, with the primary advantage being they all use a single practitioner interface. This makes creating tests that require multiple tools much easier. There are two versions, a UX Suite and Conversion Suite. The UX Suite contains click testing, web testing, survey, card sorting and a Persona Creator. Practitioners can set up web tests that provide scenarios to website visitors, who then try to accomplish the tasks and click success or abandon buttons based on whether they thought they could accomplish the task. Results are provided in data and charts. No code is required unless you choose to utilize their website intercept request on your site. The lack of tester audio and video recordings for PC and mobile is a limiter for depth of technology.  However, having the ability to utilize the tools efficiently and quickly provides a good tool impact.

Pros: A toolkit of the most common tools needed to fully understand the user experience. Having all tools in a single practitioner interface streamlines the process of creating and running multiple tool tests. In larger quantity testing scenarios, this set of tools can be more economical than using separate usability tools.

Cons: UsabilityTools does not provide audio and or video recordings of tester sessions. Mobile interaction using audio and video recordings is also not available. Pricing is not available on their website, although as of Sept. 2013 they had one time fees ($300) and monthly fees starting at $69 for the Professional version. Free panels of testers are not offered, although paid sources are available.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? No

Price: Pricing information is not available on their website.


Loop11 image from UsefulUsability.comLoop11 provides the ability to conduct unmoderated remote usability testing on PCs, Macs or mobile devices. The tool is easy to set up and fairly easy to administer (no code required). A key advantage is it can provide real-time data. Loop11 provides useful information for evaluating task flow on websites, including competitor websites. No audio and video recordings of unique test sessions are available, and because of this the depth of technology, and tool impact are lower than some competitors.

Pros: Loop11 requires no code, thus is useful for conducting comparison tests of user experience on competitor or other websites. Setting up and running tests is relatively quick and easy. Real time information can be advantageous for situations where quick results are needed.

Cons: As with other unmoderated tools Loop11 does not allow interaction with the testers to probe or ask follow-up questions. The test creator has to find his or her own test participants. No ability to record the tester via ‘think out loud’ methodology, something that exists with competitor tools. Somewhat expensive versus competitors.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? Yes

Price: $350 per project or $158 per month


UserZoom image from UsefulUsability.comUserZoom includes multiple tools including remote unmoderated testing on PCs and mobile devices (via UZ Mobile), along with other tools such as card sorts, tree testing, screenshot click testing and more. For the remote unmoderated usability testing tool UserZoom enables tasks to be created and deployed to either 3rd party panels or the practitioner’s own panel. Intercepts are also available, a handy feature when trying to reach actual visitors to a website. UserZoom does include video recordings of screen interaction, but only at the pricey $9,000 per year or higher plans, and no voice capture of testers are available, thus Think Out Loud recordings of users explaining their actions are not captured. This limits the depth of technology and tool impact when compared to some competitors.

Pros: UserZoom includes powerful reporting and analytics tools that can greatly help researchers synthesize and analyze post test data, including heatmaps, time on task reports and much more. Plans allow practitioners to pick the best price point and functionality for their needs. Plans include a free Pilot, UZ Proof of Concept at $1,000, Basic plan at $9,000, Pro plan at $29,000 and an Enterprise plan that requires a custom quote. As of the writing of this article UserZoom offers a free pilot study.

Cons: UserZoom does not enable voice and or video capture of the tester as they try to accomplish a task, a major differentiator compared to competitor services. UserZoom does not have its own panel of testers available for free, as other competitors do. Practitioners can view and edit recordings of testers while they were conducting tasks to make highlight reels, but there is no audio nor video capture of the actual tester. UserZoom is rather expensive versus its competitors, and lacking the ability to hear and see testers via session recordings while they were conducting the task is another major differentiator between UserZoom and other remote unmoderated usability testing tools.

Tester Audio and Video Recordings? No

Recordings can be edited? Yes, but only recordings of screen interaction (no tester voice is available)

Mobile testing capable? Yes

Price: $9,000 for the Basic plan and up

Niche Usability Testing Tools

Usability testing tools matrix from UsefulUsability.com


TryMyUI image from UsefulUsability.comTrymyUI.com is another of the many SaaS-based remote unmoderated usability testing tools. Testers create tasks, and videos are provided that include screen interaction and the testers voice. Written answers to post questions are also included. As of the writing of this article a free trial is included. Practitioners can use their own testers, but at a license cost of $99 for a single use test. Because video editing and mobile testing are not provided, the depth of technology and testing impact are lower than some competitors.

Pros: TryMyUI provides an alternative to other remote unmoderated tools that is slightly cheaper than many competitors. As a tool that provides video and audio recording of testers as they try to complete tasks it fits the classical definition of a usability testing tool.

Cons: All the same issues with remote unmoderated usability testing apply, with the burden being on the practitioner to create non-biased questions, and to carefully analyze the results to ensure the tester was not trying to use short cuts to complete the test. Video editing is not available, nor is mobile testing.

Tester Audio and Video Recordings? Yes

Recordings can be edited? No

Mobile testing capable? No

Price: $35 per test with bulk discount pricing available


WebEffective image from UsefulUsability.comWebEffective by Keynote Systems provides remote, unmoderated usability testing via tasks that are assigned and recorded. A panel is available, or practitioners can use their own testers. Data including click stream information is captured and displayed in a variety of reports, which can be downloaded. Individual sessions showing screen interaction are captured, however there is no audio nor video commentary captured of the tester as they go through the tasks.  Mobile testing is available. The lack of audio and video of testers as they try to complete tasks means WebEffective has a lower depth of technology and testing impact than some of its competitors.

Pros: A powerful tool to capture user interaction on websites or mobile devices, with a wide variety of reports available.

Cons: WebEffective does not provide audio and or video recordings of tester sessions. In this regard, WebEffective operates more as a click stream tool than a true usability testing tool.

Tester Audio and Video Recordings? No

Recordings can be edited? No

Mobile testing capable? Yes

Price: Pricing is not available on the Keynote WebEffective website

Ovo Solo

OvoSolo image by UsefulUsability.comThe Ovo Solo tool by Ovo Studios provides a small business version of Ovo Logger that enables screen interaction video capture along with the testers face and voice.  Simple logging controls enable analysis of the resulting test videos. However the tool is PC only. The relatively low price point makes this a good candidate for practitioners on a tight budget. Because of the PC only nature of the tool, the basic editing functions and the inability to record mobile sessions, this tool has lower depth of technology and testing impact than some of its competitors.

Pros: The relatively low price point makes this a very affordable way to capture, edit and display usability testing sessions.

Cons: This tool is PC only and is not able to record screen interaction on mobile devices. Editing tools are basic and may not meet more advanced usability tester needs.

Tester Audio and Video Recordings? Yes (but tool is PC only)

Recordings can be edited? Yes

Mobile testing capable? No

Price: $99


Silverback image by UsefulUsability.comFor Mac users, Silverback from Clearleft enables remote and in-person moderated usability testing. Silverback is not available for PC based practitioners. Session screen interaction and audio and or video of the tester are recorded, and simple editing features like task tagging are available. Recordings are saved in QuickTime, which makes editing of highlight videos relatively easy for Mac users. The lack of PC functionality and video editing capability means this tool has lower depth of technology and testing impact than some other competitors. Mac practitioners will find Silverback a very easy to use tool at a very affordable price.

Pros: A relatively low cost and easy to use tool for Mac users that provides the basics in terms of classical usability testing functionality.

Cons: Mac desktop and laptop only, does not work with PCs. No ability to record mobile device screen interaction directly. As of the writing of this article currently does not support FaceTime camera with newer Macbooks, but the app is being re-written to fix this. Lacks powerful editing tools available in other applications.

Tester Audio and Video Recordings? Yes (but tool is Mac only)

Recordings can be edited? No

Mobile testing capable? Yes

Price: $69.95

14 Usability Testing Tools Conclusion

Usability testing tools matrix from UsefulUsability.com

Any of the 14 usability testing tools reviewed here can provide excellent results if applied in a manner which best fits their unique functionality and features.

If Your Favorite Tool is Missing

Not seeing your tool here? It could be because I’m classifying it as a Feedback or Click Tracking analytics type of tool. My original 24 usability testing tools article included many of these tools, but also included some tools that can better be defined as Feedback or Click Tracking analytics tools. In future articles I will provide a matrix and reviews separately for those types of tools. And if your tool is not a feedback or click tracking tool, let me know by leaving a comment.

Note: Companies mentioned in this article may or may not be advertisers on this site. However, in no way does their sponsorship or lack thereof impact the results of this or any other editorial content on the site. In all cases, the same rubric for evaluation is used to compare tools or services and all results reflect the outcome of the comparison without regard to whether a company is advertising on the site or not.

12 Critical Conversion Science Tools You Should Use to Improve your Website Testing and Optimization

Conversion Science and Data

12-Conversion-Science-Tools from Useful UsabilityConversion science is all about using actual behavioral and task-flow data from your website or applications to make informed decisions about what to test and optimize. But where do you get the data needed? This listing of the 12 conversion science tools will provide you with many of the top tools the experts use to obtain behavioral and usability data. It is this data that is needed, and used, to create informed prioritization of conversion optimization tests. Here then in random order are the 12 critical conversion science tools…

1. Google Analytics

Google Analytics from Useful UsabilityGoogle Analytics (or GA as it is commonly referred to among frequent users) is a free data capture and analysis tool from Google that enables tracking of user behavior across websites. Having and analyzing behavioral data is critical to conversion science. Google Analytics enables conversion data tracking that is essential for making informed decisions about optimizations. Google Analytics or paid versions of similar tools such as Adobe Analytics, CoreMetrics, WebTrends and the like are essential tools for data-based conversion optimization.

Google Analytics also provides a tool to conduct simple A/B testing (call Experiments) which operates well enough for the price (free), although there are more sophisticated paid A/B testing tools available like Optimizely or Maxymiser.

Price: Free

2. Crazy Egg

CrazyEgg image from useful usabilityCrazyEgg provides click tracking and scroll maps of your website, extremely important information for conducting conversion science testing and optimization. Knowing where your visitors are clicking, or just as important where they are NOT clicking, is essential for evaluating behavior on and among pages. Additional tools like the Confetti report enable tracking of keyword data based on clicks, to help you define which audiences are interested in what topics and where in your site they go. Other tools similar to CrazyEgg are available such as Mouseflow, ClickTale and more.

Price: $9 per month and up depending on the plan, first 30 days free

3. 5 Second Test

5 Second Test image from Useful UsabilityThe Five Second Test tool offered by UsabilityHub is a great way to test your home page, or any page of your website, to learn if it is communicating effectively with your visitors. Studies have shown that most visitors make up their minds about a website well within 5 seconds. So this tool enables testers to see an image of your website page for 5 seconds, and then answer questions about it. What questions? Who you are, what you provide and why they should care are three important questions I like to test. Use this tool to learn if your website is communicating your value proposition effectively. The data captured is extremely helpful for optimization and related conversion science purposes.

Price: $20 per month and up depending on the plan, free if you earn Karma points by participating in 5 second tests other users have set up.

4. UserTesting

UserTesting image from Useful UsabilityUserTesting.com provides a powerful way to quickly capture unmoderated usability testing data. Nothing beats observing real users interacting with your website while they try to complete tasks. Seeing your actual users stumble over certain tasks will quickly help you define areas on your site that need optimization. This kind of conversion science data is often overlooked by many firms that conduct A/B testing, but in fact is extremely helpful for optimized A/B testing success. Rather than guess what to A/B test, use UserTesting.com data and analysis to find and prioritize task flow failure points. Use this information to then set up A/B tests. UserTesting enables testing of any website or mobile experience, including testing your competitors! Other tools akin to UserTesting include Usabilla and UserZoom.

Price: $49 per tester with discounts available for pre-paying larger numbers of tests.

5. DeviceAnywhere

DeviceAnywhere image from Useful UsabilityDeviceAnywhere is a very helpful tool for evaluating websites on real mobile devices. More and more websites and applications are viewed and interacted with using mobile devices. Evaluating the user interface using this mobile tool will provide extremely helpful data for optimizing the mobile experience.  This is not an emulator. Unlike other free services, these are real mobile devices, so you can quickly see how your mobile website will render across a variety of popular smartphones.

Price: The Spot-checking tool is free, paid versions for developers and Enterprise-level usage are available starting at $260 per month and up, depending on the plan.

6. iPhoneTester

iPhone Tester image from Useful UsabilityThe iPhoneTester is an emulator that enables quick checks of your website content on an iPhone interface (3G and 4G) as well as on the iPod Touch. It is quick, easy and best of all free. Just enter your URL into the emulator and what you will see is a close approximation of what your iPhone visitors will see. It is a great way to quickly validate what your mobile user experience will be like. And considering the plethora of iPhones currently being used to access website content it is a very good idea to check your site from an iPhone usage perspective. I fully expect this year to be the transition year from mobile testing being a nice to have to mobile testing being a requirement of any optimization plan.

Price: Free

7. GoToMeeting

GoToMeeting image from Useful UsabilityGoToMeeting is a very helpful tool for conducting remote moderated usability testing of websites to capture essential task flow data. It is actually part of the tool set, the other tool being a screen video capturing utility like my favorite, Camtasia Studio (explained below). By using GoToMeeting with a usability test participant, you can enable the tester to screen share his or her screen while they go through the exercises of trying to accomplish critical tasks on your website. Recording the tester screen interaction and voice as they go through the test will provide data on where tasks are failing (much like UserTesting). However, being present during testing enables you to go one better in that you can stop the tester or ask follow-up questions as you go through the testing exercise. Data from this type of testing is extremely helpful in finding and prioritizing task flow errors, which ultimately can be optimized through A/B testing of improved interfaces.

Price: $19 per month or more depending on the plan, free 30 day trial

8. Camtasia Studio

Camtasia Studio image from Useful UsabilityCamtasia Studio by TechSmith is the second half of the remote moderated usability testing toolkit. Coupled with GoToMeeting, it enables real-time video screen capturing of the tester as he or she conducts usability testing on your website. All screen interactions (and voice) can be captured to gather usability data on critical tasks. And in addition, Camtasia Studio provides powerful but easy to use video editing tools that help create professional videos quickly and efficiently. Use Camtasia Studio to record on-screen activity, customize and edit content, add interactive elements or import media, and share your videos with anyone, on nearly any device. Conversion science is all about using data, and Camtasia enables capturing of interaction data in real time. If a picture is worth a thousand words then video of actual users on your site is worth a million.

Price: $299 per license or less if multiple copies are purchased.

9. Ethnio

Ethnio image from Useful UsabilityEthnio is a website intercept tool that enables you to invite your actual website visitors to participate in a survey or website usability test. It is an excellent way to get data from those all important actual users on your website. The tool includes screeners and the ability to manage your website intercept by a number of parameters including time of day and more. Conversion science is all about real data from real users, and this tool enables finding those users. The tool is simple to use, easy to set up, and makes finding testers easy and quick.

Price: $49 per month or more depending on the plan.

10. SnagIt

SnagIt image from Useful UsabilitySnagIt is another very helpful tool in the conversion science toolkit. By capturing screen shots of an interface and editing them with the built-in SnagIt photo editing tools you can quickly highlight areas of interest, move elements around on the page to create B version mock-ups, and add helpful callouts that specify where on the page the issue is. By demonstrating issues and opportunities using actual screen shots of your website, you will be able to communicate very effectively with your teams as you create new B versions of web pages or elements on a page to test. Visualizations of optimizations are a conversion science best practice.

Price: $49.95 per license or cheaper for multi-license purchases.

11. SurveyMonkey:

SurveyMonkey image from Useful UsabilitySurveyMonkey provides the ability to capture qualitative and quantitative data quickly and easily. The science part of conversion science is based on having actionable and valid data. The best way to capture large amounts of statistically significant data is with surveys of prospects and or customers. This data is extremely useful when evaluating qualitative aspects of your web user experience, and can help guide your prioritization of conversion optimizations. SurveyMonkey enables almost anyone to be able to quickly and easily set up surveys for capturing that all important data. A note of caution about surveys, asking non-biased questions that provide actionable data is harder than it seems. It is a good idea to get second opinions from researchers or other expert survey developers on your questions, answers and scales prior to launching a survey live.

Price: Free

12. Feng-GUI

Feng-GUI image from Useful UsabilityLast but certainly not least is Feng-GUI, an automated eye tracking tool that simulates based on algorithms where a typical human might fixate as they view a page. Eye tracking provides data on what objects on the page are capturing the attention of your website visitors. Fixations on an object are noted and the data is provided in multiple useful ways including heatmaps, gaze plots, opacity maps and more. For more precise (but more costly) actual human-based eye tracking data consider paid tools such as the Tobii eye tracker.

Price: $25 and up depending on the plan.

Conclusion: 12 Critical Conversion Science Tools You Should Use

Conversion science is all about the data. And the above 12 critical conversion science tools will enable you to capture the data you need to make more informed and more productive analysis of what elements to test and optimize on your site. Using several or all of these tools is almost guaranteed to provide you with a rich assortment of behavioral data that will help you test and optimize your site on a continual basis. A word of caution, you may enjoy using these tools so much that you become a conversion science junky! Enjoy!

For more information on how to conduct conversion optimization using some or all of the above tools, read the Top 10 Conversion Optimization Tips the Experts Use.

Note: Companies mentioned in this article may or may not be advertisers on this site. However, in no way does their sponsorship or lack thereof impact the results of this or any other editorial content on the site. In all cases, the same rubric for evaluation is used to compare tools or services and all results reflect the outcome of the comparison without regard to whether a company is advertising on the site or not.

How not to throw out 125 Million dollars when creating applications

No-Usability-Equals-125-Million-Dollars-In-The-Trash-from-UsefulUsabilityDo NOT throw out 125 million dollars like Avon did after their application was deemed unusable. Instead, learn from their mistakes and protect your investment (and job). Here is how.

Avon will throw out 125 Million Dollars because it has cancelled an SAP implementation that took four years to develop, as recently reported by the Wall Street Journal.

According to the article in the Wall Street Journal (Avon’s Failed SAP Implementation Reflects Rise of Usability). Avon did so after field testing in Canada revealed that users found the software unusable, and started leaving Avon in droves.

According to the article…

“As the WSJ’s Drew Fitzgerald reported earlier today, Avon is pulling the plug on a $125 million software system rollout that has been in the works for four years after a test of the system in Canada drove away representatives the door-to-door beauty product company relies on to drive sales.

Avon began testing the new order management software system in Canada in the second quarter. While the new system based on software supplied by SAP AG worked as planned, it was so burdensome and disruptive to the representatives’ daily routine that they left in meaningful numbers. Avon relies on a direct sales model where its representatives aren’t employees, which makes it difficult to add new tasks associated with the software system.”


If you are a CIO and you like your job I have a simple tip for you…

“Conduct usability testing often and extensively when developing new applications.”

If you are a CIO you can stop reading now. Thanks for stopping by!

But for the rest of us, let us examine why Avon ended up throwing out $125,000,000.00 and four long years of work by what was probably a large team. It is called usability, or user experience. Without an easy to use and satisfying user experience your application will NOT be used by your users and you will have wasted your money.

As the article in the Wall Street Journal states…

“At a time when people are accustomed to using well-designed applications from companies such as Google Inc. and Apple Inc. in their personal lives, they have little patience for workplace applications that leave them confused. Functionality is no longer the definition of success. Usability is key.”

Just to repeat a very important phrase


Here are five tips you can use to make sure YOU do not throw out one hundred and twenty five million dollars:

1. Identify Your Personas

A Persona is a fictional representation of your typical users, and includes both behavioral and goal oriented information that is critical for design decisions. In a previous article I wrote about tips you can use to create Personas and how to avoid bad ones, but suffice it to say it is critical that you have Personas BEFORE starting any application development project.

2. Design for Persona Critical Tasks

Personas include information on the top 3 or so critical tasks they MUST do to be successful. Your application should include bold statements that failure to make those tasks brain-dead simple means failure of the application. Never forget about those critical tasks, and make sure your application design is focused on continually seeking ways to make those tasks simple, fast and super-easy.

3. Conduct Early Prototype Usability Testing

Conducting usability testing early and often is not just a catch-phrase. Early prototype testing includes testing wireframes and even conducting Card Sorts very early in the process. The data gathered from this testing will ensure your application design is focused from the users perspective. There are lots of free or pay card sorting tools that make conducting card sorting and creating information architectures easy. There is no excuse for not gathering this data. Testing paper wireframes is also a really easy but extremely helpful data point.

4. Test Often During Development

Testing often during your application development is another key to creating usable (and thus successful) applications. Using remote usability testing methods means testing can be done almost instantly, and data can be captured in hours, not days or weeks.  There are excellent books on how to conduct remote testing that make it fast and easy for even newbies to create and run usability tests. Using an agile method of application development? No problem, remote testing during sprints means never having to say you are sorry (to your spouse after you come home with a box of your stuff in hand because you just got fired for a bad application).

5. Include users in your team

An excellent idea that few organizations seem to use is including actual users as part of the application design team. Having a group of actual users that you reach out to for feedback and input will clear up disagreements and clarify your purpose as you move through your process. Panels of users are worth their weight in gold. Use the input and commentary you receive from actual users as you go through your sprints or waterfall process. Their input will ensure you are keeping your application on track from a usability perspective.

Conclusion on How to Not Throw Out 125 Million Dollars

By incorporating usability testing and conducting user-centered design as part of your application development process you will ensure your design is user-friendly and successful. Failure to do so risks the potential of your application not being used, which can waste 125 Million Dollars and four years of work.  As has been said before, Failure is Not an Option!

For more tips on how to include testing as part of an application development process read the article on 24 usability testing tools.

2 134

The Top 7 HealthCare.gov UX failures caused by poor forecasting of usage

Executive Summary: HealthCare.gov is experiencing a UX Failure due to inaccurate forecasting of usage, but this has deeper impact on attitudes about government health care in general.

Healthcare.gov UX FailureA recent Healthcare.gov UX study highlights the importance of forecasting simultaneous user sessions to ensure applications are accessible.

Failure in forecasting hurts user adoption and in this case increases already uneasy attitudes about the site specifically, and government-provided health care in general.

HealthCare.gov User Experience:

HealthCare.gov is the new government health care plan website that is now offered to U.S. citizens seeking health care insurance in the U.S. The site opened to the public on October 1st, 2013. Unfortunately, as many news stories about overloaded servers have already testified, there are serious and significant UX failures with the site.

The most grievous UX failure currently is the inability for users to access the site due to overloaded servers. As the following UX test reveals, this is potentially causing great harm in opinions about the government health care website specifically and the government provided health care plan in general.

Better simultaneous user session forecasting coupled with simple usability fixes prior to launch could have made for a much better user experience and resulting attitude about the site specifically, and the government health care plan in general.

The Healthcare.gov UX Test:

I was specifically interested in testing the Federal site Healthcare.gov, and not State specific sites such as the ones available in California, New York or other States that have their own websites.

To test the user experience of Healthcare.gov I created a simple usability test with three tasks that a typical person interested in the health care plans would need to perform. The questions the testers were presented with are as follows:

  1. Use this website to find the health care plan you think is best for you. Which plan is it?
  2. Use this website to determine how much the plan you think is best for you will cost. Can you clearly identify the differences in price, and the differences in what is covered?
  3. Use this website to start the application process, it is NOT necessary to complete the application process. Is this process easy or difficult?

I recruited testers from usertesting.com with the following demographics:

  • Testers should be located in one of the following states and should NOT have already signed up for a Healthcare.gov plan: Florida, Georgia, Illinois, Indiana, Texas, New Jersey
  • Age: 18 – 65+
  • Country: U.S. only
  • Gender: Any
  • Income: $0k – $150k+
  • Operating Systems: Any
  • Web Browsers: Any
  • Web Expertise: Any

Testing occurred across multiple days, States and times, from October 2nd through October 4th of 2013.

UX Test Results:

The results of the UX test demonstrated that even the simple processes of trying to find information about plans and costs were virtually impossible. This was due to the website being overloaded and thus incapable of providing users with the ability to perform the sign up process, the first step prior to finding plan and pricing information.

As this video below demonstrates, the ‘waiting page’ that testers landed on did not provide any information about how long they had to wait. Because of this lack of information, most testers abandoned the process after several minutes.

Click to play Healthcare.gov UX testing highlight video #1

Inadequate planning of usage by the team at HealthCare.gov is the prime culprit.

According to WikiPedia as of January 1, 2013, the United States had a total resident population of 316,783,000.

If just 1% of those people wanted to access this application after the October 1st launch of the program, that would be about 3,167,830 trying to use the site at more or less the same time.

Even if just one tenth of that 1% tried to access the site simultaneously, that would still be about 300,000 simultaneous site visitors.

Yet according to the USA Today news story, the HealthCare.gov team planned for peak sessions not even close to that number:

“U.S. Chief Technology Officer Todd Park said the government expected HealthCare.gov to draw 50,000 to 60,000 simultaneous users, but instead it has drawn as many as 250,000 at a time since it launched Oct. 1.

More than 8.1 million consumers visited the site from Tuesday through Friday, according to the White House.”

Website and Chat UX Failure Causes Consumer Confidence Issues:

Because of the website and chat system failure, most of the testers had concerns about the government’s abilities to administer health care insurance at all. This is a good example of the old saying that ‘you only have one chance for a good first impression.’

As this highlight video demonstrates, testers assumed that if the website and chat system were not working, the phone lines would be even worse:

Click to play Healthcare.gov ux testing highlight video #2

HealthCare.gov Receives a 2 out of Possible 10 Score:

Testers were overwhelmingly in agreement that the bad user experience of not being able to find basic plan information and pricing due to the non-working site earned HealthCare.gov a very low satisfaction rating. As the video below highlights, the user experience was rated extremely difficult:

Click to play Healthcare.gov UX testing highlight video #3

Other HealthCare.gov UX Issues:

HealthCare.gov has other usability issues that cause an overall bad UX score. Listed below are just a few of the other findings of this study:

1. Slow response time with no information about how long the queue is causes confusion, which leads to very high abandonment rates and re-tries, and thus adds additional server load.

 Healthcare.gov does not provide information on wait times which causes user confusion

2. Chat also experiences slow or no response time with no information about how long it will be before the person can receive an answer to a question.

3. All testers stated they were confident the phone line will be even worse, and do not attempt to use it.

4. Login instructions are confusing:

“The username is case sensitive. Choose a username that is 6-74
characters long and must contain a lowercase or capital letter, a
number, or one of these symbols _.@/-“

Several users stumbled on these instructions, and received multiple errors due to their User Name not complying with the instructions. Some wondered if one of the symbols could replace a number, because of the use of “or” for lowercase or capital letter, and a number “or” a symbol.

 Healthcare.gov does not follow typical user name UX practices causing user confusion

5. One tester who needed a User Name and Password reminder from the system received an email that had her user name in all caps, but she had entered initial cap and lower case. The system and thus the user’s password is case sensitive, the incorrect all caps User Name in the email caused the user to enter her User Name incorrectly as all caps, this caused the user to be unable to login to the system.

6. There is no detailed information letting the user know the steps in the process to create a login, or the specific set of tasks associated with creating an account and choosing a plan, or where the user is in that process.

7. All testers assumed they would be able to find plan information and pricing without having to apply. Several referred to how other health insurance companies handle plan and pricing quotes, using basic demographic data to provide coverage information prior to applying for coverage.

eHealth provides and example of what users expect when visiting HealthCare.gov
eHealth provides and example of what users expect when visiting HealthCare.gov

From the sampling of usability issues the site is already experiencing, we can only wonder about the other user experience issues awaiting those who successfully complete the application process.

This is a reminder that really smart people (and I’m sure the designers and developers of the Healthcare.gov user experience are some of the best out there) can still stumble when creating applications for large sets of users.

The best way to avoid this is with constant testing and evaluation throughout the process, and even then UX issues will still surface. Hopefully the UX team responsible for HealthCare.gov is addressing many of these issues as we read this.

Conclusion: HealthCare.gov UX Failures

Considering the impact and far reaching consequences HealthCare.gov will have on millions of Americans, it is disturbing that so many severe UX failures are present in the system at this late stage.

Unfortunately, the bad user experience the HealthCare.gov site is currently providing is impacting attitudes and perceptions about the site specifically, and about the government’s ability to adequately administer health care insurance in general.

This should be a strong reminder to everyone in the UX community that the user experience is the Brand. Without adequate planning and testing prior to launch, a bad first impression of a system will result in bad perceptions about the Brand. And sadly in this case, that Brand is the United States Government.

0 26

Interview with Dave Garr, Co-founder of UserTesting.com

Dave Garr photo, co-founder of usertesting.com
Dave Garr, co-founder of Usertesting.com

My friend Dave Garr is probably one of the most non-famous Famous usability and UX thought-leaders around. If you’ve never heard his name, you very well may have heard of his creation: UserTesting.com. As a co-founder of UserTesting.com way back in the dark days of 2007, Dave and his team have made a major impact for all usability practitioners; a fast, low-cost and useful usability testing service that provides results in a day versus what used to take weeks. When he’s not re-inventing usability testing, Dave loves writing and performing song parodies. While at Apple, he recorded these videos: “I Think We’re a Clone Now” and “Killing My Software with Windows

If not busy enough completely re-creating remote unmoderated usability testing for the entire world, Dave won a Webby for his marriage proposal.

What’s your background?

My first brush with technology came during a summer job in college with a software company who developed EasyWriter, the first word processor for IBM’s PC. Fortunately, when I went through the interview process, no one asked me if I’d ever used a computer before.

I graduated from Cal Berkeley in Marketing, and I’ve overseen websites for several companies, such as Intuit, HP, and Apple.

I live in Palo Alto, California with my wife Elizabeth who encouraged me to pursue my startup dream, and my two young daughters who don’t seem to be even remotely interested in usability testing.

How did you get into the usability field?

I was managing Apple.com and I found myself drawn to watching Apple’s user experience labs. I was fascinated with how hard they worked to improve the out-of-box experience, or to decide on the use of color in the Mac OS, for example.

Then I read Steve Krug’s “Don’t Make Me Think.” It’s fantastic, particularly his chapter “Usability testing on 10 cents a day.” He makes it so simple: “Watch some people while they try to use [your site] and note where they run into trouble. Then fix it, and test it again.” That resonated with me. Since then, Steve has been kind enough to be a mentor to me.

What is it about your job that you most enjoy, or find most rewarding?

That’s easy. It’s when we release a new capability on UserTesting.com and a customer tweets or blogs about how it’s improved their life. Like one guy tweeted: “I nominate usertesting.com for a Nobel Peace Prize for preventing warfare between designers and developers. Don’t fight, test it and see.”

What is UserTesting.com and why should someone use it?

We like to think of it as usability testing without the hassle. You create the test, and we handle everything else, including getting the testers. We record the testers using your site, so you can virtually peek over their shoulders to discover your site’s problems. It’s $39 per tester and you get the results in about an hour.

Companies commonly test:

  • Their own website and landing pages
  • Competitors’ sites
  • Semi-functional prototypes and staging sites
  • Facebook games
  • Mobile apps

What was your motivation for creating UserTesting.com?

I’ve spent an unhealthy amount of time in usability labs, and I was frustrated with how expensive and time-consuming it was. So I started doing a lot of quick and dirty usability testing with my family, coworkers, friends, neighbors, and — in those rare times that I could get up the guts–captive audiences at train stations.

So UserTesting.com solved a pain point that I had. And fortunately it’s a pain point that others have.

What have you learned while running UserTesting.com?

We launched our minimum viable product (MVP) four years ago. After every order, we emailed customers asking them “What can we improve?” We’ve received a lot of feedback because — believe me –our MVP had a lot of room for improvement. The request that we heard most often was this: “I want the participants to be my exact target market and not people on your panel.” So that’s the area we dedicated the most resources to build. Now participants don’t just have to come from our panel — they can also come from any of these places:

  • Live visitors on your website who’ve been intercepted
  • Your own customer list
  • Participants from third-party panels

Another thing we’ve learned: website owners care a lot about their competitors’ websites. As Steve Krug says, “Someone has gone to the trouble of building a full-scale working prototype of a design approach to the same problems you’re trying to solve, and then they’ve left it lying around for you to use.” In particular, our clients want to know what their competitors are doing right, so they can “borrow” it.

What advice do you have for other start-ups that wish to create an online service, whether usability related or otherwise?

If you only take away one thing from this interview, then by far and away, my biggest recommendation is: come work for us! We’re growing and want to hire more people who are passionate about rescuing the world from hard to use products.

But if you insist on doing your own thing, then here are some thoughts…

Do what you love. Since I’ve had to immerse myself in the topic of usability testing for four years, it was good that I was very interested in that topic. It would be hard for me to spend so much time every day thinking about a topic that I wasn’t passionate about.

Work with people you love. I cannot overemphasize how important it is to work with someone that you really enjoy. My co-founder, Darrell Benatar, had previously been a friend and co-worker, so I knew how well we got along.

Love your customers, even if a few take advantage of you. I’ve learned that providing great customer support is crucial. We’re trying to mimic Zappos on this one. We’ll give customers a refund, no matter what. Even if they ran tests years ago, combined the best clips into a highlight reel, and shared that highlight reel with hundreds of people; if they ask for a refund, we give it. Our customers don’t abuse our return policy–less than 1% of our customers request a refund. Best of all, by having great phone, chat, and email support, we get tons of feedback from customers about how we can improve our product.

What’s in the future for UserTesting.com? What changes or improvements are you working on?

Okay, I’ll tell you…as long as you promise to keep it a secret (laughs).

Probably my hardest job is deciding which feature to add next. We’ve learned from ConversionRateExperts.com to rate each feature (on a scale of 1-10) according to “How easy is it to implement?” and “How important is it to customers?” We multiply these two figures together to give an estimated return on investment. Next we build the feature that has the highest estimated ROI and then A/B test it.

Our DNA is about listening to customers. When a lot of customers ask for the same thing, we usually do it. The biggest things they’re asking for now are:

  • Improve our user testing of mobile devices. We’ve developed a mobile version of our platform, and making it better is a key focus of our product development for the foreseeable future.
  • Expand globally beyond the US, Canada, and UK.
  • UserTesting.com has made it easy for you to get feedback on your live website. However, as you know, the earlier in the development cycle you test, the easier it is to make changes. But we haven’t made it easy to test concepts. Shame on me for not doing a better job on that. So we’re going to try to make it easier to use UserTesting.com to get feedback in the ideation phase of the dev process.
  • Consulting services for our enterprise customers. We create the test plan, make clips of the places where testers got stuck, and recommend how to fix the biggest problems found.

What do you think the next few years will bring for usability?

Computing is moving from one screen — a computer accessing the web — to four screens: a computer, a tablet, a phone, and a smart TV.

Let me quote a usability expert named Craig Tomlin! You’ve talked about the next big UX trend is understanding that “user experience” does not mean just “web site experience” or “mobile experience” or “phone experience” or “store experience. ”

Companies will stop designing each experience in a vacuum. They’ll start putting together the holistic understanding of the entire “user experience.”

You used the example of how someone buys a car. She may go to several websites to evaluate car brands. She may use her mobile phone to schedule test drives. She may ask friends on Facebook or other social sites about their opinions. She may build and configure her ideal car on her iPad. Eventually she goes to the car lot and negotiates with the dealer. Given that she has interactions that transcend any single experience, why would car companies design the user experience she has with the website without considering the other critical interactions she’ll have during the car buying process?

What’s next for you and your career in the next year or two, what would you like to focus on?

Sentiment analysis. Sentiment analysis is software that mines text for meaning and insight. It’s often used to extract opinions and emotions from social media to help companies determine how people feel about their products.

Sentiment analysis is an extremely difficult problem, but it’s a problem that will eventually be solved. When the problem is solved, it’ll cull through a massive amount of text and automatically call attention to the biggest, most frequently mentioned issues. This will increase the value of qualitative tools like open-ended survey questions and transcripts of user testing.

Thank you Dave!


Let's Connect