7 Controversial Usability Predictions Debate


I love a good debate, especially about interesting things like usability and user experience.  That’s because having a debate about things we care about, even if we disagree with each other, helps all of us to expand our views and become more informed.

Kennedy-Nixon debate photos courtesy Travelin’ Librarian via Flickr Creative Commons License
The great Kennedy-Nixon debate

To that end, I was pleased to receive a recent rebuttal response from Charles L Mauro, usability expert and founder of Mauro New Media, in the comments of an article I wrote in December 2009 titled, ‘7 Controversial Usability Predictions.’

Loving a good debate, I was going to write my rebuttal response and post it in the comments for the original article, but feeling like this is a bit more worthy of its own posting I’ve decided to include this as a new post all to itself (SEO duplicate content issues be damned).  Therefore, without further ado here’s my response to Charles’ rebuttals.

For reference:

Charles Mauro wrote:

“Over the past 35 years that I have been working in professional human factors engineering I have seen many trends come and go with respect to methods, certifications, business models and related matters. The 7 trends cited at the start of this thread shows a profound lack of understanding of what is actually happening in usability testing at the professional level. As an owner of a firm that offers such services I find the following to be true.”

My rebuttal:

First, thank you for an alternative point of view on my controversial predictions.  Isn’t it great that we can have these debates about a topic we all enjoy?  By the way, I just wanted to remind everyone that I clearly labeled the article as ‘7 Controversial Usability Predictions,’ because, well, those predictions go against many assumptions and beliefs regarding our profession and thus are darn controversial, just to set the record straight.

Second, when referring to my writings as stemming from ‘a profound lack of understanding’ I think the description is a bit misguided.  I’ve actually been in the usability business for over 14 years, in many roles from self-employed consultant to a leader of Fortune 500 usability and UX engagements for firms like:

  • Blue Cross of California
  • Countrywide Home Loans
  • Disney
  • Kodak
  • WellPoint
  • Zurich Insurance Company and many more

I also am proud to be Certified as a usability analyst.  Although certainly not equal to Charles Mauro’s 35 years (which is impressive), it does provide me with a fair bit of knowledge about that of which I speak er, write.

Charles Mauro wrote:

“1- Usability testing costs are rising for studies that are professionally designed and that provide actionable insights that objectively impact business success. Clients pay for quality and the cost is rising not falling.”

Click here to view latest job postings

My Rebuttal:

I guess the issue with this statement is what you mean by ‘professional.’  The fact that we can go to UserTesting.com or Loop11.com (see a more complete list in my article, 24 Usability Testing Tools) and create and conduct a professional usability study of a website for the low, low cost of $39 (UserTesting) or $350 for 1,000 participants (that’s 35 Cents PER PARTICIPANT(!) via Loop11) seems to me to be pretty convincing proof that usability testing costs are decreasing substantially.  Further, these studies are a fraction of the cost of the more traditional in-person, moderated one-on-one think aloud studies (especially those involving travel).

As to the ‘professionalism’ of these tools, the fact that Fortune 500 and other major corporations are using them would seem to imply they are quite comfortable with the professionalism, aka quality of data, of these tools.  Some of these corporations as listed on UserTesting and Loop11’s websites include:

  • Disney
  • HP
  • Sony
  • Amazon.com
  • eBay
  • Dell
  • Cisco
  • Vodafone
  • Motorola
  • GoDaddy
  • IBM
  • Accenture, and many more

So I think this is proof that the cost of conducting usability testing has substantially decreased.  This is a direct result of having this ever-increasing list of online usability testing tools available for professional use.

Perhaps there are usability firms where costs are rising, but based on the success and dramatically increasing utilization of these low cost tools I think there is convincing evidence that overall costs are substantially decreasing, by much greater than the 10X I originally predicted.

Charles Mauro wrote:

“2-Remote usability testing (we recently wrote a chapter on this topic in a leading book published by Kaufman) is increasing in cost and is in fact much more complex to do well than traditional lab-based testing. The cost of these studies is increasing NOT decreasing. Some of the large longitudinal studies range well into the mid six figures. In the end more firms may be attempting remote testing but what you will find is that such methods are fraught with methodology and reliability issues which, no matter how many studies you do, limit effectiveness.”

My Rebuttal:

Yes, I actually read the book and even wrote a book review about ‘Remote Research’ (great book!).

Although the authors, Nate Bolt and Tony Tulathimutte, said remote research was slightly cheaper than traditional in-person testing, this was because they were examining it from a “billable hours” perspective, not from an infrastructure or travel perspective. I know this because I asked them as part of my book review.

When you compare the cost of utilizing remote testing tools vs. building a state-of-the-art usability testing lab, it is obvious there is major cost savings in going the remote testing route.

And if that’s not enough cost savings, consider the enormous expense of flying a usability team to a location, or several locations, to conduct in-person testing.  Add up the cost of airfare, rental cars, hotels, testing rooms, meals etc. to conduct 1-on-1 moderated testing with participants at their location.  Consider the length of time this takes as well.  Now consider the far lower costs of using remote moderated testing tools, such as Adobe Connect, and a phone to reach the exact same participants at their location, minus the travel expenses (and TIME!).

Clearly remote moderated testing is far cheaper and far faster than in-person testing at a remote location.  And again, if you add up the travel costs, especially for multiple locations, remote moderated testing beats in-person by well over the factor of 10 I originally wrote about.

As to complexity, their basic premise of the book the authors express again and again is that it is really easy to set up and conduct moderated remote research testing, and is in fact faster and simpler to set up than a lab-based test.

Here’s the actual description as provided on the ‘Remote Research’ website:

“Remote studies allow you to recruit subjects quickly, cheaply, and immediately, and give you the opportunity to observe users as they behave naturally in their own environment. In Remote Research, Nate Bolt and Tony Tulathimutte teach you how to design and conduct remote research studies, top to bottom, with little more than a phone and a laptop.”

As for me, I’ve done both in-person and remote moderated usability testing many, many times.  I can testify that the quality of the data from remote testing is just as good as in-person testing data, and better if you consider you’re actually testing users in their own environment, vs. a sterile lab.

Charles Mauro wrote:

“3- Certification is becoming very important at the top end of the market. On large projects that are competitively bid we routinely see clients asking for and making decisions based on academic background, formal certification AND proven case studies where it can be shown that a firm’s research objectively solved a major business problem. Do not think for a moment that certification is not increasing in importance at the top end of the consulting market. BTW: certification does nothing to keep consultants out of this space who have sketchy experience. These individuals or firms never win the big projects. Clients have become way too sophisticated to fall for the 1999 definition of usability expertise.”

My rebuttal:

I think we are somewhat in agreement (I wrote, ‘Without professional certification being required, more and more charlatans will be attracted to usability’) and I’m agreeing with your point that certification can help ‘sell’ a firm as experts.  (By the way, is Steve Krug certified?)

But where I disagree is in saying certification of usability practitioners will not decrease ‘sketchy experience’ consultants (I called them ‘charlatans’) from entering the space.  There are many more small and mid-sized usability projects vs. big sized projects (I’m thinking ‘big’ equals $100,000 or more) and I’m sure the charlatans don’t stand a chance in trying to bag a big project.  But they do stand a chance in the small to mid-sized projects where unsuspecting companies are looking for help, but don’t know a good usability expert from a high-school kid pretending to be a usability expert.

In my opinion, certification would do two important things to help curb charlatans:

  1. Clearly define a usability practitioner from a charlatan.
  2. Provide the means to educate companies as to who usability practitioners are, and what they do.

Charles Mauro wrote:

“4- Finally, if you are selling big projects and cannot objectively demonstrate real ROI in both statistical and conceptual terms you are not going to get the project. Any firm that does not have a very strong grasp of formal business process modeling in the context of user experience optimization is going to be dead in the water when it comes to large interesting projects. These new ROI models are vastly more sophisticated than just 3 years ago. The good thing about all this is that in the end clients who understand research speak the language of business impact. This is a dramatic help to practitioners who want to build a robust business in this research space.”

My rebuttal:

First, I don’t think I was specifically discussing only ‘big projects’ but rather the set of all usability projects.  And second, by ROI I mean the traditional sense of Return On Investment that a Chief Financial Officer (CFO) would clearly define as either increased revenue or decreased cost.  And those ROI values are expressed in actual monetary units, not; ‘we decreased time-on-task by xx%’ or ‘we reduced error rate by yy%.’

As I think we all can agree, CFOs don’t give a crap about time on task or error rates, instead, they want to know how much incremental revenue or savings you generated versus the money spent to conduct the usability project.  So ROI means money, and to my knowledge about the ONLY usability practitioner that actually publicly demonstrates an ROI example is Jared Spool and his $300 Million Button story.

My website doesn’t provide examples of actual ROI, and neither does mauronewmedia.com, most usability company websites don’t. And if there are new sophisticated ROI models that have been developed in the past 3 years it would be very interesting and informative to see them.

In this regard, I’m as they say, from Missouri, and so say to anyone that says they have ROI examples, ‘Show me!’

Show me the ROI money.

Until then, I’ll continue to express my opinion that true ROI examples garnered from usability projects, examples that can be shared publicly, continue to elude practitioners.

Conclusion – 7 Controversial Usability Predictions Revisited

So that’s Charles and my thoughts on these topics. But what are yours?

We didn’t touch on the other usability predictions I made in my original post, some of which are probably just as subject to debate, but don’t let that, dear reader, stop you!  Post your comments whether you agree or disagree with them, and let us know the all important ‘why.’  Let’s continue to have these kinds of discussions and debates, they’re good for the practice, and make for interesting and informative reading too!


  1. Craig: nicely stated rebuttal. I give you credit for fighting the good fight of cheaper, faster, lower quality as the trend in usability research. I just do not see it playing out in that manner.

    In all forms of market research (usability is just another flavor) costs have increased over time as have the reliability of results and the ability of such research to support management decision making. The larger business trend in market research services is counter your argument and always will be.

    In the end I see formal usability testing as rapidly becoming a critical research methodology employed by corporate market research teams. As usability research enters that space, fees rise as do the expectations of the client. 5 years ago usability testing projects executed by our firm were largely funded by web development executives. Today, 90% of our work is funded by corporate market research departments where we report to the VP of Market Research. Those executives demand certain levels of professionalism. They expect to pay for usability testing at a rate that is directly related to the reliability of the data and the problems it helps solve. So, my view is that the race is not to the bottom but to the top for those practitioners who want to make a career of this new research discipline. Leave the bottom to the charlatans…on that much we agree.


Comments are closed.