Tags Posts tagged with "ROI"


I love a good debate, especially about interesting things like usability and user experience.  That’s because having a debate about things we care about, even if we disagree with each other, helps all of us to expand our views and become more informed.

Kennedy-Nixon debate photos courtesy Travelin’ Librarian via Flickr Creative Commons License
The great Kennedy-Nixon debate

To that end, I was pleased to receive a recent rebuttal response from Charles L Mauro, usability expert and founder of Mauro New Media, in the comments of an article I wrote in December 2009 titled, ‘7 Controversial Usability Predictions.’

Loving a good debate, I was going to write my rebuttal response and post it in the comments for the original article, but feeling like this is a bit more worthy of its own posting I’ve decided to include this as a new post all to itself (SEO duplicate content issues be damned).  Therefore, without further ado here’s my response to Charles’ rebuttals.

For reference:

Charles Mauro wrote:

“Over the past 35 years that I have been working in professional human factors engineering I have seen many trends come and go with respect to methods, certifications, business models and related matters. The 7 trends cited at the start of this thread shows a profound lack of understanding of what is actually happening in usability testing at the professional level. As an owner of a firm that offers such services I find the following to be true.”

My rebuttal:

First, thank you for an alternative point of view on my controversial predictions.  Isn’t it great that we can have these debates about a topic we all enjoy?  By the way, I just wanted to remind everyone that I clearly labeled the article as ‘7 Controversial Usability Predictions,’ because, well, those predictions go against many assumptions and beliefs regarding our profession and thus are darn controversial, just to set the record straight.

Second, when referring to my writings as stemming from ‘a profound lack of understanding’ I think the description is a bit misguided.  I’ve actually been in the usability business for over 14 years, in many roles from self-employed consultant to a leader of Fortune 500 usability and UX engagements for firms like:

  • Blue Cross of California
  • Countrywide Home Loans
  • Disney
  • Kodak
  • WellPoint
  • Zurich Insurance Company and many more

I also am proud to be Certified as a usability analyst.  Although certainly not equal to Charles Mauro’s 35 years (which is impressive), it does provide me with a fair bit of knowledge about that of which I speak er, write.

Charles Mauro wrote:

“1- Usability testing costs are rising for studies that are professionally designed and that provide actionable insights that objectively impact business success. Clients pay for quality and the cost is rising not falling.”

My Rebuttal:

I guess the issue with this statement is what you mean by ‘professional.’  The fact that we can go to UserTesting.com or Loop11.com (see a more complete list in my article, 24 Usability Testing Tools) and create and conduct a professional usability study of a website for the low, low cost of $39 (UserTesting) or $350 for 1,000 participants (that’s 35 Cents PER PARTICIPANT(!) via Loop11) seems to me to be pretty convincing proof that usability testing costs are decreasing substantially.  Further, these studies are a fraction of the cost of the more traditional in-person, moderated one-on-one think aloud studies (especially those involving travel).

As to the ‘professionalism’ of these tools, the fact that Fortune 500 and other major corporations are using them would seem to imply they are quite comfortable with the professionalism, aka quality of data, of these tools.  Some of these corporations as listed on UserTesting and Loop11’s websites include:

  • Disney
  • HP
  • Sony
  • Amazon.com
  • eBay
  • Dell
  • Cisco
  • Vodafone
  • Motorola
  • GoDaddy
  • IBM
  • Accenture, and many more

So I think this is proof that the cost of conducting usability testing has substantially decreased.  This is a direct result of having this ever-increasing list of online usability testing tools available for professional use.

Perhaps there are usability firms where costs are rising, but based on the success and dramatically increasing utilization of these low cost tools I think there is convincing evidence that overall costs are substantially decreasing, by much greater than the 10X I originally predicted.

Charles Mauro wrote:

“2-Remote usability testing (we recently wrote a chapter on this topic in a leading book published by Kaufman) is increasing in cost and is in fact much more complex to do well than traditional lab-based testing. The cost of these studies is increasing NOT decreasing. Some of the large longitudinal studies range well into the mid six figures. In the end more firms may be attempting remote testing but what you will find is that such methods are fraught with methodology and reliability issues which, no matter how many studies you do, limit effectiveness.”

My Rebuttal:

Yes, I actually read the book and even wrote a book review about ‘Remote Research’ (great book!).

Although the authors, Nate Bolt and Tony Tulathimutte, said remote research was slightly cheaper than traditional in-person testing, this was because they were examining it from a “billable hours” perspective, not from an infrastructure or travel perspective. I know this because I asked them as part of my book review.

When you compare the cost of utilizing remote testing tools vs. building a state-of-the-art usability testing lab, it is obvious there is major cost savings in going the remote testing route.

And if that’s not enough cost savings, consider the enormous expense of flying a usability team to a location, or several locations, to conduct in-person testing.  Add up the cost of airfare, rental cars, hotels, testing rooms, meals etc. to conduct 1-on-1 moderated testing with participants at their location.  Consider the length of time this takes as well.  Now consider the far lower costs of using remote moderated testing tools, such as Adobe Connect, and a phone to reach the exact same participants at their location, minus the travel expenses (and TIME!).

Clearly remote moderated testing is far cheaper and far faster than in-person testing at a remote location.  And again, if you add up the travel costs, especially for multiple locations, remote moderated testing beats in-person by well over the factor of 10 I originally wrote about.

As to complexity, their basic premise of the book the authors express again and again is that it is really easy to set up and conduct moderated remote research testing, and is in fact faster and simpler to set up than a lab-based test.

Here’s the actual description as provided on the ‘Remote Research’ website:

“Remote studies allow you to recruit subjects quickly, cheaply, and immediately, and give you the opportunity to observe users as they behave naturally in their own environment. In Remote Research, Nate Bolt and Tony Tulathimutte teach you how to design and conduct remote research studies, top to bottom, with little more than a phone and a laptop.”

As for me, I’ve done both in-person and remote moderated usability testing many, many times.  I can testify that the quality of the data from remote testing is just as good as in-person testing data, and better if you consider you’re actually testing users in their own environment, vs. a sterile lab.

Charles Mauro wrote:

“3- Certification is becoming very important at the top end of the market. On large projects that are competitively bid we routinely see clients asking for and making decisions based on academic background, formal certification AND proven case studies where it can be shown that a firm’s research objectively solved a major business problem. Do not think for a moment that certification is not increasing in importance at the top end of the consulting market. BTW: certification does nothing to keep consultants out of this space who have sketchy experience. These individuals or firms never win the big projects. Clients have become way too sophisticated to fall for the 1999 definition of usability expertise.”

My rebuttal:

I think we are somewhat in agreement (I wrote, ‘Without professional certification being required, more and more charlatans will be attracted to usability’) and I’m agreeing with your point that certification can help ‘sell’ a firm as experts.  (By the way, is Steve Krug certified?)

But where I disagree is in saying certification of usability practitioners will not decrease ‘sketchy experience’ consultants (I called them ‘charlatans’) from entering the space.  There are many more small and mid-sized usability projects vs. big sized projects (I’m thinking ‘big’ equals $100,000 or more) and I’m sure the charlatans don’t stand a chance in trying to bag a big project.  But they do stand a chance in the small to mid-sized projects where unsuspecting companies are looking for help, but don’t know a good usability expert from a high-school kid pretending to be a usability expert.

In my opinion, certification would do two important things to help curb charlatans:

  1. Clearly define a usability practitioner from a charlatan.
  2. Provide the means to educate companies as to who usability practitioners are, and what they do.

Charles Mauro wrote:

“4- Finally, if you are selling big projects and cannot objectively demonstrate real ROI in both statistical and conceptual terms you are not going to get the project. Any firm that does not have a very strong grasp of formal business process modeling in the context of user experience optimization is going to be dead in the water when it comes to large interesting projects. These new ROI models are vastly more sophisticated than just 3 years ago. The good thing about all this is that in the end clients who understand research speak the language of business impact. This is a dramatic help to practitioners who want to build a robust business in this research space.”

My rebuttal:

First, I don’t think I was specifically discussing only ‘big projects’ but rather the set of all usability projects.  And second, by ROI I mean the traditional sense of Return On Investment that a Chief Financial Officer (CFO) would clearly define as either increased revenue or decreased cost.  And those ROI values are expressed in actual monetary units, not; ‘we decreased time-on-task by xx%’ or ‘we reduced error rate by yy%.’

As I think we all can agree, CFOs don’t give a crap about time on task or error rates, instead, they want to know how much incremental revenue or savings you generated versus the money spent to conduct the usability project.  So ROI means money, and to my knowledge about the ONLY usability practitioner that actually publicly demonstrates an ROI example is Jared Spool and his $300 Million Button story.

My website doesn’t provide examples of actual ROI, and neither does mauronewmedia.com, most usability company websites don’t. And if there are new sophisticated ROI models that have been developed in the past 3 years it would be very interesting and informative to see them.

In this regard, I’m as they say, from Missouri, and so say to anyone that says they have ROI examples, ‘Show me!’

Show me the ROI money.

Until then, I’ll continue to express my opinion that true ROI examples garnered from usability projects, examples that can be shared publicly, continue to elude practitioners.

Conclusion – 7 Controversial Usability Predictions Revisited

So that’s Charles and my thoughts on these topics. But what are yours?

We didn’t touch on the other usability predictions I made in my original post, some of which are probably just as subject to debate, but don’t let that, dear reader, stop you!  Post your comments whether you agree or disagree with them, and let us know the all important ‘why.’  Let’s continue to have these kinds of discussions and debates, they’re good for the practice, and make for interesting and informative reading too!

16 1459

The Return On Investment for eCommerce Usability Will Always Beat Online Advertising, Because of the Principle of Amortized Improved Conversion

Useful Usability eCommerce ROI articleI don’t understand why online advertising agencies don’t add usability to their service offerings. Armed with a usability component, agencies could help improve the Return On Investment (ROI) for eCommerce companies well beyond any improvement gained by conducting online advertising. Why? Because of what I call the “Principle of Amortized Improved Conversion.”

Here’s how it works:

Let’s assume that you are a business owner and you have $10,000 you want to spend to help increase sales on your eCommerce web site. You can choose to spend the $10,000 for online advertising, which will generate incremental leads and sales. Or, you can choose to spend it on a usability improvement project on your eCommerce order form, which will increase conversion and sales.

Given this choice, which would you do?

If you’re smart, and I know you are because you’re reading this, you’ll pick usability – because you know an equal amount of money spent on usability will always beat the same amount of money spent on online advertising, given a long enough period of time.

This is because the one-time spend, and subsequent one-time incremental lift in leads and sales generated by online advertising is temporary. Turn on the advertising, you increase leads, turn off the advertising, and the number of leads go back down to their prior (normal) state.

With usability, the one-time spend in improving online form conversion creates a permanent increase in conversion. This means the cost of the usability improvement can be amortized over a very long period, which decreases the total cost per sale.

Here’s the proof, we’ll look at how online advertising, and usability, impact the sales and cost per sale of a web site over a one year period of time (but first, a few assumptions):

1. This eCommerce web site generates 2,000 leads per month naturally, with no advertising expense.
2. The conversion rate of leads into sales for this site is a nice 10%.
3. We’ll ignore “lights-on” costs associated with the sales on the website, it’s a cost of doing business and will not be included in the Cost Per Lead and Cost Per Sale metrics here.

Here’s what a typical month looks like for your pretend company:

The ROI of a typical month of an eCommerce web site

You generate 2,000 leads per month through natural (not paid advertising) methods. Your 10% conversion in your online order form results in 200 sales per month. Well done.

The ROI of Online Advertising

The ROI of online advertising spend
In this scenario, you’ve spent $10,000 to conduct an online advertising campaign for one month. The good news is you’ve hired an excellent agency that provides an amazing campaign, and with your $10,000 you’ve generated 4,000 extra leads. That’s a total of 6,000 leads instead of the normal 2,000 for that month. That’s 3 TIMES (that’s 300% for you math wonks) the normal monthly flow of leads to your site. Very well done!

This provides you with an advertising Cost Per Lead of $2.50 (your $10,000 advertising cost divided by your incremental advertising-generated 4,000 leads).

With your site’s 10% conversion those 6,000 leads become 600 sales instead of the normal 200 sales. Your advertising-generated Cost Per Sale for that month is $25.00 (your $10,000 advertising cost divided by your incremental advertising-generated 400 sales).

Now let’s look at how this impacts your year (you may have to click on the graphic to see the full-size image if your eyes – like mine – are over the age of 40):

ROI of online advertising per year

Now, let’s mash-up all the monthly metrics into a final yearly total. For the year, you had a total of:

  • 28,000 total leads
  • $10,000 in online advertising cost
  • $0.36 total Cost Per Lead
  • 10% conversion
  • 2,800 total sales
  • $3.57 total cost per sale

The ROI of Usability

The ROI of usability on an average month

With this scenario, we start with the same assumptions. However, in this case you’ve chosen to spend the $10,000 on usability improvements of your eCommerce order form.

The good news is the usability work that took place over the first month resulted in a modest improvement on your site’s order form, resulting in a modest increase in conversion. Your conversion was 10%, but after the new user-friendly order form is launched in the 2nd month your conversion went up to 15%. This 15% conversion is permanent, and continues for as long as the order form (or web site) is active.

This is significant, it means that each and every month you’ll now be adding more sales, but with the same free leads, you were receiving prior to the usability improvements.

Here’s what the yearly total looks like (put your reading glasses on, or just click the graphic to see it larger).

The ROI of usability per year

Now, let’s once again mash-up all the monthly metrics into a final yearly total. For the year, you had a total of:

  • 24,000 total leads
  • $10,000 in usability cost
  • $0.00 total Cost Per Lead
  • 14.6% conversion
  • 3,500 total sales
  • $2.86 total cost per sale

The Difference Between Online Advertising & Usability

The ROI difference between online advertising and usability

So here’s what the difference looks like when we compare online advertising vs usability.

The usability improvement to the eCommerce order form resulted in an INCREASE of 25% more sales for the year, and a DECREASE of almost 20% in Cost Per Sale vs the Online Advertising scenario.

If we were to extend this into 2 or 3 years, the differences would become truly staggering, with that one-time usability improvement impressively beating the one-time advertising spend.

For those of you who are visual learners, here’s what these yearly differences look like in graph form:

The ROI difference in number of sales between online advertising and usability

The ROI difference in cost per sale between online advertising and usability

Conclusion – eCommerce Usability ALWAYS beats Online Advertising in generating more sales, and a reduced cost per sale

I’ve actually been quite conservative with the above scenarios. The reality is I’ve very seldom seen an online advertising campaign generate 3 times the normal number of leads, and I’ve rarely seen usability improvements only increase eCommerce order flows by a paltry 5 percentage point lift.

In reality, usability beats the tar out of online advertising much better than I’ve indicated above.

As was demonstrated in the case above, over time, spending money on eCommerce usability improvements provides a much better return on investment for any business that wants to increase their web site sales, while at the same time decreasing the total cost per sale.

That’s the “Principle of Amortized Improved Conversion” in action.

Smart online advertising and web development agencies will add usability testing to their service offerings.

Smart eCommerce business owners will invest in usability improvements to permanently improve their sales and cost per sale, and thus their ROI of their web sites.

If you’d like more information about adding usability to an agency service offering, or conducting a usability optimization to improve eCommerce just contact me for usability testing information and I’ll be glad to explain how easy it is to get started.

4 48

Usability & Search Engine Optimization Share the Same Problem: How to Define Return On Investment

A recent post by Search Engine Optimization (SEO) guru Bruce Clay discussed the difficulty of providing an estimate of SEO Return On Investment (ROI) for CFOs of companies. The issue is most CFO’s demand an ROI estimate for search engine optimization prior to an engagement, but as Bruce point’s out in his article this is actually not possible.

Usability ROI

As with SEO (which I consider a close-cousin to usability by the way) usability shares the exact same ROI problem. Although CFOs want and need ROI, it is not ethical or wise to provide a guaranteed ROI prior to the commencement of a project, simply because there is not enough information, or control, to accurately estimate the usability impact on ROI.

There are 4 reasons why an accurate ROI is not possible:

  1. An assessment of the severity of the usability issues facing the web site or application must be conducted. This assessment IS the usability project, including understanding Personas, Critical Tasks, and conducting a Heuristic Evaluation, and much more. Without this information any usability company guaranteeing an ROI is just guessing.
  2. The company that requests the usability study may not be able to do all the recommended usability changes. Often, due to business or technical issues not all of the usability recommendations needed to produce the ROI are able to be executed. This means money (or in this case ROI) is left on the table. Without the ability to have all recommendations made, it’s unlikely an expected usability ROI will be accomplished.
  3. ROI is compromised if other critical departments that influence the customer experience are not involved. Most often, a usability project is requested by a particular division in a company. However, other divisions that directly impact the experience (thus ROI) are often not involved. Their inability or unwillingness to make changes will impact the anticipated ROI. As an example, think in this case of a Marketing department wanting web site usability improvements to increase sales. If however, the back-end sales systems that are controlled by a different department are not engaged, the full benefit of usability improvements to increase sales, and thus ROI, may not happen.
  4. The online environment is constantly changing, and estimates based on historical data cannot forecast future ROI with accuracy. Consider the changes in the economy in the past year, would you be willing to estimate the ROI of improved sales from usability improvements for that same Marketing department I mentioned earlier if you knew their traffic, or the type of visitors to their web site would be severely changed?

For all of these reasons, and more, it’s unethical, and unwise, to provide a guaranteed ROI for usability improvements. The reality is usability improvements can and do significantly improve ROI for companies every day, it’s just that guaranteeing an ROI upfront prior to conducting usability audits is not possible, nor advisable.

So, what do you say to the CFO who’s asking about ROI?

First, provide the CFO with the above reasons why an ROI estimate is not possible prior to the commencement of the usability project. Be prepared, this may not be accepted, after all the CFO and most likely the rest of the company are held to very precise and unyielding goals in terms of numbers.

Next, talk about some examples of ROI that were accomplished from prior engagements. In the past, when I was consulting I would often mention specific examples of ROI gained, but without mentioning the name of the client (unless I had permission to use their name of course). Nothing sells like success, and mentioning other usability success stories can add weight to the argument that the usability project will be successful, but without a guaranteed ROI up-front.

I used to mention that doing nothing was the only guarantee, a guarantee of the same results. This is the same as the sage old advice that doing something over and over again, but expecting different results, is not logical. By conducting the usability project, change, and thus different (better) results are possible.

Finally, understand that CFO’s of a necessity must live by numbers, and ROI numbers are sometimes required prior to project approval. You may find that sometimes it’s necessary to walk away from the project, because ROI is a pass/fail and without it the CFO is unwilling to conduct the project.

If the CFO insists that an ROI is required, it may be tempting to start throwing out a range of ROI estimates that might be possible. Speaking from experience I can tell you this only opens the door for the CFO to start pinning exact numbers down. It’s a no-win situation if you start down this path. If you hope that the CFO won’t remember the ROI at the end of the project, or that unusual circumstances may result in the numbers being set aside, don’t count on it. The odds are whatever number you promised won’t be met, and thus you will not be trusted, and worse, bad word-of-mouth may start spreading at the executive level. Worse, your manager allies in the company may have a negative perception placed on them, causing them to be far less trustful of usability in the future, again generating bad word-of-mouth.

Conclusion: Don’t Estimate Usability ROI.

CFOs will want ROI, and sometimes a usability project will not be approved without it. My recommendation is to not try to guess at ROI. Instead, explain why ROI is almost impossible to guarantee, then use the opportunity to educate the CFO from examples of ROI gained from prior usability engagements.

Do you have favorite techniques for demonstrating the value of usability without a guaranteed ROI? If so, please share them in the comments!