Design & Usability Conflict of Interest?

Should an Agency, or Designer, Conduct Usability Testing on Their Own Designs?

Among the spam and family emails I get from my mom (hi Mom!) I sometimes receive questions from colleagues about usability. This post is about a question related to design and usability I received, here’s the question:

“Is there a conflict of interest in having the web design team who created a site also do the usability testing post launch? The web design agency has been tasked with ‘improving’ it’s own designs. They have outlined a ‘usability’ test plan that seems to be more about understanding user thoughts and reactions to a site they built for us, rather than measuring specific task-based assignments. And, even if they were to outline concrete tasks to measure, as testers, could they introduce bias into the process?”

Conflict of Interest of Design & Usability

As I sit here sipping my coffee, pushing the cat off the keyboard for the 4th time and contemplating this issue, I’m reminded of a famous saying, “Physician, heal thyself!”

I think it’s a generally accepted principle that a designer is usually too close to their own work to be an effective usability tester. Why? Because the act of designing is based on making hundreds or even thousands of decisions based on knowledge, intuition, context and experience. By making these decisions, a designer has on purpose selected and committed to what he or she sees as being the “best” outcome.

As a designer, it’s very difficult (note that I didn’t say impossible!) to remove yourself from a design 100% objectively. I would add as a side note here that unless the designer has also been trained in usability and usability testing methodology it would be difficult for him or her to conduct a test.

However, if in this case the Agency is using a separate test team for the usability work then that shouldn’t be an issue.

Design Agencies & Usability Staff

Most web site design agencies can’t afford to have full time usability gurus on staff, generally only the bigger ones can. From my experience, most agencies will farm this usability stuff out, and that’s a wise decision, it means there’s a separate trained usability team focusing on the usability testing.

Putting my agency hat on for a second, I’m pretty sure an agency would not want to come back to a client and say,

“Hey, good news! We did the usability testing of our design like you asked us to do (by the way, here’s our invoice) and we have great news! There’s no improvements we can make! The web site design we built for you is perfect!”

Let’s face it, that would cause great alarm with the client – who would probably be thinking

“BS! Now I’m nervous!”

But then again, the Agency wouldn’t want to tear apart their work either, calling it,

“Unusable crap! Holy cow, this thing is more broken than our economy!”

That would most likely cause the Agency & its designers to have a black eye, or at least a bloody nose. And the client would be wondering how they are going to explain all this to their boss.

Usability Testing Should Come Before Web Site Launch

Getting back to the question, first off, although it’s good that the agency wants to do usability testing to improve the web site after launch, it would have been MUCH better if they had done usability testing as part of the design process, from the very beginning of the project.

User-centered design means conducting quick usability tests at critical stages throughout the development life-cycle with your users, not after the design has gone live. This helps to ensure that the web site is going to be user-friendly well before it’s live for the world to see and gawk at.

Usability Testing is Not Asking User Opinions

Asking people to talk about their experiences or their opinions about web sites through focus groups is not usability testing folks, it’s opinion testing.

User thought and reactions are not “task oriented” usability testing, and although helpful cannot be relied upon as solid metrics upon which to base decisions. If the agency is recommending getting user feedback or opinions, I would suggest you should probably couple that opinion testing with task-based testing.

Task-based testing is used to determine exactly what the failure rate is for each critical task on the site, and where those failures are occurring. This is the only way to improve a web site based on actual usability data.

But don’t take my word for it. Here’s a quote from Steve Krug’s “Don’t Make Me Think” which is like the ultimate usability book and is even fun to read:

“… the conversation usually turns to finding some way (whether it’s expert opinion, research, focus groups, or user tests) to determine what MOST users like or don’t like – to figure out what the Average web User is really like. The only problem is, there is no Average User.

…it’s not productive to ask questions like “Do most people like pulldown menus?” The right kind of question to ask is “Does THIS pulldown, with THESE items and THIS wording in THIS context on THIS page create a good experience for most people who are likely to use THIS site?”

And there’s really only one way to answer that kind of question: testing. You have to use the collective skill, experience, creativity, and common sense of the team to build some version of the thing (even a crude version), then watch ordinary people [who match the Persona - Craig] carefully as they try to figure out what it is and how to use it.

There is no substitute for it.”

Powerful words. Hold on for a second while I come out of my zen-like usability trance… ok, I’m back!

Conduct Task-based Usability Testing Post Launch

So the answer to your question is this: Ask your design agency to replace the focus groups and user feedback with an actual set of 1-on-1 performance-based usability tests of the critical tasks. This will provide instant actionable data you can use to improve your web site.

The money (and time) you save in conducting the 1-on-1 performance-based testing will probably be paid back quickly with the improved conversion or related outcomes you are ultimately seeking.

Good luck!