Should I usability test my own site is a question I am often asked.
The answer to the question “should I usability test my own site” is not as clear nor as obvious as you might think.
There are multiple reasons why a quick “yes” or “no” doesn’t necessarily apply.
In this article I will explain the issues of testing your own website so you can be more informed when it’s time for you to conduct usability testing.
Usability Testing Your Own Site Definition:
So what exactly is usability testing a website? My definition is:
What I mean by that definition is “usability testing” is conducting performance-based tests on a website’s critical tasks using people who match the website’s Personas. What’s a Persona? A Persona is a representation of the most common website visitors who all share a set of critical tasks.
It is very important to remember that it is the critical tasks that are being tested, not the tester’s opinions about those tasks.
Likewise, usability testing is not about capturing survey data, or voice of the customer information (although both of those can be helpful additions to the performance-based testing).
So when people ask me,
“Should I usability test my own website myself?”
I reply with,
“If you mean you being the test participant? No. Because you can’t usability test your own site yourself conducting performance based tasks. You probably don’t match the Persona, and you are already too biased and know the correct task flow.”
Running But Not Participating In Tests:
Some of you may be wondering a similar question, which is,
“Should I set up and run a usability test on my website only (ie. not participate in the testing)?”
Here the answer is a bit more cloudy.
In general, I would caution against usability testing your own website, there are 5 reasons why:
1. Usability Training Matters:
Unless you are trained in usability testing, it’s dangerous to assume you have the knowledge and expertise necessary to very carefully write a non-biased testing protocol.
It is amazingly easy to find tools that anyone with or without training can use to conduct usability testing. But it is also amazingly easy for those without usability training to accidentally create a biased test that provides bad data.
When I was in the process of becoming certified in usability analysis I attend many classes prior to taking and passing my certification test. I would say about half the classes were focused on helping students understand biases; where they can creep in, how to uncover them, and how to write protocols that carefully eliminate them.
Without that advanced training, conducting your own testing can introduce hidden biases in the test protocol, which could make results invalid and cause resulting website usability changes to make things worse, not better.
2. Internal Usability Teams Can Be Biased:
Although less common, I’ve come across situations in which I was surprised to find that even internal company usability teams had hidden bias’ in protocols. Not always of course, but often enough over the past 20 years of my experience that I’ve decided the longer you spend time with a website or company, the more difficult it becomes for you to remove your own biases from the process.
That’s not to say that internal company usability teams can’t write non-biased protocols. It’s just that it becomes more and more difficult as domain expertise rises AND testing patterns become standardized.
Let’s face it, as humans, we are all creatures of habit. And for internal teams this can sometimes unfortunately mean picking up biased habits and sticking to those habits when writing protocols and creating tests.
3. Usability Testing Protocols Are Mandatory:
Sadly there’s a common misconception out there that anyone can usability test simply by walking down a hallway or using software to ask people to try to do something on a website and observe them.
I’m here to tell you that’s a dangerous oversimplification of how to actually conduct non-biased tests.
To ensure that bias is not in the usability test, it’s critical that a usability testing protocol be developed AND be reviewed in advance. Without the protocol, random “guerrilla usability testing” or “agile testing” can accidentally introduce bias into the process.
Some usability practitioners, entrepreneurs and usability testing software services may disagree with me.
But I’m here to tell you the primary mission of a protocol is to ensure you’re reducing bias and that you’re testing across a standard set of criteria for each test you conduct. Without it, test results may vary due to the unintended introduction of biases.
4. Moderating Usability Testing is Harder Than It Seems:
Closely associated with needing a protocol, moderating the actual usability tests without bias is not easy.
The test participant is looking for clues (after all, it is a test). They will instinctively read a moderator’s body language and other verbal or non-verbal queues. Being on guard to not generate those clues while at the same time gathering think aloud feedback plus keeping the tester focused takes skill.
The common misconception that anyone can usability test without training or bias is false.
It is best to let trained usability professionals handle moderating usability testing, else risk bad data entering the process.
Again, entrepreneurs and do-it-yourselfers may scoff, but I’ve seen plenty of bad websites made worse by erroneous data coming from badly moderated testing sessions.
5. Analyzing and Prioritizing Usability Testing Results Requires Expertise:
Analyzing usability testing results and prioritizing them also takes skill that I caution should not be handled by untrained usability practitioners.
Here’s a common scenario:
You’ve conducted usability testing and found 8 places in a task-flow with errors causing poor performance.
What’s the severity, and thus priority, of those errors?
Analyzing issues and identifying possible solutions requires a fair amount of user experience and usability best practices knowledge.
Going with gut feelings about the best way to solve an issue can sometimes work, and sometimes not. It’s better to analyze the results and make recommendations based on extensive experience in understanding UX and usability best practices.
Associated with this, I have seen plenty of bias in prioritizing results even by internal usability teams. That’s because being so familiar with their organization, they knew what could and could not easily be worked on by developers. They also knew what would and wouldn’t be easily supported in terms of workload by business owners.
How many times did the “hard to do stuff” take a back seat to the “easy to do stuff?” And, did the “hard to do stuff” have a much greater impact on usability than the “easy to do stuff?” Often it does, but often it’s not prioritized that way due to internal bias.
Conclusion: Should I Usability Test My Own Site?
So should I test my own site?
Now that you have a clearer picture of the issues around testing your own website, I think you’ll agree a quick answer of ‘yes’ is not always appropriate.
This is NOT to say that usability testing your own site cannot or should not be done!
Instead, it’s a warning that when testing your own site it is very easy to let bias into the results, thus causing bad data and potentially bad results from subsequent optimization.
Being aware of these issues, my recommendation is to consider your resources, and if it makes sense contact a professional usability consultant like me or others in our industry anytime you’re considering usability testing on your own website.
Further Usability Testing Resources:
Usability.gov – Usability Testing Methods
UsefulUsability.com – 14 Usability Testing Tools
Human Factors International – Certified Usability Analyst (CUA) Training
Nielsen Norman Group – Full Day Usability Testing Training