Monthly Archives: June 2009

1 85

Twitter’s all A-Flutter Over Nielsen’s Alertbox: Stop Password Masking

Jakob Nielsen, usability guru, doesn’t look to me like a big trouble maker. However, his latest Alertbox, “Stop Password Masking” proves he’s ready to come out punching.

He’s caused quite a stir amongst some Twitter users – who seem extremely polar, being either pro or con on the subject of password masking. But don’t take my word for it, here’s a few examples of pros and cons from Twitter:

Why All The Password Angst?

So why has Nielsen hit such an apparently tender topic with his recommendation to remove password masking? First, let’s see what he actually said the problem is, and why the solution is un-masking our hidden passwords:

“It’s time to show most passwords in clear text as users type them. Providing feedback and visualizing the system’s status have always been among the most basic usability principles. Showing undifferentiated bullets while users enter complex codes definitely fails to comply.

Most websites (and many other applications) mask passwords as users type them, and thereby theoretically prevent miscreants from looking over users’ shoulders. Of course, a truly skilled criminal can simply look at the keyboard and note which keys are being pressed. So, password masking doesn’t even protect fully against snoopers.

More importantly, there’s usually nobody looking over your shoulder when you log in to a website. It’s just you, sitting all alone in your office, suffering reduced usability to protect against a non-issue.”

Seems harmless enough, right? If you’re trying to type your passwords in before your morning coffee has really got you going, or on your tiny Palm Pre, is having the results of your fumbling fingers being un-masked (so you can see if you entered the correct string or not) such a bad thing?

Ahh, but what about that whole issue when you’re at your favorite coffee place and those strangers who are always peering over at your computer from the next table behind you are watching [shivers] – what about then? Here’s what Nielsen says:

“Yes, users are sometimes truly at risk of having bystanders spy on their passwords, such as when they’re using an Internet cafe. It’s therefore worth offering them a checkbox to have their passwords masked; for high-risk applications, such as bank accounts, you might even check this box by default. In cases where there’s a tension between security and usability, sometimes security should win.”

So, assuming all our Tweeps actually read the entire article from Nielsen (and we already know that most people on average only read about 20% of the words on a web page) why is there so much angst about removing something that is causing frustration and slower productivity?

I think it can be summed up in 5 general reasons:

5 Reasons Why We Don’t Like Changing Password Masking

1. We are creatures of habit – You and me, us humans, we like getting into routines and sticking with them. We brush our teeth with the same toothpaste every day, we go to work along the same route, and for some odd reason we always end up picking the longest queue when offered shorter or longer ones.

It’s a habit to type a password into a small box and be presented with masked characters in return. It goes against our grain to want to change this habit.

2. Change is frightening – For most of us, change is a rather frightening thing. When things change, it causes us a mild sense of cognitive dissonance. What’s that? We know we should should be doing this usual and customary thing, our brains tell us so, but now we aren’t doing that, we’re doing something different. That causes us to feel ill at ease.

For many of us change means re-wiring our brains to accept and use something new, something different, which takes work. We don’t like work (this kind of work anyway – if my boss is reading this, I LOVE work! Really!).

3. Using password masking, we have a false sense of security – Here’s what we think when entering our passwords in and being presented with a series of round black dots.

“Well, nobody can see my password, including me, so that means it’s secure and I don’t have to worry about somebody stealing my password so I’m safe.”

But meanwhile anyone watching over your shoulder, or from the cube next to you (don’t look around!) can probably pretty easily see the characters you’re typing, and pretty much be able to figure out your password. Personally, I think it’s worse with bank ATMs, they only need to watch you enter 4 digits!

And your security breach may be far worse. You have probably written down every single password you own, including the URL and account name for each and every “secure” login, on a non-encrypted file somewhere in your computer. Or worse, you’ve got your password on a post-it note somewhere near your monitor! How secure is that!?

4. We don’t understand the actual lost productivity – I know what you’re thinking, which is what I was thinking, which was:

“So what’s the big deal? So I have to retype my password sometimes, or have to contact someone because I forgot it, no big loss, right?”


According to the United States Navy, whom I trust to do their homework regarding lost productivity due to forgotten passwords;

“studies have indicated that approximately 40% of all help desk calls are for forgotten passwords.”

If you don’t believe the United States Navy, go do a search on Twitter for “password” and read all the thousands of tweets from tweeps who forgot, then sometimes found, their passwords for various logins. All that “lost my password, but now found it” stuff is lost productivity.

5. Businesses have a false sense of Control – For a business, often there’s a sense of “protecting users against themselves” by trying to manage certain processes or procedures for them. You can almost hear those in control say,

“I can manage your password and make sure you only get it if you do things I want you to do.”

This is why there sometimes are extra hoops to try to get through when obtaining a password that you forgot. In the bad old days, many times if you forgot your password the only way you could get one was to get a new password. Which of course forced you to have to write down your new password because you just knew you were going to forget it.

The Option of Seeing a Password

So here’s my take on the whole uproar issue, the people that are in an uproar may not have necessarily read the entire message.

What Nielsen actually said,

“It’s therefore worth offering them a checkbox to have their passwords masked; for high-risk applications, such as bank accounts, you might even check this box by default.”

That doesn’t seem as bad as letting everyone know your passwords by sticking them to your monitor, or putting every password and account number and URL in an unsecure file on your computer, now does it?

0 84

It’s been 10 years since Mayhew wrote “The Usability Engineering Lifecycle,” How far has usability come since then?

It sometimes surprises me to find out that my parents were right. I distinctly remember them saying “Boy, time sure does speed up as you get older, where does the time go?” I thought they were crazy. I thought this especially when me and my other 10-year-old friends spent hours on the front porch saying “What do you want to do?” “I don’t know, what do YOU want to do?”

The Usability Engineering LifecycleSo it’s rather odd that it seems to me like just a few years ago, not ten, that Mayhew brought “The Usability Engineering Lifecycle” to the world. How far have we come in ten years? Do you feel like the last ten years just zipped by? What about usability, do you think it’s come a long way down the maturity path?

I wonder what you think?

Here’s what I think…

Usability Engineering Ten Years Ago

I recall that ten years ago I felt usability testing and user-centered design principals were not very common in development projects. I would say projects that included user-centered design were in the minority, the vast majority being conducted without a thought to usability or user-centered design.

For those of you who were there doing usability at that time, remember that whole Y2K issue and all the major projects to ensure all systems would work after computer clocks clicked over to 2000? For me, it was rather difficult to get web site projects done what with all the hysteria, but when they were worked on luckily I had a senior level Boss that was a champion for usability. Together, we inserted usability testing as part of the process of the major web site redesign projects we were responsible for.

Unfortunately, it was not that easy to insert usability into web and application development projects we were not directly responsible for. There were other business owners and their supporting I.T. staff in different divisions of our company that designed and built web sites or web-based applications with zero usability or user-centered design. Most of the time usability was not even thought of, but sometimes if we cajoled or otherwise influenced them regarding the value of usability we might get lucky, and usability would be included.

The worst case for us was internally developed and utilized applications. Intranets and employee-focused internal apps had zero usability, and there was no interest in “slowing down” to make changes that might impact the delivery dates of said applications.

If you were working in a business or a design or development firm in 1999 did you have the same experiences?

At that time, after pouring through “The Usability Engineering Lifecycle,” I felt like I had a new friend at my side! It was like a breath of fresh air. In this book was a detailed description of exactly how to incorporate usability and user-centered design as part of the Software Development Lifecycle. It provided helpful war-stories of how usability sometimes did and sometimes didn’t make it into the early stages of design and development projects. It gave me lots of good ideas for “selling” usability.

Usability Engineering Today

So here we are, ten years later, and according to a recent User Experience Maturity Survey by Human Factors International 52% of the over 1,000 survey respondents said they have a usability champion at their company that supports user experience design! That’s really good news! We have indeed come a long way baby.

If you are one of those who have an executive champion that pushes the user experience design method and insists it be present for all development projects then congratulations.

Clearly as witnessed by this study, usability and user-centered design have much more support today than they did ten years ago.

For those of us without an executive champion, there’s still more work to do to get user-centered design incorporated into the company.

Even today, ten years later, I’ve seen plenty of examples where there are parts of a company where the usability executive champion has sway, but in other divisions in which the executive champion does not hold sway usability and user-centered design is still not practiced.

For example, I’ve witnessed (and been forced to use) many intranets and internally-built employee-focused applications that are designed and developed with no usability – zero user-centered design! You can tell when you attempt to use them. Ouch.

Glass Half FullSo, we’ve indeed come a long way in getting usability institutionalized into the corporate fabric. The glass is half full! But (according to the study) we still have about 50% of companies that don’t have a usability champion. The glass is half empty!

If you are one of those toiling away trying to get usability in your business don’t give up! Consider reading, or re-reading “The Usability Engineering Lifecycle.” Even though it was written ten years ago, the concepts are still as true today as they were ten years ago, and the war stories just might help give you ideas you can use at your company to get usability added to development. There is still a considerable amount of helpful information you can put to use with your situation.

Usability Engineering 1999-2009

But that’s just my opinion about the last ten years of usability engineering. What do you think? Is the usability glass half full, or half empty?

1 98

Should an Agency, or Designer, Conduct Usability Testing on Their Own Designs?

Among the spam and family emails I get from my mom (hi Mom!) I sometimes receive questions from colleagues about usability. This post is about a question related to design and usability I received, here’s the question:

“Is there a conflict of interest in having the web design team who created a site also do the usability testing post launch? The web design agency has been tasked with ‘improving’ it’s own designs. They have outlined a ‘usability’ test plan that seems to be more about understanding user thoughts and reactions to a site they built for us, rather than measuring specific task-based assignments. And, even if they were to outline concrete tasks to measure, as testers, could they introduce bias into the process?”

Conflict of Interest of Design & Usability

As I sit here sipping my coffee, pushing the cat off the keyboard for the 4th time and contemplating this issue, I’m reminded of a famous saying, “Physician, heal thyself!”

I think it’s a generally accepted principle that a designer is usually too close to their own work to be an effective usability tester. Why? Because the act of designing is based on making hundreds or even thousands of decisions based on knowledge, intuition, context and experience. By making these decisions, a designer has on purpose selected and committed to what he or she sees as being the “best” outcome.

As a designer, it’s very difficult (note that I didn’t say impossible!) to remove yourself from a design 100% objectively. I would add as a side note here that unless the designer has also been trained in usability and usability testing methodology it would be difficult for him or her to conduct a test.

However, if in this case the Agency is using a separate test team for the usability work then that shouldn’t be an issue.

Design Agencies & Usability Staff

Most web site design agencies can’t afford to have full time usability gurus on staff, generally only the bigger ones can. From my experience, most agencies will farm this usability stuff out, and that’s a wise decision, it means there’s a separate trained usability team focusing on the usability testing.

Putting my agency hat on for a second, I’m pretty sure an agency would not want to come back to a client and say,

“Hey, good news! We did the usability testing of our design like you asked us to do (by the way, here’s our invoice) and we have great news! There’s no improvements we can make! The web site design we built for you is perfect!”

Let’s face it, that would cause great alarm with the client – who would probably be thinking

“BS! Now I’m nervous!”

But then again, the Agency wouldn’t want to tear apart their work either, calling it,

“Unusable crap! Holy cow, this thing is more broken than our economy!”

That would most likely cause the Agency & its designers to have a black eye, or at least a bloody nose. And the client would be wondering how they are going to explain all this to their boss.

Usability Testing Should Come Before Web Site Launch

Getting back to the question, first off, although it’s good that the agency wants to do usability testing to improve the web site after launch, it would have been MUCH better if they had done usability testing as part of the design process, from the very beginning of the project.

User-centered design means conducting quick usability tests at critical stages throughout the development life-cycle with your users, not after the design has gone live. This helps to ensure that the web site is going to be user-friendly well before it’s live for the world to see and gawk at.

Usability Testing is Not Asking User Opinions

Asking people to talk about their experiences or their opinions about web sites through focus groups is not usability testing folks, it’s opinion testing.

User thought and reactions are not “task oriented” usability testing, and although helpful cannot be relied upon as solid metrics upon which to base decisions. If the agency is recommending getting user feedback or opinions, I would suggest you should probably couple that opinion testing with task-based testing.

Task-based testing is used to determine exactly what the failure rate is for each critical task on the site, and where those failures are occurring. This is the only way to improve a web site based on actual usability data.

But don’t take my word for it. Here’s a quote from Steve Krug’s “Don’t Make Me Think” which is like the ultimate usability book and is even fun to read:

“… the conversation usually turns to finding some way (whether it’s expert opinion, research, focus groups, or user tests) to determine what MOST users like or don’t like – to figure out what the Average web User is really like. The only problem is, there is no Average User.

…it’s not productive to ask questions like “Do most people like pulldown menus?” The right kind of question to ask is “Does THIS pulldown, with THESE items and THIS wording in THIS context on THIS page create a good experience for most people who are likely to use THIS site?”

And there’s really only one way to answer that kind of question: testing. You have to use the collective skill, experience, creativity, and common sense of the team to build some version of the thing (even a crude version), then watch ordinary people [who match the Persona – Craig] carefully as they try to figure out what it is and how to use it.

There is no substitute for it.”

Powerful words. Hold on for a second while I come out of my zen-like usability trance… ok, I’m back!

Conduct Task-based Usability Testing Post Launch

So the answer to your question is this: Ask your design agency to replace the focus groups and user feedback with an actual set of 1-on-1 performance-based usability tests of the critical tasks. This will provide instant actionable data you can use to improve your web site.

The money (and time) you save in conducting the 1-on-1 performance-based testing will probably be paid back quickly with the improved conversion or related outcomes you are ultimately seeking.

Good luck!

Human Factors International Recently Published their “User Experience Maturity Survey 2009” Report, and there’s good and bad news about usability – especially about usability champions

User Experience is Not Yet Mature, Based on the Lack of Usability Champions

Reading the User Experience Maturity Survey that HFI recently conducted is a lot like reading your 401k statement. Sure, the good news is you have money in there, but the bad news is it’s a lot less than you were hoping for. Likewise, the User Experience Maturity Survey has good news and bad news, most especially about Usability Champions.

Over 1,000 survey responses were received (1,123 to be exact) which is a nice surprise, as I didn’t realize there were so many companies out there with usability practitioners on-board who were willing to share their experiences in usability with the rest of the world. Hurray for us and for usability!

More good news! The majority of survey respondents reported that User Experience (aka UX) fit into their business for:

#1 – Building web sites (excellent, usability in websites is very important)
#2 – Creating Applications & Software (great, usability helps there too)

Conducting usability work on web sites and applications is like putting a nice warm fuzzy blanket around the web site and giving it the tender loving care it needs to truly be successful out there in the big cold world. Well, not really, but you get the idea.

Half of Us Don’t Have a Usability Champion

But now to the bad news, almost half (48% to be exact) of the survey respondents said their organization does not have a visible, committed Usability Champion supporting user experience design.

Let’s stop right there for a minute. Let’s you and me put down our Blackberrys for a second and focus.

Most of the people who responded to this survey are doing usability work, and they are doing usability work for important things like building websites and creating applications. But barely half have a usability champion! Why, that’s like NASCAR drivers without a Pit Crew! It’s like a horse race without Jockeys! It’s like your wedding without a piece of wedding cake you can mash into your Spouses face (even though you were warned NOT to do that).

Folks, without a usability champion it’s going to be very hard to have a clear and actionable usability practice that operates as one of the core business units for your company! You’ll be stuck doing last second usability testing as an after-thought on almost completed projects that someone is realizing only now is totally screwed up and needs some sort of help, any sort of help, for the life of your career at your company! That’s sad, it’s like having to live in Seattle and constantly be rained on.

Finding a Usability Champion

So, if you are one of those unlucky 48% of usability practitioners that don’t have a usability champion, or you live in Seattle, here’s 3 things you can do about that (um the usability part, for the Seattle part your on your own – just kidding all you Emerald City dwellers, I heart ‘ya and your City, especially the Space Needle, it’s cool):

1. Find a Usability Champion – Most marketing and sales VPs are looking for ways to get their bonus achieve their vision for making their ever increasing web site sales goals. If there was a way you could show them that by improving the usability of their eCommerce web site their sales would increase, then you would have an instant Usability Champion (and probably be invited to some of those cool “Marketing/Sales after work parties”). Try researching “usability roi case studies” on Google, you might come across some helpful ammo with which to approach your Marketing VP.

If you don’t have a sales or marketing VP (really? remind me again how your company sells things and makes money?) then you might have a Product or Product Development VP. Here to, if you can demonstrate that by improving the usability of the product more people will find the product easier to use, and that will help increase use (sales?) of the product, then you might have a new friend.

HINT: The product development VP is probably worried about getting their bonus achieving their vision based on delivering a product on time. You’re going to have to prove to a rather skeptical person that you won’t slow him or her down in getting that product out the door while conducting usability work. I provided some hints on how to do faster usability testing which might come in handy here.

2. Become a Usability Champion – Nature abhors a vacuum (so do I when my wife tells me it’s time to vacuum the stairs!) so here’s a chance for you to step in and fill that usability void. Push hard for usability projects to anyone who’ll listen (at your company, your mom already knows you’re the best usability person in the whole wide world – whatever the heck “usability” is).

If you’re brave, you might actually conduct your own usability evaluation of your web site or product and provide an executive summary to some well placed VPs – just for reading material the next time they have to fly somewhere. Try to make friends with the coders and developers of the web site or application and see if they’ll become “usability friendly” – try taking them to lunch, that might help.

Also, read the post I wrote on the 7 enterprise usability tips for ways you can apply usability projects beyond just a web site, if that avenue is closed to you. There’s lots of potential usability projects lurking at your Company, in many different divisions. Consider IVR messages, customer service web sites or applications or even internal applications that all employees have to use.

The point here is perception is reality, and if all you do is talk about usability and how it helps increase sales or interactions or whatever, eventually people will believe you are the Usability guru and champion.

By the way, it’s important that you are a usability Champion that can actually get usability projects approved. Being a usability champion without the authority to use the company pen to sign a company check to pay for a usability project means you don’t have the authority to make usability projects happen, and thus you’re probably not a usability champion. Some of your co-workers in your office might be calling you just a “usability nut,” the mean ones a “usability chump-ion” (ouch).

3. Leave the Company – Guess what? Sometimes an old, non usability friendly door must close before a new, usability friendly door can open. If the corporate culture is dragging you down, and you’ve tried everything you can (see numbers 1 and 2 above) to get a usability champion at your company, it might be time to leave and find a company that does have a usability champion.

Don’t think of it as failing, being a quitter or throwing in the towel. Think of it as your company is not able to incorporate usability as a part of it’s core function, and thus you must find a better environment where you can make a difference practicing usability. Those losers.

According to USNews & World Report, you, yes you, as a usability experience specialist are in one of the Best Careers of 2009. If your current lame unresponsive dimwitted ignorant company doesn’t truly value and support usability, more than likely there’s another company out there who will.

So dust off your resume, dry-clean your interview suit, shine up those shoes and hit the pavement you usability experience specialist you! You’ve got a whole new exciting opportunity just WAITING for you to get started! What are you waiting for? There’s a better opportunity for you and all you have to do is take that first step!

Conclusion – User Experience is Not Mature, Especially without a Usability Champion

According the results of HFI’s recent “Usability Experience Maturity Survey 2009” almost half of those of us who completed the survey are without a usability champion. Without a usability champion at your company it’s going to be very difficult to achieve usability maturity, and include usability as a core attribute of your company’s operations.

If you’re not comfortable trying to accomplish usability without a usability champion, you can either find one in your organization, become one, or worst case leave the organization and join one that does have a usability champion.

0 274

Web site managers have many choices when testing their sites, including usability, A/B or Alpha/Beta testing, but which one is best, and why?

Part III, The Pros and Cons of Usability Testing

This is Part III, the Pros and Cons of usability testing, of a three part series on web site testing methods. Part I, the pros and cons of A/B testing started the series, and Part II, on the pros and cons of alpha/beta testing, well, was in the middle.

A recent Tweet from Eric Harrigan asked an intriguing question:

“If you could implement only one website testing program: A/B, Alpha/Beta or Usability, which one would you pick and why?”

I found myself thinking about this question and I think (and by the way I need to place a brief disclaimer here and tell you; I’m a bit biased what with being a Certified Usability Analyst and all, so you have to consider the source) that the answer is…

“It depends”

Lame answer you say? Hardly!

You see, there are positives and negatives with each web site testing method. When applied in certain circumstances, one choice may actually be a better pick than another (assuming you have to pick one).

At first, I was going to blog about the pros and cons of each of the three types. But, my post kept getting longer and longer and longer, so I thought I better cut this post back. Therefore I’m chopping this into 3 parts:

Part I  Pros and Cons of A/B testing
Part II  Pros and Cons of Alpha/Beta testing
Part III  Pros and Cons of Usability testing with a summary comparison of all three.

I’d like to stop you here and mention that a smart web site manager will use all three testing methods, plus additional sources of testing, when optimizing a site, but that spoils the fun of being forced to mentally pick one, so let’s proceed.

Pros of Usability Testing a Web Site or Application:

1. Tests Real Users – A core attribute and strength of proper usability testing is – real users are the test participants. If usability could flex its biceps at you, this would be the reason why it would do it. Usability is a user testing stud!

Now this assumes that Personas were developed prior to testing, and that users who match the Persons were invited to participate in the testing. What’s a Persona? It’s a fictional representation of a typical user. Personas include behavioral and needs-based information about the typical user, enough such that any design or testing decisions can be made based on the information that comprises the Persona.

Testing the actual people that the web site or application was originally designed for in 1-on-1 testing sessions provides unparalleled opportunity to observe actual user behavior. You’ll discover where the road-blocks are in a task flow for those users almost immediately.

2. Tests Reality, Not Opinions – Usability testing, specifically 1-on-1 usability testing, is typically used to test a task-flow, to determine where in the process there are issues that either slow down, or completely stop a user from completing a task.

Unlike other forms of testing this means you’re getting measurements based on actual behavior and actual events, not opinions or assumed intentions based on secondary data.

This also means there is no need for large numbers of usability testing participants, because usability testing is not opinion-based survey testing, and doesn’t rely on statistically-significant numbers to produce valid results. If 6 out of 7 usability test participants fail to complete a task, you don’t need to test 300 additional participants to see if the results are valid. The task has failed reliably and usability testing documents this.

3. Can test throughout design & development – If you fly from London to New York, or any long distance, your pilots are making minor course corrections as you journey along the way. Consider what might happen if they didn’t! By the time you are nearing the end of your trip your airplane could be hundreds, perhaps thousands of miles off course! You might have to land in Canada, or worse, New Jersey (just kidding all yous guys from Joisy, I heart ya!).

The point is minor course corrections while traveling are easy, major course corrections near the end of a trip are hard.

Your web site or application development process is also a trip. At the end of this trip, you’re supposed to have this amazing web site or application that your users find so easy to use, they just love it and use it and you make lots of money have lots of satisfaction!

Consider usability testing as your opportunity to conduct little course corrections along the development process journey.

If you conduct usability tests during key points in the design and development process, it allows minor (aka low cost) corrections to be made to improve the user experience. This ensures your web site or application will not have to make any major (expensive!) corrections at the end of your development process, or worse, have your application end up in Joisy (just kidding again)!

No other form of testing enables you to have specific user-based data that is so clear, and so actionable, for the entire length of your development process.

4. Provides the Why of user behavior – All those mysteries around the numbers in log files, like; “why does everyone click on A and not B?” can now be answered.

The usability testers will (if prompted correctly) explain their thoughts and actions to you. This provides the behavioral insights as to why certain choices are being made, or not being made.

Of all the testing methods out there, including by the way tracking software, this is the only way to learn why users do the crazy things they do interact with your web site the way they do.

Cons of Usability Testing for Web Sites and Applications

1. Trained Usability Professionals required – Unlike Doctors and Lawyers, you don’t need a license, degree or ice-cold waiting room to practice the art and science of usability testing. This means anyone who wants to claim they are a usability practitioner can do so. Without extensive knowledge of usability testing theories, methods and best-practices, you might end up with a usability test that provides poor, or worse, bad advice.

The best way to ensure you’re dealing with a reliable trained usability professional is to do your homework before you hire them. Ask for referrals and call those referrals. Research the usability practitioner and their company. Have they been in business a long time? Are they active in Associations like the Usability Professionals Association, SIGCHI (Special Interest Group on Computer-Human Interaction) or related groups? Do they have training and/or a degree in the Human-Computer Interaction field? Do they blog about usability? Do others in the usability field blog about them?

As with plastic surgery and iron-clad prenuptial agreements, your usability fate lies with the expertise of the practitioner, make sure you hire a good one.

2. Testing won’t reveal all issues – Usability testing is not going to find every problem with a web site or application. There are great debates that rage from time to time in the usability field about how many 1-on-1 usability test participants it takes to do a proper test.

But that’s not the correct question. The correct question is, “How many problems do I think I need to find?” Is it 50%, 75%, 80%? Based on that estimate, the proper number of test participants can be gathered.

Even with lots and lots of test participants, usability testing simply won’t find all the problems with a web site or application.

The important thing to remember about applying usability testing to find problems is that you’ll probably find most of the more significant issues, and fixing those probably will get you much closer to an optimal user experience.

Just realize going in that it’s unlikely you’ll find all the problems, at least in one testing session.

3. Usability Testing results can vary – A while back some studies of usability testing methods and results were conducted. A single web site was tested by groups of usability professionals. You may be surprised to learn that there were just as many test methods, and test results, as there were groups of testers.

This is because usability is part art (dealing with humans and design) and part science (using standard methods of conducting tests that generate repeatable results). Meld the two together and you have an inexact science.

Remember that whole “make sure you do your homework about your potential usability professional” message above? Ditto here! Your usability testing results can and will vary based on the expertise of the usability professional.

4. Testing & fixing are different things – A common misconception among usability clients is that the act of conducting usability testing will fix their web site or web application. In reality, as with other forms of testing usability testing only finds the usability issues and makes recommendations for improvements. It takes a separate effort to actually do the work to make the recommended changes.

Problems arise when the recommendations for usability improvements cannot be made, either due to business or technology issues, or due to the costs in terms of resources or time necessary to make the recommended changes.

When this happens, the dance that is called “trade-offs” happens, meaning sometimes important usability optimizations are not done. Sadly, it’s a rare usability engagement that has all of the suggestions for improvements done. Sometimes some easy changes (the low hanging fruit so to speak) are completed, but the more difficult but important changes are left undone.

This might leave a bad-taste about usability with some business owners. After all, wasn’t the usability testing supposed to fix our problems? Um, no. Find the problems and suggest solutions? Yes! Actually fix them? No.

Be sure to include time and resources at the conclusion of usability testing to make the recommended optimizations. This ensures your usability testing money is not wasted.

Conclusion – Usability Testing Web Sites and Applications:

usability testing

So here’s a summary of usability testing for web sites or application you can print out and carry with you in your purse or wallet. Collect them, trade them with your friends! Well, maybe not. Just memorize these as there will be a test later.

Summary of A/B, Alpha/Beta and Usability Testing Methods:

As promised, here’s your handy-dandy summary of testing methods that you can print out and share with your boss, friends or the guy that you see jogging every day and wonder how come he doesn’t have shin-splints already. I hope you’ve enjoyed the series, please let me know if you did, or even if you didn’t!

And don’t worry about the test, I was just kidding about that. The real test is will you actually USE these testing methods to improve your web site or application!

A/B testing summary

usability testing

Prior testing articles in this series:
Part I  Pros and Cons of A/B testing
Part II  Pros and Cons of Alpha/Beta testing