Review of the usability book ‘Remote Research’ by Nate Bolt and Tony Tulathimutte
For those of you who’ve been following my blog for a while (hi Mom! hi @kailualisa!) you know the risk Nate, Tony and Lou Rosenfeld took asking me if I’d like to read and potentially review their new usability book, “Remote Research.”
As you know, I don’t pull any punches, and I calls ’em like I sees ’em.
Although I was provided by the authors with a free evaluation copy of the book – I am fully prepared to skewer it like a hot knife through butter if I think it’s a waste of your and my time (and money!).
The news for you, Nate and company is: I have read the book cover to cover and I can state in no uncertain terms that this is a usability book that usability practitioners, or those learning about or just interested in usability, should buy and can use over and over.
Simply put: You’ll use this book, a lot.
For a nominal fee of US $36 (softcover and DRM-Free PDF, or $22 for 2 editions of PDFs) you will have a very handy how-to / reference guide for conducting remote research you will use again and again.
Here‘s some of the key points covered in the book:
- What remote usability research is (and what it isn’t)
- Pros and Cons of conducting remote research
- Step by step how to (for first timers)
- Detailed list of tools (both moderated and un-moderated)
- Tips and tricks for each tool (experienced remote researchers take note!)
- A companion web site with additional tools and information is included
Gosh, Nate and Tony even go so far as to include a rebuttal from remote research h8r opponent Andy Budd, creative director at Clearleft, as to why remote research should NOT be used.
I think Nate and Tony have done a good job of being fair about this subject, sharing both the pros as well as the cons of remote usability testing.
Review of ‘Remote Research’
The writing is light and easy to read, you won’t feel at all like you are reading a manual or dictionary. Jargon is kept to a minimum, and for each new term they use Nate and Tony explain what it means in plain English.
Rather than putting me to sleep at night (I’m looking at you, ‘The Complete History of Taxation’) I found the writing to be fun, engaging and sprinkled with humorous yet relevant dialog.
Reading this book is very much like having a fun yet informative lunch with a remote research guru – you learn a lot, and enjoy the experience along the way.
Remote usability testing for first timers:
For some of you, it’s probably not necessary to read the entire book cover to cover (but you should, because you will be missing some humorous comments and very helpful pointers). Those who have not yet conducted remote usability testing should read Chapters 1-5 and 7 (I’ll break out more information about each chapter below).
Those chapters are perfect for first timers, as they provide the overview of what remote testing is, along with step by step instructions for conducting your first remote usability test. Included is a detailed listing of all the tools (there’s not many) you’ll need.
Remote usability testing for old pros:
For those of us who’ve already been conducting remote usability testing sessions, you’ll probably want to focus more on Chapters 7-9, and potentially Chapter 6 (Automated tools). That’s not to say you shouldn’t read the other chapters, I learned some new ideas and tips in the overviews that I’m anxious to try, just that if you are pressed for time you may find those the most actionable.
So here then is a brief overview of each chapter with some of my observations about it included:
Chapter 1 – Why Remote Research?
The book starts with a very helpful overview of the various types of remote usability research methods. It also includes a very helpful case study. I think this chapter provides some useful tools you can use to “sell” remote usability testing internally, if you are in an organization that is resistant or unaware of the benefits. Likewise, if you are selling remote usability research to prospective clients I think you will want to read this and use the information.
However, I do have one point of contention. In Chapter 1, and later in the book Nate and Tony refer to the cost of remote researching being almost equal to in-person moderated testing, thus not having a big cost savings. This in my opinion is not exactly accurate.
They were apparently basing their cost from the perspective of a consulting agency using remote vs. in-person testing, and thinking mostly about the billable hours of moderation and analysis – not from a Corporate (i.e. Client) infrastructure perspective.
At several companies I worked at, I created detailed studies comparing in-person vs remote testing costs if the company were to:
- Build and maintain a usability lab
- Rent usability labs
- Use remote testing technology and methods
My analysis revealed that remote testing was the clear winner for lowest cost, by a landslide.
Building a lab for in-person testing vs. using remote methods only made sense if the company would either use the lab every day (none of the companies was big enough to do that) or rent the lab out to 3rd parties during down time (none thought that was a good idea).
Renting usability labs for in-person testing was far more expensive than using remote methods. When you look at the total costs after just one year of usability testing remote easily won. And if you compared both after several years of usability testing, the costs of paying for travel, hotels, per diem and lab rental fees for rental usability labs was astronomically more expensive than the costs for using remote research methods.
My personal opinion? There’s almost no difference in results for conducting usability test sessions remotely vs. in-person with a usability lab – the methodology and tools for conducting the testing are almost identical, and the task-error results will be the same. My belief is unless usability labs can add additional value beyond straight usability testing, over time they will go the way of 8-track tapes and VCRs (technologies that became cumbersome and too expensive vs. newer tools).
Again, that’s just my opinion and I hasten to add is not mentioned in the book.
Chapter 2 – Moderated Research: Setup
In this chapter Nate and Tony provide a detailed step-by-step process for actually conducting a real remote usability test. I believe the information is perfect for beginners, and even seasoned pros may find some of the information new or different, I know I did. I found the table of screen sharing tools at a glance (on page 34) to be very handy, as well as the detailed scripts and observer instructions. You’ll probably dog-ear the example script pages here, and use them later, I did.
Chapter 3 – Recruiting for Remote Studies
Recruiting for remote usability testing is potentially one of the trickiest parts of the whole process – and must be done right – else the study can be flawed. Nate and Tony do a great job of providing plenty of detail and tips on how to recruit, and how to avoid using bad recruits (fakers and professional survey takers).
Chapter 4 – Privacy and Consent
Legal stuff is boring. But I think you should really, really read this chapter – you’ll thank me later when you avoid a multi-million dollar privacy lawsuit. The legality and privacy issues of recording a participant is a critical element of usability testing, and there’s very little information out there about what to watch out for. This chapter provides excellent information and should, after you read it, make you reach for the phone to call your lawyer and get more information about the dos and don’ts of legal language for your consent forms.
Chapter 5 – Moderating
Moderating is tricky, and Nate and Tony do a good job providing details and insight into how to be a better remote testing moderator – along with very handy example scripts. Unless you do a lot of moderating (and by a lot I mean doing it every day) I think you’ll really appreciate the guidance and tips in this chapter.
Chapter 6 – Automated Research
Nate and Tony do a great job explaining what automated research is (also called un-moderated), what it’s not, how to use it and when not to use it. I’ve dog-eared plenty of pages in this chapter – especially page 128 which contains a really great chart of the major automated research tools and where they sit in the dimensions of Qualitative vs. Quantitative and Concrete vs. Conceptual. I think I may blow this chart up and place it on my wall.
Chapter 7 – Analysis and Reporting
It’s a funny thing about usability testing, there are lots of standards for how to do it, but almost no standards for how to analyze and especially report on it to clients. This chapter covers both moderated and un-moderated analysis and reporting, and provides some really nice examples of ways to report effectively on all the data captured. Also included are examples and tips of how other usability experts report results to their clients – good stuff.
Chapter 8 – Remote Research Tools
This chapter is worth the cost of the book all by itself. It provides the best list I’ve seen of remote tools, including screen-sharing, recording and automated tools and services. And here’s the really great part, Nate and Tony provide tips and tricks for how to use each of these tools. From Adobe Connect to Webnographer, this is a comprehensive list and cheat-sheet of testing tools. I am almost certain that you’ll dog-ear almost every page of this chapter, just like I did.
Chapter 9 – New Approaches to User Research
Nate and Tony provide some very interesting new tools and techniques, including one of my favorites, reverse screen sharing. There’s also some good advice and guidance on mobile device research and research on video games. They’ve even included information on conducting remote testing in cars. Considering the rapid expansion of multi-modal mobile devices I fully expect in the next year or two to see a lot more research and research information about testing these devices in their “natural” environment such as the car, train, home or bleachers at the kid’s softball game.
Chapter 10 – The Challenges of Remote Testing
This chapter is an excellent listing with details about the issues and potential problems associated with remote usability testing. Everything from legitimacy to getting the right recruits to persistent negativity is covered. I think Nate and Tony did a good job keeping it real, so to speak, with being up-front about when to use, and when NOT to use, remote testing, and what some of the problems and pitfalls are.
Conclusion – Book review of “Remote Research” by Nate Bolt and Tony Tulathimutte
So, with “Remote Research” I believe you have a very useful and practical guide to remote usability testing. This book is easy to read, is written in a open, honest and humorous way, and will provide you with a large amount of useful information you’ll be able to use again and again. I have a special place for books I refer to often (down low on the wall bookshelf, where I can reach them) and that’s where this book is going.
I suspect you’ll place this book in an easy-to-reach place as well.
You can buy the book “Remote Research” at Rosenfeld Media, or at your favorite online bookseller.
- “Remote Research” By Nate Bolt and Tony Tulathimutte
- Copyright 2010, Rosenfeld Media, LLC
- First published, February 2010
- ISBN: 1-933820-77-2
- 266 Pages
- Available in paperback and DRM-free PDF (US $36.00)
- 2 digital editions (screen-optimized and print it yourself PDFs) (ISBN 1-933820-44-6) (US $22.00)
In case you were wondering if I’m making any profit from this review, I’m not. The authors/publisher of this book provided a free evaluation copy of this book for my personal use. I was not compensated for this review nor required to write this review. The views, content and opinions expressed in this book review are mine alone. Links from this review to the book web site are not affiliate links and I receive no money for traffic sent from this site to the book site nor for sales of the book. I’m honest and non-biased. Broke, but non-biased. Hey, that’s how I roll.