Monthly Archives: March 2011

SXSW 2011 – Long After the Thrill, Sustaining Passionate Users


Stephen Anderson

SXSW 2011 Stephen AndersonHow do get people to fall in love with our application and keep sustaining passionate users? All the talk in the past few years would seem to imply the answer is “make it a game!” Maybe, reasoning goes, it should be more gamelike! A quote Stephen recently saw was:

“Motivating consumer behavior through game mechanics.”

But, he says, that’s actually a wrong way to think about it.  The right way is it’s human, not consumer, it’s psychology, not game mechanics.

“Motivating consumer human behavior through game mechanics psychology.”

He mentions the game motivators that incentivize and excite us humans, things like:

  • appropriate challenges
  • variable rewards
  • pattern recognition
  • curiosity
  • reputation
  • social proof

Stephen says let’s focus on appropriate challenges, the heart of most gamelike activities. What about sustaining passionate users through delightful challenges?

Let’s go back to the classroom. When I was teaching, I saw three attitudes about teaching and motivating students:

1. “This stuff is boring. I’ll make the best of it, but you’ll have to work and apply yourself to get something out of this.” This equals “Apply Yourself” typical web design, we’ve seen this plenty, like find the FAQs, read the “Read Me” instructions, etc.

2. “This stuff isn’t all that interesting. But I’ve added activities to the content that will make this a lot more fun for everyone.” This is called “Sugar Coating” and in my opinion is what most gameification is all about right now.

Why has gamification taken off? Because whatever the thing is that we are making a game, now has a layer of fun, people are likely to engage with this thing. Right? But this is flawed. Games are play and challenges. Goals and rewards reinforce the goals and challenges. But the right way of making something a game is to go learn about game design, which is what I’ve done. I read things like, “Theory of Fun for Game Design” and “The Art of Game Design.”

Gamasutra is a great website to get more information about games. “Drive” by Dan Pink is a great book that talks about motivators, not necessarily just about games, but has implications for game design.

PRO Tip: read those books, or for speed just follow along with these notes.

Exercise, we’ve all been given 3×5 cards as we came in, now he wants the audience to do a fun gamification exercise – our subject for gamification will be time tracking. Stephen believes the best time tracking tool is Harvest. So, we need to track our time, but we don’t do it. We lack the motivation to track the time, so let’s start with something easy to use, and try to make something a bit more gamelike.

So, think of any type of game, and write the game on one side of a card – but make it simple – not complex – don’t’ write World of Warcraft for this exercise.  (I wrote “Trouble.”) Now flip the card over. Consider FarmVille or Pictionary, it has limited time, teamwork, self expression and pattern recognition.

Look at your game and try to identify three or so items from:

  • Limited Time
  • Group Competition
  • Teamwork
  • Self-Expression
  • Pattern Recognition

(On my card I wrote: “Group Competition, Limited Time, Pattern Recognition.”)

Doing this helps you see design from a different perspective.

We need to move beyond the usual points and badges part of gamification, to get to the real part of designing games.

But, with gamification there’s a big flaw: He notes The Golden Age, listen to a song as part of an assignment. Another example, Old Navy hid coupons on the site, hiding Easter eggs on the site is a great game device. That was very effective. He shows a sign up form of “Fastest sign up 16 seconds, can you beat that?”  Here’s the issue; after doing something several times the interest starts to wear off. It’s a problem if nobody really wants to do it or care about it.

One of the motivators, “Set Completion” can work. Stephen says we are compulsively oriented to finishing an already started activity. Are there sets that we can work with in time tracking? Stephen talks about his use of Gmail priority inbox. A way to get to an empty inbox by using sets to categorize activities by priority.

And the third teaching motivation?

3. “This stuff is really quite interesting! I’m going to show you why this is important. But first, I’ve got a challenge for you…”

He tells us an example, about 50% of elementary kids skipped recess to watch an education video. Stephen says we like a challenge of completing something.  He notes there is a difference in Performance Vs Learning goals

Performance goals – Getting an A in French

Learning goals – Desire to learn how to speak French

Another audience exercise, this one is called Laddering – The idea is we pair up, one person starts by asking a question to the other: Why are you here at SXSW? The Five Whys. When the second person responds, the first person rephrases the response into another “Why” question.  We do that about four or five times. We have two minutes to do this as an exercise.  So, now do the time tracking gamification with the Five Why’s.

“What’s a core challenge for your time tracking app?”

For Status, it’s often confused as reputation, but the two are different. Status is about us vs others, but also us vs. ourselves. Gowalla does this with number of checkins, can I beat my own record? Over at they use the same element of status to not break the chain of completing red X’s to say you’ve done something, accomplished something.

So going back to time tracking, what about status?

  1. To know how long I’m taking
  2. To learn where I’m underestimating
  3. To get better at estimate my time (this is where I focused)
  4. To have a more balance life

For a day, what about an estimated plan of the day, vs the actual number of hours I spent during the day, the challenge is to get to a 100% accuracy rating.

Play & challenges + goals & rewards = game

Most games have; Play & Challenges, then Conflicts & Choices, then Goals & Rewards, then Feedback Loops. Consider a target circle with Play & Challenges in the center, then going outward we have Conflicts & Choices, and outside that ring we have Goals & Rewards, and at the outer-most ring Feedback Loops.

Example of Scarcity in designDribbble – You can only select a few shots to upload, which makes scarcity force the behavior of only uploading great, high quality shots.  That’s a great way to force quality.

Examples of Feedback loops – Aza Raskin, in a session here on healthcare and using gamification, said that what is the one secret to changing human behavior? Feedback loops! Think about a speed limit sign on the side of the road showing how fast you’re going, if you’re going too fast what do you do? Usually it’s slow down.

Attaching a measure to something turns it into a game.  Adding a number to something, for example miles per gallon gauge in a car, makes it a game if you try to improve your gas mileage, or see what the very best it that you can get. Another example is virtual flower on a pedometer that is a powerful metaphor for growing. Imagine if your credit score was a balloon that was either inflated or deflated depending on your score.

Consider creating a table:

Play & Challenges          Make lots of money

Conflict & Challenges    Pressure to make choices

Feedback loops              Stock fund goes up or down

Goals and rewards        Make more money!

Rewards motivate people to get more rewards.

How do we get people to stay in love with our applications? Make a really good game, but the problem is most games eventually END.

Sustaining passionate users takes more than just making games.

Stephen did a survey; he asked, what are some web apps services you’ve used for more than 3 years? Why, why stick with them? The answers were, Facebook, Skype, Twitter and several others.

When asked why, the answers were “it works” “reliability and ease of use” “very reliable and affordable” “It’s not complicate” “my friends use it” but nowhere did people mention games or fun!

We must PROVIDE a service that is trustworthy AND rewarding.

What do you use for more than 6 months that continues to delight, not just satisfy you, and there were no responses. Apps are not there yet.

Think about the Kano Model: four squares together in a large square, Satisfaction is the vertical axis, with low on the bottom and high satisfaction at the top. Features are on the left-right axis, with not fully implemented on the left, and fully implemented on the far right.  Delight represents the upper left corner of this model. Basic needs are in the lower right.

Most apps have a line going bottom left to top right. They work, but there’s not a lot of delight.

Delighters are those applications (or companies!) that are in the upper left half to middle upper right half of the model. Delighters are not required, but when added brings great value.

The example Stephen provides is he feels United Airlines is in the lower right corner, due to many issues. It’s a basic service with little delight for him. He believes American Airlines is in the middle, it’s ok in his opinion, but doesn’t delight him. They serve his needs but he’s not delighted. Then he mentions Virgin airlines with a smile. They delight him, they are in upper left corner. He went so far as to decide to fly Virgin from DFW to SFO even though he had free mileage points with American Airlines!

Other reasons people Stay in love with applications?

Social Proof is very powerful and important. “My friends use it, so I use it.”

Stories. The stories are another great way to help people stay in love with applications.

Sustaining Passionate Users Summary:

  • Look for the game already in the activity
  • Focus on intrinsic motivations
  • Goals and rewards should be in service to something meaningful

Whew, he talked fast and flew through his slides, but you can find them at his Slideshare page: Sustaining Passionate Users.

2 155

SXSW 2011 – Conserve Code Storyboard Experiences with Customers First


Joseph O’Sullivan and Rachel Evans

Having had a full 7 hours of sleep last night I’m feeling far more refreshed and energized for this morning session. As I entered, I was handed stickie notes and a piece of storyboard paper with 6 cells to draw in.  I’m guessing I should get my pen out of my case to write with, we’ll see.  And a quick personal note – THANK YOU to the Austin Convention Center for having these rooms at normal human operating temperature, not the deep freeze of most normal ballrooms.  Thank you Austin!

SXSW 2011 - Joseph O’Sullivan and Rachel Evans Storboard with Customers FirstAnd we begin. Both our speakers are involved in design at Intuit. Design thinking is all about Deep customer empathy, Go broad to go narrow, rapid experimentation with customers (which is what we’re focusing on for this talk). They use storyboards for the entire context of use, not just the actual design.

Storyboards came from Hollywood, Hells Angels by Howard Hughes was the first film to use them, they used sketches due to the complexity of aerial footage.  Disney took up storyboards shortly after. Today, designers tell the story of the customer via storyboards. Agile design has also lent itself to the use of storyboards. Scott McCloud has also got our attention as someone advocating storyboards.

At Intuit, we use storyboards for:

  • Web application
  • Mobile applications
  • Customer care
  • Human resources
  • Community support

They show a video demonstrating their mobile tax tool, Snap Tax. At the beginning of the project they realized they needed to drive excitement to get doing taxes on the phone. Refunds are fun, so they wanted to show the refund in the first two minutes of using the app to continue using the tool to complete a return. They storyboard the customer well before using the tool, while worried about whether they are getting a refund or not. The tool would provide a few questions, and then the customer would see the refund amount, and be excited.

Pro Tip: It’s easier to get true feedback (i.e. “boy, this stinks” or “this is good”) when designs look really rough, and very crude. More refined drawings seem to cause people to not be so inclined to share their true feelings, perhaps for feelings of not wanting to “hurt” the design team on something that seems rather formalized. In fact, very rough, hand drawn looking designs work best for getting true customer feedback. Also, by not focusing on the actual interface the team can feel more unrestricted in interface development. Finally, rough (fast) drawings mean they only put a few moments into the storyboard so the customer could give the feedback. No large amount of work or time is lost if the concept is deemed not worthy of producing.

So, what did they find by doing these tests? Turned out, the Intuit team’s assumptions were totally wrong. The actual question people had was, can this really do my taxes? Customers were not excited about the idea. Intuit scrapped the idea. Imagine if they had built a working prototype, they would have lost huge amounts of time.

Storyboard benefits:

  • Storyboards are a mirror, you’re telling the customer story, from the customer’s perspective. You are identifying the user’s problem, the context in which the will use the solution and how they will be personally benefited.
  • Pity begets honesty, the cruder and rougher the more HONEST than with finished versions. People reviewing your storyboards are not criticizing your art, you can get to the sense of what they think. This is extremely helpful information early in the process.
  • Narcissus Antidote, people never become attached to a storyboard, so if things don’t work, it’s OK to throw away. But if coding was done, there’s a tendency to want to hold onto it.

Storyboard structure is made of:

Problem, what’s the customer problem?

Solution, what’s my solution

Benefit, does my solution provide a happy outcome that they tell their friends about?

You can’t test the idea in isolation, it needs to be in the context of the customer experience, what’s the customer problem?

With Snap Tax, Accountants get tons of information, faxes, phone, paperwork, email. The solution was to test an online checklist the accountant could send to the client. The benefit would be they would save 2 hours per return.

An important point in storyboards is every cell has intention and a purpose.

Problem: The key thing to understand is what is the problem? What you need to learn is, do you understand the problem, is it an important problem? The example shown has 4 cells to highlight the problem.

Solution: For the solution, what you need to learn is does your solution solve the problem completely? As the customer goes through and comments about each cell, does this solution work for them?

Benefit: As to the benefit, what you can learn is what is good about your idea from the customer perspective, will it delight them? The example shows one cell. Will they use the product and tell others?

Now there’s the part where we use the paper and stickies. We are going to create a storyboard, fun! The storyboard concept is created using:

  • Script
  • Visuals
  • Structure

Storyboard-MaterialsFor our 6 cell storyboard, two stickies are for the problem, next three are for the solution, and last one of the 6 is for the benefit. (Everyone starts by placing a stickie over each cell, the drawing in the cell will come later. So first, be thinking about the problem and write them down on the stickies.



1. First is what’s the project you’re interested in or working on, what’s your goal? Our example project is a guy who wants to crowdsouce a taxi ride home after being out after hours.

2. Next, what are they like? In our example case the customer is 21 – 30 year old, male, urban, smartphone, tech savvy, likes to drink but aware of the social consequence.

3. Next, what’s the problem, from the customer’s point of view. We use… “I’m trying to, but I can’t because…” Careful, don’t insert your solution as the problem!

Next step in the process is to start structuring the script:

4. So what is the solution? What are the important moments that need to occur for the solution to execute? For our example guy, it’s texting a cab, then the driver gets a text while driving home, and finally a confirmation text exchange.

The script storyboard should be 2 problem cells, 3 solutions cells, 1 benefit cell.

5. Finally, the benefit. This is not a benefit of the product or service or features. Instead, this is what would the customer say is great about your solution. DO NOT regurgitate the feature list. In our example, it might be; “I didn’t have to fight for a taxi” or “It confirms my belief that people are good.”

Once this is done, we must evaluate our assumptions by asking some potential or existing customers about our ideas. What do you want to learn about this thing? If you take this to a customer, what do you want to learn? Your goal is to gather as much new feedback as possible. What do you want to learn about each section of your script?

  • Customer problem, do you understand the problem, is it an important problem?
  • Solution, does your solution solve the problem completely?
  • Customer Benefit, what is good about your idea from the customer’s perspective?

After reviewing our assumptions with customers, and making adjustments that might be necessary, we are now (finally) ready for drawing:

Drawing, you don’t have to be an artist to draw a storyboard, it’s actually helpful for the reasons above to NOT have a nice drawing.  To imagine how rough we’re talking, think about your 3rd grade drawing that mom placed on the fridge and you’ve got it.  In the example, phone, head, face, body and UI equals a cell.

For each cell, pull the stickie (2 problems, 3 solutions and 1 benefit) off one at a time and draw in the cell the item written about on the stickie.

Now, our whole audience here in the ballroom is in complete and deep silence as everyone is thinking and drawing.

Finally, they ask each person in the audience to share their storyboard with their neighbor.  And suddenly the silence is broken by the sound of 1,400 people all talking to each other.  Each person in the audience shares their designs with each other.

And with that, our exercise is complete.  We now know how to create a storyboard.  The key learning is the amount of thought that goes into the problem, solution and benefit well before pencil or pen ever start drawing figures on paper!

Joseph and Rachel have done a fabulous job and I personally believe this is one of the best sessions I’ve attended at SXSW 2011.  Congrats to you both!

And now we move into Q&A.  This was a VERY useful session and a great way to learn about, and practice, creating storyboard experiences with customers first, all to conserve code.

0 24

SXSW 2011 and a few early morning thoughts

Morning-SXSW-2011SXSW 2011, Monday, March 14.  It’s early morning, 8am to be precise, and at this time of day SXSW is a different world, stirring in me thoughts of long ago and a comparison with Internet World.  This morning there’s just a few people strolling the empty halls. Perhaps they’re from the UK or Europe (I met a nice couple from Belgium yesterday) in which case this is more or less the afternoon for them. Or maybe there’s those rare somewhat odd “morning” people that can wake up at 5am chipper, with a cheery and annoying “Good morning!” as soon as you even think about looking in their direction (I’m one of those sad group of individuals).

As I stroll through the emptiness, the air of expectation and hidden power of Potential energy that will in a few hours be the reality of SXSW 2011 Monday is all around.  A hidden but real feeling pervades the echoing halls. A memory flashes, I remember the same feeling I had in a long-ago time when the Internet was “new,” 14 years ago, and a massive show called Spring Internet World 1997 was the happening spot in the Los Angeles, California convention center. Strolling through the Internet World exhibition halls, the massive exhibition halls of Internet World, vendors plied their trades, then as they do today.  I was there, drinking in the excitement that was this thing called the Web.  Cool items like “PointCast” and “BackWeb” were the new hot technology everyone talked about.  HotMail was a brand new free email tool, it exchanged “free” for “advertising” what a concept!

Today, the exhibit hall will open (note to SXSW, you’ve lost a major monetization opportunity not having that hall open starting Friday, I was really looking forward to wandering the halls for several days) and all the usual suspects will be there, but suspects of 14 years later.  New technologies we’ve never heard of, vendors pitching the next big thing, the promise of cool and popular things to come.   What will be the next big thing?  I guarantee it’s someone in that exhibit hall, in a small booth tucked away from the main alleyway, hastily printed business cards and tattered displays hide the new hidden, soon to be amazing opportunity.

SXSW-2011-afternoonSlowly, the masses start to arrive, in a few hours this place will be jostling with the excitement of over 100,000 people, all focused on interactive ways and means.  We are all different, yet we all share the common thread of energy, excitement and expectation for what comes next.  It’s what drew us to the Spring Internet World 1997 14 years ago, and it’s what draws us to SXSW 2011 now.  It will draw us 14 years in the future.  I hope to be there, early in the morning, awaiting it, I hope you’ll be there too.


0 96

SXSW 2011 – Congratulations! Your Brand is About to Become Obsolete

Andrea Ring, Managing Director of Planning, R/GA

William Charnock, Chief Strategy Officer, R/GA

It’s Sunday morning, and I think I’m hitting the wall big-time for this conference, feeling pretty worn down. Damn that Lisa Barone, she makes this look waaay too easy. Two cups of Starbucks didn’t even make a dent.  Now I’m in the packed conference room and they are testing the mics. BUZZZZ RIIINNGGGG!! Wow, massive sound feedback problems assaulting our ears! We’re awake, we’re awake! Please make it stop! We’ll buy whatever you’re selling!

SXSW 2011 - Your BrandAndrea Ring William CharnockWilliam starts: Our agency has a digital bias. Our relationship with our clients has changed over the past two years. Rather than being project focused, clients are looking for help in terms of the future of brand from the digital perspective.

Who is their brand, in the digital perspective. But people are stuck thinking about Brands using the old model from 30 or so years ago. A brand is the money you get that people pay that’s more than just a commodity.

ANDREA: Interbrands notes in a chart that strong brands beat the S&P 500. However, Ford is reducing in value year over year. Kodak is the same issue, falling off the face of the earth, Borders was another. So why is this happening?

A common perception is obsolesce is just what happens and there’s nothing you can do about it. Cunard cruises disappeared after airplanes came out. Fur stores used to be everywhere, but that almost disappeared due to PETA and other social influences. Kodak had trouble shifting out of the film businesses, when everything went digital. Ford said they were blindsided by competitors.

But who survived? There are some brands in the Interbrand chart that survived. We think they were so big they were able to block and own channels. In the 60s in broadcast, brands that owned the airwaves ruled, relying on their scale to dominate the channel.  Scale used to be the advantage, but.

WILLIAM: We now have universal distribution of communication, everyone is now a broadcaster. There’s no longer a competitive advantage in this space. Also, everyone now has ubiquitous information, so people have the tools they need to get information from experts, get data, and make up their own minds about Brands, not just what Brands were saying.

Reality shift, we’ve moved from perception to reality, we have emotional relationships with Brands, these need to be what’s going on in the realworld. I believe marketers are spending too much time outside the real world. Consider, gathering data takes 6 months for qualitative and quantitative data, but that makes it out of date. Focus groups are not real data, there are customers you can gather data from now in realtime. This era is not about perfection and Broadcast quality standards, instead create something and let it iterate over time.

So, the closest point to obsolescence is when the Brand is at the highest point of success (middle of a bell-curve chart). We’ve been trained to maintain consistency when you’re at the top, but this is the point where Brands need to change most. When you are most successful, that’s when you need to drive change.

ANDREA: The biggest point of obsolescence is when Brands can’t recognize the need for change, that’s a sign. Ideas like bank aggregation, which became Application store became iTunes.

WILLIAM: The root cause of obsolescence is Brand itself, the structure of it and the obsession of being rear-view focused. So ask a Brand:

1.       What are you doing that’s new and interesting? Those brands that were thinking in the future were able to answer that, backward brands couldn’t.

2.       Why are you in business? You’re not in the business of making money, What’s your purpose beyond selling a product or service?

These two questions are more helpful for defining a brand.

In traditional marketing, there was a lot of talk about Brand as a story, it was metaphor based, overly so.

In digital marketing, there is a lot of talk about brand systems, utility, information.

Where these two worlds come together, between system thinking and story thinking, Brands can have success. Advertising is more about showing people what can be possible, and what’s available somewhere else. Sites are moving away from intensive and heavy to lighter and easy consumption of information.

We think where both worlds come together is “play.” Storytelling doesn’t go into this, but this is where we are going. Play is the glue that sticks us together as a community. A playground is about people, purpose and objects. This idea of play is not fixed, but something that can build and grow over time. What’s the territory of the playground for the Brand?  The purpose helps us define that. What objects and tools and people we invite into the playground is the digital platform.

WHO: Audience Participant User
WHAT: Intention Action Information
WHY: Empathy Solidarity Utility

The center of the spectrum above, Participant, Action, Solidarity is where to be.

If we can get people playing together we can, especially with participant, action, and solidarity (think of Call of Duty) that can become a very powerful Brand platform.

Brand Purpose = Why you are in business, helps identify the direction of innovation. This pushes people into understanding the business they are in, beyond products and services.

ANDREA: Products become obsolete, don’t focus on products, that takes you down the backside of the obsolescence curve. By focusing on purpose you can move above and beyond the obsolescence curve. Example of Ford, what if they are in the mobility business, not just car business. What about transport like trains, public transport, other mobility playgrounds. Barnes and Noble, don’t focus on books, see yourself as being in the reading industry. That’ how Nook was born, unlike Borders. Google is doing it right. They are not about search engines, they are about organizing the worlds information, about useful information. Nike is doing this very well, they are not about sneakers, they are about performance.

What is it about Brands that people will pay for? It’s about purposeful play, everyone in the brand needs to get behind the idea of what the business is about, and create something customers can do with the brand to play and build together.

WILLIAM: The brands that are doing well in Interbrands study are Google, Netflix, Amazon. These companies change in realtime, they use data more than any other brands. You see it in their everyday decisions, they are not doing 6 or 12 month iterations. You can invite more people to participate and play, and that grows the brand.

5 rules of play, seeing your brand as a system:

1. Be interesting: Example, Chapstick, created a facebook fan page. We wanted to figure out what interests people about the Brand, what’s interesting to them? We invited people to a contest of Sing Your Love, we got hundreds of amazing videos on Youtube from people (shows videos that have very high production value from regular people).

We wanted to know who the fans were and what they thought about the brand.

2. Be Surprising: Play actually gets better when unusual things happen. That’s why play is a learning experience. Surprising is better than being expected. Try new and unexpected things. The digital space allows small, easy to use things to try. I’d like to see Brands doing things that are a little unexpected.

3. Get real, in real time: Don’t’ wait for labs, get out in the real world with realtime data and real people.

4. Open your experiments: Netflix gave out a $1M prize for a better recommendation. Not only were they experimenting, they were reaching out to their greatest users. Look at your power users for any Brand, they are probably already out there experimenting already, get involved with them.

5. Shift before shift happens: Just when you’re feeling comfortable, it’s already too late. Example being Nike +, it’s been around since 2006 but now it’s so different than what it was. There were shifts that happened back then that would have rendered them obsolete, but they changed. (shows video from Nike+ about the new Nike with GPS to track progress. Accelerometer to track stationery running, crowd yell when FB likes your comment, other cool engagement things).

And with that, this session is over and we’re heading into questions.  And I’m looking for an energy boost.

SXSW 2011 – Stop Listening to Your Customers

Nate Bolt and Mark Trammell

Stop listening to your customers? Why, that’s CRAZY talk to a guy like me with a marketing and usability background!  But if Nate says there’s something to it, then so be it (he’s been known to be right sometimes, just kidding Nate, you rock).  I know Nate rocks because I read his book, Remote Research Testing and wrote a brief book review about it. Let’s find out why it doesn’t make sense to listen to our customers, shall we?

Nate starts, how do people get information from their customers he asks the audience: Various answers including focus groups, research labs, and surveys, all the usual suspects.

Nate says, in 1995, there was the Decision Theory & Adaptive Systems Group in Microsoft. Folks were playing around with such interesting concepts as Bayesian methods and Animated pedagogical theory – the idea was to infer when users need help, and offer it (seems easy to me, what could possibly go wrong?).

So, they conducted lots of research by asking people, “Hey, pretend you were in trouble. Would you like an animated helper to pop-up and give you some assistance?” And guess what, people nodded their heads and said, “yeeesss, of COURSE I would like that!”

Clippy, a user experience failure caused by bad research.All that work lead to one of potentially the most hated computer features on earth, “Clippy” the animated Microsoft paperclip helper. What went wrong? They asked people, would a friendly character help you? People said yes, why didn’t that work?

Nate says it’s because it turns out asking customers what they think is a bad way to listen to customers. In addition, not having an actual tool or working prototype to test was the second “aha” key learning.

Nate says, here are two bad ways to listen to customers:

1) Ask hypothetical questions

2) Provide a false premise

Mark says, at Twitter, for research we observe users with the goal of ultimately trying to remove friction. Everything at Twitter is about watching how people use it, then changing the interface to improve it.  Mark provides an example of trying to figure out a way to better handle retweets.  They observed the users having difficulty with the concept, and tested various concepts to see if they could reduce that friction point.

Same issue for making groups – people needed different identities for different needs like sports, work, etc. They observed this, and created classification of user accounts to help users classify by groups.

Another interesting function that came from observing users was when Atlanta had gasoline shortages.  They observed users Tweeting about locations of gas using hastags.  An example being #atlgas.  The researchers watched users solving categorizing and bookmarking tweets using the hashtag. They didn’t invent that, the users did.

Mark continues, coming up with a handle on twitter was caused by people doing the @ sign in front of their handle.  Again, Twitter didn’t create that, the users invented it.

So, Mark summarizes, these are examples of good research, look at CURRENT behavior and observing users interacting with systems.

Nate: But this is not monitoring metrics, this is watching behavior. There is no easy way to monitor system behavior. He refers to Malcolm Gladwell and the 80s research. Nate mentions Howard Moskowitz, who had a client, Ragu. Ragu was losing sales of their spaghetti sauce and wanted to know what to do about it. Which sauce among 40+ potential sauces do people like? They had used focus groups, but Moskowitz said we need to not ask in focus groups, but actually rent a hall, get hungry people and feed them and watch them eat it. Each person got 10 bowls with some of the 40+ variations of Ragu. The findings? Moskowitz said there IS no one favorite. Turns out people mostly like one of three main types: Plain, Spicy, Extra Chunky (and extra chunky was something Ragu had never heard of). So Ragu went off and made extra chunky and it took off, and concentrated on the other big three flavors to great success.

The big idea is: it used to be that we ask people what they want, now we watch them.

Three things to learn from this:

So user research needs to be…

1.       Contextual

2.       In the participant’s Timeline

3.       Behavioral in focus

Nate then cracks the audience up by showing a video cartoon making fun of people doing intercepts in malls.  Yeah, heh, mall intercepts are so 1980s.

Mark: If you are creating a survey, don’t bore people with the survey – nobody cares about demographic information, so don’t get it, it doesn’t matter about age, race or gender. Ask broad open ended questions, then get those answers to use for a quick multiple choice survey. Assumptions in your survey may be false, so based them on broad open ended questions.

Nate talks about; The Truth about Download Time.  A study was completed on Amazon to see whether users felt it was or wasn’t fast. Even though the site wasn’t the fastest site, participants said it was. They confused the experience of shopping on the site with speed, that’s a key finding as to why surveys are unreliable.

Mark: What Twitter does is we do sprints to a functional prototype. We identify the quickest way to build something, then start testing using that. Could be a static site, wireframes, paper prototype, index cards, card sorting – if those work to answer the questions. But use real data. Twitter builds functional prototypes that work with actual user behavioral information. Then we iterate. Test over and over with quick rounds of testing.

Nate: How many people have heard of screenblocks? He shows Sifteo – interactive blocks that were the discussion of a recent Ted talk. They are blocks that sense other blocks around them. They are aware of each other. This started as research at MIT, they knew they couldn’t have wood blocks when conducting research, they need real data, not false data. They used blocks with wires hanging off it, so they got away from “What about if we did this” questions to users, to which people might say sure, we would use it, but without actually using it.

Mark: The new version of Twitter started as a prototype, we observed and iterated as we watched users interacting with it. We emphasize and focus on the moment. We try to provide our users with a simply as possible user experience.

Mark recommends testing using four simple steps, much of which came from the book he HIGHLY recommends: “Observing the User Experience

1.       Define the audience and their goals. Who’s using it and what’s their motivation?

2.       Crate tasks that address those goals.

3.       Get the right people. This is critical. And don’t give them false motivation.

4.       Watch them try to perform the tasks.

Simple, but highly effective!

Mark shows a pic of their usability testing set-up. It’s a small table with three computers; a screen for a user, a screen for him (as moderator), and a screen for an observer (by the way, he uses Silverback). We use a little webcam to stream video and audio back to the developers in a big room of developers and designers, even projected on a giant screen so everyone can see the users going through the usability test.

It’s a very fast way to test and iterate. Everyone can see what’s broken, instead of waiting for a report a few days later, and people can start brain storming immediately with ways to fix.

He adds, we did our testing with four people – we knew the problems were so large that we could find them with only four people. We tested in one day, then did building the next day. We did this for 60 hours.  We did this with friends and family. We were moving really quickly. We could try really crazy stuff and not worry about it.

Was he worried about testing being biased because of using friends and family? No, he says, I only care if they have biases that I don’t know about. With friends and family at least I know their biases.

Nate: People lie less at home. Some researchers feel labs are better, but in the past few years there’s been a bazillion new products coming up. Can these tools make us better researchers?  Yes, but you have to pick and choose, never use just one tool by itself. We set up with a participant elsewhere, we use GoToMeeting, get everyone of the observers in the room in person  – designers and developers. It’s way more valuable for the testing participant to be home, but everyone else in the room. Designers and developers should be beating down the door to attend these sessions.

For, we talked to five people by intercepting them live on the rdio homepage using ethnio. We did the same testing with Usabilia, and used clicktracking to map clicks plus had users enter why they clicked there. We then did 10 users from Cost was minimal.

  • GoToMeeting: 5 users
  • UserTesting: 10 users
  • Usabilla: 50 users
  • Ethnio: 468 recruits from home page

We did all this in one day, using a Guerrilla method.

For Mobile research, Nate says, we just followed people in homes and stores using an interface with a webcam and stream live. We take two webcams, one for streaming and one for recording. Design folks could chat with us live, so we could follow up with questions. We did a test of QR in stores and learned that the plastic that wraps QR codes was stopping people getting the QR codes.

The big myth is: geniuses have genius ideas that turn into genius products. That’s not true, anyone can create genius products if the watch their users and iterate often.

The Truth:

  • Great ideas come from other great ideas.
  • You need to get to the motivation to understand the why of behavior.
  • Imaginative research facilitates imaginative products.

Nate saysExpand the notion of what research is:

  • Build
  • Behavior
  • Time-aware

Nate finishes by saying: Time aware is the overlooked part of research, be on the participants timeline, do not force them to be in your timeline.

And they head into questions, and I introduce myself to Nate in person and then run off to my next session!

0 11

Saturday March 12 – Craig’s SXSW Schedule:

Welcome back to SXSW 2011 all you SXSW fans!

Today is Saturday, March 12 and here’s my schedule.  If you are here at SXSW today, and happen to be in one of the seminars I’m attending then let’s meet up!  And if not, well this virtual meeting is better than nothing, right?

9:30 AM

Stop Listening to Your Customers
Salon C

11:00 AM

Believe Me or Your Own Eyes: Eye-Tracking Entertainment
Design / Development
Ballroom C

12:30 PM

Chicken or the Egg? What Search Activity Conveys
Salon H

2:00 PM

Keynote: Seth Priebatsch
Featured Speaker
Ballroom D

3:30 PM

User Experience Meet Up
Meet Up
Room 616AB

5:00 PM

Metrics-Driven Design
Design / Development
Ballroom A