An Event Apart: Just Enough Research

by Luke Wroblewski April 2, 2013

In her presentation at An Event Apart in Seattle, WA 2013 Erika Hall walked through the importance of research and how to apply it to making product decisions. Here's my notes from her talk on Just Enough Research:

  • In 2001 the Segway promised to revolutionize transportation. Despite all the hype, it fell flat when it was released. The Segway completely ignored existing transportation infrastructure and didn’t fit into the existing conventions of how people moved through cities. The Segway ignored people with children, or any passengers, anyone without 5,000-6,000 dollars to spend, anyone who lived in a harsh (even just rainy) climate, anyone who needed to go more than 5-6 miles to work, and more.
  • The team behind the Segway failed to do some basic research about the world.
  • There’s usually a gap between your assumptions of how the World works and how the worlds actually works. This gap is filled with hype, wishful thinking, and risk.
  • To bridge the gap, you need research that gives you insight into how the world really works.
  • Innovation can improve the world but you need to take reality into account when you plan and design. When blue sky thinking meets reality -reality wins.

Why Research?

  • Objections to research: we don’t have money, we don’t have expertise, we already do A/B testing. These don’t hold up.
  • Why do research if you can learn through failure? Learning through failure leaves needs unmet on the table. Unmet needs are what you should be focusing on.
  • Don’t waste people’s time and effort on untested assumption if you don’t have to. Failure is an expensive way to learn. There are faster, cheaper ways of learning.
  • What is Research: simply asking questions and doing a good job of interpreting answers.
  • Pure research: carried out to create new human knowledge. Answer an unknown question through observation and rigorous. This is science.
  • Applied research: serves a specific goal but uses a lot of elements of pure research. If your research helps you meet your stated goal, then your research was successful.
  • Design research: focused on understanding the people for whom we are designing. It can become too focused on the process itself: exercises and deliverables.
  • Design research is not enough. You can understand people’s needs and have a product that meets those needs well. But real world context can still cause you to fail.
  • The real world context in which you are authoring your product is essential to understand. Things that make perfect sense in the abstract might not make sense in reality.
  • Research is not asking people what they like. Like is not part of a critical thinker’s vocabulary. Liking something is subjective and empty. Research is not asking what people hate. Many people engage in activities they claim to hate.
  • Research is not a political tool. Don’t use it to push your agenda. Applied research is not science. Avoid arguments on statistical significance. Focus on learning useful facts.

Research Process

  • The research process is simple: form questions, consider them, observer things, interview people, and analyze the data.
  • The research process generates insights that influence your ongoing work, which helps you be more successful. Insights are just one input into what you are doing.
  • The most important part of research is forming the right questions. Good questions are: simple, specific, answerable, and practical.
  • Once you articulate questions, think critically about them. This process challenges assumptions and is: disciplined, self-correcting, clear, and logical.
  • Observation is getting good at noticing things and telling stories about what you saw. To make observation effective, you need to de-personalize from observations. Instead of what it means for me, ask what it means for the person you are observing.
  • Interviews: you have to get out and talk to people, especially those that are different from you. Know your question (what do you want to find out), Warm people up (get them talking), Shut up (and let the person you are interviewing talk) and listen.
  • Interviewing is not talking, it is listening. Get the interviewee into their comfort zone. Get the interviewer out of their comfort zone, so they can learn.
  • Analysis: make sense of and collect data. Get all your findings and your team together and start to organize your observations on a white board.
  • An observation can be a reported behavior, a direct quote from a participant, or an observed behavior. Then group and label these observations.
  • Turn your insights into actionable inputs. Draw conclusions and work on them.
  • Recruiting: use a tool like Ethnio to cast a wide net and get people on the phone. Its not hard to find people to talk to.
  • Incentives can help reduce biases in your study. You can get a lot of people to spend time with you for $50. If they are there because they like you, they’ll have an inherent bias for talking to you.
  • Personas: start with one person. Explain to your target person why they should care about the feature or product you are making. Imagine them asking you “so what?”. Have an answer for this question.

Research Types

  • Organizational research: important to uncover requirements (what will this do for us), politics (the more you know the more you can work with it), workflow (determine what everyone can do), capabilities (what can you actually do), goodwill (talk & listen to people about what matters to them).
  • The organization on paper is very different than how people really work together. You need to change how you work to support different ideas.
  • Talk to people one-on-one. People will tell you things in person that they won’t say in a group.
  • User research: what do people do and why do they do it?
  • Fight the urge to stay in your tidy office and get out into the real world where things are messy. Take a deep dive into people’s daily lives. Get into the mundane lives of people and pull out the stories.
  • Focus groups are a waste of time. They do not provide insight into real people in the real world. They are research theater. At best, they can be the source of ideas to follow-up on with real research.
  • Competitive research: who are your competitors from the perspective of your users. How else might your target customers solve the same problem? How is your solution better than the alternatives? How is what you are offering better than nothing?
  • Anything people could be doing instead of using your service is your competition. People are naturally lazy and hate change. They have to love what you are doing more than they hate change.
  • Quantitative research: it can tell you which option performs better but it can never answer the question “why”. This gets you to a local maximum. It optimized what your current system can do for you.
  • Don’t just ask how much but why?

Who should do research?

  • Who should do research? Everyone. Even if they can be involved just a little bit. Everyone likes to be part of solutions and feel smart because they had insights.
  • Use the tools and methods of applied research to get answers. We benefit when more people ask more questions, and think critically about the answers.
  • What is Just Enough? What problem do we want to solve for our target customers? That’s usually the key question to answer.
  • The complete answer, however, depends on you organization and what it takes to to help you understand your users and
  • Cultivate a culture to be proven wrong as quickly as possible and at the lowest cost.