“Just Give Us the Takeaways”
9 people from the product team had gathered around the conference table: a mix of designers, developers, business analysts and management.
After some technical challenges with the big-screen TV, we were ready to play the first video recording of someone using a new version of the company’s mobile app.
Then it happened.
The product manager, who’d arrived a few minutes late, looked at me and said “Can’t we just skip the videos and you tell us what your biggest takeaways were?”
This was 6 years ago. It was the first time I heard this question, but it wouldn’t be the last. Since then, it’s come up at some point from nearly every UX client. In fact it happened yesterday, again. Different company. Same setting. Same question, this time from an executive: “Can’t you just share the insights?”
Expert Research vs. Team Research
When we started doing usability testing and other UX research about 10 years ago, we used the Expert Research approach. This is what the product manager above was hoping for.
With Expert Research we go off and do everything ourselves — planning, recruiting, facilitating, observing, analyzing, brainstorming — and come back to the client with a report of our findings and recommendations. Often we accompany this with a presentation that includes a few video highlights of the top findings. Yes, there’s some discussion, and the team plays a role after the report handoff, but it’s largely a top-down, expert-driven process.
This is still how most companies do user research. And it’s easy to understand why. While the research process often faces challenges, for those with experience it is straightforward and predictable. The researcher has a lot of control over the deliverables. For everyone else, it’s easy: show up to the presentation, or read the report.
These days, we still do Expert Research from time to time, but we actively try to avoid it. That’s because, around 6 years ago, we discovered a better approach: Team Research.
We first discovered it in Steve Krug’s book, Rocket Surgery Made Easy — which has a chapter called “Make it a spectator sport.” I’m still a fan of his simple approach and believe it’s the right place to start.
After trying his approach multiple times and seeing some limitations, we began experimenting with Jared Spool’s affinity mapping approach (aka KJ-Technique). That’s mostly what we use now, with some adaptations based on what we’ve learned, in particular for a remote setting.
Team Research is harder to pull off than Expert Research. The process and results are messier. And stakeholder buy-in is challenging. But it’s worth it. And the more we use it, the more we love it, and the harder we push back on Expert Research. Here are 5 reasons why.
Why Team Research Beats Expert Research
1. Believability
Telling a designer that people can’t figure out his wizard feature doesn’t compare to having him watch 4 of 5 users struggle to use it. As Krug says, when it comes to user research, “seeing is believing.”
Steve Blank has found the same thing when it comes to the Lean Startup’s version of user research: customer development. A researcher telling a startup founder that people don’t have the problem she’s trying to solve doesn’t compare to her hearing this directly from customers.
The founder and the designer have a hard time thinking their ideas could be wrong. And they have a hard time appreciating how different they are from their target customer. Only through in-depth exposure can they come to accept that some of their assumptions about user’s needs and behaviors are wrong.
Quick highlight reels are better than nothing, but they pale in comparison to extended observation, which allows stakeholders to truly understand customers and their broader pain points and goals. Among other things, when you hear enough up front from a user to say “This is the exact type of customer we want …”, it’s harder to shrug it off later when you see them struggling on the site. And that brings us to …
2. Stickiness
For teams that truly embrace Team Research, we often hear members of the team talking about specific users and their experiences long after the review session.
Months later, we could be talking about a completely new project or feature, and someone will say “Remember that lady who was trying to plan a trip to New Zealand? And she couldn’t figure out where to look for hotels? Would this change solve her problem?” No posters necessary. Just the experience and memory of sitting in a room learning about a user in depth and watching them use the site. And when you have the full team involved, this leads to …
3. Shared Understanding
User Interface Engineering has done extensive research on what makes some design teams great. The “closest thing [they’ve] found to a silver bullet” is user exposure hours. Specifically, the top design organizations spend a minimum of 2 hours every 6 weeks watching users. Not just researchers, but the everyone who influences the product. For instance, at GOV.UK even the top executive participates.
Why is this so important to producing high-performing products?
Having the whole team watch users gets everyone on the same page about what the biggest problems and opportunities are. And doing it regularly ensures that the team stays focused on fixing the users’ frustrations, rather than veering off in many different directions based on opinions.
4. Better Findings & Solutions
Even if they buy into the benefits above and embrace having the team observe research sessions in depth, many executives remain skeptical of the affinity mapping process as a replacement for expert analysis.
“The group approach is fine. But we really want to know what you thought were the biggest problems, and what your recommendations are,” they’ll say.
While we do share our thoughts, we stress that in our experience, the group usually does a better job of catching and prioritizing problems — and generating creative solutions — than 1 or 2 experts would. That’s because the group has a variety of perspectives and areas of expertise. As a team, they have a better handle on the business’s needs, the customer’s needs, what’s been tried in the past, and what’s easy or hard to implement. The trick is channeling the group’s expertise effectively, and so that’s where we focus our expertise — through affinity mapping and other collaborative exercises.
5. Humility
Finally, there’s a more subtle, but more powerful reason to embrace the Team approach, and it has to do with humility and risk mitigation. On normal days in most organizations, people spend countless hours designing, writing, coding, and debating things that are based on assumptions about users’ needs and behaviors.
The Team Research approach does ask more of these team members in terms of the time they spend observing users. And since they’re already busy, this can seem like a lot to ask.
But given how much time they spend designing and building for users, and how costly wrong assumptions are, is 2 hours every 6 weeks really a lot to ask? Or even 20 hours every 6 weeks?
Taking time to watch and understand users really just acknowledges that we don’t have all the answers. Or as Steve Blank says, “there are no answers inside the building.”