Discovery Research Case Study:
Enterprise Experimentation Intelligence Research: Finding Best Practices Through Expert Interviews
“Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day.”
— Jeff Bezos
Challenge: How to Find Best Practices in a Niche Field
An enterprise company was eager to advance its digital experimentation practice. Their experimentation team was conducting more and more A/B tests each month, but senior management wondered what might help them be more like Amazon, Netflix, and other companies with a true culture of experimentation. Their questions included:
- Among the top experimentation companies, what are the shared best practices?
- How do they structure their experimentation teams to support an enterprise?
- How do they scale their experimentation practice?
- Do they use a centralized, decentralized, or hybrid model?
- How do they prioritize experiment ideas?
- What types of experimentation platforms and tools do they use?
Some members of this company’s experimentation team had found some promising articles in sources like Harvard Business Review, but the articles didn’t answer their questions or provide the depth of insights they were seeking.
A meeting with senior management was around the corner; the team needed answers and insights quickly. The company reached out to Marketade to see if we could help.
Approach: In-Depth Interviews with Expert Practitioners
2 of our researchers recruited and interviewed 6 professionals with deep, senior-level experience in digital experimentation and optimization at the enterprise level. From these interviews, we saw clear patterns and opportunities that we delivered in a short report. From start to finish, the project took exactly 2 weeks. Here were the key stages of our process.
Discovery
We kicked off the project by leading a short remote meeting with members of the experimentation team. We used the call to reach alignment on the project goals and research questions.
Based on what we learned in the kickoff meeting, we wrote an informal research plan that included a series of possible questions to ask during the interviews. It also identified the types of people we wanted to interview.
Recruitment
We decided to primarily go after expert practitioners for the interviews. We wanted people who had done experimentation and had done it inside enterprise environments. Consultants or analysts might have useful insights on some of our questions, but they wouldn’t have the same in-depth expertise as in-house practitioners, especially on the organizational questions.
The ideal candidate looked something like this:
- Has primarily worked in-house at top experimentation companies in their career
- Has worked at multiple enterprise companies in different industries
- Has worked in at least one leadership role in an experimentation group
While practitioners were our primary target, we also wanted 1 or 2 experts who had worked for one of the top experimentation products that are used by enterprise companies. Specifically, we wanted a consultant and/or a product owner — someone with a strong understanding of the needs and habits of their customers.
With both groups, we had a target list of companies. The catch was that we wanted people who had worked there recently but weren’t currently working there. Why? Most professionals won’t share much, if anything, about the companies where they currently work; they’re much more comfortable talking about companies where they worked in the past.
Even with past companies, many professionals — especially those in leadership positions — are hesitant to participate in interviews in which they will be asked about what went on inside those organizations.
All of this made recruitment for this project very tricky. We were targeting a small universe of candidates (maybe 100) and we needed to make them comfortable about talking with us.
While we tried a few approaches, we quickly found a formula that worked: use LinkedIn search tools to hand-pick qualified candidates; narrow the pool down to the most promising; use LinkedIn InMail to contact them and invite them to participate; contact batches of 3 at a time to figure out our success rate; expand as needed.
We customized part of the InMail messaging to make it clear to invitees that we’d hand-picked them based on their unique background; they weren’t part of a big email blast. We offered a good incentive. And we provided a link to our interview calendar so interested people could schedule their interview right away.
In the end, we had a 50% success rate; of 12 people that we contacted, 6 scheduled interviews. 4 of those were from the primary audience (in-house practitioners); 2 were from the secondary audience.
Research & Analysis
We conducted the interviews over the course of a few days. Then our researchers collaborated to analyze the sessions and identify patterns.
Despite the small sample size, clear patterns and best practices quickly emerged. We wrote these up in a 6-page report and delivered it to the client. Finally, we jumped on a 30-minute call to discuss the report and answer questions.
Our report provided 7 research-based recommendations for our client. Within each recommendation, we shared expert insights and quotes, in addition to answering the research questions identified at the start of the project.
More Case Studies
How Tonal Turned Customer Journey Mapping into a Team Sport
A fitness tech company visits its customers’ homes to gain a deep understanding of their needs — paving the way for actionable personas and journey maps.
Creating Research-Based Personas for Herman Miller
We conducted 10 contextual interviews with customers of Herman Miller’s Design Within Reach furniture brand. Later we facilitated a 1-day workshop to generate personas and innovation opportunities.