UX Research Case Study:
Prototype Usability Testing of Billing & Leak Scenarios for American Water
The largest publicly traded U.S. water utility was trying to improve the experience of customers investigating a high bill, conducting a leak analysis, or requesting an adjustment. We conducted prototype UX testing and led a 1-day workshop that “crunched all the customer feedback in record time.”
Goals
Ever get an unusually high utility bill? If you logged into your online account, what might you try to do? What information and tools would be most useful?
A team at American Water was exploring these types of questions. Their goal: improve the experience of customers investigating a high bill, conducting a leak analysis, or requesting an adjustment.
American Water (NYSE: AWK) is the largest publicly traded water utility company in the U.S. Started in 1886, the company has over 6,400 employees and provides services to over 14 million people in 24 states.
In two previous projects for American Water, Marketade led early-stage usability testing of a redesigned customer portal. Those studies covered a wide range of critical tasks, from paying a bill to looking at usage projections.
A year later, having made significant progress on the new portal rollout, the team was focused on a narrower set of customer pain points related to billing analysis and leak adjustments. Their UX designers had prototyped improvement ideas. Before implementing and launching the new designs within the portal, the team wanted to catch and fix UX problems within the prototype.
Approach
Marketade conducted moderated usability testing with a small sample of American Water customers. We interviewed users and observed them as they attempted to complete tasks using an InVision prototype.
We then facilitated a collaborative stakeholder workshop outside Philadelphia to identify and prioritize findings and brainstorm solutions. The team identified 5 high-priority problems and generated solution directions for each problem.
Two of our researchers collaborated on this 5-week project. We spent 1 week on each of these 5 steps: designing the research, recruiting participants, conducting the sessions, designing and leading the workshop, and writing the final report.
Here are additional details from the project.
Qualitative Research
- We designed a study around 6 realistic scenarios related to high bills and leak adjustments.
- We wrote a detailed interview script with background questions, usability tasks, and post-test questions.
- We launched a web-based recruitment screener survey; American Water sent an email campaign to a subset of its customer base, linking to the survey.
- We phone-screened the most qualified candidates and scheduled 1:1 interviews.
- We conducted and recorded moderated research sessions via Zoom.
- We selected the most useful research sessions and lightly edited them.
During a research session, an American Water customer (bottom) thinks aloud while completing a leak adjustment task using a prototype. Our moderator (top) asks probing questions as needed.
Analysis Workshop
Once the research was complete, we designed a 1-day analysis and ideation workshop. We helped American Water pick the right team to participate — a mix of staff from customer experience, design, and customer service.
Two Marketade team members then traveled to American Water’s headquarters in Camden, NJ to facilitate the workshop. Here were the key steps of our workshop:
- We instructed the team on how to observe research sessions and how to take notes.
- We spent the morning watching the sessions.
- After lunch, we facilitated an affinity mapping exercise.
- After a series of structured, collaborative steps — posting, grouping, naming, and voting — the team arrived at a consensus on the biggest UX barriers with the new designs.
- We helped the team divide the problems into 3 broad buckets: “No Brainers”, “Larger Problems”, and “Parking Lot Items”.
- For the 4 No Brainers, we led a “lightning round” exercise in which the group identified a solution and next step for each item within 2 minutes.
- For the 5 Larger Problems, we divided into 5 small groups and assigned each group one of the problems. Through a series of steps, each group created a poster with a detailed problem definition, research references, and solution recommendations.
- Each group presented its solution poster and fielded questions from the larger team.
- We then led a Who/What/When exercise to collaboratively brainstorm the next steps that were needed in order to implement each viable solution idea.
American Water team members take structured notes while observing a moderated usability testing session.
Problem prioritization: workshop team members place votes on the most important UX barriers they observed in the research.
Outcome
By the end of the workshop, the American Water team had agreed on 15 action items and assigned owners and deadlines for each.
A week after the workshop, we delivered an 11-page report that included:
- Executive summary
- Research process
- Research participant information
- Workshop steps
- Key findings
- Solution ideas
- Next steps
After the project, American Water stakeholders shared the following feedback:
“Love the approach and the guided process”
— Yimi Kierman | Senior Director, Customer Experience
“Crunched all the ideas and customer feedback in record time. Now we can build a product tailored to customer needs and expectations.”
— Penny Farmer | Project Manager
“We were able to find key problems fast. [The impact will be] a more streamlined approach for customers.”
— Heather Holting | Billing/Collection Manager
Excerpt of a Who/What/When next steps table created at the end of the workshop and digitized in the project report. The table included 15 next steps across the 5 top problems identified in the workshop.
More Case Studies
A Year in the Life of an Enterprise User Research Program: GEICO
Learn how we worked with a long-term retainer client to turn their research projects and programs into valuable tools that help meet their goals.
Product/Market Fit Research for Verizon’s Innovation Group
A Verizon team needed to test key assumptions to move closer to product/market fit for a new enterprise software concept. Our rapid research cycle delivered new user insights that helped answer their questions and validate/invalidate their hypotheses.