Our clients are really excited about the potential of AI to transform the customer experience and create tons of new business value - and so are we!
But, in the rush to roll out AI, conversations may focus on how AI features can benefit the business rather than customers, and assume faster and newer capabilities will “improve the customer experience” without defining how - or whether the results are meaningful for either customers or the business.
Similarly, the business value of AI products - like staff efficiency gains, higher conversion, or better forecasting - can seem obvious, but the right KPIs are often not never defined or tracked, and no one can definitively say how the model is performing and whether the business should continue to invest in and scale it.
We’d like to push product, design, and tech leaders to be explicit about where the value comes from in these features and how it will be measured, and use those insights to make decisions about the rollout of AI experiences.
We’re sharing our best practices here to help others avoid the worst case scenario - where a business over-invests in a powerful data product that doesn’t solve a user need or have a real business impact - and worse, may actually hurt the experience.
But first...what's the difference between customer and business value?
By customer value we mean the experience helps the customer achieve their goal (or jobs to be done) - like finding what they need, finishing a task, or having fun. We can also create customer value when we make the same experience easier or more enjoyable.
Adding customer value means the customer appreciates the experience of interacting with your product and brand, and may choose it over others. Customer value improvements may lead to measurable business value, like higher conversion (when barriers are removed) or repeat visits (when brand loyalty is developed), but it's not the same thing.
By business value, we mean growth metrics that are tied to customer or staff behavior. For example, customer acquisition, conversion or task completion rates, or the productivity of staff workflows.
Important note! We believe it’s valid for the goal to be to affect either business or customer value, but not both… as long as it's measurable and everyone is on board!
Real talk: sometimes stakeholders have already decided to build an AI-driven experience without defining or measuring the customer or business value it will create. In that case, researchers still need to use this framework to measure, tease apart, and tell a compelling story about the two.
Let’s take a closer look how you could assess the business and customer value at each stage of product development:
Discovery & Concepting
This stage is all about proving there is real, measurable value for end users before building anything. Unfortunately we see all too often that this stage is skipped.
If you hear someone saying that an AI feature will “improve the customer experience”, we challenge you to ask them - "and how would you measure that?"
Research Methods: Good old qualitative research with target customer or staff users.
Hopefully the idea for this data or AI feature came from a real user need that you observed in your customer or staff experience. If it didn’t (for example, if the product idea came from a business need or available data), it’s still critical to study if there is a problem to solve in this part of the experience and how high it is on the list of priorities in the entire journey.
The biggest question to answer in this phase is what specific value your feature could create for users as it relates to that need. For example, we found that an AI feature that generated personalized product recommendations could help users meet one of three goals in the shopping journey:
- better product discovery: this helps me find things I didn’t know I wanted
- increased savings: this helps me get an even better deal
- enhanced convenience: this helps me finish my task faster and easier
It’s important to do the work of talking to users to hear this kind of value in their words - not make assumptions about the value users will see.
If you’re adding this feature where no user problem exists or no new user value is created (only business value), your qualitative research must show that it’s not making things worse for users. This can be an uncomfortable insight to deliver!
MVP
If users say they find your concept valuable, now the challenge is to prove that they will use it in real life and that it will lead to the behaviors you want… repeatedly and predictably.
Research methods: quantitative measures of behavior change, plus qualitative research on the reasons behind it.
In this phase you’re answering three questions “in the wild”:
- Does this feature lead to the intended behavior?
- Does it lead to that behavior repeatedly?
- What’s the business value of this change?
To answer #1, you could launch a quantitative A/B test to see if the new experience led to a behavior more often than in the old experience. If not, your qualitative follow up with users should answer where it went wrong - are there usability issues? Are AI generated answers not complete or relevant? Or is there just not enough perceived value there?
To answer #2, you’ll have to prove that users exposed to this more than once choose to use it when there's another option because they see value. If previous users abandon or choose a different option the second or third time, that’s a great clue. You might end up following up with three groups: adopters, rejectors, and “triers” who don’t repeat use.
Defining the business value means choosing specific KPIs and getting metrics before and after the MVP. For example, in an A/B test, there were 500 inbound call center requests per day without the self-service chatbot, and 300 requests when the chatbot was available. However, use a combination of qual and quant to investigate those results a step further - did the ~200 customers who used the chatbot actually resolve their issue or did they just abandon?
Scaled product
If you’ve proven that your feature or product reliably leads to the desired behavior and that there’s business value in that change, later on in the lifecycle the challenge is to prove the macro effect on the business or customer experience.
Research methods:
Researchers can use before and after measures of the customer experience like NPS scores or sentiment analysis of customer comments, but those can be hard to attribute to a specific part of the journey. Narrower metrics on behaviors related to the feature itself could be more useful if they show a meaningful change over time - such as adoption rate, completion rate, staff productivity increases, etc.
Also worth noting is that there can be value in the data collected when customers or staff use a feature, even if it doesn’t change their behavior, for example, building a corpus of customer questions to train a large language model, or gaining a better understanding of customer needs at different points in the journey.
About Ann
Ann is the founder of Amplify and has a background in product innovation and service design. She is experienced in leading research and strategy projects on complex questions, designing data-driven experiences, and developing user-centered mindsets and skills within organizations.
Ann is the founder of Bird, a coaching platform for everyday runners, that was a member of TechStars 2020 and acquired in 2023. Ann lives in Chicago with her family and their two dogs, Ruthie Bader Ginsburg and Cosimo de' Medici.
