English
German
In this article, we share some of what we’ve learned about conducting Mystery Shopper research on Chatbots.
You mean there’s Mystery Shopper research on Chatbots?
We recently completed our fourth Mystery Shopper research program on a Client’s chatbot offering for Customers.
And when we share that we conduct Mystery Shopper research on Chatbots the first thing we usually hear is this –
“There’s such a thing as Mystery Shopper research on Chatbots?”
Because almost any time you hear someone in the CX or Contact Centre industry talk about Chatbots, it’s on things like AI, machine learning, natural language processing and the like.
All cool and important stuff.
But inwardly focused. You’ll never hear a Customer say –
“You know I used that NLP enhanced Chatbot at the bank/telecom/insurance company today and I have to say that they have the best decision tree ever.”
For a Customer it’s another channel. One they expect will help them achieve their goals.
And Mystery Shopper – when well designed – is a great tool to help ensure the Customer perspective is considered.
Some of our Clients prefer the term ‘Digital Contact Audit’ rather than ‘Mystery Shopper Research’.
But whatever you call it the goal is to improve the Customer Experience of the Chatbot.
Why did these organizations pursue Mystery Shopper research on their Chatbots?
Firstly a big thanks to the four Clients – all in Asia Pacific – for engaging in the Chatbot research.
And even though the Mystery Shopper research briefs came from four different organizations, there was significant commonality across all the briefs.
In summary, the commonality sounded like this.
Client: “We’re known in the market for service. And we’ve got a Chatbot. So the Chatbot has to represent our service focus well or it could hurt our brand.
And of course, the better the service delivered by the Chatbot, the better the containment rate will be within the Chatbot channel.
But we think we’re suffering internally from knowing ‘too much’ about our own organization.
Too much about the Chatbot tech, about the products & services we offer and about we do things around here.
So we’re looking for a Customer Experience based Mystery Shopper program that’s going to look at the Chatbot from the Customer perspective – not our own.
And it’s been great to see how some organizations are raising the bar on their Mystery Shopper research.
Moving the research beyond typical compliance measurements to research that embraces the Customer Experience.
The parameters selected for the Chatbot research
To achieve the research objectives, we set the following key parameters for the Chatbot research. And these parameters have served us well.
The use of Journeys
We defined, with the Client, which Customer journeys to study. So each Mystery Shopper program consisted of a defined number of specific journeys.
To help explain and define what a Customer journey looked like – and what we would be studying – we used this simple script’ as a guide.
Customer: “Because _______I wanted to find out ________ so that I could _________. (all blanks to be filled in)
Customer: “Because I’m bringing my family on holiday in June, I wanted to find out about the opening hours & entry fees at the theme park, so that I can understand the budget. if any promotions apply and better plan for the trip overall.”
Customer: “Because I’m travelling overseas next week, I wanted to find out how to avoid unnecessary or unexpected roaming charges on my mobile phone.”
And the individual journeys that were studied, were selected for different reasons.
Sometimes the Client chose to study the journeys with the highest number of visits.
Or they chose the journeys with the highest number of opt-outs to other channels.
Often the selected journeys were those that had attracted the lowest satisfaction rating.
Because one of the great things about Mystery Shopper research is that you can choose what you want to learn. And that approach worked perfectly for the Chatbot research.
The Customer Experience lens we used
To bring structure and clarity to assessing the Customer Experience delivered by the Chatbot, we used the 3 Levels of Customer Experience model.
A model that any Certified Customer Experience Professional (CCXP) or Customer Experience professional will be familiar with.
The 3 Levels are:
- Effectiveness (Met my needs)
- Ease (Was easy)
- Emotion (what I thought and how I felt)
We assigned a scoring mechanism to each level, as well as to the overall performance and we developed a ‘dashboard’ to indicate which journeys needed improvement and at which levels.
And of course all scores were supported and informed by qualitative input. Because when you’re assigning a score related to Customer Experience, that score must be backed up by the qualitative rationale.
The Customer Experience Mystery Shopper Program – are you on track?
A key learning that came out of the Mystery Shopper research
A key Customer Experience learning that came out of the research across all the Clients was this.
Much of the language built into the Chatbot assumed high Customer familiarity with the various product names, industry terms and context in use at that organization.
All things that a dedicated Employee of the organization would intuitively understand. But not necessarily Customers.
When the Customer didn’t understand the language presented by the Chatbot, it took them 3 -4 times as long to complete the chat – either successfully (they got what they needed eventually) or unsuccessfully (they needed to rollover to human assisted service).
And in Client discussions, we learned that some of the Chatbot script writing had been delegated out to different departments.
So for example, if the Chatbot was going to answer a Finance-related question, the ‘answer’ to that question was developed by the Finance Department.
On the surface, the approach seemed logical.
But it resulted in approaches and language that varied by department. And by the competency within each department to write in a Customer-oriented manner.
Terror in the Boardroom – and the impact on your Mystery Shopper research
In closing
We have found that our Mystery Shopper work on Chatbots has been as rewarding as the work we do on other channels.
And a few parting tips:
- Bring Customers into the picture – ask them to evaluate their journeys with you
- Think deeply about the need that ‘drove’ the Customer to use your Chatbot
- Consider where the Customer goes ‘after’ engaging with the Chatbot – is it all done?
- Keep in mind that sometimes the very best Employees have the most difficult time thinking like a Customer – and understand that doesn’t come from a lack of Customer-centricity
Thank you for reading!
Daniel and Marcus
[email protected] / [email protected]
