What we’ve learned about conducting Mystery Shopper Research on Chatbots

We share some of what we’ve learned about conducting Mystery Shopper research on Chatbots.

You mean there’s Mystery Shopper research on Chatbots?

We recently completed our fourth Mystery Shopper research program on a Client’s chatbot offering for Customers.

And when we share that we conduct Mystery Shopper research on Chatbots the first thing we usually hear is this –

“There’s such a thing as Mystery Shopper research on Chatbots?”

Because almost any time you hear someone in the CX or Contact Centre industry talk about Chatbots, it’s on things like AI, machine learning, natural language processing and the like.

All cool and important stuff.

But inwardly focused. You’ll never hear a Customer say –

“You know I used that NLP enhanced Chatbot at the bank/telecom/insurance company today and I have to say that they have the best decision tree ever.”

For a Customer it’s another channel. One they expect will help them achieve their goals.

And Mystery Shopper – when well designed – is a great tool to help ensure the Customer perspective is considered.

Some of our Clients prefer the term ‘Digital Contact Audit’ rather than ‘Mystery Shopper Research’.

But whatever you call it the goal is to improve the Customer Experience of the Chatbot.

Why did these organizations pursue Mystery Shopper research on their Chatbots?

Firstly a big thanks to the four Clients – all in Asia Pacific – for engaging in the Chatbot research.

And even though the Mystery Shopper research briefs came from four different organizations, there was significant commonality across all the briefs.

In summary, the commonality sounded like this.

Client: “We’re known in the market for service. And we’ve got a Chatbot. So the Chatbot has to represent our service focus well or it could hurt our brand.

And of course, the better the service delivered by the Chatbot, the better the containment rate will be within the Chatbot channel.

But we think we’re suffering internally from knowing ‘too much’ about our own organization.

Too much about the Chatbot tech, about the products & services we offer and about we do things around here.

So we’re looking for a Customer Experience based Mystery Shopper program that’s going to look at the Chatbot from the Customer perspective – not our own.

The parameters selected for the Chatbot research

To achieve the research objectives, we set the following key parameters for the Chatbot research. And these parameters have served us well.

The use of Journeys

We defined, with the Client, which Customer journeys to study. So each Mystery Shopper program consisted of a defined number of specific journeys.

To help explain and define what a Customer journey looked like – and what we would be studying – we used this simple script’ as a guide.

Customer: “Because of _______I wanted to find out ________ so that I could _________. (all blanks to be filled in)

Customer: “Because I’m bringing my family on holiday in June, I wanted to find out about the opening hours & entry fees at the theme park, so that I can understand the budget. if any promotions apply and better plan for the trip overall.”

Customer: “Because I’m travelling overseas next week, I wanted to find out how to avoid unnecessary or unexpected roaming charges on my mobile phone.”

And the individual journeys that were studied, were selected for different reasons.

Sometimes the Client chose to study the journeys with the highest number of visits. Or they chose the journeys with the highest number of opt-outs to other channels.

Often the selected journeys were those that had attracted the lowest satisfaction rating.

One of the great things about Mystery Shopper research is that you can choose what you want to learn. The design process can be intensive – to the benefit of the entire research process and outcomes.

The Customer Experience lens we used

To bring structure and clarity to assessing the Customer Experience delivered by the Chatbot, we used the 3 Levels of Customer Experience pyramid.

A model that any Certified Customer Experience Professional (CCXP) or Customer Experience professional will be familiar with.

The 3 Levels are:

Effectiveness (Met my needs)
Ease (Was easy)
Emotion (what I thought and how I felt)

We assigned a scoring mechanism to each level, as well as to the overall performance and we developed a ‘dashboard’ to indicate which journeys needed improvement and at which levels.

And of course all scores were supported and informed by qualitative input. Because when you’re assigning a score related to Customer Experience, that score must be backed up by the qualitative rationale.

The Customer Experience Mystery Shopper Program – are you on track?

A key learning that came out of the Mystery Shopper research

A key Customer Experience learning that came out of the research across all the Clients was this.

Much of the language built into the Chatbot assumed high Customer familiarity with the various product names, industry terms and context in use at that organization.

All things that a dedicated Employee of the organization would intuitively understand.

But not necessarily Customers.

When the Customer didn’t understand the language presented by the Chatbot, it took them 3 -4 times as long to complete the chat – either successfully (they got what they needed eventually) or unsuccessfully (they needed to rollover to human assisted service).

And in Client discussions, we learned that some of the Chatbot script writing had been delegated out to different departments.

So for example, if the Chatbot was going to answer a Finance-related question, the ‘answer’ to that question was developed by the Finance Department.

On the surface, the approach seemed logical.

But it resulted in approaches and language that varied by department. And by the competency within each department to write in a Customer-oriented manner.

Terror in the Boardroom – and the impact on your Mystery Shopper research

In closing

We have found that our Mystery Shopper work on Chatbots has been as rewarding as the work we do on other channels.

And a few parting tips:

Bring Customers into the picture – ask them to evaluate their journeys with you
Think deeply about the need that drove the Customer to use your Chatbot
Consider where the Customer goes after engaging with the Chatbot – is it all done?
Keep in mind that sometimes the very best Employees have the most difficult time thinking like a Customer – and understand that doesn’t come from a lack of Customer-centricity