Mystery Shopper Research: Scenarios and The Magic 20% (Part 4)

In Part 4 of our Mystery Shopper Research Series, we explore the role of Scenarios and introduce the concept of The Magic 20%.

In this Part 4 post we will –

Share where we are in our 7-Step Design Process for Mystery Shopper Research
Look at Mystery Shopper Scenarios – what they are, how they work and how to classify them
Recommend why the Design process should be cross-functional
Explain what we mean by ‘The Magic 20%’ – the powerful unexpected learnings that you should expect from your Mystery Shopper program
Share three case studies on The Magic 20%

Here are the links to the earlier posts in this series:

https://www.omnitouchinternational.com/mystery-shopper-research-a-window-into-customer-experience-part-1/

https://www.omnitouchinternational.com/mystery-shopper-research-definition-and-value-part-2/

https://www.omnitouchinternational.com/mystery-shopper-research-research-objectives-part-3/

Let’s refer back to our 7-Step Design Process for a Mystery Shopper Program

In this post we cover Step 2 in our 7-Step Design Process.

As a reference here is a summary description of each of the 7 Steps:

What do I want to learn: Define and document what you aim to learn from your research program – your Research Objectives
What Scenarios do I want to test: Define specific journey and/or touchpoint scenarios to be tested / Categorize each scenario as either existing Customer, prospective Customer or Observation
Special Profiling & Selection Needs: Consider special profiling needs for the Mystery Shoppers as to what they need to ‘be’ or ‘have’
Select Performance Standards: Select and define the Performance Standards to be scored and exactly how they are to be scored
Plan Volumes & Timing: Develop a fieldwork calendar that lays out the finalized volumes and timing of Mystery Shopper interactions
Design Reporting: Layout report design and flow – decide how will results be presented to stakeholders both when finalized and as they come in (red flags)
Design & Execute Training: Design the paths and scripts for Mystery Shoppers to execute as well as the tasks and equiprment along the way

Organizational metrics can’t tell the whole story

As you’d imagine, any Organization will already have a number of metrics in place to study the Customer’s experience as they progress through a specific journey or set of touchpoints.

But these metrics – even when assembled on a dashboard – aren’t great when you want to share a narrative perspective of what a Customer actually goes through.

It’s like trying to read and understand a novel where every other paragraph on the page is missing. Or where entire pages and even chapters are missing.

Making it difficult to truly ‘see’ the experience. And if you can’t see it, you can’t make it better.

That’s where Scenarios come in.

What Mystery Shopper Scenarios do I want to test?

With Mystery Shopper Research you can test nearly anything you want to. By sitting down with the Team and scripting out the specific Scenario you want to test.

A Scenario describes the the journey and set of touchpoints you want to study. The ‘flow’ or path the Mystery Shopper is to take.

And it describes the specific nature of the interaction to take place. The ‘script’ and/or set of actions the Mystery Shopper is to act out.

Remember that you’re the Director of this play.

And your Mystery Shoppers, like actors in a Broadway play, will follow the set of actions and responses that you’ve designed. And act out your pre-written script.

Which will help the Organization uncover and analyse what it’s like for real Customers to go through that same Scenario.

Mystery Shoppers never just go out there and ask or do whatever they want. That’s not professional Mystery Shopper Research.

Three examples of Mystery Shopper Scenarios

Here are three examples of Scenarios we’ve researched for Clients.

Example 1 – The Luxury Watch Retailer:

Have the Mystery Shopper visit a luxury watch store at the airport and express interest in purchasing a watch from Brand X. Observe how the Salesperson responds to the potential watch Buyer.

The impetus for this Scenario were a handful of complaints received from actual Customers who said that they were discouraged by the Salesperson from buying any other watch than one from Brand Z.

The Management at the retailer knew that the commission structure payout for Brand Z was higher for the Salesperson and wanted some data on what was really happening on the retail floor.

They also wanted to use the Scenario to gather data on how well Salespeople were using the trained sales techniques and brand language.

Example 2 – The global Courier Company

Have the Mystery Shopper call the Client courier company and ask about the price of shipping a pre-determined item from Country A to Country B. This is only an enquiry and no actual shipment need be made.

The global Management at this well known courier company wanted a set of service benchmarks across 38 countries (internal comparisons) and also with 2 competitors selected in each country (for external comparison).

The ‘Booking Enquiry’ was one of their most common and revenue generating enquiry types and they wanted a complete picture of how that enquiry was going now – on a global scale.

Example 3 – the 5 Star Hotel

Have the Mystery Shopper stay 2 nights in our hotel. To study our goal to be ‘seamless’ for our Guests, as the Mystery Shopper to act out the following scenario(s) at the prescribed time during their stay –

- At the Front Desk make a Room Service order to be delivered later that same day
- At the Restaurant tell the Greeter that the pillows in the Guest’s room are too hard

Hotel Management were eager to study how well their Customer Experience vision was being brought to life for hotel Guests.

Their CX Vision consisted of 7 Brand Values in all.

One of those values they called ‘Seamless’ – which meant that a Guest could make a request or drop feedback at any point or place during their stay – and not be redirected or pushed to another department or function.

Here are a few more Mystery Shopper Scenario examples

With each Scenario you choose you will need to sit down and write out a complete ‘flow’ – the path the Mystery Shopper should take – and a complete ‘script’ – what the Mystery Shopper should say or do as they go travel along the path.

But this simple listing of different Scenarios – all of which we’ve studied for Clients – should highlight the nearly endless possiblities you have when deciding what you want to test –

How well does a Museum Tour Guide handle answer questions about the latest exhibition?
How successfully can the Customer use an online portal to register and receive a response to a question about a particular product or service?
Does the Customer Service Representative at the University Services Contact Center fully share the latest scholarship options when a parent calls in with questions?
How well does a Real Estate Agent help a prospective Home Buyer who visits a show property?
In what manner does Airline Staff respond when a Customer claims to have lost their luggage?
Has the Staff at the Visitors Center or Club Lounge fully restocked the magazines, brochure rack and snacks for Guests?
Are Theme Park Employees delivering the theme park’s brand story in everything they say and do?
How clear is the Customer Service Team at the government pension fund in explaining how the calculation of retiree payouts work?

No two Organizations test the same things

There is no such thing as an ‘industry-standard’ Mystery Shopper program.

Ultimately each Organization decides what they want learn and the specific Scenarios they want to test.

I remember one retail company that sold jeans calling us up and saying, “We want the industry standard CX Mystery Shopper program please.”

It took a bit of back and forth to clarify what it was they wanted to learn.

Which turned out to be a current state evaluation of their Customer Service delivery.

How many Mystery Shopper Scenarios should you test?

Here is a ‘pro tip’ on how many Scenarios you should test.

I don’t recommend trying to test a lot of Scenarios with a low volume of interactions. For example, testing 20 different Scenarios – each 5 times.

Let me give a clarifying example.

A Client might say, “Dan, we want to test 20 Scenarios with 5 interactions each for a total of 100 Mystery Shopper interactions.”

“Our first Scenario is to send an email to our Correspondence Team and ask them about the benefits of our latest product X. Send that email 5x please.”

Technically the Client is expressing their Mystery Shopper needs in the correct way. This many Scenarios (20) this many times (5 interactions per Scenario) for a total of 100 interactions.

But after hearing this request, I’d probably tell the Client this

That 5 interactions is a rather low volume per Scenario
That a focus on a lesser number of Scenarios with an increased number of interactions per Scenario would yield deeper insight
When program results come out, it’s going to be hard to make improvements to up to 20 different Scenarios at the same time – it’s too much for any Organization to take on
You can run different ‘waves’ of Mystery Shopper over time – perhaps 5 scenarios in Q1, 5 different scenarios in Q2 and so on

Across some of our biggest Mystery Shopper programs, our Client focused in on as few as 3 – 5 key Scenarios to test.

And what’s impressive here is that the Client in these cases, really knew what they wanted to learn. They had prioritized what they wanted to learn.

Of course there is no industry standard for how many scenarios to test. But in my experience, it makes a lot of sense to do fewer Scenarios (at a time) and do those in more depth.

And you can always roll out additional Scenarios in a future ‘wave’ of Mystery Shopper Research.

Categorizing your Scenario(s)

We find it helps to categorize your scenario. For the work we do with Clients we typically share (3) main categories as follow:

1. Prospective Customer Scenario

We see this especially with sales environments. The Mystery Shopper shows interest in trying the shampoo, purchasing the apartment, or test-driving the new BMW.

A prospective Customer has questions. They’re making a decision as to whether to ‘go for it’ or not.

2. Existing Customer Scenario

An existing Customer already has an account or record with the company.

Perhaps an existing insurance Customer who has questions about a policy renewal. Or a telecoms Customer who wants to know when there might be new promotions to upgrade their phone.

With airlines, they sometimes ask us to find Mystery Shoppers who belong to their premier membership program.

Or for a doctor’s clinic, a patient who already receiving treatment for a certain ailment.

3. Observation Scenario

An Observation Scenario is exactly what it sounds like. The Mystery Shopper goes in to look for something. There may be no interaction with other people.

Or even where this interaction with people those interactions are not evaluated.

Are the TV screens showing the latest corporate video? Have the aisles been kept clear in the grocery store? Has the new logo been put up? Were the saftey barriers put up properly at each of the renovation sites? Is the signage in front of the theme park rides clear?

And as you’d expect, observations can be included in the Scenario design for either Prospective or Existing Customer categories as well.

Make your Mystery Shopper design work cross-functional

A common error in the design of a Mystery Shopper Program is to limit the design discussion to one or two key departments within the Organization. Such as Customer Experience or Market Research.

For some reason, the Mystery Shopper program is kept ‘secret’ from others in the Organization.

That’s not a good idea or a good look.

Because when the program results do come out and are shared across functions, it is typical for the (now somewhat shocked) functional Managers to resist and question the legitimacy of the program.

And even more, they resent that such a program was run without their prior knowledge or involvement.

One of the outstanding benefits of making your Mystery Shopper program design cross-functional – from the very beginning – is that the process of answering ‘what do we want to learn’ and ‘what Scenarios should we test’ results in healthy discussion, refinement and even alignment across Stakeholders.

The benefits of these discussions, refinement and alignment far outweigh the time taken. And help build program acceptance, not to mention deepen cross-functional relationships.

Because when program results do come out, we want folks to accept them – so that they can get to work on them.

‘The Magic 20%’ – expect the unexpected in your learnings

Now we come to one of my favorite aspects of any Mystery Shopper program. We call it The Magic 20%.

The Magic 20% is a term that refers to the unexpected but powerful learnings that you should always expect from your Mystery Shopper program.

‘Unexpected’ learnings simply because you don’t know what is going to pop up
‘Powerful’ learnings because this deep insight that can help you transform the way you do things
‘Expected’ learnings because you should expect some level of The Magic 20% in every Mystery Shopper program you run (even though you don’t know yet what they will be)

I should mention that the 20% isn’t an official figure. Some programs have more unexpected learnings. Others have fewer.

We coined the term The Magic 20% simply to make it memorable.

And to remind everyone involved that an advanced Mystery Shopper program should consist of more than scores and comments on those scores.

What’s the source of these unexpected but powerful learnings?

The source of these wonderful learnings? Comments that we receive directly from our Mystery Shoppers.

While it’s important for Mystery Shoppers to accurately score the Performance Standards they are assigned (which we cover in the next post in this series), we also ask them to provide their subjective feedback.

And great Mystery Shoppers deliver. Because they’re not just box tickers and score givers.

They’re out there observing and taking in what they see and hear. Which they give back to us in their subjective feedback. Feedback that can be surprisingly insightful at times – and unlock new ideas and thinking.

And perhaps the easiest way to explain them is to share real case studies.

Case Study – The Magic 20% for a major Financial Magazine

This case study is based on a global Mystery Shopper Program that we conducted for a major financial news magazine.

When analyzing the Contact Center results, we observed that the rate of putting a Caller on hold was significantly greater in Country A compared to Country B.

At first glance, it seemed that the service performance must have been ‘worse’ in Country A because of the high rate of holds.

However, after speaking with our Mystery Shoppers and doing more analysis, it turned out that the accuracy of the information provided by the Agents in Country A was significantly greater than Country B because the Agents in Country A made more of an effort to get the right answer.

Even though that meant putting more Callers on hold.

In Country B the Agents did not place the Caller on hold to verify or validate the answer to the Customer question. So they ended up providing a lot of (different) wrong answers. Though in a very convicing manner!

While the handling time and the rate of holds in Country B was less than Country A, the inaccurate information given could results in both dissatisfaction and unnecessary repeat calls.

The Magic 20% Learning: Avoid making superficial assumptions about outcome data. Conduct analysis carefully – especially when comparing performance across different groups.

Case Study – The Magic 20% for a major International Bank

In a Mystery Shopper program for an international Bank – which involved comparing and contrasting the bank’s performance against selected competitors – we uncovered a key learning on how the Counter Staff was ‘transferring’ or handing over the Mystery Shopper from the Counter Staff to the Wealth Manager.

The program planners designed it so the Mystery Shopper would share with the Counter Staff about a recent financial windfall. This ‘hint’ was meant to trigger a handover from the Counter Staff to a Wealth Manager.

The scoring design for the program included how the Counter Staff handed the Mystery Shopper over to the Wealth Manager. They called the Performance Standard ‘Polite Handover.’

But while the ‘Polite Handover’ performance scored 100% for one of the competitor banks, our Mystery Shoppers shared some subjective feedback which seemed to contradict the high score.

The Counter Staff would show the Mystery Shopper to a seat in a public waiting area while they went over to brief the Wealth Management Staff, excluding the Mystery Shopper/Prospective Client from the introduction.

The two banking staff would talk about the Mystery Shopper in full view, without including them in the conversation, which created a negative impression.

Comments from Mystery Shoppers essentially revolved around the theme “…I don’t care how nice they were – it was rude to talk about me and not include me – why did they not bring me over and introduce and explain in person? I would never sign up with this bank…”

The Magic 20% Learning: Be careful not to only look at scores. Behind a ‘high’ score could be a process or behaviour that could still drive Customers away. The bank subsequently made sure they included their Customer in the handover.

Case Study – The Magic 20% for a major Indian Airline

In a Mystery Shopper Program for an Indian Airline, the client asked us to evaluate all touchpoints on a flight journey. Starting from the valet at the curb to the collection of luggage at the final destination.

Within just five Mystery Shopper interactions, valuable subjective feedback emerged from our Mystery Shoppers.

When our Mystery Shopper dressed in a Western style business suit, they perceived the level of service they received as high. However, when the same person took another flight and wore ‘local’ or traditional garb they perceived the level of service standard they received was less.

The Mystery Shopper perception was that the ticketing and counter staff were making personal judgements them based on what they were wearing.

Which went against the image of this airline which was that anybody can fly and fly well with this particular brand.

When we alerted senior management on this finding, they reacted quickly.

And later on our Client contact told us that there was a sense that this was happening but it helped them to have the actual ‘proof’. The airline also updated their orientation, training and coaching programs.

The Magic 20% Learning: Don’t judge a book by its cover. Sometimes the wealthiest Clients have no one to impress.

https://www.omnitouchinternational.com/should-your-cx-head-be-a-contact-center-expert-too/

Thank you for reading!

In this Part 4 post in our Mystery Shopper Research series we –

Shared where we are in our 7-Step Design Process for Mystery Shopper Research
Looked at Mystery Shopper Scenarios – what they are, how they work and how to classify them
Recommended why the Design process should be cross-functional
Explained what we mean by ‘The Magic 20%’ – the powerful unexpected learnings that you should expect from your Mystery Shopper program
Shared three case studies on The Magic 20%

In the next post in this series we will –