Welcome to Part 4 of our Mystery Shopper Research Series — where we cover Scenario Design and ‘The Magic 20%’ — the unexpected qualitative learnings you should expect.
This article is part of our Mystery Shopper Research Series — practical insights on designing, running, and learning from Mystery Shopper programs.
In Part 4 we:
- Locate where we are in the 7-Step Design Process for Mystery Shopper Research
- Explore Mystery Shopper Scenarios — what they are, how they work, and how to classify them
- Specify specialing profiling needs for Mystery Shoppers
- Explain why the design process should be cross-functional
- Define The Magic 20% — the powerful, unexpected learnings you should expect
Let’s Revisit the 7-Step Design Process for a Mystery Shopper Program
In this post we cover Steps 2 & 3:
- What do I want to learn? Define and document your research objectives
- What scenarios do I want to test? Define specific journey and/or touchpoint scenarios. Categorize each as existing Customer, prospective Customer, or observation-only.
- Special profiling & selection needs: Specify what shoppers need to be or have (such as a status or product type)
- Select performance standards: Define the standards to be scored and the scoring logic (binary and scaled) with rubrics
- Plan volumes & timing: Build a fieldwork calendar with finalized volumes and cadence
- Design reporting: Decide how results will be presented—both as they come in (such as red flags) and at the final readout
- Design & execute training: Provide paths and scripts for shoppers, plus the tasks and equipment (such as recording devices) needed.
Organizational Metrics
As you’d expect, Organizations already track many metrics across journeys and touchpoints. But even a metrics-rich dashboard can’t convey the narrative of what a Customer actually goes through.
It’s like reading a novel with missing pages. If you can’t see and interpret the experience, you can’t improve it.
That’s where well-designed scenarios come in.
Mystery Shopper Scenarios
A Mystery Shopper Scenario is a narrative of the journey and touchpoints you want to study. You write the story and define the script and actions the Mystery Shopper will carry out.
You can test almost anything — by selecting and writing out the specific scenario you want to study.
Think of yourself as a film director. And your 
Mystery Shoppers as the actors — who follow the actions and responses you designed.
It is this intentionality of design that enables you to study what it’s like for real Customers who go through that same scenario.
Here Are Three Examples of Scenarios
We conducted each of these scenarios for Clients. As you read them, you’ll understand why the Client chose the scenario and see the specificity with which each scenario was designed.
Example 1 – The Luxury Watch Retailer
The Scenario: Have the Mystery Shopper visit a luxury watch store at the airport and express interest in purchasing a watch from Brand X. Observe how the Salesperson responds to the potential watch Buyer.
The impetus for this scenario was a handful of complaints received from actual Customers who said that they were discouraged by the Salesperson from buying any other watch than one from Brand Z.
The Management at the retailer knew that the commission payout structure for Brand Z was higher for the Salesperson and wanted some data on what was really happening on the sales floor.
They also used the scenario to gather data on how well Salespeople were using the sales techniques and brand language from a recent training intervention.
Example 2 – The Global Courier Company
The Scenario: Have the Mystery Shopper call the courier company and ask about the price of shipping a pre-determined item from Country A to Country B. This scenario is an enquiry only; no shipment needs to be made.
The global management at this well-known courier company wanted a set of service benchmarks across 38 countries (for internal comparisons) and also with two competitors selected in each country (for external comparisons).
Example 3 – The 5-Star Hotel
The Scenario: Have the Mystery Shopper stay 2 nights in our hotel. One of our brand values is to be ‘seamless’ for our Guests, so the Mystery Shopper should act out the following scenarios at the prescribed times during their stay:
- At the Front Desk, place a Room Service order to be delivered later that day
- At the Restaurant tell the Host/Hostess that the pillows in the guest room are lumpy
Testing ‘seamless’ was one just one of their brand values. Hotel management were eager to study how all seven of their brand values were being brought to life.
To accomplish the goal, we ended up designing over 20 scenarios to test.
More Scenario Examples
With each Scenario you choose you will sit down and write out a complete ‘flow’ — the path the Mystery Shopper should take — and a complete ‘script’ — what the Mystery Shopper should say or do as they go travel along the path.
This simple listing of different scenarios below — each of which we’ve studied for Clients — highlights the endless possiblities for deciding what to test:
- How well does a Museum Tour Guide handle answer questions about the latest exhibition?
- How successfully can the Customer use an online portal to register and get a response to a product/service question?
- Does the Customer Service Representative at the university Contact Center fully explain the latest scholarship options when a parent calls in with questions?
- How well does a Real Estate Agent help a prospective Home Buyer who visits a show property?
- In what manner does Airline Staff respond when a Customer claims to have lost their luggage?
- Have the staff at the Visitors Center or Club Lounge fully restocked the magazines, brochure rack and snacks for Guests?
- Are Theme Park Employees delivering the theme park’s brand story in everything they say and do?
- How clear is the Customer Service Team at the pension fund in explaining how the calculation of retiree payouts works?
Keep in mind that sometimes your Mystery Shoppers will need to ‘be’ or ‘have’ something particular. We call this special profiling needs.
They may need to:
- Have a current account with a certain bank
- Own a particular device or piece of equipment
- Be part of an existing loyalty or VIP club
- Live in a certain location
- Speak a certain language
Special profiling needs are based on scenario.
No Two Organizations Test the Same Thing
TThere’s no such thing as an ‘industry-standard’ Mystery Shopper program. Each Organization decides what they want to learn and what scenarios to test.
I remember one retail company that sold jeans calling us up and saying, “We want the industry standard CX Mystery Shopper program please.”
It took some back-and-forth to clarify what they wanted to learn. Which turned out to be a current state evaluation of their Customer Service delivery.
How Many Scenarios Do You Test?
Avoid testing too many scenarios with low volumes per scenario.
Let me give a clarifying example: A Client might say:
Dan, we want to test 20 scenarios with 5 interactions per scenario for a total of 100 Mystery Shopper interactions.
100 interactions may sound like a lot but not when it’s spread across 20 scenarios. Here’s what I’d likely say to this Client:
- 5 interactions per scenario is typically too low to learn much
- Focus on fewer scenarios with more interactions per scenario
- It’s hard to improve many Scenarios at once—focus in on what matters most so you can take action
- Run different waves of research to introduce new scenarios: e.g., 5 Scenarios in Q1, 5 new Scenarios in Q2, etc.
Categorizing your Scenarios
We find that it helps to categorize your scenarios. We typically use three main categories, as follows:
1. Prospective Customer Scenario
In a prospective Customer scenario, the Mystery Shopper expresses interest in a product or service by asking questions and getting answers.
It could be about trying a medication, purchasing a new apartment, or test-driving the new BMW.
2. Existing Customer Scenario
An existing Customer already has an account, service, or status with the company.
- Ask an existing insurance Customer to go through a policy renewal.
- Ask an existing mobile Customer to ask about new promotions.
Airlines sometimes ask us to find Mystery Shoppers who belong to their premier membership or frequent flier program.
3. Observation Scenario
An Observation Scenario is exactly what it sounds like. The Mystery Shopper goes in to look for something.
There may be no interaction with company employees. And even when company’s employees are present, that aspect of the experience may not be measured.
Here are examples of observations:
Are the TV screens showing the latest corporate video?- Have the aisles been kept clear in the grocery store?
- Has the new logo been put up everywhere?
- Were the saftey barriers properly installed at each of the renovation sites?
- Is the signage in front of the theme park rides clear?
- Have hand-written signs been used?
Are the TV screens showing the latest corporate video?){kind=link}
And as you’d expect, observations can be included in the scenario design for Prospective and Existing Customer categories as well.
Design Work Should Be Cross-Functional
A common error is limiting program and scenario design to one or two departments (e.g., CX or Research). Sometimes the program is intentionally kept hidden from others in the business.
That’s not a good idea.
Because when the program results do come out and are shared across functions, it can result in shock and resistance.
Stakeholders who are suddenly hit with Mystery Shopper outcomes question the program’s legitimacy — and resent that it ran without their awareness or involvement.
There are outstanding benefits when you make your Mystery Shopper program design cross-functional.
The processes of answering ‘what do we want to learn’ and ‘what Scenarios should we test’ result in healthy discussion, refinement, and alignment across Stakeholders.
And it helps to build trust in the process and the outcomes. Because when the program results do come out, we want our Stakeholders to accept them — so that we all can get on with taking action.
The Magic 20% – Expect the Unexpected in Your Learnings
Now we come to one of my favorite aspects of any Mystery Shopper program. We call it The Magic 20%.
The Magic 20% refers to the unexpected and powerful learnings you should expect from a Mystery Shopper program.
- ‘Unexpected’ because you don’t yet know what will surface
- ‘Powerful’ because the insights can help you transform the experience
- ‘Expected’ because every program should yield some of these
I should mention that the 20% isn’t an official mathematical figure. Some programs have more unexpected learnings while others have less.
We coined the term The Magic 20% to make it memorable — and to remind teams that an advanced program yields more than just scores.
What’s the Source of these Unexpected but Powerful Learnings?
The Magic 20% comes from qualitative comments provided directly by Mystery Shoppers.
While it’s important for Mystery Shoppers to accurately score the Performance Standards assigned (which we cover in the next post in this series), we also ask them for their subjective feedback.
And great Mystery Shoppers deliver — not just ticking boxes on a form, but observing keenly. Because their personal perceptions can be surprisingly insightful at times.
The Magic 20% for a Financial Magazine
We conducted a global Mystery Shopper program on the Contact Centers of a financial news magazine.
When analyzing the Contact Center results, we observed that the rate of putting a Caller on hold was significantly greater in Country A as compared to Country B.
At first glance, it seemed that the service performance must have been ‘worse’ in Country A because of the higher rate of holds.
However, after doing more analysis, it turned out that the accuracy of the information provided by the Agents in Country A was significantly greater than Country B. The Agents in Country A made more of an effort to get the right answer.
Even if that meant putting more Callers on hold.
In Country B the Agents did not place the Caller on hold as often to verify or validate their answer to the Customer question. But when we looked at the accuracy Agent answers in Country B we found that the rate of inaccuracy was higher.
Magic 20% learning: Don’t make superficial assumptions about outcome data. Analyze carefully — especially across groups — before formalizing conclusions.
The Magic 20% for an International Bank
For an international Bank we uncovered a key learning on how the Counter Staff was ‘transferring’ the Mystery Shopper from the Counter to the Wealth Manager.
The Mystery Shopper was instructed to share with the Counter Staff that they had a recent financial windfall. This ‘hint’ – about having some extra money – was meant to trigger an introduction from the Counter Staff to a Wealth Manager.
The scoring design for the program included how the Counter Staff handed the Mystery Shopper over to the Wealth Manager. The behavior was called ‘Polite Handover.’
But while the ‘Polite Handover’ performance scored 100% according to the Client’s definition, our Mystery Shoppers shared subjective feedback that added insight to what was really happening.
The Counter Staff would show the Mystery Shopper to a seat in a public waiting area. And then walk over to brief the Wealth Management Staff. The two employees discussed the Shopper in full view of the Shopper — without including them.
Comments from our Mystery Shoppers essentially revolved around the theme “…I don’t care how nice they were – it was rude to talk about me and not include me – why did they not bring me over and introduce and explain in person? I would never sign up with this bank…”
Magic 20% learning: High scores can mask issues. Pair scorecards with narratives to spot behaviors that can drive Customers away (e.g., handovers that exclude the Customer).
The Magic 20% for an Indian Airline
For an Indian Airline, the client asked us to evaluate all touchpoints on a flight journey. Starting from the valet at the curb to the collection of luggage at the final destination.
Within just five Mystery Shopper interactions, valuable subjective feedback emerged from our Mystery Shoppers.
When our Mystery Shoppers dressed in a Western style business suit, they perceived the level of service they received as higher. But when they wore traditional garb from their region, they felt the service standard degraded.
In essence, Mystery Shoppers felt that airline staff were making personal judgements about them based on what they were wearing. Which went against the stated brand values of the airline.
When we alerted senior management on this finding, they reacted quickly. They told us the objective evidence we had gathered helped. They updated their orientation, training, and coaching materials across all employees.
Magic 20% learning: Don’t judge Customers by appearance. Bias in treatment undermines brand values — and the wealthiest Customers often have no one to impress.
Where You Are in This Series
In this Part 4 Article we:
- Located where we are in the 7-Step Design Process for Mystery Shopper Research
- Explored Mystery Shopper Scenarios — what they are, how they work, and how to classify them
- Specified specialing profiling needs for Mystery Shoppers
- Explained why the design process should be cross-functional
- Defined The Magic 20% — the powerful, unexpected learnings you should expect
In Part 5 we will:
- Define Performance Standards—how to select, score, and report them
- Share practical examples
Thank you for reading!
I regularly share stories, strategies, and insights from our work across Contact Centers, Customer Service, and Customer Experience. If this resonates, I’d love to stay connected.
You can drop me a line anytime, or subscribe via our website.
Daniel Ord
[email protected]
www.omnitouchinternational.com
