6 Dysfunctions of Using a Survey as a North Star Metric
We love our data and we need it to make decisions. Surveys can be a valuable tool, when used sparingly and directed to the right people and under the right circumstances. However, when their results are used to try to systematically guide the behavior of your team, they don’t work very well. Surveys are subjective proxies that can lead to dysfunction for many reasons. Let’s explore a few of them.
1. They are disruptive 100% of the time.
It is tempting to want to survey everything as often as possible, but it takes a toll on your customer’s experience. Surveys can be a useful tool, but they are often abused. In the worst cases, companies are interrupting their customers during or after every other experience in the name of the latest statistical trends, like the Net Promoter Score. The voice of the customer and understanding the customer experience is an important and necessary part of innovation, but surveys are almost always intrusive and annoying.
When I ask the leaders in my workshops to raise their hands if they love to get surveys, no hands go up. The industry even has a term for this. It is called “survey fatigue.” The moment you have people who decide not to take your survey, you end up with survivorship bias. Nassim Taleb, Ph.D., calls this “silent evidence,” in his book, The Black Swan.
2. The mere act of surveying impacts the results.
When researchers studied employees in a Western Electric plant known as “Hawthorne Works” in Cicero, IL — they demonstrated that people can’t be studied without influencing their behavior. The researchers coined the phrase the Hawthorne Effect. This phenomenon is also well known in science as the Observer Effect. We can’t possibly measure people without impacting how they experience us — surveys are no exception. Your customers may not want to let you down, and they may inflate or will at a minimum adjust their response just because you are paying attention. They may see themselves as good people who don’t want anything bad to happen to the people in your organization. Alternatively, they may give you lower results because they are annoyed, simply because you are surveying them.
The worst example of this problem is when that greasy salesperson stalks you as you are leaving the showroom to tell you about the survey you are going to get and how it will impact their personal income. I highly recommend against using survey data in compensation schemes.
3. What People Say is Not Always Correlated With What They Do.
Margaret Mead was noted for believing that there is a big difference between what people say they will do and what they will actually do.
What people say, what people do, and what people say they will do are very different things. — Margaret Mead
This was an early lesson in the early days of building software applications for users on the internet. My teams would survey users to find out what features they would use, in our “discovery phase,” and get all sorts of ideas. When we implemented many of them, they would never get used. If it were so easy to just ask your customers what they wanted, wouldn’t everyone be doing that? If that were the case, everyone would be building the exact same things. Survey instruments that ask people what they will do, like the NPS (Net Promoter Score) have been proven to be poor predictors of actual human behavior.
4. They only measure a single point in time, and it is from your perspective.
If you are deploying a survey instrument to assess how your customer is feeling, you have to choose where in your journey you want to interrupt them. Your customer’s journey, however, is never as smooth as you think it is. We would like to believe that most of our customers go down the “happy path” with our products and services, but they will, most likely, tell you a very different story. So where do you assess their satisfaction or likelihood of recommending you? If you only assess them at the point of product delivery, you are missing a bigger opportunity to look at the broader experience that they are having.
5. They are error-prone, subjective self-assessments.
Some people are never going to recommend anything to anyone. It is just how they are built. Other people are promiscuous with their recommendations. When you ask a question like: “How likely are you to refer a friend or family member?” Some people include all of the people they have ever said “hello” to, while others only consider their three best friends and their mom. However you might word your surveys, they are always open to interpretation by the subject, leading to all sorts of misleading errors and biases.
6. Other things, besides your performance, impact their answers.
Everyone has a mood. They wake up, proverbially, on different sides of the bed, have different experiences throughout their day, and the decisions that they make are deeply impacted by their environment and the events of the day. In one famous study by Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso, for example, judges were found to be more lenient on criminals after lunch. If professionals at that level are impacted, it is highly likely that people responding to your surveys are going to do the same thing. They will give a lower score if they were cut off at the traffic light in the morning than they would if they were surprised that morning by someone bringing donuts into the office.
Before you roll your next survey out into the wild, take a step back and ask yourself if there is a better way to derive the insights that you are looking for. For example, If you want to know if your users are recommending you to their friends and family, make it easy for them to recommend you and measure how often they actually do it. If you want to know if they are happy with your product, measure how often they are actually using your products or services. Repetitive engagement at appropriate intervals is a great measure of satisfaction and loyalty for your organization. Consider looking for ways to objectively measure actual human behaviors.
All of that to say that surveys have a place. They can provide powerful qualitative insights that will help you drive more efficient innovation cycles if they are engineered well. Game-changing ideas only come from ideas and ideas only come from people. In fact, I think the absolute best place for surveys is with your advocates. Customers who are already advocates are usually willing and often eager to share constructive criticism, feedback, and great ideas. In this case, however, they should be used sparingly and thoughtfully so that it is a positive experience for them and not a step down the relationship ladder.
Even better, the people that talk to or have access to many of your customers are more likely to know where the sources of both frustration and joy might lie. Your customers have only one set of experiences with you — theirs. Customer service, marketing, sales, and service people, however, observe hundreds or thousands of customers going through their journeys with your organization. They know where all of the skeletons are hidden. You are often better off surveying those folks than you are surveying your customers.
If you liked this, clap, share as widely as possible and give me some feedback, I would be honored. Also, check out this article on the dysfunctions of using profit as your north star metric.
References:
Survivorship Bias on Wikipedia
The Black Swan, by Nassim Taleb, P.h.D (2010)
Net Promoter Score Considered Harmful, by Jared Spool (2017)
The Hawthorne Effect on Wikipedia
The Observer Effect on Wikipedia
Margaret Mead, by Jane Howard (1989)
Measuring Loyalty, by Sean Flaherty (2020)
Extraneous Factors in Judges Decisions, by Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso (2011)
