How Focus Group Questions & Opinion Questions Hurt Your Business

You’re hurting your business. Every time you ask your customer’s opinion, you’re imperiling your business. And science proves it.

You’ve probably heard that Steve Jobs hated focus groups.

It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them.” – Steve Jobs

But there’s more to the story. A focus group almost killed one of the most successful ads of all time. It was in Apple’s early days and you’ll see how in a minute.

Opinion questions ruin your focus groups and customer feedback. It’s proven in science. And we’ll explain why.

Our testing was flawed

It became obvious many years ago. We love testing concepts before deploying them. But we never got useful feedback from opinion questions. The responses were all over the map and provided no clear direction.

So, we quit asking opinion questions. But a couple years later, we discovered the science that explained the fallacy of opinion questions.

Here’s why: The part of the brain that explains the decision is not the part of the brain that made the decision. And the explainer part of the brain was never told why. So, it makes up a reason.

Seem strange? It will blow your mind when you see how it works.

A great thing almost never happened

Apple tested their legendary 1984 ad. And the focus group flatly rejected it. Apple’s board read the results and pulled the plug on the project. There is only one reason the 1984 ad ever got played.

Steve Jobs stood in front of the board and said he and Woz would pay for the ad themselves. The world pivoted on that one moment.

That ad launched the era of iconic Superbowl ads. People who weren’t even born in 1984 know about this ad and its legendary impact. Did you know …

That ad only ran that one night. But more than 3 decades later, it continues to build Apple’s brand.

Thankfully the focus group results were discarded.

Here’s a video of the focus group testing. (Aren’t you glad you’re not caught on video shooting down one of the best ads in history!)

Research and testing led to the wrong decision

Do you remember New Coke?

$4 million in testing and development and 200,000 test subjects proved it was the right decision. Blind test results pointed to a clear winner.

New Coke was the preferred choice over both Pepsi and Classic Coke. The data was clear. What could go wrong?

The response was immediate. Almost a half million people sent hate mail and demanded their original Coke[1]. That forced Coca-Cola to make another costly decision.

Coca-Cola wrote off $30 million in inventory while they switched back to original Coke. That’s about $75 million in 2019 dollars.

That size of mistake would bury most small and medium-sized businesses in America. Diligently seeking customer feedback was a costly mistake. So, what went wrong?

Science has the answer

It began by cutting the brain in half. Yes, surgically severing the brain in two … on a live human.

Like many breakthroughs, it was discovered by mistake. In this case, doctors were pioneering a treatment for epilepsy patients when they noticed this effect.

Patients with the split brains could function normally with no impact on hand-to-eye coordination. But it got interesting when they had to explain their actions.

Their actions were correct, but their explanations were complete and total nonsense. But only for information processed in the left field of view.

Let’s grab a beer some time and I’ll happily explain why it only impacted the left field of view. For now, let’s stay out of the weeds and focus on the actionable idea this experiment revealed. It’s both simple and important.

You have your fast brain and your slow brain. The fast brain makes most of your decisions. It forms most of your opinions. It effortlessly and continuously processes and categorizes information. But it has a major limitation.

Your fast brain can’t process any question that begins with “why.” So, it summons the slow brain to give an answer. But there’s a problem.

The slow brain was never involved in the decision. So, it starts looking for the pieces to assemble an answer. When some of the pieces are missing, the slow brain fills in the gaps with its best guess. And this leads to the problem.

Your slow brain never tells you that it’s guessing. It reports the information to you as gospel truth. Then you stand and rigorously defend this choice because you believe it to be rational and thoughtful.

That’s what happened in the split-brain experiments. They defended their nonsensical answers because they believed them in the depths of their hearts.

Are you beginning to see how 200,000 taste testers led Coca-Cola to a $30 million mistake?

They asked the wrong question

We don’t buy soda based on blind taste tests. There are many factors that influence the choice.

What if one of your most powerful memories of your grandfather was the time you sat on the pier together and drank a Coke while you talked about something important? Every time you drink a Coke it reminds you of that time. Is your soda choice about the best flavor?

Think about how the test results would have changed if they put a small sample cup in front of clearly marked cans of Coke, Pepsi, and New Coke and they gave the tester $1. The tester was told they had to spend the dollar to purchase one can of soda. And they could taste a small sample of each before making the choice. How would that have changed the outcome?

This moves you in the direction of understanding how people will spend their money. And that’s what matters.

If you want to know how people will spend money, create tests that observe action – not opinions.

If possible, figure out what people feel – as opposed to what they think. Feelings drive more purchases than rational thought. That’s true with the most rational of thinkers.

Engineers pride themselves in their rigorous process of rationality. But after you spend a few minutes with an engineer, you will discover high risk aversion. They are afraid something will go wrong. And guess what – fear is an emotion. So, yes, feelings drive more purchases than rational thought.

The testing environment is flawed

As soon as people know they’re part of a test, their behavior changes.

A great example is the lighting experiments from the industrial revolution. The goal was to determine how lighting impacted productivity. So, they had a control group with normal lighting. And they had a variation group with enhanced lighting. What happened?

Productivity for both groups increased because they knew they were being tested.

Focus groups are fundamentally flawed. And Apple’s 1984 commercial makes that clear.

You measure how people want to be perceived more than how they really are or will make choices. It’s not that they’re lying. At least not lying intentionally. This takes us back to the split-brain tests.

The split-brain tests reveal that we’re influenced by factors we can’t explain. And the brain generates reasons to explain our choices.

We need to change the way we test.

Create decoy tests

The best test is always measuring how people spend their time and money when given a choice. But that’s not always practical. So, use a decoy test.

You know people change their behavior when they know they’re being tested. So, create a test that lets you measure how they behave on factor “A” while making them believe they’re being measured on factor “B.”

Here’s an example: We had a financial management client, and their chief investment officer was previously an active gambler.

If you were a part of a focus group, how would you answer this question:

You are looking for someone to manage your retirement savings. If you found out that your financial manager had a gambling background, would it increase or decrease your trust?

Most people would answer that it decreases trust. Someone else with a need to be edgy and controversial would answer differently. In the end, the focus group testing would conclude it was too risky and not proceed.

But none of those answers would reveal true results. So, here’s what we did.

This content was going on the company’s About Us page. We mocked-up a page and put it to test. We weren’t subtle about his gambling background. In fact, here is a paragraph that is 100% devoted to revealing that background.

John’s only financial background was a gambling fascination and exploring ways to use computer science to make him a better gambler. He split his time between college, IT support and Vegas.

There you go. A paragraph with two sentences and both sentences reference gambling. Not subtle at all.

This may sound counter-intuitive, but the purpose of this disclosure was to create trust. The method is steeped in science and psychology. But we needed a test that could accurately measure if it increased or decreased trust.

So, here’s what we did …

We created a test that made the testers think they were being tested on comprehension and retention. This was important to make sure they read and remembered the gambling bit. We told the testers to take extra time to make sure they read it thoroughly. That was our decoy.

Once they read it, we asked them to tell us 3 key facts or events from the company story (About Us page). And everyone dutifully told us things that struck them about the story.

Then we gave them a series of other tasks we wanted them to complete on the website. We didn’t really care about these tasks. They were just part of the decoy. Again, our goal was to make them think we were measuring them on something totally different.

Lastly, we asked them to list the strengths and weaknesses of the company. Here’s where it gets interesting.

• Most couldn’t think of a weakness.
• Those that could, focused on other issues. (i.e. lack of diversity of investment strategies)
• All testers found the company to be honest and trustworthy.
• Several mentioned that they wanted to investigate the company more after the testing. Because they wanted to move their investments there.

 

In one case, we tried to talk the tester out of her decision. We highlighted the gambling background. But with every argument, she defended her choice. Finally, she said to be a good investor, you need to be a gambler.

We call that the irrational defense of a decision already made by the fast-brain. It’s the decision process that the split-brain testing revealed. That when the fast-brain makes a decision, and the slow-brain needs to explain it, it will generate answers on the fly. And they truly believe it was a rational choice.

The benefit for the client was we found a way to increase their trust. And we found a way to measure that.

Putting it to work

Thankfully Steve Jobs pushed through and persuaded the board of directors to run the ad. It changed Superbowl ads for ever, and it continues to build Apple’s brand 35 years later.

The marketing team behind New Coke must feel horrible. They tested 200,000 people and still made the wrong choice. That’s disheartening.

But you can have a different outcome. Quit asking your customers opinion questions. They don’t deliver reliable results. Change the way you do focus group testing.

Use testing decoys and measure action. Those will reveal the path forward.

[1] https://www.business2community.com/consumer-marketing/market-research-fail-new-coke-became-worst-flub-time-01256904