When my wife and I were considering leaving London and moving back to Warsaw, we felt a lot of cognitive dissonance. On one hand, most of our friends were sharing clickbaity headlines of despair. Depending who was sharing it, either the whole country was turning into a totalitarian state or it was in a state of moral apocalypse driven by the EU. On the other hand, when we visited grandparents during the summer at their log cabin, the lazy afternoons on the porch felt very cozy, safe, and relaxed. The only thing that was clear: articles shared on social media were exaggerating on both ends of the political spectrum.
The Netflix special "The Social Dilemma" has tried to popularize what most of the tech industry has been aware of for a decade. We life in a world of social media bubbles. Each of us are subject to massive confirmation bias. Confirmation bias refers to discounting any facts which are inconsistent with our current beliefs. So we only believe facts which are already consistent with our current beliefs, accumulating unfounded confidence in how right we are. Social media amplifies this dynamic. If our friends keep repeating our own views back to us, we radicalize in our opinions.
This dynamic is similar to that of cults. Once you are on the inside, any information sourced from outside the cult is discounted as unreliable. So despite being able to think for yourself rationally, you end up believing that staying in the cult is the only reasonable choice you have–even as they serve pink Kool-Aid to thousands of naked people in the desert.
So why am I writing about confirmation bias on an innovation blog…of all places? Because it’s the underlying problem behind the most common reason startups fail: building a product that nobody wants. And despite Lean Startup and similar reasonable approaches, 40% of new product failures from startups and corporates have been occurring for this reason since studies the 1970s.
I would even say that the real problem isn’t rational. We need to go deeper to understand why they haven’t reduced the frequency of this problem. The founder is so convinced of their vision, they avoid gathering feedback or they discount any feedback inconsistent with their vision. Even if it comes from prospective customers. Just like believing every headline you read on Facebook.
Bias and false beliefs
The nature of early stage innovation is that small changes in trajectory can have massive implications years into the future. To borrow an aeronautics analogy, this is similar to the proverbial autopilot mechanism on most planes. For most of a long haul flight, a plane is a few degrees off course. But as long as it constantly corrects and readjust for the right destination, it will get there.
If the autopilot didn't correct in the early stages, it could miss the mark by a massive margin. This is why disconfirming evidence is useful (necessary in my opinion) in this context. If the autopilot mechanism doesn't know you are two degrees of course when flying from Europe across the Atlantic, you'll end up in Florida when you expect to land at JFK airport in New York. Let's say the autopilot mechanism breaks and the pilots fall asleep once the plane reaches a cruising altitude. Some of the planes will arrive at JFK regardless. Most won't; early diversions off course (even small ones) will have the largest impact on where the plane "arrives".
Full disclosure time. For years, I have struggled to articulate exactly why experiments are so valuable, despite my enthusiastically foisting them on unsuspecting founders. The scientific process makes a lot of sense in an academic research context. But what exactly is it's value in business, other than being trendy in the Lean Startup crowd? Being super rigorous and adhering fully to the publishing standards of academic journals is impractical and usually a waste of resources for founders.
Beyond that there are many "shades of gray" of how the scientific method can be valuable, so it's difficult to isolate exactly how to simplify experimentation enough to make it valuable without making it impractical. For example, let's say a teenager's car emits a strange noise when driving. As he is an enthusiast, he has already watched a lot of YouTube videos about car engines. He opens the front lid. He confirms he can replicate the sound when the car stands in place. He loosens a screw and tries turning on the engine again, to see if it changed the problem. No. So he tries another small adjustment, and tries revving the engine. And so on. Essentially, he follows the scientific process–just without an experiment log and paperwork. The thought process of testing one thing at a time and holding everything else constant is the scientific process. Just outside of a lab.
In that context, how do you get rid of the "baggage" of rigor which scientists need to follow in order to be published while still gathering actionable insight? Starting a business and getting published in an academic journal have different requirements for experimentation.
Seeking out counterfactuals
More than anything else, the value in experiments lies in forcing you to seek out the counterfactual evidence as you take action. This grounds you better in reality as it is. First you have to open yourself to potentially being wrong, which is "cold shower" uncomfortable. Then you have to go and spend resources on trying to prove yourself wrong, to get to the truth of the situation–before you make major decisions like ones related to the existential risk of your startup idea. Both are inconvenient yet necessary prerequisites for innovation and change.
The basic idea with startup experiments is that they are a learning goal. They direct action, attention, and resources, in the same way as a regular goal. If you are right your achieve what you set out to do; as a side effect, you prove your assumption was correct. Or they give you an opportunity to learn something useful at the time of finishing the experiment. Ideally you are running experiments in areas that matter to your business. If you do disprove some aspect of your vision, it's just time to adjust your direction. For the "future you" this is the best possible outcome, i.e. you found out early, before you invested too much financially or emotionally in a product vision. In effect, you are borrowing pessimist's hat temporarily, in order to strengthen your conviction.
With experiments, you construct a positive feedback loop which sends you in the right direction…turning your product into a "heat seeking missile" that targets market segment "heat". If you design an experiment to prove your assumption (where most people start) you try gathering evidence around important assumptions you are making, to validate your hunch. If you design an experiment to disprove your idea, and it still passes, then you "earn" more confidence in the idea. In either case, it takes time and effort to run experiments (and that's to be expected, because they are goals).