When my wife and I were considering leaving London and moving back to Warsaw, we felt a lot of cognitive dissonance. On one hand, most of our friends were sharing clickbaity headlines of despair. Depending who was sharing it, either the whole country was turning into a totalitarian state or it was in a state of moral apocalypse driven by the EU. On the other hand, when we visited grandparents during the summer at their log cabin, the lazy afternoons on the porch felt very cozy, safe, and relaxed. The only thing that was clear: articles shared on social media were exaggerating on both ends of the political spectrum.

The Netflix special "The Social Dilemma" has tried to popularize what most of the tech industry has been aware of for a decade. We life in a world of social media bubbles. Each of us are subject to massive confirmation bias. Confirmation bias refers to discounting any facts which are inconsistent with our current beliefs. So we only believe facts which are already consistent with our current beliefs, accumulating unfounded confidence in how right we are. Social media amplifies this dynamic. If our friends keep repeating our own views back to us, we radicalize in our opinions.

This dynamic is similar to that of cults. Once you are on the inside, any information sourced from outside the cult is discounted as unreliable. So despite being able to think for yourself rationally, you end up believing that staying in the cult is the only reasonable choice you have–even as they serve pink Kool-Aid to thousands of naked people in the desert.

So why am I writing about confirmation bias on an innovation blog…of all places? Because it’s the underlying problem behind the most common reason startups fail: building a product that nobody wants. And despite Lean Startup and similar reasonable approaches, 40% of new product failures from startups and corporates have been occurring for this reason since studies the 1970s.

I would even say that the real problem isn’t rational. We need to go deeper to understand why they haven’t reduced the frequency of this problem. The founder is so convinced of their vision, they avoid gathering feedback or they discount any feedback inconsistent with their vision. Even if it comes from prospective customers. Just like believing every headline you read on Facebook.

Bias and false beliefs

The nature of early stage innovation is that small changes in trajectory can have massive implications years into the future. To borrow an aeronautics analogy, this is similar to the proverbial autopilot mechanism on most planes. For most of a long haul flight, a plane is a few degrees off course. But as long as it constantly corrects and readjust for the right destination, it will get there.

If the autopilot didn't correct in the early stages, it could miss the mark by a massive margin. This is why disconfirming evidence is useful (necessary in my opinion) in this context. If the autopilot mechanism doesn't know you are two degrees of course when flying from Europe across the Atlantic, you'll end up in Florida when you expect to land at JFK airport in New York. Let's say the autopilot mechanism breaks and the pilots fall asleep once the plane reaches a cruising altitude. Some of the planes will arrive at JFK regardless. Most won't; early diversions off course (even small ones) will have the largest impact on where the plane "arrives".

Full disclosure time. For years, I have struggled to articulate exactly why experiments are so valuable, despite my enthusiastically foisting them on unsuspecting  founders. The scientific process makes a lot of sense in an academic research context. But what exactly is it's value in business, other than being trendy in the Lean Startup crowd? Being super rigorous and adhering fully to the publishing standards of academic journals is impractical and usually a waste of resources for founders.

Beyond that there are many "shades of gray" of how the scientific method can be valuable, so it's difficult to isolate exactly how to simplify experimentation enough to make it valuable without making it impractical. For example, let's say a teenager's car emits a strange noise when driving. As he is an enthusiast, he has already watched a lot of YouTube videos about car engines. He opens the front lid. He confirms he can replicate the sound when the car stands in place. He loosens a screw and tries turning on the engine again, to see if it changed the problem. No. So he tries another small adjustment, and tries revving the engine. And so on. Essentially, he follows the scientific process–just without an experiment log and paperwork. The thought process of testing one thing at a time and holding everything else constant is the scientific process. Just outside of a lab.

In that context, how do you get rid of the "baggage" of rigor which scientists need to follow in order to be published while still gathering actionable insight? Starting a business and getting published in an academic journal have different requirements for experimentation.

Seeking out counterfactuals

More than anything else, the value in experiments lies in forcing you to seek out the counterfactual evidence as you take action. This grounds you better in reality as it is. First you have to open yourself to potentially being wrong, which is "cold shower" uncomfortable. Then you have to go and spend resources on trying to prove yourself wrong, to get to the truth of the situation–before you make major decisions like ones related to the existential risk of your startup idea. Both are inconvenient yet necessary prerequisites for innovation and change.

The basic idea with startup experiments is that they are a learning goal. They direct action, attention, and resources, in the same way as a regular goal. If you are right your achieve what you set out to do; as a side effect, you prove your assumption was correct. Or they give you an opportunity to learn something useful at the time of finishing the experiment. Ideally you are running experiments in areas that matter to your business. If you do disprove some aspect of your vision, it's just time to adjust your direction. For the "future you" this is the best possible outcome, i.e. you found out early, before you invested too much financially or emotionally in a product vision. In effect, you are borrowing pessimist's hat temporarily, in order to strengthen your conviction.

With experiments, you construct a positive feedback loop which sends you in the right direction…turning your product into a "heat seeking missile" that targets market segment "heat". If you design an experiment to prove your assumption (where most people start) you try gathering evidence around important assumptions you are making, to validate your hunch. If you design an experiment to disprove your idea, and it still passes, then you "earn" more confidence in the idea. In either case, it takes time and effort to run experiments (and that's to be expected, because they are goals).

Essentially the purpose of experiments is to systematically test your beliefs, ideally turn it into a habit, to identify opportunities or threats early. Ones which you would miss otherwise. Beliefs themselves influence what we notice and see. And if you are starting a completely new venture, your own beliefs can get in your way. They are a major source of "unknown unknowns" of two varieties:

1. being absolutely convinced about something that isn't actually true (common wisdom and best practices)

2. not noticing evidence to the contrary because it would undermine a bias that you have, like your social media confirmation bias or a sunk cost (that's the way we do things around here).

To be clear, you don't need to do this when starting any business, just a high risk business model. If you are starting a dentist's office, the overall risk is pretty low. The service is easy to explain. And the variables that will influence your success are relatively well known. There are off-the-shelf books, courses, and advisers which can walk you through all of the important decisions, because all of the variables are known.

But what if you don't even know what the important variables are? This is the scenario with most high tech businesses. You are subject to more risks, many which you don't even see. And being hard-headed and opinionated is more of a liability, especially in this early stage. Because you avoid gathering disconfirming evidence, to expose your own biases. And you don't course correct, like the airplane on autopilot.

Later implications for your company

A really important implication here is the emotional contagion of being a leader. As a founder or a member of a founding team, your beliefs form the basis of your company's culture. And therefore, if you don't install the rigor of systematically questioning your beliefs into the culture, then it won't be there. One of the commonalities of the major tech companies is that they continue to foster this type of open debate and feedback.

Alex Kantrowitz (@kantrowitz), author of Always Day One, says Facebook is probably the most famous for this. On an upcoming episode of the Align Remotely podcast, he recalled the case of when Mark Zuckerberg was pushing strongly for a web only app, which would also be used on mobile. This clearly simplified the (already massive) technical challenge for his company; however, a web-only interface had a sub-par user experience. A native approach from within the mobile operating systems gave developers more tools and components to orchestrate a great experience. When one of his technical czars came to Zuckerberg with feedback like this, Zuckerberg suggested that they do a prototype of how this will look, both within their company and technically. They assembled a team and got on with it. And a few months later, when they created a mobile prototype, Zuckerberg saw that his vision was wrong. The user experience was so much better when Facebook was an app (also it allowed for gathering a lot more data about each user, thus making advertising more targeted). With this disconfirming evidence, they continued to pursue Mark's overall goal but changed the approach.

Stepping back, this is a great example of how irrationally holding on to a false belief could have cost Zuckerberg his company. It’s almost ironic that the social media giant is such a good example how to deal with confirmation bias, given its role in making it so widespread.

Ultimately, incorrect assumptions can be both risky and expensive. So rooting them out, as quickly as you can, lies in your best interest, even though it can be unpleasant and feel counterproductive (at first). The same holds regardless of company size. It's true if you are an early stage founder or riding a bucking growth bronco like Zuckerberg was at the time.

Wait, but isn't this too negative of a mindset to achieve your goals?

Think the danger here is that you only do experiments. If you keep questioning everything, you never feel like you are making progress. If you only focus on finding counterfactuals, isn't that like visualizing failure then expecting it to happen? Not if you get reliable counterfactual evidence. A useful variant of the thought process (especially at small sample sizes) is: assume your are wrong. Gather evidence to prove prove yourself wrong. And then hopefully be happily surprised.

The point here is not to be right. It's to be successful, as quickly as you can. And do that you have to prove the proverbial naysayers wrong, by taking their concerns seriously.

In this context, I find it useful to think of risks as blockers which you address with experiments. If you identify the major risks, and run experiments, you will unblock yourself. If you have addressed all major risks around the product, then customers will be clamoring for your product or service and happy with it when they use it. Once you address major risks around growth, you will have a repeatable and scale-able way to access more of your existing customers in a large market. In both cases, execution should be simple and rote. You know exactly what to do, and you just need to get on with it to make money as a company.

Case Study

My "signature failure" was starting a website with video courses for financial professionals about Microsoft Excel in 2008. In today's terms, the idea was something like a super-focussed Udemy on that particular topic for that audience. After spending 11 months building it (and a lot of personal savings), I realized nobody wanted in my market segment actually wanted it. Their day-to-day problems and challenges were quite different, and couldn't be addressed with a =VLOOKUP() tutorial.

A few years later, as a retrospective test done out of curiosity, I set up a landing page offering a similar product to the same audience. I promoted it on social media and bought ads. Unlike many similar tests which I have run, I had minimal signups and interest. So if I had sourced counterfactual evidence for a few days, I could have saved myself from a painful, long running, and expensive failure. And moved on to find an idea actually worth pursuing.

Next week, I’ll be writing about an unlikely tool I used to build a habit of counterfactual thinking from way outside the startup world. If you’d like to be notified, grab my hero canvas, and you’ll automatically get an email update when that post comes out.

Key Takeaways

• Your biases and beliefs affect the assumptions you make when starting a new business
• Unproven assumptions can be expensive and/or risky, especially if you are trying something new
• Common knowledge and hidden biases can easily sidetrack you through "unknown unknowns" or not realizing the full implications of "known unknowns"
• By using learning goals to seek out counterfactual evidence actively, you establish a solid baseline for building a groundbreaking new business