How Many Experiments Do I Need To Run?

A teacher said to her student, “Billy, if both of your parents were born in 1967, how old are they now?”
After a few moments, Billy answered, “It depends.”
“It depends on what?” she asked.
“It depends on whether you ask my father or my mother.”

It’s also like that with the number of experiments you need to run, in order to build a new product.

It depends.

On how much of a breakthrough you want your product to be.

Photographer: Rafael Pol | Source: Unsplash

Soviet inventor Genrich Altschuller created the “Theory of Inventive Problem Solving”, or TIPS. In the original Russian, it’s called TRIZ. TRIZ’s a massive topic in its own right, providing a systematic methodology for solving problems creatively.

Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.

Apparently even creativity can be systematized.

Anyway.

While Altschuller reads quite heavily in the translated English version, he shares nuggets of wisdom–which are very relevant for product entrepreneurs.

For example, he pored over about 40,000 patent filings, mining for insight and inspiration. It turned out, that there were clear patterns in how inventors approached problem solving. Depending on the type of challenge being addressed, the number of experiments the inventors did made the technological breakthrough even greater.

Number of experiments

This shows exactly what’s required if you want to create a breakthrough product. Most commercialized products which are “revolutionary” will require at least 10,000 well-thought out experiments.

You don’t know which of your experiments will reveal insight. If you aren’t doing thousands of them, then you ain’t gettin’ none.

About to start a greenfield project?

Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.

Contact Us

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

The takeaway here is clear. Everyone wants quick results. Sometimes they do happen. (I’m all for being optimistic.)

More often than not, though, you’ve got a lot of experiments to run. Systematically. In order to really get beyond conventional wisdom. If you want to discover a counter-intuitive truth. So get started already!

Create an explainer video for your complicated new product. Make sure your audience understands it, without being overwhelmed by technical details.

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

Taking this into the Lean Startup realm, before you even start your technical experiments like Altschuller, you can run marketing experiments.

Figure out what your market wants. What problems they have. What keeps them up at night.

Then, when you put your product creator hat on, you’ll make a product which has a chance at commercial success. That’s the bit the Soviets missed.

If you want to find out more about how to run these types of marketing experiments, I’ve got your back. My upcoming book Your First Startup Experiment goes into a lot more detail.

Read it.

Apply it.

Profit from it.

Why the Launch Tomorrow process actually works

Recently I showed up on the Predictable B2B Success podcast. Vinay interviewed me. He originally worked with Dan Norris as his content guy, and is now running a content repurposing agency.

Vinay Koshy

To be honest, Vinay did a great job pushing me for details with a healthy skepticism to what I was saying. This dynamic turned or conversation into a good interview, highlighting on the ins and outs of the Launch Tomorrow method as it now stands. And also how it's different from both Lean Startup and online direct marketing, when looking at growth.

The Predictable B2B Success podcast

We cover a lot of ground including:

  • Why the Hero Canvas exists
  • Why there is so much focus on the customer segment at the super early stage in the Launch Tomorrow method
  • Why you can't separate the product from marketing, especially when first starting out, and what connects them
  • How to prioritize in the super early stage when you are probably feeling most overwhelmed
  • Why founder-market fit seems implicitly obvious, yet often overlooked (including a case study of what that means)

Worth a listen regardless of whether you’ve just found Launch Tomorrow, or following it for a while. This podcast episode is good listen, as are many of the other episodes on Vinay’s podcast.

Why Founder Market Fit Matters…A Lot

Back in the day, I was looking into entering the weight loss market with a SaaS solution or coaching services. In theory, it sounded like a great idea. I based it on a “signature success” story of my own. Based on that success, I thought I could provide value, help others achieve the same. But I overlooked the importance of founder market fit.

What I didn’t consciously realize was that most of that actual market was very different from me. I was a young, somewhat nerdy guy, the type of guy who got excited about techie stuff and spreadsheets. And that’s how I achieved what I did in the context of weight loss.

Hmm…business opportunity?

At the time, you could easily buy paid Google advertising. It was a learning environment where I tried a lot of the techniques that are now part of the Launch Tomorrow method: driving traffic to landing pages, getting sign ups, interviewing customers and gathering surveys, and using it all to pre-test and build up a marketing message. Use spreadsheets and numbers to figure it out. After some desk research, I put up a survey and bought google ads on keywords related to weight loss.

But the weight loss market, the people who were actively spending money and trying to change the fact they were overweight, were women roughly twice my age. Even though being overweight equally affected both men and women across the age ranges, the _market_ was very different. The ones willing to spend money. After doing all this market research, I realized my survey responses mirrored market research reports. Roughly 90% of the market was female. Not 50-50, as I implicitly expected at first. Then I connected the dots that 89% of the survey responses were from women twice my age living in small towns.

Weight loss market breakdown

Given that was true, I was at a natural marketing disadvantage right from the beginning because I was harder to relate to:

Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.

  1. I’m not a doctor and I’m not planning on going to medical school just to look credible
  2. I’m not a woman and I’m not planning on becoming one for the purposes of this startup idea
  3. my approach of apps and spreadsheets wouldn’t have naturally appealed to this market anyway, so they were likely to struggle with my process and not have it solve the problem.

It would have been a hard sell, and also likely to be less effective than it was for me. I’d lost faith that I’d be able to achieve results sustainably for this market once I realized the nature of the market. These natural disadvantages I had in this market were ones that I couldn’t easily change about myself. Best to cut my losses and get out. Founder-market fit wasn’t there. In this case, it had marketing implications for me as a as a founder and being perceived as a relatable expert entering a market, even though I knew my process/product was effective. It was a great market, just not a great market for me to enter, at least as a solo founder.

Hence the importance of “founder-market fit”

Josh Kopelman, co-founder of First Round Capital, posits that:

6/ Most founders spend <5% of their time on idea [market] selection, yet I believe that “the pick” accounts for >50% of startup success/failure 7/ Observation #1) Many founders rush “the pick”. If you’re spending the next 5-10 yrs of your life doing something, pick your idea wisely.

Choosing the right market to pursue is one of the key decisions an early stage founding team makes. Even though there are a lot of considerations, fortunately you can iterate to the right combination for you and your team with systematic testing. And the founders themselves are an important part of that decision, which they often miss (as I did above).

The startup world bandies around the term “founder/market fit” (FMF). Founder/market fit: the founders have a deep understanding of the market they are entering. They have relevant knowledge, skills, and experience. Besides that, they have strengths or characteristics. Ultimately, these founder factors help them gain a natural competitive advantage. Build a moat. Theoretically, anyone can test a market test in the abstract or hire an agency to do it for them. But there are markets which a specific founding team will be more likely to succeed in, based on founder/market fit.

And as venture capitalist Chris Dixon says, founder/market fit can serve as a leading indicator “of whether a startup will achieve product/market fit”. Successful founders use their experience to prevent premature scaling. They find it easier to test a value hypothesis using customer discovery, because they have ready access to customers.

About to start a greenfield project?

Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.

Contact Us

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

There’s three major reasons why getting founder-market fit right makes a difference:

  1. Strongly tied to how founders expect to acquire customers
  2. Choosing a high growth market
  3. Getting the right founding team composition for your business model

Strongly tied to how the founders expect to acquire customers

Here’s why: founders often default to what they know. They expect to acquire customers from market segments they can reach easily, especially if short on time. And if a segment fits well, growth is easier to execute. If founders can build on pre-existing contacts and specialized knowledge, for example enterprise purchasing processes, this can be a major feather in their caps. At least compared to founders who are green. The marketing and sales knowledge in particular is important, particularly in relatively established markets:

Create an explainer video for your complicated new product. Make sure your audience understands it, without being overwhelmed by technical details.

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

Often new ventures can operate in the same domain as previous experience (i.e. founder-market fit), but the way customers are acquired is different. For example, elephant-hunting enterprise software sales leaders aren’t really a good fit as self-service hosted software founders, even if the core business software vertical is right in their wheelhouse. A killer paid online acquisition guru isn’t a good fit for a venture which requires fostering an open source community for adoption. And sometimes domain expertise can’t be relied on at all when markets are entirely new it’s much more challenging to have founder-market fit for a VR startup when the category didn’t meaningfully exist a few years ago. However, you can still have founder-go-to-market fit in an emerging or entirely new market because the methodology for customer engagement can rhyme with what founders have done in the past. (genuinevc.com)

Ditto on the consumer side. Someone with experience in a particular industry will sense patterns, how marketing works in that space. And also will have industry friends who can help, if needed.

You need not commit up front. Before you know anything. Which is also why it’s best not to obsess over the choice. There are techniques like Justin Wilcox’s now classic SPA scorecard to help you get out of the building and to start speaking with prospects. The customer discovery process will not only tell you a lot about the prospects but also about your own ability to acquire them. If you can’t acquire customers for interviews, how are you going to build a high growth business in that market segment?

High growth markets will buoy up the business

If you are entering a new business, there are several potential approaches. The most “long-term” approach to starting a business is to enter a market hockey-sticking into a high growth phase:

If you take this criterion of growing markets to heart, you significantly increase your overall chances for success. Many of your potential product ideas won’t pass this criterion. Filter them out at the beginning. You’re left the ideas which have better shot at high growth. You buy market share at a discount.

It also means the product is likely to be around for longer. Demand for a new product tends to be highest at the beginning of its life-cycle. If the demand is growing, it’s still early days for the product category.

Founder core competencies are surprisingly a strategic decision

In a small business single-owner scenario like my weight loss one, the founder’s strengths and weaknesses matter disproportionately for a given market. Yet tech startup investors like Paul Graham (@paulg) advise having multiple founders. Each with different relative strength. That strength can come from experience, background, personality, or some other source. This co-founder owns a function in that startup, where they are naturally strong. In addition to playing to cofounder strengths, this approach divides up the seemingly endless workload with accountability, both among themselves and also to the external world. Each relative strength maps to a “key activity”. The key activities box (on a @strategyzer business model canvas) defines how you deliver the value proposition:

The key activities box on your business model canvas

The basic principles is: one founder per key activity. Depending on exactly who you have in your founding team, you’re much better equipped to go after different markets.

For example, you’ve invented an IoT device for agriculture. It gathers data about weather and soil conditions. The obvious starting point is manufacturing and selling devices. But is that your best option? For example, you can give the device away, yet sell services and data. Use machine learning or statistics based on the data that the devices gather. If that’s the case, then you need data expertise in your founding team. The main technical person is ideally a machine learning and software specialist, as opposed to a hardware manufacturing expert.

In these scenarios, a market opportunity exists. You just need appropriate people, each with a different strength. One in each part of the business. To deliver. And ideally each expert trusts the decisions made in other areas. To move fast as a company. Your competitors operating in the same market will also be trying to achieve the same. Your ability to deliver is often a differentiator.

What about industry outsiders as successful founders?

For every founding team who bring a wealth of experience to a successful startup, there is a story of a founding team industry outsiders. They go on to raging success. They bring a new perspective, and as a result, they notice opportunities that incumbents take for granted. The sacred cows aren’t sacred to them. Often, they don’t know any better. Or they deliberately go contrarian.

For example, Laura Behrens Wu is the founder and CEO of Shippo, a logistics and shipping software platform for e-commerce merchants. Laura had:

Yet according to @bussgang, Wu has navigated Shippo to becoming one of the leading online shipping platforms in the world, with 80 employees and $30 million raised from top investors such as USV and Bessemer.

These startups are often the outliers in terms of growth, as they are systematic with testing and discover opportunities that everyone else misses. They don’t know industry best practices, so they set out to discover and form their own.

Or like the case of AirBnB. First they ignore you. Then they ridicule you. And then they attack you and want to burn you. And then they build monuments to you. This is a much harder and more expensive path, but also significantly more lucrative when you create a new product category—and proceed to lead it.

Summary

In short, pay close attention to the market you choose. This one decision can significantly affect the outcome of your startup journey. Fortunately, you can iterate and learn; however, many founders miss the strategic importance of this choice, particularly with respect to the composition of their founding team. It’s also a decision you need to need to commit to early on, so dithering too long can also have unsavory consequences for your business.

Essentially, either your founders need to be capable of generating rapid growth within a well established vertical, or be willing to create a new vertical. This is true from a marketing perspective, but also from an operational perspective. Both require specialized knowledge in many industries. Founder-market fit ultimately ensures you can deliver a meaningful experience for that chosen market.

Key Takeaways

Ah, You Got Me… (The 4M framework)

Ketchup, chips, and chlorine.

Ketchup, chips, and…

Take any one of them away, and it wouldn't be the same experience.

That was the smell of the local Drexelbrook, PA pool when I was growing up. Technically, that's called a lido, in British English.

Chlorine…

Do you remember the kids game Marco-Polo? Usually played in a pool, one kid shuts their eyes and yells "Marco". The rest of the cub-pack yell "Polo".

"Marco"
"Polo""Polo""Polo""Polo"
"Marco"
"Polo""Polo""Polo""Polo"
"Marco"
"Polo""Polo""Polo""Pol-ah you got me"

Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.

Based on sound and touch, "Marco" chases a "Polo". Once touched, that "Polo" becomes a "Marco".

Touching anyone who isn't playing doesn't do anything, though.

Sculptural bust of traveler Marco Polo.
Photographer: Egor Myznik | Source: Unsplash

Named after a 13th century explorer, who bumbled his way in the dark from Europe to China (the center of all civilization at the time), the children's game hooks into something primal. It keeps kids engaged for hours at a time.

Marketer and designer Nathan Barry has previously pointed out that this game also serves as an analogy for call-response.

Direct response, even. You only want to be chasing after the Polos, and avoiding everyone else in the pool.

That's my kind of marketing.

About to start a greenfield project?

Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.

Contact Us

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

Yep, it’s you I want…

The Marco-Polo analogy holds really well for what it's like to enter a new market. You know there are people interested. You want to catch them, yet avoid everyone else.

You just need to figure out how to reach the "Polos". Speak to them. Hook into what interests them.

Create an explainer video for your complicated new product. Make sure your audience understands it, without being overwhelmed by technical details.

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

There's three more twists, though, compared to shouting "Marco" in a pool:

  1. You can use any medium you want to reach them, not just yelling at the top of your voice: advertising, free content, phone calls, events, or anything else.
  2. You can optimize what you say, not just "Marco", to attract & identify the "Polos".
  3. You can be really smart about choosing the first "Polos" you want to pursue.

This is exactly the situation you're in, when launching a product. I call it medium-market-message match. The 4Ms. 😀

Choosing a channel and message and market can feel like choosing an elevator

Choosing media is an interesting game. Essentially, you're tapping into other people's existing audiences, and putting a message in front of them. Depending on how relevant that message is for that particular audience, more people will "convert". And then you can be smart about who you actually go after.

Pol-ah, you got me.

And the best way of testing these combinations?

Why, never thought you'd ask.

Landing Page MVPs.

There's a good book on this topic called Launch Tomorrow. Arguably, I'm not the best person to judge that, since I wrote it. But I'm right anyway–this time.

It goes into constructing constructing tests around the 4Ms. Put all of your ducks in a row.

Catch the "polos" who are keen to hear what you have to say.

Even better, build products which are popular with lots of "polos".