Here’s Marc Andreesen’s take in the original article which coined the term “product-market fit”:
The only thing that matters is getting to product/market fit.Product/market fit means being in a good market with a product that can satisfy that market. You can always feel when product/market fit isn’t happening. The customers aren’t quite getting value out of the product, word of mouth isn’t spreading, usage isn’t growing that fast, press reviews are kind of “blah”, the sales cycle takes too long, and lots of deals never close.And you can always feel product/market fit when it’s happening. The customers are buying the product just as fast as you can make it — or usage is growing just as fast as you can add more servers. Money from customers is piling up in your company checking account. You’re hiring sales and customer support staff as fast as you can. Reporters are calling because they’ve heard about your hot new thing and they want to talk to you about it. You start getting entrepreneur of the year awards from Harvard Business School. Investment bankers are staking out your house. You could eat free for a year at Buck’s.Lots of startups fail before product/market fit ever happens.My contention, in fact, is that they fail because they never get to product/market fit.
He goes on to say that you can pretty much ignore everything else until you have product market fit, because that’s the only thing that matters to the business.
up and to the right: how we like our curves
Once you genuinely have product market fit, you stop working on it. So there is no balance.
Or to be even more precise: first get product market fit. Then work on scaling. Any time you spend on trying to scale a product that doesn’t have product market fit is kind of a waste of time.
The style of working also changes once you begin to scale. For example, Brian Chesky of AirBnb says:
At scale, you have to learn how to move from intuition to data. When you are first starting a company — data is not the most important thing. Pre-product market fit — data wasn’t important for us — it was much more about the person-to person interactions with customers.
Be ready for a totally different challenge once you begin to scale. But it doesn’t make sense to prepare for it or start early, because you might be scaling a product turd. Or not have the right team. Just because it might be tempting to jump to scaling, heading to the quantitative stage before you have exactly the right product for your market is just premature optimization. And a waste of resources—(your) time.
Icarus and his father Daedalus, the architect of the Labirynth, attempt to escape by air from Crete. Daedalus had become a political refugee. He had helped Theseus, an enemy, break into the Labirynth to kill the Minotaur.
Icarus falls, assuming the plan so good he doesn’t need to watch himself implement it
Daedalus then constructed wings from feathers and wax. Icarus’ father warns him first of complacency and then of hubris. Daedalus asked that Icarus fly neither too low nor too high. The sea’s dampness could clog his wings. The sun’s heat could melt the wax. Icarus ultimately ignored his father’s latter instruction. He flew too close to the sun. When the wax melted, Icarus tumbled out of the sky, fell into the sea, and drowned.
How is this relevant when launching new products?
Entrepreneurs have a strong sense of vision, particularly when creating something new and magical like the wings in the legend above. That’s why we love them. Working alone or as a small team, they execute. They see something no one else does, and they pursue it relentlessly, until it happens. In one form or another.
Conversely, corporate innovators often have to plan a lot up front, to align across multiple departments and stakeholders. Which means that detailed planning results from significant negotiations. And that any change often requires re-negotiation. But there is a great deal of vision and planning required to create something out of nothing.
Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.
These two scenarios have more in common with one another. Vision and planning is crucial, but having too much of an attachment to any plan is counterproductive.
Believing too much in the initial plan is clingy (and can feel unproductive)
For years, I was curious about whether there was any data to back that up. If you want to create a successful product, usually you want high growth as an ultimate metric of success. Recently, I found that data point.
In The Origin and Evolution of New Businesses, professor Amar Bhide of Harvard interviewed the founders of an entire year’s-worth of Inc 500 companies in the late 1980s, a ranking of high growth startups. This is independent of sectors and also not influenced by any of the more recent lean startup/small batch thinking. At the time, most startups needed to raise a significant amount of money to launch anything new. From a funding & risk perspective, the success criteria were close to that of corporate product development today.
Bhide reported that approximately 2/3 of the founders he spoke to pivoted away from their original concept:
More than one third of the Inc 500 founders we interviewed significantly altered their initial concepts, and another third reported moderate changes.
The majority of the founders shifted away from their original vision. Was it out of necessity? Or is being both willing and able to adapt is actually a requirement for new product success. One third of the founders were lucky to have guessed exactly what their customers wanted. But most needed to adapt and pivot to the best possible business model.
Plans are worthless but planning is everything –Eisenhower
In other words, the most helpful mindset & approach is to:
assume that you will be wrong about something when launching a new product
make sure you have the option to pivot
often, this happens once you collect data that challenges your assumptions…so collect data first!
This is how innovators work. This is also how “search” for a new business model differs from executing a known business model and aiming for efficiencies only.
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
This post is part of my “7 Lies of Innovation” email course. Click here to sign up if you’d like to know when the common wisdom around innovation doesn’t actually hold up in the data.
Richard Branson, the billionaire adventurer, has a complicated relationship with risk. On the one hand, he loves trying outdoorsy stunts and attempts to break world records.
Branson as an expert at managing the risk of getting punched in the face professionally
For example, he managed to cross the Atlantic, Pacific, and then circumnavigated the globe in a balloon. He even succeeds at doing so, despite the occasional crash:
On the other hand, he’s brilliant at managing his risk professionally, when starting new ventures. For example, thirty five years ago, he was stuck for hours on Peurto Rico. He wanted to fly to the British Virgin Isles to join his girlfriend, and came up with the following scheme:
Branson walked to the back of the plane and asked for a chalkboard and a writing implement. He figured out how much it would cost to charter a plane to BVI — and how much it would cost each passenger if the expense were pooled. And then he went for it, walking up and down the aisles of the grounded airplane selling tickets. On the chalkboard he wrote, “$39 one way to BVI.”
In one move, he confirmed that there was demand for an alternative. So he charted it. That got his appetite wet. Piggy-backing on his record store chain, Branson went on to sell plane tickets. He entered into a conditional lease of a plane from Boeing, where he insisted on having a one year break clause, to be able to return the plane if necessary. He aimed to fly it between the UK and the Falkland Iles. So Virgin Atlantic was born.
Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.
In short, he made a couple of moves which significantly limited his downside while giving him the possibility of evaluating whether it was worth soldiering on.
With new products, your primary goal is to manage your risks
Denying the existence of risks only makes you more susceptible to them. Particularly when you are faced with a high risk of failure. Like 50-90%, depending on the study you look at. No joke.
“Jed’s dead. Dead dead.”
It’s really important to note that this is not about being pessimistic. Or questioning your vision. This is about figuring out your vision’s fastest possible path from your head and into reality.
The “magic pixie dust” resides in prioritizing what you test. By testing your riskiest assumptions first, you reduce the chance you go belly up. Regardless of where you are in your new product’s journey, you will always face unexplored or unmitigated risks.
Yet, identifying this riskiest assumption is hard. The key factor to consider when managing your resources (including your time) is risk. All of that requires you to have a solid grasp on what your biggest risk actually is at that moment. Which isn’t always easy, particularly if you are doing something new.
What makes this particularly hard: it’s likely there are some “unknown unknowns”. So even if you do deliberately try to identify problem areas, you’re likely to miss something.
About to start a greenfield project?
Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.
Essentially, the idea is that you want to generate a large number of ideas about what’s risky. And then prioritize to identify what is riskiest (and were you need to start).
Ways to get inputs on what’s riskiest:
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
This is hard to describe. Sometimes you just have a gut feel. Pay attention to it. And run with it. Essentially, you’re looking for what feels like it could have really big & bad implications. And it is likely to be a problem.
Personally, I feel this with tension in my gut. So it’s just a question of tuning into that sufficiently, especially before making bigger decisions. And that’s often good enough. But it has taken some effort to be aware of this. Moreover, in a larger company, you may still need to convince others and create alignment, and “I feel it in my gut” is not a strong enough argument. Even if it’s true.
2. Who is the customer and does the customer actually have problem you think?
This is a great insight I got from Lean Startup Machine specifically. Running around London like a headless chicken with some friends, trying to interview young parents. (admittedly before I was one myself). We had a product idea which we though would be attractive for this group. We made some guesses about their biggest challenges. And then we hit the streets and looked for anyone pushing a buggy/pram.
This was the net result:
hitting a low point somewhere during 67 customer interviews in one weekend
After a few failed hypotheses, we had to acknowledge that we actually had no idea what parents of kids 0-2 years old struggled with. And while there were some patterns in what we were hearing, it was clear that it was still too wide of a market to have clear patterns. The tool on the wall helped us systematize testing:
who is the customer?
what is their biggest problem or challenge?
would the solution be an appropriate solution to that problem?
In the end, we had to acknowledge that we the pie chart we were drawing had too many possible answers.
survey of product managers
So we weren’t going small enough to have a clear pie chart, with a significant share of answers being the same:
looking for well ordered market segments like this
In other words, we needed a consistent easily identifiable & reachable group of people who share a particular challenge. That is a good market segment on which you can build a business.
This is almost always true. And often overlooked up front. Without it, you’ll have a hard time building something that will be relevant for enough people to make a dent. At least in a B2C business, where you need a lot of sales to get started.
In a B2B business, that breadth is less important, as much as depth. You need to be certain that whatever you do build and launch fully addresses the customer’s problem. And has positive and measureable side effects around:
increasing revenues
reducing costs
lowering risks
improving their customers’ experience.
At a high level, most businesses will only pay for those outcomes. But each industry and business is a bit different, so you will need to dig into the details of how to achieve it. But the starting point is really deeply understanding the problem from your prospects’ eyes.
3. Analogs
In short, ask people having similar success in different areas or industries doing things similar to what you want to achieve. This was an insight from Getting to Plan B by Komisar and Mullins.
For example, let’s say you discover a really powerful yeast which helps give red wine a unique and attractive flavor. Then you buy wine in bulk, add your magic dust, and then try to resell it to large supermarket chains.
Coming up with the yeast was the hard part. To figure out how to sell the supermarkets, you can probably speak to any salespeople who also sell to supermarkets in your country. It doesn’t have to be other wine merchants or wholesalers. You will learn a lot from a few conversations with people like that, enough to get your head around how this sales cycle works. So even though you don’t have in depth experience doing it yet, you’ll have a clear picture of how to proceed. In this case, you are using an analog to discover any hidden assumptions that could derail the use of direct sales as a distribution channel.
Channel testing example: selling superior wine into supermarkets
This is a good technique to turn “unknown unknowns” to “known unknowns”. Possibly even “known knowns”. Even though it’s less likely to have a company killer impact, it is still possible that you are making assumptions about that initial sales process which could derail you.
4. Antilogs (or Picnic in the Graveyard)
Some industries, especially software and internet, have a lot of public information about startups that have tried something. And failed. There are lots of writeups by previous founders of why they think their startup failed, such as:
If you are thinking of building something similar, you are just at the beginning. Those founders had been through years of trying to get a similar idea off the ground. So it’s quite likely you will learn something from them, if you can read about their failed attempts. Or even better, call them up and speak with them directly.
The idea here is to formulate a good reason why you think you will be successful this time around. It might be a different feature or market. It might be market timing. It might be any number of reasons. Because if you can’t articulate it, it questions whether it’s an idea worth pursuing. Not to mention, it will be hard to convince investors, too.
5. Risk Scorecard
This is kind of the left-brained alternative to #1 above. Basically, you can create a spreadsheet scorecard, where you:
list all of the risks
assign each a weighting from 1 to 5 for both likelihood and impact
multiply the weights together
reverse sort by the highest number going down
The outcome is that you have 1 specific risk you need to address and start focusing your experimentation on.
Here is a more detailed writeup of how to do this. Or here.
6. Pre-Mortem
Imagine your business failed 4 years into the future. You meet with your cofounders over beers or tea. As your tears flow, and drip on your Business Model Canvas (BMC), you discuss what you wish you would have known up front.
Which is the likeliest to have caused the failure?
Write down all possible additional reasons you come up with onto post-its.
After a general session, you can also think through any assumptions in each box of the BMC which might be relevant for this particular business.
Am I going after the right customer–for me or my company? Is it a strategic fit?
Can I reach my customer cost-effectively?
Is this customer type willing to do anything about the problem? Because if they aren’t, they aren’t likely to spend any money.
So for example, it’s a good idea to build a business around a group of people you can easily relate to and who you like. I tried building a business in the weight loss industry despite not having a medical background or being a middle-aged woman. So while I did have a signature success story in this context, my marketing would always be less effective than if either of those requirements were fulfilled. And I wasn’t keen on going back to school to become a doctor first.
All of these do matter in certain contexts. And they’re probably good “spot check” questions anyway, before you proceed with a specific idea.
8. Talk to industry experts
Once I get outside of the internet and software worlds, there is a lot I don’t know. Lately, I’ve been working with AgriTech and MedTech startups, for example. And while I know the concepts behind innovation, I honestly don’t know have that much experience in how they apply to those sectors.
For example, agriculture has a “crossing the chasm” like curve. However, it’s heavily skewed to the right. So basically, there are many more people in the late majority or even the laggards. In other words it takes a lot longer for a technology to diffuse. But when it does, it’s everywhere.
negatively skewed tech adoption curve
Anything related to medicine has to go through regulatory approval. In this case, you can’t just pooh pooh the regulators like various marketplace startups did (Uber, AirBnB). People’s lives and health are potentially at stake. In practice, for founder this means it takes a decade before you can go to market. Which means two things:
you need a lot more money up front
you have less attempts at building a business during a founder’s lifetime
In short, these industry specific details significantly change how you’d look to innovate in these areas. And I learned both of these points from experts with decades of expertise in the industries (thanks Huw and Anthony).
From a risk management perspective, expert’s ideas are simply ideas that require testing. You don’t want to ask experts for their opinions and treat it as gospel either. As that would mean you replicate their biases. Gather your own data and proof. But at least you know what to pay attention to.
9. Red team, Blue team
Finally, to stress test your business model, you can use this approach that the British military came up with. This is a strategic exercise, to try to identify any holes or oversights in a plan. The red team tries to penetrate a well designed defense plan.
Essentially nominate a few well-meaning friends.
As them to “red team” your current business model.
Looking together at the current BMC together with your “blue team” consisting of the co-founders or delivery team, ask the “red team” to identify any major assumptions you might have missed. Holes that need to be covered. Things which could easily go wrong and derail your product launch.
There was a good example of this in the HBO serial Newsroom. The news team was working on a high stakes story. Certain people were intentionally kept in the dark about the big scoop. When the team producing the story was ready to go live, they assembled the people who had been intentionally left out to form a Red Team. The Red Team’s job was to poke holes in the story, as the paper was risking its reputation by going live with this story. People kept in the dark are more objective. They aren’t invested in the story, so they see it from a different perspective.
What you are after
Basically the goal is to uncover as many assumptions as possible. And to prioritize them. So you have 1 specific assumption to start testing. This focuses your experimentation on company killers and existential risk, the most meaningful place to start managing risk. This is, in fact, what successful founders like Branson excel at–like when he put up a cardboard sign and started confirming there was enough of a market to create a challenger airline.
If you can take smart risks while being better at managing them, then you are much more likely to do better over the long run. Any particular risk might still go sour. But this ability to manage risk will ultimately define your career as a startup founder and business builder.
“First they ignore you. Then they ridicule you. And then they attack you and want to burn you. And then they build monuments to you.” Nicholas Klein commenting on a strike at the Amalgamated Clothing Workers of America conference in 1914
This quote is often misattributed to Ghandi. In fact, there was a festering conflict between thousands of garment workers and the clothing industry. The strike action pitted workers against management and against Chicago police on horseback. It also exposed divisions in the union. The organization did not support its unskilled members. This culminated in striker deaths by gunfire, wounding, and mobbing of a 10 year old over a period of a few years. The industry bosses made a risky assumption. They assumed their employees will accept difficult working and financial conditions.
Also, they assumed that employees would remain productive under these conditions indefinitely. In practice, the workforce was ignored and ultimately attacked. Angry, the employees fought back. Once the dust settled, a social shift occurred. Employees earned the right to fight for reasonable conditions. In this case, the change went through these four phases, as Klein observed in his quote.
And it’s a pattern for social change which has been followed by tech startup outliers. If you think about startup unicorns from the last three decades, many of them have undergone a similar process . If you look at the ex-China unicorns since the dot com crash, most of them came about by following that pattern. A pattern that would seem counter-intuitive to the average person in 2001. When they first started out, the very idea in its basic form seemed crazy. “Access your files from anywhere” by Dropbox. Send cyberpayments via PalmPilots for PayPal. And so on. Many of these startup ideas are based on a social or behavioral premise. One that would have been hard to believe for the average person at the time. As a result, there is an implicit social conflict many of these companies had to overcome. This is in addition to all of the other business and technical challenges they had. At the heart of this conflict, you’ll often see critical assumptions about customer behavior.
Let’s look at AirBnB, to see why risky assumptions matter in practice
AirBnB is a two-sided marketplace. They match landlords and homeowners with spare capacity to short term travelers. In this case, there is at least one risky assumption for each side.
Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.
The travellers
The homeowners or landlords.
“Air” stands for air mattress. In the very early days, the founders monetized their spare capacity using an air mattress.
You should have imagined when I called home, and told my mom about this idea. She said, “OK, so you built a website so that strangers could sleep in your home, because you don’t have enough money for rent? I guess you don’t have that job with health insurance anymore.” I said, “No, I don’t, but that’s not why I’m doing it.” –Brian Chesky
Using a paraphrase of his mother’s words, Brian articulated their risky assumption about landlords. Well pretty close. The idea of letting strangers sleep in your home was quite counter-intuitive back then. Particularly given that you had very little to go on. And that you needed to trust you’d be safe.
Proving the risky demand side assumption (as they were ignored)
At the time, the website looked approximately like this:
source: airbnb.com, waybackmachine.com
They wanted to prove that consumer travelers would be interested in paying for such an arrangement. Just because the founders were keen to earn a few extra bucks at conference, it didn’t mean they would be able to find a consistent stream of prospects. Ones who would be willing to not stay at a hotel. According to Brian Chesky, a co-founder and CEO of AirBnB:
We look on the hotel section, and all the hotels they were recommending are sold out. It’s like the Marriott’s sold out, the Hyatt’s sold out, sold out, sold out, sold out. Joe and I, we look at each other, and we say, “Well, why don’t we just create a bed and breakfast for this conference? That seems like it makes sense. We’ve got all of this extra space.” We’ve got a living room, a kitchen, two bedrooms, and no furniture. There was no furniture in the apartment. A bed and breakfast without a bed, it’s like a floor and breakfast, and we couldn’t really afford breakfast, so it wasn’t really super romantic. I said, “Joe, this is a problem. We don’t have any beds.” Joe said, “Don’t worry. I just went camping.” He brought some air beds out of his closet. I have no idea why he had all these air beds. He said he went camping. We pulled these three air beds out of a closet. We inflated three air beds, and we called it the “Air Bed and Breakfast.” We eventually cooked people Pop‑Tarts, yeah. They were really, really good Pop‑Tarts.
Even if these same tourists travel under “air mattress” conditions when visiting their own family, would they be willing to pay for the right location? (but not necessarily a hotel). This also turned out to be true.
About to start a greenfield project?
Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.
To my surprise, three people wanted to stay with me. They broke every one of my assumptions. The first was a woman, a 35‑year‑old woman that wanted to stay in our apartment. If you knew anything about me growing up, you would be very surprised that a woman would want to stay in my apartment. That was the first thing that totally surprised me, or surprised everyone who knew me. The second thing that surprised me was a 45‑year‑old father of five from Utah ‑‑ he was Mormon ‑‑ came, and he wanted to sleep on an air mattress in our kitchen floor. The third person that wanted to stay with us was from India. At this point, I’m like, “This is like a United Nations. This is a wide range of people. What is going on here? Why do they all want to stay with me?”
In other words, despite all of the reasons it could go wrong, the end consumers were fine with it. With the idea of renting someone’s spare room. And sleeping on a mattress and eating Pop Tarts for breakfast. Regardless of what they said their preferences were, clearly they were ok to act as if it was ok. Because they wanted the freedom to be at the conference or in San Francisco, despite limited hotel space.
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
For the supply side, the team needed to confirm that they could acquire landlords. But also that landlords would be interested in following the model the founders pioneered. An extra income on the side might not entice enough landlords to make it all work. AirBnb found conferences to be a clear case of insufficient supply. So they continued down that route. When they launched the company “officially” at SXSW in 2008, they made 2 bookings. But then when they managed to get blogger coverage around the Democratic National Convention in 2008, they made 80 bookings. (official company history). That was enough to confirm that–under the right conditions–people were happy to rent out extra space. Yet the marketing was still a struggle:
And we don’t blow up. No one wants to use the site after the DNC. Most people think we’re crazy. My mom thinks I’m out of my mind. All of our friends later tell us they thought we were crazy.
So with risky assumptions on both sides of the market validated, it made sense for the team to shift gears. To start investing time and effort into the product.
Now imagine if either the demand or supply side of those assumptions had turned out false. For example that homeowners didn’t want to do short term lets. And to be fair, most people in 2001 would probably have felt awkward about it. The team could have built the product, yet they would have found it next to impossible to grow it. Because they wouldn’t be able to offer tourists anything. And vice-versa.
Building something nobody wants is one of the most common reasons for product failure. If not the single biggest reason (depending on which survey result you look at). At the time, it would have been somewhat hard to believe. Part of commercializing the product is convincing everyone that counter-intuitive beliefs are true.
Then when estimating market size while raising funding, they were ridiculed some more…
For example, this comes up when raising funding. Initially, the founders thought the breakfasts were the draw. But breakfasts were a smaller market than beds.
How to you get serious investors to believe that the market size for “people willing to sleep on air mattresses” is a multi-billion dollar market? Or that it will be in 10 years, even if it isn’t that big now. Chesky quips, “Suddenly, the market size was huge, except we were still selling air beds. Like, “We’re going to make $30 billion with these air beds. Everyone’s going to live on air beds.” This is a classic conundrum for these wonky startups which navigate a social change cycle as they grow. How do you estimate the future market size of a market that doesn’t exist yet, but may end up being huge.
Then AirBnB were attacked as they drew attention scaling up
For a long time, hotels claimed they were serving a different segment than AirBnb. They didn’t need to acknowledge the presence of the rapidly scaling startup. AirBnB maintained close to 100% growth annually for 10 years. This growth came from having proven the risky assumptions before everyone else. Then built a product and a platform to capitalize on that insight. This ensured they were relatively unique.
Suddenly, hotels woke up to the fact that this startup is here to stay. And they attacked, primarily taking a regulatory angle, according to the Boston Hospitality Review:
American Hotel & Lodging Association (AH&LA) attacked Airbnb by sponsoring research to demonstrate its negative impacts on the economy and lobbying governments to impose taxes and regulations on homesharing. The association is arguing for a level playing field between homesharing and hotels.
They were implictly trying to frame AirBnb as running illegal hotels, and therefore as a company that was ignoring regulation.
Finally, they built monuments to AirBnB.
Now let’s zoom back to AirBnB today. They have about 6% market share of the hotel industry. Their potential capacity totally flexible based on demand.
This is due to the inherent flexibility of their two sided marketplace. A marketplace freed them from needing to own hotel buildings. Among many other successes, they’ve normalized what seemed like risky behaviors. These are ok because there are thousands, millions of people already using it. There are strictly defined parameters for this exchange. And it’s a pleasant alternative to a hotel. For example, families want to cook healthy for themselves, so a hotel can feel limiting.
Hotels accepted that AirBnB is here to stay. They try to get in onthe action. They have bought up, partnered with, or tried to create direct competitors:
The hotel industry has acknowledged the longer secular trend that airbnb was riding. And started cashing in on it. AirBnB’s valuations are comparable to Marriott’s.
But most importantly, there is an aikido like paradox. Because AirBnB capitalized on these risky assumptions, these same assumptions now shield them from disruption.
They have finely optimized systems which are tuned to this. Eating their lunch requires a lot more resources.
Key Takeaways
Even though many of the major breakout successes have this kind of social and behavioral change trajectory, most won’t. Assumptions about people, particularly those in the target audience, are critical. But it’s fair to say that:
every startup will be making assumptions, before they know exactly how their business will work.
Thinking through and deliberately checking any big assumptions, especially ones related to human behavior, you’re making is the most level-headed way to begin a startup journey.
“One thing I learned is big ideas sound stupid in the beginning. I’ve always heard that if your idea is any good there is no problem with sharing it because people will dismiss it. ” Chesky
Because it’s highly conceptual, software development is notoriously difficult to manage with data:
On one hand, it’s very clear when something that’s been built–works. And when it doesn’t. And even though it’s digital, it follows a logic almost as objective as those of physics: gravity, thermodynamics, etc. It will be painstakingly obvious to pretty much anyone with a pulse that something’s off.
On the other hand, building software is a creative process. There are many ways to approach building a piece of code. At a high level, it’s about solving a business problem or addressing a user need. But when you get into the details, there can be big variations based on what’s easy to make, what’s possible, and what’s attractive to the user.
In an effort to get a bit more quantitative, I started peering into our source code control system. For each person on the team, I noted down the number of commits, or changes to the code, they made over a monthly period.
I wasn’t really looking to create some kind of monitoring system. I just wanted to identify patterns in the work being completed. And to find anything that’s obviously wrong. It turned out that a few developers were averaging one small change per week, whereas the rest were managing to submit an equivalent number of changes over the course a day. So it kicked off some useful discussions.
But that said, if this became the only metric to evaluate developer performance, it would ultimately distort overall productivity. I think the fastest way to increase productivity is not to make the individual contributors as productive as possible, but to make sure the team is productive. And commits/developer would result in a local optimization, not the entire system. In addition to commits, factors like how quickly and how well code is reviewed matter. How quickly it’s tested after it’s done also matters a lot. And without this data, it would be hard to truly operationalize anything. Because it would probably skew behavior in the wrong direction.
Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.
At the moment, in order to address this problem, we’re implementing gitprime at the client site. It sounds good on paper, where we’ll have greater visibility into team dynamics. And also systemic metrics based on what’s going on in the code base. If the primary output we need is working software, then metrics around how this arises are a high leverage activity. Will report back when I have a clearer view of how it helped (or didn’t).
Here are a series of potential traps I’ve fallen into around choosing operating metrics in the past, when looking at managing software teams. This is for my own benefit and hopefully you’ll also find it useful:
Hard to observe/measure
If it’s possible to generate a metric, but it takes a lot of effort to generate the number, then it’s not a good metric to use. Or alternatively, if you can only observe the metric on a monthly basis, then it’s useful strategically, but not operationally. Because the team won’t have feedback loops to guide their efforts.
If you are using a manual process, the gold standard for this is what I found in the EOS approach:
source of data given by link which clearly defines how a metric is “calculated”
Have a spreadsheet with a number of metrics which are recorded on a weekly basis. See above for a google docs example. For each of those metrics, have it documented exactly how that metric is generated. It should be so clear and easy, that you could have an intern or receptionist follow the instructions. In my case, since I was mostly gathering stats around my writing, this included URLs/links to specific reports in google analytics. And then describing which boxes needed to be transcribed into the operating spreadsheet.
This was also why I started just aggregating commit data manually from git. Being forced to look at each team member’s profile on a regular basis meant that I became aware of what was going on. And the spreadsheet gave me a cross team view. Which I updated weekly, so it worked over time.
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
Focusing on efficient output while losing sight of outcome
This is a really common one in the context of waterfall project management. The three underlying variables that are optimized there are:
% utilization: how much time each person spends working
% completion: a (usually highly subjective) estimate of how much of the work is done
how this relates to predetermined dates that were agreed at the beginning of the project
While in theory all of this sounds like a great idea, these metrics are devoid of a measure of output and more importantly outcomes. Don Reinertsen ( @DReinertsen) has a concise summary of why this is madness: “In product development, our problem is virtually never motionless engineers. It is almost always motionless work products.” All of the above metrics focus on what the people are doing or not doing. Not on whether the product is getting built.
If you are interested in efficiency, motionless work products matter more than motionless engineers.
Also notice how all of the above are true of projects in general, but there is nothing business or industry specific in them. According to these numbers, whether or not the project finished on time is what matters. So if that is what matters, that is what should be tracked, not efficiency. Because efficiency can easily dominate everyone’s attention, so that you lose sight of what you’re trying to accomplish.
Working with both larger companies and entrepreneurs, I suspect founders and startups get this intuitively. But beyond a certain size, the bigger companies get so obsessed with efficiency, that they end up spending very little time talking about effectiveness and strategy. So everyone just focuses on efficiency.
Independent of customer needs
This is one of the common traps I tried to address with the Hero Canvas. For established companies, there is an increasing internal focus over time. A lot happens, but customers don’t see it. And to be fair, they might not want to see it all anyway. 🙂 Any work which doesn’t contribute directly to what a customer or prospect might want is “waste”. Some of this waste is necessary, to produce the intended outcome. But at least it’s deliberately added.
hero canvas
For example, in a larger company, there are few organizational onion layers between the product teams and customers. There can be lots of reasons for this. One of the most common ones are functional silos. Because sales and CRMs own the customer relationships, they are unwilling to risk sharing access to anyone else. Conversely, the technical staff might prefer to deal with technology, and to let the sales and marketing guys deal with the messiness of people & relationships.
As a result, there is a disconnect between the market facing side of the company and the technology facing side. And lots of un-sellable features are developed. Or the timing is off, and the sales teams want a faster time to market. To some extent, a well curated roadmap can help alleviate this problem. But even then, the point of the roadmap is not the roadmap. It’s the conversations and openness that lead up to creating it. The roadmap itself is just a side effect.
Metrics as simple as nr of customers affected or some kind of measurement how they are affected would be really useful to stay on track with any new products or changes to existing products.
Multi-colinearity, e.g. spending with a project calendar
Multicolinearity is a pesky attribute of many complex systems. Basically, this means that a number of variables or metrics you think are independent of one another actual vary together. In particular, this is most important for the variables that help define what the output of the system is.
The most famous example in financial markets relates to what happens during market crashes. When the market falls like a knife, all of the variables you were trying to use to figure out where the market will be…fall together. So suddenly all of your fancy math and statistics stop being useful. Right when you need it to work the most.
On a project level, there are a lot of metrics that just don’t change much. They simply go up the same amount, month by month. For example: total development cost seems like a good metric to track. Yet as long as the team composition stays stable over the life of the project, you’ll have a pretty good predictor of what your product development cost will be. Each month will cost you a certain run rate. Beyond that, spending time and effort, e.g. on a monthly basis, analyzing has much less value.
Because project budget is co-linear to the calendar. And to percent of original plan completed. If any one of these variables (total development cost, percent scope completed, earned value) is 63% complete, all of them will be. So you can just track one of them and know it’s roughly similar for all of them. Or even better, make sure your metrics are really related to something that matters.
source: @clcondron
Or as Chris Condron (@clcondron ) put it recently, you’re: “measuring progress based on how much gas you have left in the tank.” It’s better to measure progress based on what’s happening in the near or immediate future. Not based on a view you took months earlier when you knew very little about what needed to be done. Which leads me to the next observation…
Budgets and earned value are up-front guesses, which assumes nothing changes over the life of a project
In anything high tech related, most of the assumptions you make at the beginning of a project will be a guess. And many don’t matter, but a few assumptions might matter a lot. This includes both business, operating, and technical assumptions. And most financial metrics are just not useful, when you have such low confidence in them in the first place.
Given that’s the case, you shouldn’t be surprised that 35-50% more effort or scope is added to projects on average, relative to the best efforts of the product team to define what needs to be done. This is a really tricky one. Because you’re knee-deep in the “fuzzy” size of innovation, you don’t really know exactly how to build the product. You want the team to have enough flexibility to explore the problem domain. At the same time, if you don’t have a clear scope, you QA team won’t know what to test. Because you don’t know what you’ll deliver at a more detailed level.
All of this creepiness wreaks havoc to more traditional financial metrics for the project. Annual budgets are defined on an annual cadence. Product development work happens on a weekly to monthly cadence. So by month 3, what the product team does might be radically different than what was in the budget. Because realizing the budget was discovered to be pointless.
Source: Bogsnes, Beyond Budgeting
Budgets themselves also tend to fall in two categories according to Bjarte Bogsnes of Beyond Budgeting fame, based on the demand and market context in which they operate:
Same weather tomorrow as today: if this happens, you can just pretty much repeat what you did this year + an annual inflation adjustment. So not much will change anyway, and it’s not worth spending much executive time and effort on planning and politicking.
Different whether tomorrow than today: if this happens, pretty much everything you did this year is irrelevant and you need to redesign your budget from scratch (zero based budgeting)
And finally earned value is even more pointless for truly high tech products. Multiplying a subjective (and usually unvalidated) estimate of sales activity by the subjective estimate of % completion. Both of those are easy to game, hard to measure objectively, and the people doing the measurement have incentives to skew the numbers in their favor.
Not related to a decision either immediately or in the future
Finally, tracking metrics for general purpose knowledge is not a good use of anyone’s time. A great litmus test for whether or not a metric is useful is whether or not it would trigger action or a decision at a threshold value. If it wouldn’t, then realize that you are paying a cost of monitoring it (one which most likely doesn’t exceed the value you get from it).
It doesn’t really matter if it’s actionable for the product team or for the stakeholders around the project. The main point is that it should be really obvious when you need to intervene. So that you don’t intervene otherwise and you minimize management meddling.
Key Takeaways
Metrics need to drive a decision or change of behavior. Otherwise they’re pointless.
Metrics are ideally tied with how you deliver customer outcomes. Or prospect outcomes.
Financial numbers such as budgets give an air of precision, but are typically not useful in the context of new product development.
Branson as an expert at managing the risk of getting punched in the face professionally
“Jed’s dead. Dead dead.”
hitting a low point somewhere during 67 customer interviews in one weekend
survey of product managers
looking for well ordered market segments like this
Channel testing example: selling superior wine into supermarkets
negatively skewed tech adoption curve
