Is premature optimization a warranted concern?

Here’s a wonderfully geeky question from a forum I’m on, related to lean startup:

I have a product that I want to test. I’ve built the one-pager website and set up some Facebook advertising to figure out the messaging / features that makes people sign up to determine the direction I will eventually build towards.

I have three ads: A, B, A+B
and three website versions: A, B, A+B

My question is whether I should test the advertising AND the website versions at the same time. I want to test quickly, but I’m worried that if someone clicks advertisement A and then lands on website B, it won’t be relevant and it will skew my data.

Is this a warranted concern? Should I test the advertising first, then the website version? Or should I do both and just figure that, with enough users, the data will smooth itself out faster then if I test one at a time? Any insight would be greatly appreciated.



Couple of things:

  1. Yes it is a warranted concern. When you send traffic from ad A to landing page B, you lose them. You can set up ad A to point to landing page A, ad B to landing page B, and so on. Then by rotating through all ads, you’ll implicitly rotate through all of your landing pages.This “scent” which a prospect follows is really important. You may be losing people. They land on a LP which doesn’t correspond to their post-click expectations. This may skew your results.The goal here isn’t necessarily to convert, i.e. get a lot of signups. It’s to measure user behavior and preferences. Discover where you have latent/unmet demand.
  2. The right test depends on what you’re trying to learn and what your hypothesis is exactly. For example, why are you even split testing in the first place? If you start split testing too early, it ends up being premature optimization. Understanding why you are split testing will help you formulate a meaningful hypothesis. Split testing only tells you if option A is relatively better than option B.You’re better off first choosing a threshold for conversion, then running ads to see if you hit that threshold. It’s a different type of statistical test and way of thinking about the problem. Then you’re also learning more about demand.
  3. You will achieve statistical significance faster focusing only on user behavior around ads. Clicks. Based on that you can “test before you test”, and find out what larger/full tests are worth doing in a full environment.

As a rule of thumb, understanding your users better will usually be more profitable than understanding your product idea better.

If you want to go into greater depth about using landing pages rigorously to learn about your market, then check out Launch Tomorrow. It clears up a lot of the confusion around new products and landing page MVPs. Why blunder your way through? Launch Tomorrow‘ll take care of ya.

The #1 mistake of Lean Startup newbies

lean startup mistakes launch tomorrow

Validating your product idea is very important.

Yesterday I was speaking with a gaggle of early stage tech entrepreneurs about Lean Startup at Google Campus in London.

They were eager to learn more about about hypothesis testing. Newbies wanting to learn about the Lean Startup approach.
Yet they were falling prey to what I call the “solutionizing bias”. They only wanted to talk and think about solutions. And making sales. It’s so tempting, easy and natural for founders to fall for that trap.

As entrepreneurs, we’re naturally optimistic go-getters. We have a solution for every problem. We want to help out.

Yet when you’re building a new product, you don’t know whether your solution is important for your customer. If you don’t prove that first, then brace yourself for a long uphill battle.

Your solution solves a problem your customer has. The customer cares about their problem, not your solution. Before you think about solutions, you need to know whether the problem your solution solves is important. In your customer’s eyes.

That’s why validating your product idea is so important. First ensure that the problem you’re addressing affects a large percentage of your target market.

If you focus on your solution only, you won’t know which problem is worth addressing. It’s blinding.

Imagine you’re reading a market research report about the biggest problems of your target market. In that report, there is a pie chart.

Let’s say your prospects have 4 different problems: A, B, C, D. 40% consider A their main concern, 30% B, 20% C, and 10% D.

Yet, you’ve already built a solution. You realize it addresses problem D.

About to start a greenfield project?

Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.

Contact Us

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

For the same amount of marketing effort, you’ll get 4 times less results than another founder who addresses problem A.

In my experience with building products, I’ve managed to build products that addressed problem E.

Create an explainer video for your complicated new product. Make sure your audience understands it, without being overwhelmed by technical details.

or call us now at:

US/Canada: +1 202 949 4478
UK: +44 773 952 7708
EU: +48 692 870 297

In other words, it was a problem I thought the market should have, but actually didn’t. I was dead in the water. I write about an example like that, and how to prevent it from happening to you, in Launch Tomorrow.

While it’s critical to think in terms of sales you’re going to make when you start a business, first check if you are addressing a meaningful problem first. One that a big chunk of your market thinks they have. And that they’re willing to pay for a solution. Then you set yourself up for hockey-stick growth.

Only then do you build a money-making machine.

Otherwise, you might as well be a missionary.

If you follow the one-day launch sequence in Launch Tomorrow, you’ll get a decent blueprint to do exactly that. Build what your customers want.