Completing tests quickly will help you run multiple tests in succession
Your success will be a function of the learning cycles you fit into
your runway.
— Tom Hulme, Ideo
The best way to discover what makes your prospects tick is to run
an experiment. If you are pressed for time, the best way to
discover what makes them buy is to run many experiments over a very
short time period.
Assuming these two would cost exactly the same amount, which would
be more valuable to you as a product manager?
1 test, which runs for one week, which gives you 99% certainty
on your results
30 tests, which take one hour each, with 90% certainty each
Clearly the second set of tests is much more valuable. When you
finish the first test, you can form another hypothesis and test
that. One set of results directly feeds into the rest. If you want
to identify surprising results quickly, as this is most likely to
give you outsized results, option B is a much better way to enter a
market.
To some extent, it’s reversing the iterative nature of agile,
and applying it during pre-development when performing product
research. Feedback loops are enormously powerful when dealing with
prospects and customers too.
“Market Testing Velocity” can mean a lot to you in a
high-pressure situation if you want to get something going quickly.
If you complete tests quickly, you can run multiple tests in
succession.
In this case, you are doing marketing research on your
prospects or existing customers, in order to map out what they
need. Your goal is to identify what they need most, a
“bleeding neck”, as quickly as possible. You are trying
to discover a cognitive and emotional map of their needs, in order
to orient your own efforts most effectively.
If this tickled your fancy, and you’d like to read more about
market testing velocity, you’ll find out more in Launch Tomorrow.
When developers start talking about test-driven development and automated testing, most product managers get antsy. They want to invest lots of time, without actually creating any new features. Oodles of cash disappearing, with seemingly nothing to show for it. Sounds like a terrible business idea.
Or is it?
By getting rid of waste in your product development, there are three main business reasons why automated acceptance testing will make you more money. Not only does acceptance test driven development (ATDD) help reduce really important structural costs and risks, automated tests bestow an existing product with a lot of sales and marketing mojo. A decent automated test suite helps the developers execute really fast on new ideas. Magic.
Enforcing well thought out specifications
First of all, a test can be used to enforce a specification. These specifications exist to be sure that the functionality isn’t changing, once it’s understood. Conversely, missing tests highlight missing understanding at the initial stage.
Historically, specifications have been written by reams of business analysts (BAs), negotiated, and nailed down as the ideal solution to a particular problem worth solving.
The spec would be tossed over the wall. Developers then try to make sense of it all. They turn it into code. Then testers use those same specs to confirm (manually) that the specifications been hit. Sounds great in theory.
In practice, you play Chinese Whispers. What the client said isn’t quite what the BA’s heard, which isn’t quite what the developers heard, which then is not what the testers heard.
This is hard work, particularly if you aren’t really clear your big-picture business goals. If it’s not clear what the main goal was, then it’s very difficult to justify having the specification handed over in this way.
In contrast, if those initial specifications are turned into acceptance tests, you are forced to take apart this bigger vision into specific functions in the code. Once the developers write code which passes those tests, the tests are essentially enforcing the original scope as specified. By converting the specification into code, the original spec becomes immortal like the code itself. Acceptance tests are testing the code from the point of view of the client or end user, typically exercising code all the way through the system. Unlike unit tests, they aren’t testing individual methods or classes.
Whatever it is the customer wanted, automated acceptance tests confirm that that’s what they get. Each time the tests are run. There could be edge cases which were missed in the original specification, but if there wasn’t a test for it, then it wasn’t initially specified.
Alternatively, the acceptance tests can be modified or expanded, as you discover new requirements, but then that’s a very conscious decision. Chris Matts has previously pointed out that writing your specifications as automated acceptance tests increases the amount of communication on a team, which maximizes learning and the embedded real options in your development process. By minimizing the Chinese Whispers between BAs and the manual testers, you are much more likely to deliver what you expect, and everyone is clearer on what needs to be done.
About to start a greenfield project?
Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.
When you could use acceptance test driven development, you get a high level of certainty in the moment. You’re delivering what you expect. If you manage to build out your acceptance test suite, you genuinely verify the relevant edge cases. It’s much more likely that you’ll capture side effects of changes you didn’t consider earlier.
In the end, you also deliver a much higher quality product.
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
Second of all, releasing software, particularly when regression tests take a long time, can be quite problematic. Not only is the release itself quite a headache, it costs some extra time. But that’s not the real problem.
The real problem lies with unreleased features. If you have a big backlog of unreleased features, it’s like having a pile of a pile of boots without soles in a shoe factory. You can’t sell it, yet you’ve already bought the raw materials and started the work. Those boots have near zero value, because they aren’t done. The same thing happens with unreleased features. Any feature which is dev-done but not released is like a sole-less boot.
Typically, the cost of release is a big reason why companies don’t release half-finished software. What’s the fastest way to reduce that cost? Automated acceptance tests, and an automated release process. While it will take a lot of time and aspirin for the related headaches, having this in place can put an entire software business on a completely different footing (You got sole, baby!).
Moreover, with a regression test suite, you can have much great certainty that none of the changes in a release candidate have broken pre-existing functionality. With automated unit and acceptance testing, you can completely eliminate the regression test component of a release. As soon as the manual testing is done for a given new feature, and/or any new test for that feature, you’re ready to release. Your cost of release goes down significantly in terms of time, developer time and everything. And your turn around time is much, much faster.
Lowered lead times
Third of all, this big business benefit ties the above together. Typically, customers expect to wait a long time after requesting new software or a new feature. In many cases, they may be willing to pay for it. Even if not, it’s clear that this particular request would have a lot of value in their eyes.
There’s an easy way to measure and quantify this. Lead time is the amount of time between a customer’s request and when that customer actually gets what he wants. While a lot of effort goes into producing this, ultimately this is all the customer really cares about. They want to know what’s in it for them.
Yet, it’s inevitable that your lead time will grow as your code base gets larger. The more parts you have, the more interconnections you have. This grows exponentially, unless each part has it’s own set of tests.
Tests keep your lead time linear. They’re an insurance policy. If customers do request a particular feature, with relatively high level of confidence, you can give them what they want without breaking anything else. That new feature is contained behind a test suite, so you don’t need to worry your pretty little face about that.
Imagine a big company being as nimble as a startup. Most of the real life examples have famously large test suites. In addition to massive test suites, Google even wrote and open-sourced their own testing framework.
If you can minimize lead time, that tends to put a lot of upward pressure on revenues (that’s a good thing). At this point, we aren’t just talking about containing costs, we’re talking about earning more revenue. Generating new business. To be blunt, this is what all of your costs are subtracted from when calculating profits.
The investment that pays for itself
As with any business decision, deciding to build an automated test suite is an investment. Depending on the details, it will take your geeks some time to build this automated test thingy. There is a lot more to automated testing than well-intentioned “software craftsmanship”. At the same time, you can be sure to reap significant bottom line benefits in an existing software product business.
Even if users are clamoring for new features, dedicating some time to creating automated test suites will eventually make your product more profitable. The trick is to focus your testing efforts on areas where you expect to get high pay-offs. Prioritize these efforts rigorously, to make sure that this test suite pays for itself each time you release.
When releasing a new product, the first step is to find product customer fit quickly. The minimum viable product (MVP) encompasses the essence of the Lean Startup ethos. An MVP helps go through one cycle of the Build-Measure-Learn loop. Lean Startup author Eric Reis warns “Customers don’t care how long something takes to build. They only care if it serves their needs.”
<whistling wind>
You also need to choose a specific customer, in order to address a customer’s specific need. The main goal of an MVP is to learn about the customer and the market. You want to validate or reject your hypotheses. Once you know what your market wants, you are in a completely different position.
In order to do this, you need to:
1. decide on an ideal client
2. get access to such a client through marketing activity
3. identify their most pressing need or problem
4. get paid to solve it
At an early stage, you can solve the problem in any number of ways. It’s often easiest to provide a service. That way, you learn more about your clients. You gather useful information about how to deliver a solution, which can then be automated via software or other type of product.
At the same time, it’s an extremely high-bandwidth way to learn about your customers’ needs. Most importantly, it’s useful for them. Once you have some experience delivering this type of service, you have a much better chance of successfully prototyping a solution which addresses that customer’s need.
You identify your customer’s primary need. You learn what the customer thinks about it, how they dream they could overcome the problem. You hear them vent about their frustration. You dig deep into specific aspects. You seek out find you can address.
You find out how your customer thinks about the problem. This is direct marketing gold. It helps you identify where to focus your efforts, so that you address what your customer finds the most vexing.
By focusing on the must-have features only, you release a product or a service that addresses that particular need. It’s rudimentary. Yet it works. You might not even require a line of code to prove your concept. Must-have features are essentially all related to specific changes you want to induce. Your target customer will not consider the product valuable otherwise.
About to start a greenfield project?
Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.
In order to be successful, you really need to dig deep into one particular problem. You want to know the logical reasons why it’s attractive for your customer. You want to know why would benefits from the product. You even want to find out more about any emotional benefits they might get from the product.
This approach correspond’s to Ken Schwaber’s scrum value burndown charts. Develop the highest value features first. If the product ends up being successful, then in fact, these are the extremely valuable core product features. They are must have features for this particular problem. Without them, you can’t claim you have a workable solution to that specific problem. Each one potentially generates massive incremental value in users’ eyes.
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
Often a few features, if packaged together well, which work reliably, is enough to interest the early adopters in a market. While they realize they may not be getting a complete solution to the problem, they like being first, having access to the inventors, and contributing feedback.
These must have features define the product. Often, just a handful of features suffice for your product type to be attractive, such as copy and paste in word processors, compared to the typewriter alternative. That’s the kind of gap you want to find.
If your minimum viable product is not attractive to your target segment, move on. Next. Another niche. Another need. Another customer type. The faster you discard the unattractive options, the faster you’ll achieve product-customer fit.
I dragged my feet up the cracked cement staircase. Next to the stairs, wild, thorny shrubs grew out of control. No one had bothered with them for years. A metallic pipe with black paint peeling off served as a fence. It was a snappy cold February day.
As we got to the top of the staircase, two massive guys got out of a parked sedan with an engine humming to keep them warm. In black, furry sport jackets, they tumbled up the stairs, with stubby legs sticking out from underneath rather big beer bellies. We were meeting with a builder named Slawek. Slawek was the last entry in our “beauty contest” of building contractors.
My wife and I had just bought our first apartment. We’d never made such a big purchase in our lives- ever. We had serious “skin in the game”. The housing market already started spectacularly nose-diving. We wrangled down the purchase price by quite a bit, relative to the area. The owner was desperate to sell an apartment he had rented out for over twenty years. It showed.
Unfortunately, neither of us had any experience with DIY, other than hanging up a picture frame, much less construction work. At least, I knew “Project Management”. Moving in from remote parts, we had to have the renovations done well, and finished on time. Otherwise, we’d be paying a both a mortgage and rent. We also knew that we’d be living with any mistakes for a long, long time.
Everything had to be redone (electricity, plumbing, floors, walls, ceilings-basically everything), in one month at most. The scope was staggering, at least for a small apartment. It was probably cheaper just to build a new one.
We also wanted everything to be certified, and in accordance with the local regulations, since we know how important that would be when we’d be selling the place, based on recent experience as house-hunters.
Most of Slawek’s peers had a sheer look of terror when we laid out the scope and timeline-they started losing nose hairs. In contrast, Slawek and his buddy just smiled.
Slawek said, “Yes, sure, no problem. Three weeks should be enough, actually.”
“Even with the regulatory certifications?”
“Yes, sure”
“Ok,” I said, “in that case, you won’t mind a penalty for lateness, if you run over one month,” I suggested.
He flinched a bit.
I thought giving Slawek a buffer was benevolent move. At minimum, I wouldn’t feel guilty about imposing a penalty. He had some space for risk-for the unexpected. At the same time, he had to know we were serious about not missing our deadline.
“Yeah, one month. No problem,” he repeated.
About to start a greenfield project?
Have Launch Tomorrow run an in-house "riskiest assumption workshop". Remote delivery options also available. Discover where to prioritize your validation efforts, to get to market fast.
As the work progressed we visited every few days. Rather than actually being a builder, Slawek himself turned out to be a sales guy and an extremely talented project manager. While anything PC-related clearly overwhelmed him, as I had to help explain what an RTF file was, it didn’t stop him at all. He wouldn’t even install MS Project even if I paid him a few thousand extra.
Create an explainer video for your complicated new product.
Make sure your audience understands it, without being overwhelmed by technical details.
At the same time, he had an intuitive grasp on timing, i.e. when to curse out one of his hired hands to “motivate” him. In his next breath, he was back to dishing out highly technical advice to us, packaged as stellar customer service.
I had severely underestimated how street smart the guy was. He was a real rainmaker. There was clear progress every single time we visited. This guy ate milestones for breakfast.
After a few slips of the original schedule, the builders gave back the keys exactly one month after starting; taking full advantage of the buffer I had given them. The work was well done. Visually, the apartment was everything we had imagined it would be. We could move in. We were happy. All our boxes were ticked.
A few months later, we were eating dinner at our dinky IKEA table, chatting away and…the room lights in the entire flat go out-all of the lights in every room, and many of the other lamps, including outdoor lighting.
Dropping my fork mid-bite, I ran to the circuit breaker. One circuit had popped. I realized most of our lights were on one circuit, which I had missed during the handover from the builders. Not only that, the circuit breaker kept popping back when we tried to set it. There was a serious underlying design flaw in our electricity.
I quickly realized the only way to fix it was to redo the wiring in the whole flat, but now-we actually lived in the apartment. It was painful thinking about it.
I flipped out, and started looking for Slawek’s phone number. I couldn’t get through. At the time of the renovation, he had said he was planning on leaving the country soon. We only wanted to get the electrician’s phone number, but no luck.
After scratching our heads, we realized we had received a certificate from the electrician. I desperately paged through old paperwork, and thrilled, I pulled out the flimsy-looking thing. All hope was not lost.
To add insult to injury, I realized, only now, that the certificate looked photocopied-many times. There were some scribbles on the paper, and I could make out a phone number. A bit suspicious, I jumped online to find out more about the exact nature of the certificate. It looked like a forgery. Not only that, it definitely wasn’t the gold standard certificate we thought we had bought. He had given us a no-name certificate, and we hadn’t even noticed it.
Of course, the electrician didn’t pick up the phone either, only responding to our queries with nasty text messages. To be honest, he had his cash, and he had no interest in providing us service “under warranty”, even if we paid for it. He knew exactly what the problem was, so he didn’t even want to try fixing it.
While we could temporarily introduce a few floor lamps, we had expected the lights to work after a fresh rewiring of the entire flat. That was unacceptable. By my standards, the ultimate result was not functional.
At the same time, I could only really blame myself, or to be more precise, the approach that I took. I had slyly slipped in a late penalty for delays. As a result, Slawek couldn’t let the electrician mess around any longer than the first week, if he wanted any chance of avoiding the penalty. I wanted to minimize my own risk, forcing them to take shortcuts.
Those shortcuts ultimately “broke” the deliverable in the long term. I had no idea about the interdependencies among their tasks. I wanted them to manage it, and I just pressured them to do work faster.
If the above was a software project, it would have been a massive success:
• Everything in scope was delivered
• Everything was delivered on time
• To be honest, we actually got a pretty good price given the initially perceived quality
• Didn’t need to use metrics
• Didn’t need to use specialized project tracking and handoffs, as popularized by MS Project
At the same time, the builders introduced a big, undetectable, architectural problem, which became a big, unpredictable, and nearly unfixable problem for the “users”.
The main cause?
The rush, the hurry-which I introduced as the moneybags.
Of course I could be angry at this electrician, but I had insisted on finishing fast. As a result, when they had to cut corners, they did…because I had clearly defined the criteria they need to satisfy.
When launching a new product, you are typically faced with the same problem. You want to deliver something quickly, in order to be first, in order to make a splash. You want first mover advantage. It has very concrete financial implications for you. In highly competitive markets, especially online markets, there are typically only a few large players.
In the late nineties during the dot-com boom, as new business models were being introduced, many of the segments had three players, and a handful of also-rans which typically didn’t have more than 10% market share. Nowadays, it’s not much different.
Whether you realize it or not, delay is usually the biggest cost you bear when launching a new product. It can mean the difference between a majority market share, and a minority market share for your product.
You risk making fewer sales if your competition beats you to a market. Even if they have more bugs, they will already be earning cash.
This is particularly true for startups. When there is big disruption, and you create a new segment, customers will only remember 3 three players in a segment at most.
In most markets, according to Ries & Trout’s booklet the 22 Immutable Laws of Marketing, there is space for three players in your users’ mind. How many brands of toothpaste can you come up with, without checking a store?
More poignantly, do you know who the second person to cross the Atlantic was, without checking Google or Wikipedia? (Hint, it wasn’t Charles Lindberg or Amelia Earhart).
You are up against the limits of perception and memory, at least within your niche, and the clock’s started ticking.
