How does Automated Testing make anyone more money?
When developers start talking about test-driven development and automated testing, most product managers get antsy. They want to invest lots of time, without actually creating any new features. Oodles of cash disappearing, with seemingly nothing to show for it. Sounds like a terrible business idea.
Or is it?
By getting rid of waste in your product development, there are three main business reasons why automated acceptance testing will make you more money. Not only does acceptance test driven development (ATDD) help reduce really important structural costs and risks, automated tests bestow an existing product with a lot of sales and marketing mojo. A decent automated test suite helps the developers execute really fast on new ideas. Magic.
Enforcing well thought out specifications
First of all, a test can be used to enforce a specification. These specifications exist to be sure that the functionality isn’t changing, once it’s understood. Conversely, missing tests highlight missing understanding at the initial stage.
Historically, specifications have been written by reams of business analysts (BAs), negotiated, and nailed down as the ideal solution to a particular problem worth solving.
The spec would be tossed over the wall. Developers then try to make sense of it all. They turn it into code. Then testers use those same specs to confirm (manually) that the specifications been hit. Sounds great in theory.
In practice, you play Chinese Whispers. What the client said isn’t quite what the BA’s heard, which isn’t quite what the developers heard, which then is not what the testers heard.
This is hard work, particularly if you aren’t really clear your big-picture business goals. If it’s not clear what the main goal was, then it’s very difficult to justify having the specification handed over in this way.
In contrast, if those initial specifications are turned into acceptance tests, you are forced to take apart this bigger vision into specific functions in the code. Once the developers write code which passes those tests, the tests are essentially enforcing the original scope as specified. By converting the specification into code, the original spec becomes immortal like the code itself. Acceptance tests are testing the code from the point of view of the client or end user, typically exercising code all the way through the system. Unlike unit tests, they aren’t testing individual methods or classes.
Whatever it is the customer wanted, automated acceptance tests confirm that that’s what they get. Each time the tests are run. There could be edge cases which were missed in the original specification, but if there wasn’t a test for it, then it wasn’t initially specified.
Alternatively, the acceptance tests can be modified or expanded, as you discover new requirements, but then that’s a very conscious decision. Chris Matts has previously pointed out that writing your specifications as automated acceptance tests increases the amount of communication on a team, which maximizes learning and the embedded real options in your development process. By minimizing the Chinese Whispers between BAs and the manual testers, you are much more likely to deliver what you expect, and everyone is clearer on what needs to be done.
When you could use acceptance test driven development, you get a high level of certainty in the moment. You’re delivering what you expect. If you manage to build out your acceptance test suite, you genuinely verify the relevant edge cases. It’s much more likely that you’ll capture side effects of changes you didn’t consider earlier.
In the end, you also deliver a much higher quality product.
Tests reduce the cost of release
Second of all, releasing software, particularly when regression tests take a long time, can be quite problematic. Not only is the release itself quite a headache, it costs some extra time. But that’s not the real problem.
The real problem lies with unreleased features. If you have a big backlog of unreleased features, it’s like having a pile of a pile of boots without soles in a shoe factory. You can’t sell it, yet you’ve already bought the raw materials and started the work. Those boots have near zero value, because they aren’t done. The same thing happens with unreleased features. Any feature which is dev-done but not released is like a sole-less boot.
Typically, the cost of release is a big reason why companies don’t release half-finished software. What’s the fastest way to reduce that cost? Automated acceptance tests, and an automated release process. While it will take a lot of time and aspirin for the related headaches, having this in place can put an entire software business on a completely different footing (You got sole, baby!).
Moreover, with a regression test suite, you can have much great certainty that none of the changes in a release candidate have broken pre-existing functionality. With automated unit and acceptance testing, you can completely eliminate the regression test component of a release. As soon as the manual testing is done for a given new feature, and/or any new test for that feature, you’re ready to release. Your cost of release goes down significantly in terms of time, developer time and everything. And your turn around time is much, much faster.
Lowered lead times
Third of all, this big business benefit ties the above together. Typically, customers expect to wait a long time after requesting new software or a new feature. In many cases, they may be willing to pay for it. Even if not, it’s clear that this particular request would have a lot of value in their eyes.
There’s an easy way to measure and quantify this. Lead time is the amount of time between a customer’s request and when that customer actually gets what he wants. While a lot of effort goes into producing this, ultimately this is all the customer really cares about. They want to know what’s in it for them.
Yet, it’s inevitable that your lead time will grow as your code base gets larger. The more parts you have, the more interconnections you have. This grows exponentially, unless each part has it’s own set of tests.
Tests keep your lead time linear. They’re an insurance policy. If customers do request a particular feature, with relatively high level of confidence, you can give them what they want without breaking anything else. That new feature is contained behind a test suite, so you don’t need to worry your pretty little face about that.
Imagine a big company being as nimble as a startup. Most of the real life examples have famously large test suites. In addition to massive test suites, Google even wrote and open-sourced their own testing framework.
If you can minimize lead time, that tends to put a lot of upward pressure on revenues (that’s a good thing). At this point, we aren’t just talking about containing costs, we’re talking about earning more revenue. Generating new business. To be blunt, this is what all of your costs are subtracted from when calculating profits.
The investment that pays for itself
As with any business decision, deciding to build an automated test suite is an investment. Depending on the details, it will take your geeks some time to build this automated test thingy. There is a lot more to automated testing than well-intentioned “software craftsmanship”. At the same time, you can be sure to reap significant bottom line benefits in an existing software product business.
Even if users are clamoring for new features, dedicating some time to creating automated test suites will eventually make your product more profitable. The trick is to focus your testing efforts on areas where you expect to get high pay-offs. Prioritize these efforts rigorously, to make sure that this test suite pays for itself each time you release.
Product Customer Fit Only Requires Must-Have Features
When releasing a new product, the first step is to find product customer fit quickly. The minimum viable product (MVP) encompasses the essence of the Lean Startup ethos. An MVP helps go through one cycle of the Build-Measure-Learn loop. Lean Startup author Eric Reis warns “Customers don’t care how long something takes to build. They only care if it serves their needs.”
<whistling wind>
You also need to choose a specific customer, in order to address a customer’s specific need. The main goal of an MVP is to learn about the customer and the market. You want to validate or reject your hypotheses. Once you know what your market wants, you are in a completely different position.
In order to do this, you need to:
1. decide on an ideal client
2. get access to such a client through marketing activity
3. identify their most pressing need or problem
4. get paid to solve itAt an early stage, you can solve the problem in any number of ways. It’s often easiest to provide a service. That way, you learn more about your clients. You gather useful information about how to deliver a solution, which can then be automated via software or other type of product.
Let’s say you want to build a software company helping people learn foreign languages. Entrepreneur Derek Sivers points out that you can get started by just scheduling a language teaching session. It’s very manual. It’s not automated at all.
At the same time, it’s an extremely high-bandwidth way to learn about your customers’ needs. Most importantly, it’s useful for them. Once you have some experience delivering this type of service, you have a much better chance of successfully prototyping a solution which addresses that customer’s need.
You identify your customer’s primary need. You learn what the customer thinks about it, how they dream they could overcome the problem. You hear them vent about their frustration. You dig deep into specific aspects. You seek out find you can address.
You find out how your customer thinks about the problem. This is direct marketing gold. It helps you identify where to focus your efforts, so that you address what your customer finds the most vexing.
By focusing on the must-have features only, you release a product or a service that addresses that particular need. It’s rudimentary. Yet it works. You might not even require a line of code to prove your concept. Must-have features are essentially all related to specific changes you want to induce. Your target customer will not consider the product valuable otherwise.
In order to be successful, you really need to dig deep into one particular problem. You want to know the logical reasons why it’s attractive for your customer. You want to know why would benefits from the product. You even want to find out more about any emotional benefits they might get from the product.
This approach correspond’s to Ken Schwaber’s scrum value burndown charts. Develop the highest value features first. If the product ends up being successful, then in fact, these are the extremely valuable core product features. They are must have features for this particular problem. Without them, you can’t claim you have a workable solution to that specific problem. Each one potentially generates massive incremental value in users’ eyes.
Often a few features, if packaged together well, which work reliably, is enough to interest the early adopters in a market. While they realize they may not be getting a complete solution to the problem, they like being first, having access to the inventors, and contributing feedback.
These must have features define the product. Often, just a handful of features suffice for your product type to be attractive, such as copy and paste in word processors, compared to the typewriter alternative. That’s the kind of gap you want to find.
If your minimum viable product is not attractive to your target segment, move on. Next. Another niche. Another need. Another customer type. The faster you discard the unattractive options, the faster you’ll achieve product-customer fit.
The Software Construction Metaphor is Broken
