Launch Tomorrow

Landing Pages for your Lean Startup

Archives for June 2019

How to use metrics to manage delivery with a human touch

by Leave a Comment

At one client site, the CTO from the head office wanted to introduce metrics to help him monitor what was going on across all of the new product initiatives. Like the other delivery managers at the time, I understood his need. But when I opened up the first spreadsheet defining the template I needed to report to–my jaw dropped.
It seemed like most of the numbers were about compliance to the centralized directives, particularly with respect to timeliness. With very little focus on whether or not what we were doing will actually make the company any money. If we were successful at hitting those timelines, we wouldn’t necessarily make anything until those delivery dates were hit.
Months and years away.
unsplash image dbfcdbcbeb
Photographer: Aadhar Sharma | Source: Unsplash
Not everyone likes numbers. I get that. Once you grow a program beyond just one delivery team, though, the firehose of information coming at you becomes overwhelming. So it’s perfectly natural to start looking to numbers to help figure out what needs attention.
As a leader, just like this CTO, you have a very simple concern: anything that goes wrong will land in your lap. So the more proactive you are, the fewer crises you deal with. And your calendar stays in your hands.
The flip side, though, is if you try to manage this from a “control” and “risk avoidance” mindset, you’ll stifle the ability for your team to respond and learn. Even more so if you’re only focusing on schedule risks. If your schedule is at risk, it’s likely you have too much scope and not enough client interaction to drive product development.
So the below insights are the most useful that I have read or have discovered myself when choosing or constructing operating metrics around new product development.

1. Information and data is everywhere. Any given data point has low value. In contrast, the ability to contextualize data has enormous value.

The amount of potential data doubles expands exponentially, making it increasingly more valuable to find signal in all that noise. In other words, interpreting data requires structure. That structure requires context. And interpreting context requires wisdom.
    • Just because there is a lot of data, doesn’t mean you can pull up a meaningful table using exactly the summary numbers you need.
    • And even if you have that table, you need to be able to interpret it subjectively.
    • Not every number exists to be added, aggregated, maximized or minimized numerically and you need to know the real-world meaning of every number.
    • And even if you do know this real-world meaning, being able use discernment and wisdom to interpret it usefully.
For example, I had this problem before I had taken accounting classes during my Masters in Finance degree. At the time, I was poring over financial statements of hundreds of dotcom Internet companies. The truth was there were lots of numbers to look at, but I was kind of guessing what they meant at the time. I could focus on just net profits and work back from there. Comparing lots of net profit numbers across companies across time would be a form of structured data as a table. Maybe useful. It kind of depended what was behind the number. It also varied significantly, as the real world meaning was quite varied across the hundreds of “dotcom”. And finally, at the time I didn’t have enough wisdom and discernment to know that actually looking at net profts for early stage companies was just premature. What actually mattered was learning and operational progress.
For new products, the key context is whether you are learning something useful as that will shorten your path to profitability (even if that learning process is initially a net loss at first in pure cash flow terms). In my opinion nowadays, “will the project deliverables arrive on time?” is the wrong question to be asking. They have to arrive (or at least be defined) early enough for a prospect to want to pay for it. Yes, dates are important, just let’s keep them in the right context.
This goes much deeper. My friend Perry Marshall has said that the #1 success skill in the 21th century (one that almost nobody talks about) is discernment. Identifying the “vital few” factors that actually matter. The ability to discern is based on wisdom. Applying timeless principles to your specific situation. Building to last, not just hitting next quarter earnings targets.

2. The high bar for metrics: what decision would this metric help me make? And under what circumstances?

I interned in IT at GMAC mortgage, General Motors’ home mortgage subsidiary, while still in college in the mid-1990s. My boss at the time had me look into setting up the first ever company webpage and intranet. I had a late night discussion with my boss at the time about whether or not we should have a “hit counter” displayed directly on the site. This was a pre-built component displayed directly on the page, which went up by one each time somebody browsed to it.
The higher the hit counter was, the more popular that page or the company’s website looked. But in and of itself, having a higher number of hits didn’t mean anything. Certainly not to us. It wasn’t a number that helped make a decision or indicate a change of tactics was needed. What mattered was who was coming to the page, why, and whether they were achieving what they wanted.
web analytics cebaccfdaccbebc
source: Avinash Kaushik
This nugget of wisdom came from Avinash Kaushik’s book Web Analytics 2.0. The example he used was website hits. Yet his point goes much deeper than just interpreting website traffic. If you are tracking a operating metric that doesn’t help you or your organisation make a decision of some kind, then it’s just not useful.
There’s lots of numbers you could be tracking. Just log into your company’s various systems to get a taste: sales pipelines, ticket tracking, financial metrics. But remember you will get the most value by identifying the vital few that actually matter most to your company. Which by definition are specific to your company. They will require some deeper discussion to get right. And effort at aligning everyone in the company around them.

3. The GPA effect…

In the mad rush to the real world, most of my fellow students competed for the most prestigious jobs based almost solely on their grade point average (GPA). The conventional wisdom applied by investment banks and consulting firms was that a high GPA meant that you were smart and hard working. You stayed up and studied, so that you earned good grades, and therefore you’d be reliable.
Ultimately, for the most popular gigs, GPA became a first sorting mechanism. As recent grads, we were a commodity on the job market. If hundreds of students applied for the same investment banking analyst position, then the banks only interviewed the people with the highest GPAs.
In practice, though, this was a classic example of a heuristic which had a lot of potential faults:
    1. Did someone with a 3.94 GPA show more promise than a 3.92 GPA in a role that largely involved the ability crunch spreadsheets for 50 hours straight?
    2. What about relative strengths and preferences? Would someone gregarious and social with a slightly lower GPA really be a worse salesperson at a bank?
    3. Didn’t this skew even further towards people already coming from privileged and wealthy backgrounds who had benefited from and could continue to afford nearly infinite tutoring? Didn’t this reduce the diversity of new hires?
    4. Wouldn’t this skew towards people with a perfectionist streak? And reward them for high compliance? Always looking for the “right” answer in an academic context doesn’t translate that well in all business context (such as new product development for example).
In contrast, there were guys on the crew team which clearly only had crew and beer on their mind. So a low GPA was just a lagging indicator of their relative lack of maturity more than anything else. And in this case, it was probably a useful metric for hiring purposes, particularly as a way to identify bad choices.
gpagraph faedafbb
source: New York Times
So GPA was useful as a metric to help prevent a complete blowup in a hiring scenario, but downright dangerous as the only data point that matters.
I suspect the same is true of many operating metrics in a larger business. Use them to highlight problems or potential problems, but beware of optimizing for them to the exclusion of all else.
For example: optimizing for velocity at all costs. You can cut corners with quality. You can run your team into the ground. You can create lots of badly documented features. But you went fast, right?
Sadly, the most common reason for new product failure isn’t missing a timing window. It’s building something the market doesn’t want or care about. In which case, pursing high velocity for its own sake is a fool’s errand.
That said, having very low velocity isn’t good either. It means it’s time to go digging and figuring out why the teams are struggling. And something needs to change or to be fixed.

4. When there is still too much data, give the most weight to data around your first order priorities.

If there is too much data, revisit your goals and priorities to figure out which metrics to pay attention to. Quite often operating metrics are unclear or not useful, simply because the stated goals are too nebulous or inaccurate to be useful. Or each stakeholder has their own goal(s) to achieve and therefore it’s difficult to agree on a common set of metrics.
Quotefancy aeafccdaafddba
source: Stephen Covey
Reid Hoffman, founder and scaler of LinkedIn, had a much more eloquent way of articulating this concept:
“When there is too much potential data… What are the key things that would change my decision? What is the data around my first order priorities? After you have some idea, then you consider the impacts for 2nd and 3rd order priorities.” — Reid Hoffman on Jordan Harbinger’s podcast
If your company has clarity and alignment on goals, then it should be clear what data sources to pay attention to. Or to create.
This is particularly true when you need to narrow your focus on what works in order to scale an effort. Figure out what numbers are the most meaningful, and then optimize for that. If that means re-assessing the strategic tradeoffs you’re making, then do that.

5. Accept that metrics on their own are not the full story, and they never will be.

The metrics themselves are not the story. While they help you hone in on the the underlying mechanics of delivery system, this system is just an abstract structure.
In fact, the metrics only matter to help drive conversations to pull out the full story. In other words, they aren’t a replacement for a conversation. But they are best used a trigger for a conversation. An early warning sign, to be able to act preemptively if needed. And also to provide feedback that might not be apparent right on the front line.
For example, when running a bigger team, it’s hard to keep track of what everyone is doing. It’s hard not to like everyone involved. But when you look at output metrics, even if they aren’t perfect, it gives you an alternate view of the performance. While not perfect, even something as simple as keeping track of number of commits for software developers over time will at least highlight anyone who isn’t putting in the work. Which is still only the basis to start a qualitative investigation. But it gives you something concrete to work with.
As an introvert, I like using numbers to track what’s going on. I feel better about this than just using a subjective or intuitive approach, which doesn’t scale up that well for me.
But…
Empathy will always supersede any metric, because people underlie the system. For example, life happens outside work. A great developer on my team had a cancer surgery in the family. It was enough to knock him off his feet, for example. But after he had the time to attend to that rough patch, this person came back with stronger performance for a while. All this was reflected in the metrics, both the down and the up. So trying to act on the fall of the metric would have been premature and shortsighted. And lacking in empathy.

Aiming for self-accountability at work

All that said, i think the right balance is achieved when there is a system for self-accountability at work, which everyone buys into. From the front line busy bee to the senior decision-maker. This would be the ideal culture. Of course, it is much easier said than done.
A good set of metrics prevents you from having a blame culture on one extreme, an abdication culture on the other. It requires a good amount of self-awareness as an organisation, which is often the hard part. But it’s totally worth it if you want to keep everyone involved engaged and happy.

<< Help Yo' Friends

The top 10 “rules” of adaptive innovation management

by Leave a Comment

Adaptive innovation focuses on building and releasing new products in high levels of uncertainty, and optimizing resources as you go. Not just up front at the planning stage. Studies of high growth Inc 500 startups have confirmed that firms adapting to market needs achieve growth more frequently, compared to companies sticking to an original vision and plan. Established companies looking to innovate, would be wise to allow for such flexibility as they expand their product portfolio. Corporate innovation shares the same market context with startups when launching new products.

Diving deeper into why high growth startups  were successful will help increase the chances of succeeding as innovators. In contrast to startups, larger firms have access to much greater resources, although they struggle with the complexity this causes. Moreover, their starting point is often one of being cost efficient. This can lead to challenges in the early stage. Evolving products requires multiple rounds of financing, in order to minimize risk rather than using traditional budgeting to help new products succeed.

The following observations are based on my innovation experience in a B2B software context. While this is a very particular type of environment, with high technology and quality constraints but no manufacturing costs, the principles can be cross-pollinated into other environments. Also it’s worth noting that software has the ability to be packaged as products (MS Office) or as services (Google Suite). In short, the model and its insights can be applied to other types of new product development.

*NgjBtEotLNq
Photographer: Markus Spiske |

1. In practice, the full cost of time is dictated by sales and external market conditions. This is because you are nearly 100% certain not to make a sales forecast, if the relevant and sufficient scope for a customer does not exist yet. This holds regardless of your confidence in your sales forecasts and estimates.

2. Any planned internal costs will always be lower than cost of lost sales. They have to be over the longer term, otherwise you wouldn’t have a viable business.

Your riskiest assumptions are probably related to your prospects and customers. Establish empathy quickly with your target prospect, figure out what's valuable, and get your innovation into the market.

3. You can use linear approximations as an adaptive method to quantitatively arrive at a current “cost of time” to help drive optimal decision making. This is done by comparing the effect on sales if the planned scope was available immediately vs if it exists on the estimated delivery date. Based on this you can derive a cost of time.You probably won’t have the luxury of directly observable continuous functions. In effect, you’re starting to think in terms of calculus and deltas, as opposed to solving for the absolute values in algebra.you move from “overall cost” to d (cost) / dt.

 
*gDpvsDvLXBabt

4. Costs and Revenues follow different probability distributions. and these distributions are skewed costs:

  • inflexible especially in the case of legacy, large, or infrastructure
  • relatively certain once agreed (near 100%)
  • fixed in advance in accordance with high level strategy
  • once agreed, they tend to stick around for a while due to the sunk cost fallacy

Revenues:

  • Highly variable
  • Depend on how well the immediately available scope matches to the customer problem
  • Can make a big $ sale with a small incremental add of $ (in scope terms) — true even for a new product, there is a 20% of functionality that covers 80% of the value
px Comparison mean median mode svg dadfbaaaafddfbabedab

Why delivery velocity is a broken metric

by Leave a Comment

“For by the ultimate velocity is meant that, with which the body is moved, neither before it arrives at its last place, when the motion ceases nor after but at the very instant when it arrives… the ultimate ratio of evanescent quantities is to be understood, the ratio of quantities not before they vanish, not after, but with which they vanish” –Principia Mathematica by Isaac Newton

I’m not a mathematician, but I’m not afraid of numbers. I’ve drifted through the standard high school and college math requirements, admittedly with curiosity. And I’ve hung around the hedge fund industry, software engineers, and actually trained mathematicians for long enough that math’s kind of rubbed off on me. Yet my skillset is probably best described as a mix of product and, more recently, delivery management with an academic foundation in accounting.

Over time, I’ve noticed that cost accounting and traditional/waterfall project management operate primarily on absolute values assuming they shouldn’t change. Especially with respect to budgets and time. In fact, variance or change is almost a dirty word in that context. Today’s markets, especially in the context of new product development, are different from the heyday of cost accounting in the 1950s. Back then, you could easily assume stable incremental growth in demand in many markets for 15-20 years. And actually be right. Usually you wanted to establish a budget to control unnecessary costs, because everything else was likely to stay the same.

px Finite difference method svg cffefecaa