How to analyze the impact of velocity on your release date

While “agilefall” has many well documented downsides, I’ve found a counterintuitive bright side to the following aspect of it: “We have a product backlog with priorities, but we start working on a release with a long list of features already committed to the business.”
In this case, we have a pretty detailed view of scope as well as developer estimates of complexity. There is a lot of data to crunch.
From a pure business perspective, the key variable that matters is the release date. Dates have a significant implication for the rollout across the company and existing client base. Here’s a few business reasons why:
  • marketing and sales plans need to be made (since they’re separate “workstreams”)
  • overall costs and ROI of the program changes
  • coordination and resources need to be shared/reassigned with other internal teams
And most importantly, the date is completely non-technical. Anyone who graduated from primary school can understand it. So the nice thing is that the release dates help focus the discussion on what matters, while only needing to go into a minimal amount of technical detail.
Making the relatively liberal assumption that scope will not change, it’s possible to generate a pretty decent forecast of the date.
Moreover, it’s possible to model the impact of a higher or lower average future velocity on release dates. In other words, if there is a need for a plan, you can make the entire plan using the following approach:
  • dependent variables: incremental release dates
  • independent variable: hypothetical velocity
And then look at what happens to the release dates as you vary velocity.
unsplash image feeefeedbedb
Photographer: Waldemar Brandt | Source: Unsplash
The entire plan fees like an accordion which can be pulled out or pushed in, depending on your assumptions about velocity. If velocity is high, the releases are close together. If it’s low, they’re quite spread apart.
Here’s an example of a quick model I did by exporting my data from Jira and using Excel’s scenario manager:
velocitysensitivityanalysis ebeebaccdfcbcc
Type caption (optional)
 
By varying that one number and holding scope & team constant, the dates which are hit for each release change significantly. Let’s get into how to do it for yourself first.

Case Study

We’ve defined, broken out and even estimated scope for this program. It contains hundreds of stories (tasks really) related to the delivery of a big infrastructure. All of this is being tracked using a standard agile tool called Jira, with relevant details attached to each story.
Here’s the type of analysis we’ll need to do as an interim step:
versions ecbcfaacfac
Type caption (optional)
 
The unit of effort measurement here is the “Story Point“, an abstract measure of relative complexity of a piece of work. This is quite common in agile circles, to help give the delivery team a safe way to discuss and size the work, without biasing it towards what stakeholders “want to hear” and to help them “save face” if a complex task takes a long time (even though it wouldn’t necessarily be obvious to a non-technical person).
The key “number behind the number” here is the velocity. It’s expressed as the story points completed per unit of time. In this case: one week. Velocity tracks increments of fully complete work that has been specified, developed, checked, and signed off.
If we estimate the story points for each task up front, it’s possible to observe velocity as the team works on the scope. Basically, you see performance as the change in that number over time.
On this project, we are tracking velocity in weekly increments. This gives us more data points and increases our confidence in the numbers faster than if we were calculating it on a monthly basis for example. More observations, higher confidence–statistically speaking.
In the example above, the actual team doing the work has been delivering at a rate of 27 story points per week. But what if the average future velocity was higher or lower than that?

Step 1: Pull out the scope sizes from Jira

For me, the fastest way to get to the real numbers is to pull up reports. At the moment, most of the modelling we are doing is based on the Jira versions (fixVersion). Previously, I’ve used Jira epics using different reports.
reports fdcafaaccead
reports on left sidebar
Then choose the version or epic report:
versionreport eedbcafeaaada
here in a drop-down
And finally pull out the actual story points completed in the version:
completedstorypoints feefcbbeeffe
Type caption (optional)
 
And incomplete:
incompleteSP dfebbcbfbfde
Type caption (optional)
 
And finally the number (count) of unestimated stories, as delivery teams tend to push back on spending too much time estimating rather than doing the work:
unestimatedissues cdceebaeecdaddcc
Type caption (optional)
 
Remember to exclude any known bugs as these will typically have 0 story points anyway, and use your judgement for other issues.
Type these into Excel (yay data entry!):
inexcel step befadefad
typing in numbers from that report

Step 2: Create a key assumptions section

Hah! The ‘A’ word.
In your spreadsheet, put in an assumed value for the unestimated stories as the average size (in story points) of issues and the expected velocity for your delivery teams:
aumptions afbacaafcfcde
Type caption (optional)
 

Step 3: Fill out the remaining formulas to estimate your release date

Essentially, once we have the estimated total scope and velocity, it’s just a matter of dividing the former by the latter, to generate the number of weeks of work remaining:
firstversionestimate dfffdbddbadb
Type caption (optional)
 
Here are the actual excel formulas I used to generate the magic date:
firstversionestimate formulas cffadbfeaacfafdfb
Type caption (optional)
 

Step 4: Rinse and repeat for remaining versions on your backlog

Essentially, you are holding the team, scope, and velocity constant and using your estimates to figure out when you will be ready to release each version. This is, of course, based on what you know now, which is subject to change. 🙂
versions bfdbccacbaaaa
Type caption (optional)
 
The formulas are more or less the same throughout the versions. In this case, I’m assuming the team will finish version 1 and move immediately to working on version 2, so column I needed some adjustment.
versions formulas eccfbcdeceddd
Type caption (optional)
 

Step 5: Kick up the What If analysis using Excel’s scenario planner

First move your cursor to the variable you want to model, in this case your velocity assumption:
movecursor ecdbcaccdadcf
Type caption (optional)
 
This is the “independent” variable in statistical terms that you want to vary, in order to see how it affect the dependent variables you care about: the completion dates.
whatifanalysis dddadefdc
Type caption (optional)
 
This is probably one of Excel’s most powerful non-quantitative easter eggs. While allowing you to vary individual values, you can extrapolate impacts that reach far beyond what is possible (for most humans) intuitively.
selectscenariomanager afdbfdadeffacaaefcb
Type caption (optional)
 
Within there, choose the Scenario Manager, and the following comes up:
blankscenariomanager defdffc
Type caption (optional)
 
Click add, and then use the cell you selected previously as the main variable to vary:
baselineveloctity ddcbeafeed
Type caption (optional)
 
You can call the scenario whatever you want that has business meaning to you and your company. Then enter a value for this scenario in the next prompt:
enterscenariovalue ceafaba
Type caption (optional)
 
And then do this again for a few other scenarios. In this case, I created a higher velocity at 60 and a lower velocity at 20. This range is so wide, simply because I have no data to go on, so I’d like to explore the optimistic and the pessimistic scenarios.
When you have all of this done, and the scenario manager is correctly set up, click on summary:
scenarios bdaacdfdb
Type caption (optional)
 
Based on that, Excel asks you which cells in the sheet you actually care about when you are varying scenarios (the dependent variables):
dependentvariables dbcebddbddc
Type caption (optional)
 
And after you hit OK, Excel hangs for a bit and returns with the following magical table:
scenariosummary bfcfdabfedefb
Type caption (optional)
 
This gives you a sense of how different average velocities will affect your delivery dates and plan. It gives you a qualitative feel for what would happen anywhere within your range, too, by looking at the outer boundaries. It’s enough for you to take and present to stakeholders in terms of what the velocity number actually means, what tradeoffs might need to be adjusted, what resources added, and all of the other things which might need to be changed to influence velocity.
Of course, you’d want to make it a bit prettier and clarify what these things mean:
velocitysensitivityanalysis aacdeedcbdeb
Type caption (optional)
 
Ok, now you’re talkin’! So we are finally at the velocity accordion state. Depending on what velocity we end up getting out of the team, these are the dates we expect to hit for each incremental release. As you can see, the difference between doing the entire scope at a velocity of 20 vs 60 is over an entire year out. Even if all of the details of what is delivered are exactly the same, done in exactly the same order, by exactly the same people, there more than an entire year of difference in the final release.
In other words, detailed planning this far into the future should not include commitments to dates–even if you do have your entire backlog defined and specified and you’re hoping it’s “accurate”.
Figuring out your velocity and how to increase it has much greater business value (bringing release dates in) rather than trying to specify and estimate everything up front to the utmost detail.
This is particularly true in a corporate context with large budgets and a need for “saving face” in case the product isn’t fully perfect before it’s even discussed with prospects or existing customers. It’s much better to commit to small increments of work that are then tied to specific customers needs, as that is the shortest path to revenue in an enterprise sales environment. On top of that, focus on managing velocity and not detailed planning, as that will get your product out there faster. Ultimately your customers could care less how detailed your release plan is; they just want a product that addresses their needs.

Keep following the story

I’ve written a book on speeding up innovation. Check it out over here.

How to use metrics to manage delivery with a human touch

At one client site, the CTO from the head office wanted to introduce metrics to help him monitor what was going on across all of the new product initiatives. Like the other delivery managers at the time, I understood his need. But when I opened up the first spreadsheet defining the template I needed to report to–my jaw dropped.
It seemed like most of the numbers were about compliance to the centralized directives, particularly with respect to timeliness. With very little focus on whether or not what we were doing will actually make the company any money. If we were successful at hitting those timelines, we wouldn’t necessarily make anything until those delivery dates were hit.
Months and years away.
unsplash image dbfcdbcbeb
Photographer: Aadhar Sharma | Source: Unsplash
Not everyone likes numbers. I get that. Once you grow a program beyond just one delivery team, though, the firehose of information coming at you becomes overwhelming. So it’s perfectly natural to start looking to numbers to help figure out what needs attention.
As a leader, just like this CTO, you have a very simple concern: anything that goes wrong will land in your lap. So the more proactive you are, the fewer crises you deal with. And your calendar stays in your hands.
The flip side, though, is if you try to manage this from a “control” and “risk avoidance” mindset, you’ll stifle the ability for your team to respond and learn. Even more so if you’re only focusing on schedule risks. If your schedule is at risk, it’s likely you have too much scope and not enough client interaction to drive product development.
So the below insights are the most useful that I have read or have discovered myself when choosing or constructing operating metrics around new product development.

1. Information and data is everywhere. Any given data point has low value. In contrast, the ability to contextualize data has enormous value.

The amount of potential data doubles expands exponentially, making it increasingly more valuable to find signal in all that noise. In other words, interpreting data requires structure. That structure requires context. And interpreting context requires wisdom.
    • Just because there is a lot of data, doesn’t mean you can pull up a meaningful table using exactly the summary numbers you need.
    • And even if you have that table, you need to be able to interpret it subjectively.
    • Not every number exists to be added, aggregated, maximized or minimized numerically and you need to know the real-world meaning of every number.
    • And even if you do know this real-world meaning, being able use discernment and wisdom to interpret it usefully.
For example, I had this problem before I had taken accounting classes during my Masters in Finance degree. At the time, I was poring over financial statements of hundreds of dotcom Internet companies. The truth was there were lots of numbers to look at, but I was kind of guessing what they meant at the time. I could focus on just net profits and work back from there. Comparing lots of net profit numbers across companies across time would be a form of structured data as a table. Maybe useful. It kind of depended what was behind the number. It also varied significantly, as the real world meaning was quite varied across the hundreds of “dotcom”. And finally, at the time I didn’t have enough wisdom and discernment to know that actually looking at net profts for early stage companies was just premature. What actually mattered was learning and operational progress.
For new products, the key context is whether you are learning something useful as that will shorten your path to profitability (even if that learning process is initially a net loss at first in pure cash flow terms). In my opinion nowadays, “will the project deliverables arrive on time?” is the wrong question to be asking. They have to arrive (or at least be defined) early enough for a prospect to want to pay for it. Yes, dates are important, just let’s keep them in the right context.
This goes much deeper. My friend Perry Marshall has said that the #1 success skill in the 21th century (one that almost nobody talks about) is discernment. Identifying the “vital few” factors that actually matter. The ability to discern is based on wisdom. Applying timeless principles to your specific situation. Building to last, not just hitting next quarter earnings targets.

2. The high bar for metrics: what decision would this metric help me make? And under what circumstances?

I interned in IT at GMAC mortgage, General Motors’ home mortgage subsidiary, while still in college in the mid-1990s. My boss at the time had me look into setting up the first ever company webpage and intranet. I had a late night discussion with my boss at the time about whether or not we should have a “hit counter” displayed directly on the site. This was a pre-built component displayed directly on the page, which went up by one each time somebody browsed to it.
The higher the hit counter was, the more popular that page or the company’s website looked. But in and of itself, having a higher number of hits didn’t mean anything. Certainly not to us. It wasn’t a number that helped make a decision or indicate a change of tactics was needed. What mattered was who was coming to the page, why, and whether they were achieving what they wanted.
web analytics   cebaccfdaccbebc
source: Avinash Kaushik
This nugget of wisdom came from Avinash Kaushik’s book Web Analytics 2.0. The example he used was website hits. Yet his point goes much deeper than just interpreting website traffic. If you are tracking a operating metric that doesn’t help you or your organisation make a decision of some kind, then it’s just not useful.
There’s lots of numbers you could be tracking. Just log into your company’s various systems to get a taste: sales pipelines, ticket tracking, financial metrics. But remember you will get the most value by identifying the vital few that actually matter most to your company. Which by definition are specific to your company. They will require some deeper discussion to get right. And effort at aligning everyone in the company around them.

3. The GPA effect

In the mad rush to the real world, most of my fellow students competed for the most prestigious jobs based almost solely on their grade point average (GPA). The conventional wisdom applied by investment banks and consulting firms was that a high GPA meant that you were smart and hard working. You stayed up and studied, so that you earned good grades, and therefore you’d be reliable.
Ultimately, for the most popular gigs, GPA became a first sorting mechanism. As recent grads, we were a commodity on the job market. If hundreds of students applied for the same investment banking analyst position, then the banks only interviewed the people with the highest GPAs.
In practice, though, this was a classic example of a heuristic which had a lot of potential faults:
    1. Did someone with a 3.94 GPA show more promise than a 3.92 GPA in a role that largely involved the ability crunch spreadsheets for 50 hours straight?
    2. What about relative strengths and preferences? Would someone gregarious and social with a slightly lower GPA really be a worse salesperson at a bank?
    3. Didn’t this skew even further towards people already coming from privileged and wealthy backgrounds who had benefited from and could continue to afford nearly infinite tutoring? Didn’t this reduce the diversity of new hires?
    4. Wouldn’t this skew towards people with a perfectionist streak? And reward them for high compliance? Always looking for the “right” answer in an academic context doesn’t translate that well in all business context (such as new product development for example).
In contrast, there were guys on the crew team which clearly only had crew and beer on their mind. So a low GPA was just a lagging indicator of their relative lack of maturity more than anything else. And in this case, it was probably a useful metric for hiring purposes, particularly as a way to identify bad choices.
gpagraph faedafbb
source: New York Times
So GPA was useful as a metric to help prevent a complete blowup in a hiring scenario, but downright dangerous as the only data point that matters.
I suspect the same is true of many operating metrics in a larger business. Use them to highlight problems or potential problems, but beware of optimizing for them to the exclusion of all else.
For example: optimizing for velocity at all costs. You can cut corners with quality. You can run your team into the ground. You can create lots of badly documented features. But you went fast, right?
Sadly, the most common reason for new product failure isn’t missing a timing window. It’s building something the market doesn’t want or care about. In which case, pursing high velocity for its own sake is a fool’s errand.
That said, having very low velocity isn’t good either. It means it’s time to go digging and figuring out why the teams are struggling. And something needs to change or to be fixed.

4. When there is still too much data, give the most weight to data around your first order priorities.

If there is too much data, revisit your goals and priorities to figure out which metrics to pay attention to. Quite often operating metrics are unclear or not useful, simply because the stated goals are too nebulous or inaccurate to be useful. Or each stakeholder has their own goal(s) to achieve and therefore it’s difficult to agree on a common set of metrics.
Quotefancy  aeafccdaafddba
source: Stephen Covey
Reid Hoffman, founder and scaler of LinkedIn, had a much more eloquent way of articulating this concept:
“When there is too much potential data… What are the key things that would change my decision? What is the data around my first order priorities? After you have some idea, then you consider the impacts for 2nd and 3rd order priorities.” — Reid Hoffman on Jordan Harbinger’s podcast
If your company has clarity and alignment on goals, then it should be clear what data sources to pay attention to. Or to create.
This is particularly true when you need to narrow your focus on what works in order to scale an effort. Figure out what numbers are the most meaningful, and then optimize for that. If that means re-assessing the strategic tradeoffs you’re making, then do that.

5. Accept that metrics on their own are not the full story, and they never will be.

The metrics themselves are not the story. While they help you hone in on the the underlying mechanics of delivery system, this system is just an abstract structure.
In fact, the metrics only matter to help drive conversations to pull out the full story. In other words, they aren’t a replacement for a conversation. But they are best used a trigger for a conversation. An early warning sign, to be able to act preemptively if needed. And also to provide feedback that might not be apparent right on the front line.
For example, when running a bigger team, it’s hard to keep track of what everyone is doing. It’s hard not to like everyone involved. But when you look at output metrics, even if they aren’t perfect, it gives you an alternate view of the performance. While not perfect, even something as simple as keeping track of number of commits for software developers over time will at least highlight anyone who isn’t putting in the work. Which is still only the basis to start a qualitative investigation. But it gives you something concrete to work with.
As an introvert, I like using numbers to track what’s going on. I feel better about this than just using a subjective or intuitive approach, which doesn’t scale up that well for me.
But…
Empathy will always supersede any metric, because people underlie the system. For example, life happens outside work. A great developer on my team had a cancer surgery in the family. It was enough to knock him off his feet, for example. But after he had the time to attend to that rough patch, this person came back with stronger performance for a while. All this was reflected in the metrics, both the down and the up. So trying to act on the fall of the metric would have been premature and shortsighted. And lacking in empathy.

Aiming for self-accountability at work

All that said, i think the right balance is achieved when there is a system for self-accountability at work, which everyone buys into. From the front line busy bee to the senior decision-maker. This would be the ideal culture. Of course, it is much easier said than done.
A good set of metrics prevents you from having a blame culture on one extreme, an abdication culture on the other. It requires a good amount of self-awareness as an organisation, which is often the hard part. But it’s totally worth it if you want to keep everyone involved engaged and happy.

Why delivery velocity is a broken metric

“For by the ultimate velocity is meant that, with which the body is moved, neither before it arrives at its last place, when the motion ceases nor after but at the very instant when it arrives… the ultimate ratio of evanescent quantities is to be understood, the ratio of quantities not before they vanish, not after, but with which they vanish” –Principia Mathematica by Isaac Newton

I’m not a mathematician, but I’m not afraid of numbers. I’ve drifted through the standard high school and college math requirements, admittedly with curiosity. And I’ve hung around the hedge fund industry, software engineers, and actually trained mathematicians for long enough that math’s kind of rubbed off on me. Yet my skillset is probably best described as a mix of product and, more recently, delivery management with an academic foundation in accounting.

Over time, I’ve noticed that cost accounting and traditional/waterfall project management operate primarily on absolute values assuming they shouldn’t change. Especially with respect to budgets and time. In fact, variance or change is almost a dirty word in that context. Today’s markets, especially in the context of new product development, are different from the heyday of cost accounting in the 1950s. Back then, you could easily assume stable incremental growth in demand in many markets for 15-20 years. And actually be right. Usually you wanted to establish a budget to control unnecessary costs, because everything else was likely to stay the same.

px Finite difference method svg cffefecaa

Most choices in a big company don’t have the same implied cost of time. The value of time is assumed to be incalculable, more philosophical than practical. And it is, if you are stuck using “absolute” values.

Yet you can value time and derive implications for velocity if you are using finite difference methods. Thinking in relative terms enables you to think of relative profitability increases (in the moment) rather than always using an annual or quarterly budget yardstick determined much earlier, when you knew less and which is often likely to be out of date.

An invitation

I am just discovering the outer edges of this at the moment. Please let me know if you have any suggestions or feedback, and sign up below if you’d like to follow me as I share more on this topic.

Keep up with me by signing up over here for updates about my upcoming book on adaptive innovation.

How to create an actionable client profile

daniel day lewis launch tomorrow

Daniel Day Lewis, method acting maestro

He’s the first man to ever win three Oscars. Daniel Day Lewis, that is.

For the entire filming of My Left Foot, he didn’t leave his wheelchair, sound coherent, or even feed himself. For Last of the Mohicans he became a survivalist. He lived off the land. For In the Name of the Father, he lived in a prison cell. He starved himself. He asked the cast to insult and abuse him.

When playing Abraham Lincoln, he even signed his texts “A.”

Daniel’s style of acting, called method acting, expects him to become his character. To live in their skin. Which is a potent skill to have as a marketer.

Why?

[Read more…]