Estimates are crack cocaine for managers and executives. Estimates are an emotional crutch that give managers an illusion of control over an inherently risky process. As well as being an expensive waste, estimates can drive unhealthy behaviour in teams. The problem with estimates is that there are different forms of estimate that are appropriate for different decisions and different levels of understanding, and unfortunately managers do not always use the appropriate estimates.
To begin with, an estimate is a “tea bag”, it is part of the solution needed to create an output for the manager. The output is required to satisfy a need for the manager. There are two outputs from an estimation process:
- How much will it cost?
- When will it finish?
In actual fact, you need to answer the second question before you answer the first.
The cost of an IT investment is the duration of the investment multiplied the day rate of the investment. Consider a team of eight developers who spend half of their time on support and half on new investments. In addition to the developers are a product owner, UX, managers, etc. amounting to a further sixteen head count on the project. If the investment is estimated to run for a further thirty days, then the cost will be 30 * 20 person days = 600 person days. i.e. All the people involved in the investment rather than just the four developers. That is why an understanding of Theory of Constraints is so important to managers, especially those who have been trained in cost optimisation which tends to create constraints and increase costs.
So the cost of the based on the duration of the investment, or rather, when will it finish. How do we determine that? There are two measures we can use:
- Count of stories
- Story Points
According to the work of Todd Little and Troy Magennis, there is little extra accuracy using story points over the count of stories. However there is a real advantage of the story count as it is available much earlier in the process than the story points. A reasonable estimate of the count of stories is normally available very early on in the process. Story point estimates are normally available about a sprint before the story is developed. That is, after the product owner has worked with the UX and Data dudes in the Product Owner Squad to create the prototypes, test them with users, write the gherkin format acceptance criteria and then slice the epic into stories that can be estimated by the team in the product refinement session. The alternative is to create a whole bunch of pure waste to create faux story point estimates much earlier than they should be available. Faux story point estimates that are not following the normal process and are thus not really story point estimates, and provide a false level of comfort.
Estimating when an investment will finish is best done by looking at a Cumulative Flow Diagram and projecting based on the rate stories are being created and completed. Consider the following Cumulative Flow Diagram. The black dotted line is “now” and the red dotted line is when the investment is due to finish. The red dotted line is calculated by projecting the rate at which stories will be created and finished. (Troy Magennis’s “Cool and Fabulous Spreadsheet” does something similar with numbers instead lines, and in a much more grown up way using statistical confidence intervals).
A week later, the manager looks at the CFD and sees that the estimated finish date has moved because the rate at which the stories have completed has increased. This may be because the amount of support work has reduced due to higher code quality, or because the team are improving and going faster. The manager can investigate and potentially identify changes to the process to improve the estimate of the number of stories.
A week later (in an alternate reality), the manager looks at the CFD and sees that the estimated finish date has moved because the number of stories has increased. This may be due to unforeseen functionality, or because a story was more complicated than expected and needed to be broken down. The manager can investigate and potentially identify changes to the process to improve the rate of delivery of stories.
A week later (in yet alternate reality), the manager looks at the CFD and sees that the estimated finish date has not moved because the number of stories has increased but that has been offset by an increase in the rate that stories are completed.
As well as understanding that there is a change in the estimated finish date, the manager can understand the source of the uncertainty that has created the change.
Now lets consider the same graphs drawn using story points. To begin with, we need to get some “experts” in room to estimate all of the stories, because we would not want to get the whole team to give an estimate as that costs too much. That means we have guess what the stories will be (before user testing and all that). Our “experts” will only be too pleased to provide a story point estimate, even though it is not a story point estimate. It is an expert “estimate” rather than one by the team doing the work, and it is not based on a “Definition of Ready” story, its a faux story point estimate. This means at some point during product refinement, the estimate will change to a real story point estimate and we will have some stories estimated using story points and some using faux story points. This mixture of story point methods is misleading and as a result, the manager will pressure the product owner to complete all the stories to a state of “definition of ready” so that they can get a reasonable estimate. There is no sensible* intervention that a manager can do to improve the quality of the estimates.
( * We could adopt a waterfall approach to creating stories. In order to improve the estimates, we could increase the risk and cost to the organisation substantially which isn’t really sensible ).
Even worse, there are now three reasons for a change in the estimated finish date, as well as the number of stories created and completed, we now need a third dimension for the change in story point estimates. In other words, to understand what has happened, the manager needs to study two graphs** and the difference between them. What was obvious on the CFD, has now become complicated to understand and needs an “expert” consultant to explain. This lack of transparency leads the manager to fail the Agile Risk Framework Audit that requires them to know and understand what is going on in their investment.
( ** I tried to think how you would represent in a useful way this but gave up as it was too complicated, and it was silly anyway)
One day, the manager has a revelation! Instead of using faux story point stuff to estimate the finish date (and hence cost), lets encourage the product owners to write stories of a broadly similar size. From now on, the largest story allowed will be an “eight” (or a “five” for teams with one week sprints). No more stupidly massive “thirteen” or “twenty” stories. That will mean I can get my estimates for free without any stupid and wasteful “faux story point” estimation sessions.
So if you consider yourself a competent manager, go read Todd Little and Troy Magennis stuff on estimation and prediction.
Todd Little Key Papers
The IT Risk Manager 
December 29th, 2016 at 2:30 pm
Hi Chris! Nice article. One problem I’ve seen with just the PO creating the stories is sometimes they don’t know if they’ve created a 13 / 20 story – they need the eng team’s input to figure that out. Is that just a risk you take, or would you assume some early tech lead / eng team input at least, even if its not to point-level estimation?
December 29th, 2016 at 3:56 pm
Hi Mike, If the PO is creating stories that turn out to be too big (13s/20s), this will be evident in the CFD from the increase in stories being created. As such, someone might suggest an intervention such as a developer (and tester if you have such separate beasties) join the product owner squad to help generate the stories in the first place. Personally, I would expect the PO to be working with a developer (and tester) to break down the stories anyway. I would not expect the whole development team to be involved in breaking them down but I would expect the whole team is needed to give the story point estimates (based on a story that meets definition of ready).
December 29th, 2016 at 4:02 pm
Cool, that’s what I would expect too.
December 30th, 2016 at 11:15 am
I’m also a big fan of just counting the cards, but I’d still like to see more explicit mention of engagement with the technical team. I see teams where this production-line approach to creating and breaking up features misses critical feedback (and challenge) from the development team. It’s a good approach for consultancies that can just assign people to plug-compatible roles, but for in-house teams it’s less than the sum of the parts.
December 30th, 2016 at 12:08 pm
Hi Steve
I was assuming proper engagement between the product squad and the development team, i.e. that one or more members of the development team were involved in product decisions.
The post was really aimed at people who need an estimate, and the fact that studying CFDs of story count (or using Troy’s spreadsheet) was better for satisfying their needs than using estimates and story points. Further trying to help them understand that story points make things harder when you are looking beyond the team because they make things extra complicated.
Chris
December 30th, 2016 at 5:08 pm
I am all up for #NoEstimates, but some businesses heavily depend on some sort of high level estimates to help them prioritise work for their customers. Usually T-Shirt sizing is good enough in most cases.
Once the work is correctly prioritised, the estimates/size of each story doesn’t really matter, does it. It needs to be delivered regardless of it being a 5, 8 or 13.
January 2nd, 2017 at 10:47 am
Hi Iqbal
From my experience, product owners have never prioritised using story points or T-Shirt sizing. The story point estimates are for the benefit of the team so that they can decide how much to commit to during the current sprint. In reality, product owners prioritise based on the value that they think a story will deliver. If you look at Todd’s paper you will see that the relationship between the actual cost and the estimated cost is a log normal (Wiebel) distribution with an average of double and four fold in ten percent of the cases. That means from a cost perspective, a 5, 8 and 13 are the pretty much the same. On the value side, actuals versus estimate follow a power law distribution. I have worked on a project where the value was out by three orders of magnitude (my cost estimate was spot on because I had doubled it). So the value is where all the risk is but the cost estimate is where all the focus is. A bit like the drunk looking for his keys under the street lamp when he dropped them in a field because it easier to see under the light.
The business need for estimates is a cultural, organisational and funding problem as it assumes a project (funding per item) focus rather than the funding of long lived teams. Imagine the situation where 99% of the effort is in one team and 1% is in a team that is a constraint. Now imagine that the item is way down the organisational backlog. The problem isn’t budget, its getting access to the constraint. However if we spend 99% of the budget, the business then screams “Sunk Cost Fallacy” at everyone until the get their own way and the organisation invests in the wrong thing.
If the organisation does need an estimate, they can get a SWAG (Sweet Wild Assed Guess) from each team (which is not a commitment on the team) and calculate the cost accordingly. It does not mean they should get a green light on the investment because the investment needs to prioritised alongside other investments at the constraints. Just because the investment has the highest return, does not mean that it is the right investment. Consider the situation where an investment will result in one million new customers which is ten times the current number of customers (100,000). However churn is currently 90% and reducing churn would only increase the number of customers by 90,000. Investing in new customers is pointless until you have fixed the churn problem.
Does this help?
Chris
January 2nd, 2017 at 5:22 am
[…] All Estimates are Waste, some are useful Written by: Chris Matts […]
August 14th, 2017 at 8:40 am
[…] Great article here by Chris Matts, All Estimates are Waste, some are useful. […]
April 15th, 2018 at 5:47 pm
[…] Story Points are crack cocaine for managers who want to believe in a world where they have more accuracy than they really do. Story points, if used at all, should only be used by the development team and the product owner to communicate relative effort. In order to create a story point estimate, the product owner must present a story to the development team that meets the definition of ready for the team. Typically this occurs one or two sprints before the team work on the story. Often the definition of ready includes the product owner taking the story through a “Three Amigoes” session with a developer and tester. This is after the product owner has detailed the story including acceptance criteria, often in the Given-When-Then format. In other words, a large amount of effort is normally required before a story is ready for the team to give a story point estimate. […]
July 1st, 2018 at 3:56 am
[…] Story Points are crack cocaine for managers who want to believe in a world where they have more accuracy than they really do. Story points, if used at all, should only be used by the development team and the product owner to communicate relative effort. In order to create a story point estimate, the product owner must present a story to the development team that meets the definition of ready for the team. Typically this occurs one or two sprints before the team work on the story. Often the definition of ready includes the product owner taking the story through a “Three Amigoes” session with a developer and tester. This is after the product owner has detailed the story including acceptance criteria, often in the Given-When-Then format. In other words, a large amount of effort is normally required before a story is ready for the team to give a story point estimate. […]