Billy Beane, the manager of the Oakland A’s baseball team was in trouble. After a successful 2001 season he was losing three of his best players but he had no money to replace them. As Plato said: “Necessity is the mother of invention”. With his back to the wall, rather than rely on the instinct and gut-feel of experienced scouts, for the 2002 draft Beane took a radically different approach – one which was ridiculed by the experts. They put together some objective numbers based on real statistics of what players actually did. Then they picked players based on the estimated value they would bring to the team – ignoring the gut-feel driven advice of the experienced scouts. Everybody thought this was crazy.
It took some time for the new team of misfits and has-beens to fit together, and not everyone was on board. The new system required tuning and coaching to get the best out of them, but the outcome was better than anyone expected: despite having one of the lowest salary budget’s in the baseball league, the Oakland A’s went on an incredible 20-game winning streak. Seeing the value of this approach, other teams quickly changed the way they picked their players to match the system Beane had introduced. All MLB teams now make more use of objective data and science in the way they manage their teams. This same approach is increasingly being used in other sports, like the NBA – where it was once believed this approach wouldn’t work. Until, of course, it did.
Unfortunately, the software world is firmly entrenched in the old system – where gut-feel about value is what primarily drives decisions. Scrum (the most popular framework for developing software) reinforces the gut-feel approach by anointing the “Product Owner” as the person responsible for decisions about what gets built. Some enlightened Product Owners seek a more data-driven approach, but it’s safe to say that this is the exception rather than the rule.
‘I’ll just write my numerical estimate of business value on the story card then’ – no product owner, ever #agile
— Peter Hilton (@PeterHilton) August 21, 2014
To make better prioritization decisions and make trade-offs more visible we also need to estimate value. The subjective approach to value works well if you have a small team and a charismatic leader who can articulate their vision and make all the necessary trade-offs. If you’ve got lots of teams with lots of demand coming from multiple fronts you’ll probably find that this breaks down fairly quickly. Making value estimates is hard, but it is ultimately worthwhile. Having a number is better than having no number at all, even if the only output is to learn more about what is valuable or not. In a similar way to the Oakland A’s, we should perhaps question the conventional wisdom that the best way to prioritize is by deferring to a single person.
Patterns of resistance
Beane found significant resistance to his ideas from people with a fixed-mindset. It is our prejudices and perceptions, fears and uncertainties that stop us from seeing what might be. We tell ourselves it can’t be done and so we don’t even bother. We also preach these beliefs to others, with convincing arguments based mostly on anecdotes and FUD. Usually the reaction is a learned behaviour, the result of cost and date estimates being badly abused and turned into contracts. They expect the same will happen with value estimates. I understand this, but I’ve also experienced first-hand lots of situations where estimates are simply used as a rough aid to help make difficult prioritisation and trade-off decisions.
No matter how we dress up the problem, we simply can’t have it all – and at the very least we have to decide where to start. For this, I can’t think of an alternative to some form of estimation. How else are we to make sensible decisions? Eventually, the nay-sayers can be convinced if they see the change not only works, but that it changes the conversation. Rather than obsessing about cost, instead people are focused more on the value side – the very place where most of the variability and uncertainty lies. And if you don’t?
In the absence of information about Value, of course the system optimises for other things. Why should this surprise anyone? — Joshua J. Arnold (@joshuajames) January 5, 2014
Perhaps there is a better way?
Maybe, instead of spending inordinate amounts of time and effort obsessing about cost estimates, perhaps we could change the focus of the conversation. Maybe we could get people talking about and comparing notes and surfacing assumptions about value and urgency and using Cost of Delay to improve decision-making. What may surprise you is that the transition actually comes quite naturally to those who are used to talking mostly about cost and dates.
Cost of Delay is a wonderful head fake. It gets people who normally talk mostly about cost, talking about value. Changes the conversation — Joshua J. Arnold (@joshuajames) January 5, 2014
Even better though: by expressing value in terms of “Cost of Delay” we are actually tapping in to people’s natural bias towards loss aversion and using that to talk about value and urgency. Just as it was for Billy Beane, the hard part is taking on the romantic belief that value is “intangible” – that the benefits cannot be quantified, or in any way estimated. I completely agree that it would be wrong to focus primarily on profit or money. We are spending real money doing this though, and we are making trade-offs every day that cost real money. We seem perfectly happy to do this based on ‘gut-feel’, even though our gut-feel about the cost of delay is typically out by a factor of 50.
We simply have no business making the tradeoffs we do – which have real monetary impacts – without some understanding of the tradeoffs we are making. Focusing on delighting customers and end-users doesn’t mean we shouldn’t attempt to make these trade-offs measurable and consider the likely economic effects of our decisions. This is often presented as a false dichotomy. If we express value in dollars or some other measurable thing then we will become robots, blindly following a GPS and forgetting to look out the window.
This view fails to account for the fact that it is our System 1 that feeds our System 2, not the other way around. My view is that these things are orthogonal to one another. And it is possible: Douglas Hubbard wrote an interesting book called “How To Measure Anything: Finding the Value of Intangibles in Business. In it, he makes the following case for why we should push through the barriers and give it a go:
“Often, an important decision requires better knowledge of the alleged intangible, but when a [person] believes something to be immeasurable, attempts to measure it will not even be considered. As a result, decisions are less informed than they could be. The chance of error increases. Resources are misallocated, good ideas are rejected, and bad ideas are accepted. Money is wasted. In some cases life and health are put in jeopardy. The belief that some things — even very important things — might be impossible to measure is sand in the gears of the entire economy. Any important decision maker could benefit from learning that anything they really need to know is measurable.”
The issue, as with many things, is the presuppositions we carry with us all the time. It is our mindset, our belief system, that tells us it can’t be done. It is System 1 feeding System 2 with anecdotes where System 2 got it wrong. What it tends to heavily discount is all those times when System 1 was just as wrong. Retrospective coherence has a way of explaining after the fact why that was, and that this time it will be different.
It’s not that we intend to rely entirely on System 2, but that we know that our gut-feel alone is a terrible guide. It is AND, not OR. Ignoring what just a little analysis might reveal at least about where there key assumptions lie enables us to more quickly go and test those assumptions. This is the essence of Lean Startup: Build > Measure > Learn.
“Is a month of delay worth $1m or $1k? Approximately 85 percent of product developers cannot answer this simple question.” – @DReinertsen — Joshua J. Arnold (@joshuajames) July 28, 2014
“Since goals often conflict with one another we must express them in the same unit of measure to make tradeoffs between them” –@DReinertsen
— Joshua J. Arnold (@joshuajames) July 29, 2014
Changing this isn’t impossible, I’ve seen it many times. It just takes one person brave enough to get over the allergy, discard the romantic notion of value not being quantifiable, and willing to give it a go. Just like for the Oakland A’s, it may seem crazy at first – and it may take some time to see the effects – but it will be worth it.