SAFe and Weighted Shortest Job First (WSJF)

In 2012, when Dean Leffingwell launched the Scaled Agile Framework (SAFe) it was obvious the impact that Don Reinertsen’s teachings had on elements of the design. In particular, SAFe specifies Don’s recommended method for scheduling: Weighted Shortest Job First (WSJF). Whatever you think of the rest of SAFe, it really should be commended for encouraging organisations further along in a few specific areas:

  1. It’s fabulous that a much wider audience is being exposed to the importance of scheduling methods and that Weighted Shortest Job First (WSJF) will be more widely used for prioritisation and queue management. Perhaps organisations implementing SAFe will become less blind to queues, and the cost of queues in their organisation?
  2. SAFe also suggests that WSJF should be applied at feature level, which matches our experience. Perhaps when they start to see the distribution of value in their backlogs they will start to realise that large-batch projects are a terrible vehicle for developing software.
  3. SAFe also suggests using Cost of Delay as the weighting in WSJF. Perhaps this means that Product Owners will surface their assumptions about where the value lies, and design better experiments to test and learn about what is valuable, rather than relying on their gut-feel? Perhaps managers around the world will start talking about, and focus more on Cost of Delay and less on far less interesting things like productivity, velocity, and estimates of dates and cost. And perhaps teams will finally have a way of making vastly better trade-off decisions and more quickly discover, nuture and speed up the delivery of value.

I’m super excited about the last one in particular. In my experience, this is a really powerful lever. I’m sure Don agrees, as he could hardly have stated it more clearly when he said:

If you only quantify one thing, quantify the Cost of Delay.

In fact, this very quote of Don’s is at the top of the page where SAFe explains Cost of Delay, so I’m sure they’ve paid attention to it. Lets take a look at how SAFe explains and recommends that organisations go about quantifying the Cost of Delay. I’ve had some experience doing this in a number of settings, so I’m really interested to see how they approach it…

SAFe’s interpretation of Cost of Delay

SAFe teaches that Cost of Delay is made up of three primary elements, which add up to the Cost of Delay. Let’s look at each of these and I’ll do a little thinking-out-loud along the way:

User-Business Value: Do our users prefer this over that? What is the revenue impact on our business? Is there a potential penalty or other negative impact if we delay?

We are on fairly solid ground with “revenue impact”. (This maps well to the Increase Revenue and Protect Revenue buckets in the value framework we use to get people quantifying Cost of Delay). I’m not so sure about the users preference question. If you are talking about internal users (a common case in SAFe’s Target Market) this is often a fairly weak indicator of value. Asking if there is a potential penalty “if we delay” seems to overlap with the next parameter – but perhaps they are talking about fines, loss of license or ability to operate?  This could be more clear, I think.

Time Criticality: How does the user/business value decay over time? Is there a fixed deadline? Will they wait for us or move to another solution? What is current effect on customer satisfaction?

The first three questions are all great questions. I would put customer satisfaction under the previous heading of Business value. What’s not clear though is how the answers to these questions should be treated. Is a “fixed deadline” high criticality, or low? The way this should work is that the Cost of Delay for something that has a fixed deadline is initially zero – right up until the point when you need to have started it. After this point, the Cost of Delay is possibly all of the revenue related to that date. In many cases, leaving it until the option expires is too risky, so the Cost of Delay may ramp up some time beforehand to reflect the risk. All work should have some degree of urgency, or “time criticality” though. If there’s no urgency, you’re better off investing elsewhere. It’s not clear from this how SAFe treats time criticality – I’d love to see some examples of how they’ve actually used this model in the wild.

Risk Reduction-Opportunity Enablement Value: What else does this do for our business? Does it reduce the risk of this or future delivery? Is there value in the information we will receive? Will this feature enable new business opportunities?

Again, these are good questions, but they are all questions that are a subset of, or affect the likelihood of delivering value in my view. Surely it would be easier to simply add these to the first parameter? Reducing risk is primarily about avoiding cost or in some way affecting the asymmetry of the payoff function  We can also put a price-tag on information. (Some might even argue that all value is simply a filtration of a set of information). Enabling new business opportunities seems to be about Increasing Revenue. By treating these types of value as a separate thing to Business Value the risk is that they will be treated differently to “Business Value”. This seems convoluted and confused to me, but perhaps that’s just me.

Overall, in my view, the definition of these parameters could do with some simplification and clarification. I have a couple more questions though:

Why is it additive?

In my understanding, Cost of delay is a way of combining Value and Urgency. If either of these two parameters are zero then the Cost of Delay is zero. Adding “Time Criticality” to “Business Value” doesn’t make economic sense to me. I could in theory have something with zero business value, but highly time critical – and this would somehow score? The result of this is that using the SAFe algorithm will almost certainly give you a suboptimal scheduling of options.

Why Relative?

More worrying is the stance of “we needn’t worry about the absolute numbers”. I can only assume this decision was made by people with experience primarily in I.T. It is a shame that the Lean and Agile community seems to think that the primary purpose of Cost of Delay is for prioritisation. There is much more to it than that. Relative estimates simply don’t give you any of the potential benefits I hoped for above. Even very roughly quantified estimates (perhaps into fairly large ranges) would be better than this. The are a few other issues with relative estimates of value to consider:

  1. It’s very difficult to remember the thinking behind the relative positioning of more than 20 or so items. Applying Fibonacci doesn’t make sense to me either. For estimates of cost or effort we have an obvious optimism bias, which gets worse the larger the task primarily due to conflation. This is why the probability distribution of lead-time conforms to Weibull – it’s the conflation of uncertainties. There is no evidence (that I’ve seen) of this same skew applying to the value side of the equation though. Our overestimates tend to be balanced out by our underestimates – it is actually the Black Swans that dominate the value side of the equation. On what basis is Fibonacci used? (I fear this obsession with Fibonacci is turning us all into sons of simpletons).
  2. Relative Estimates don’t help much if you have more than one stakeholder. One person’s relative score of 2 is another person’s 8. If you’re managing three or four stakeholders then relative estimates becomes either an escalation to the HiPPO or a painful horse-trading exercise. In my experience, this is a huge pain point for most large organisations – which I believe is the target market for SAFe.
  3. Relative Estimates don’t help encourage us to surface assumptions. Actually quantifying the Cost of Delay has the wonderful effect of forcing us to switch into the more analytical System 2 thinking in order to question our gut-feel System 1. As we teach in the workshops we have run with various clients and at conferences, the point of this is not really the resulting number, although any quantified Cost of Delay is better than none. It is more about surfacing the assumptions and identifying the most uncertain part of the whole enterprise: “where is the value?”. Talking about, writing down and making visible our Cost of Delay estimates is like sunlight disinfecting our backlogs. It’s not that we are shooting for certainty, it’s that we should at the very least share those uncertainties far and wide so that people can design more effective experiments to discover whether the value is there or not.
  4. By hiding behind relative estimates we do nothing to change the focus of the conversation. Abdicating from quantification on the value side of the equation means that the focus tends to drift toward the duration or cost estimate side of the equation. As we all know, this is mostly a waste of everyone’s time and effort, not least of which is because of the information asymmetry, let alone the uncertainty.

As Jason Yip has already pointed out the key ingredient in all of this is the quantified Cost of Delay. Without it, most of the benefits evaporate. You simply can’t make trade-off decisions with relative value estimates. The cost of queues will still be invisible. The cost of large batches will still not be obvious. Assumptions about value will likely remain hidden. We will still be stuck negotiating estimates of dates and cost and obsessing over pointless things like “velocity”. In short, this is not the Cost of Delay I know.

In conclusion

As I said at the beginning of this, it’s great that SAFe has adopted WSJF and Cost of Delay. Unfortunately, it seems much of the potential positives I’ve listed above have been watered down or muddied – to the point where, in my opinion, we are mostly left with muddy water and very little substance. To be clear, I’m all for simplifying: I’ve been doing this for real with organisations and at conferences over the last 5 years. If SAFe really is interested in making its Framework better, then I’m sure they’ll reach out and ask for input to make this better – we’d be more than happy to help.

UPDATE: I’ve made the smallest, simplest suggestion to improve SAFe’s approach to Cost of Delay here.

If you’re interested, I’ve also published a Qualitative approach to Cost of Delay, for those who are afraid of numbers.

Comments 25

  1. Great post and thought provoker; in my experience many organisations are so far away from understanding value that doing a relative cost of delay is extremely challenging. There’s nothing in SAFe that says you shouldn’t go beyond relative CoD if you can. Sounds like a great place to be!

  2. “This is why the probability distribution of lead-time conforms to Weibull – it’s the conflation of uncertainties.” – early delays compond and make things later. There are more things against you than for you in delivering without any delays.

    Its hard to apply this thinking to value. Its hard to know which way value would be distributed. It would seem to me that value is boundless. Mostly additve, not compounding. I’m sure its got some exponential decay in the range estimates, but not the final value (if anybody cared to measure).

    Its a good question though. If we get a range of value estimates. What distribution should be used for risk calculations (if anyone does them). I know Log Normal is the general go-to, but i’m suspicious.

    Be good to have a historical dataset for $ value of features in a live product. Wonder if we can get some. My bet would be 80% revenue comes from 20% of th features. And WSJF is of course trying give us the greatest chance of releasing that 20% ASAP.

    Nice article.
    Troy

    1. Post
      Author
  3. Obviously I am a couple years late to the discussion, and while I’m not Dean or from Scaled Agile, Inc., full disclosure, I am a SAFe Program Consultant. Here’s my observation:

    The SAFe WSJF approach is intended to be lightweight and integrate well with the rest of the framework. The intent is for everything to “hang together” far more than have any specific aspect of SAFe be locally or otherwise independently optimized separate from the rest of the framework.
    SAFe uses story points (Fibonacci scale) as a proxy for duration and aims for a process that emphasizes speed and “good enough” ballpark estimates over deep precision. It’s rationale for this is that humans are better at estimating large things in contrast with other large things than estimating large things with much precision, and they want “just good enough” analysis performed “just in time”.
    As you detail in your post, SAFe uses Fibonacci scale for relative weighting of the components of CoD and makes it a simple additive approach, but it certainly does not preclude fine-tuning for your local context. Instead, it seems to me the intent is again aiming for general usefulness and broad applicability. For example, in my consulting with government customers who operate within the realm of “mission outcomes” as much or more so than “dollars and sense”, defining CoD and value is quite a different experience than when it’s a commercial firm’s profit-loss considerations, and I’m fully aware SAFe intends its approach to be generic enough to work in both contexts.

    In short, I would contend SAFe is designed to be shu-level prescriptive and attempts to avoid sacrificing “good enough” at the alter of “perfect” (not to mention “good enough” tends to be more generally applicable, while “perfect” tends to be more contextually dependent). I think (nay, I know) that this approach has proven successful in many different markets, customers, and contexts.

    Could teams use t-shirt sizes or some other non-Fibonacci way of estimating stories/features/epics? Perhaps — but keep in mind the overwhelming need for a common estimating approach across teams and levels for SAFe to work, so SAFe adopters who wish to use an alternate estimating methodology must account for that when deviating.

    Similarly, might certain customers benefit from managing their portfolios and prioritizing their feature implementation using more quantifiable CoD estimates than the additive, Fibonacci-based BV + TC + RR/OE formula? I’ve no doubt. But even in your own writings (http://blackswanfarming.com/qualitative-cost-delay/), you seem to endorse more relative and qualitative approaches (at least for shu-level implementation scenarios, no?). So why is the SAFe WSJF any less appropriate, applicable, or useful than e.g. your qualitative CoD matrix?

    Last but not least, other than Don Reinertsen himself, I’m not aware of a more potent, published (inclusive of internet blogging), and widely respected authority on the subject of CoD than yourself. In that vein, rather than waiting for Dean or someone else for SAI to reach out to you for suggestions on how to improve SAFe (and indirectly impugn their integrity if/when they don’t), why not go ahead and publish a post (or amend this one) explaining exactly what SAI should do to improve its framework? I’m quite sure they’d notice if/when you do. And maybe even formally respond. But unless and until you do, as they say, “everyone’s a critic”.

    1. Post
      Author

      Thanks for the comment, Andrew. Better late than never! I’m always happy to discuss, learn and help if that’s something you’re interested in.

      Let me attempt to respond. I may well be misunderstanding what you’re saying, so feel free to correct me:

      The SAFe WSJF approach is intended to be lightweight and integrate well with the rest of the framework.

      Lightweight is fine, but you seem to be asserting that the alternatives are heavy? What evidence do you have for this? This doesn’t match my experience, and I have helped lots of different organisations, public and private, to quantify Cost of Delay and use CD3 for scheduling across their portfolio. It’s true that it takes some time and effort to develop the basic patterns of value and urgency, but you don’t need a PhD in economics to do it. Nor does it take a lot of time. The fact is that there will be someone in the organisation who is already making assumptions about these numbers anyway – all you’re really doing is surfacing those assumptions and taking into account the effect of time. If numbers are just too scary though, I have published a qualitative model that at least makes sense.

      Integration with the rest of the framework is also fine, (I guess?). If SAFe is so reliant on using Fibonacci and relative estimates of value and duration for it to work though I worry about how safe SAFe really is? Making an entire framework reliant on a particular way of prioritisation would fit my definition of fragile. I don’t believe this is the case though. I think (and hope) that the need for integration is a red herring.

      In short, I would contend SAFe is designed to be shu-level prescriptive and attempts to avoid sacrificing “good enough” at the alter of “perfect” (not to mention “good enough” tends to be more generally applicable, while “perfect” tends to be more contextually dependent).

      My contention is that SAFe is being prescriptive with a model that is flawed. It is trivial to show examples (see above) where adding together fibonacci numbers as prescribed results in obviously poor results. I’m sure anyone using it is sensible enough to work around these flaws by either fudging or ignoring the prescription. If you’re going to be shu-level prescriptive, this requires mastery of the topic to determine the “one way” that the student will apply (safely) without needing to understand the underlying theory. I’m not seeing a mastery of the subject in this particular prescription.

      Could teams use t-shirt sizes or some other non-Fibonacci way of estimating stories/features/epics? Perhaps — but keep in mind the overwhelming need for a common estimating approach across teams and levels for SAFe to work, so SAFe adopters who wish to use an alternate estimating methodology must account for that when deviating.

      Again, if SAFe has an “overwhelming need for a common estimating approach across teams and levels for SAFe to work” then I really can’t see how SAFe is sustainable. It may work for a while, (while the consultants are still around?) but I expect the system will figure out some sort of fix that doesn’t rely on this. Not only is a common estimating approach unrealistic, it’s also quite unnecessary in my view. Why is the need so overwhelming? Again, I hope this a red herring?

      …you seem to endorse more relative and qualitative approaches (at least for shu-level implementation scenarios, no?). So why is the SAFe WSJF any less appropriate, applicable, or useful than e.g. your qualitative CoD matrix?

      Firstly, if you have read the link you’ve pointed to I’m sure you noticed that both the intro and the conclusion strongly encourage quantifying Cost of Delay. Cost of Delay is not just useful as an input to CD3 scheduling, it has a whole bunch of uses, which is why Don says it’s the “one thing” to quantify.

      Secondly, if you compare the basic logic of the two approaches, I’m hoping you can see for yourself the difference between the two? Even if you insist that quantifying Cost of Delay is hard, that doesn’t mean that you can ignore the underlying components and how they interact. True simplification is hard because it requires deep knowledge, understanding and experience. I’m not saying my qualitative model is E = mc^2 simple, but at least it makes logical sense! I also hope that it acts as a first step on the ladder towards a more useful quantification of Cost of Delay.

      why not go ahead and publish a post (or amend this one) explaining exactly what SAI should do to improve its framework?

      This post was a probe to see if anyone was interested. I think I’ve made it pretty clear what the problems are. Explaining exactly what they should do would preclude a conversation and (from my perspective) a better understanding of the design criteria that matters to SAFe. If they’re not interested in either a conversation or help to improve it that’s perfectly ok. It’s entirely possible that this just isn’t urgent or valuable enough for SAI :)

      But unless and until you do, as they say, “everyone’s a critic”.

      Indeed, and here you are critiquing my critique and somehow making this my problem to solve. What I’d love to hear is a response from SAI or a SAFe consultant that actually addresses the points I’ve raised, rather than explaining to me why they don’t matter. If they do matter, I’m more than happy to help make this part of SAFe more safe.

      As I said at the beginning, I’m always happy to discuss, learn and help if that’s something you or SAI are interested in…

      1. I have little interest in doing a point-counterpoint-countercounterpoint extended rebuttal/analysis. I’ll just what I think both you and Jason are overlooking with the SAFe approach.

        SAFe WSJF is strictly limited to prioritizing backlog items for implementation; it is not applied to e.g. economic calculations of whether to delay a release to incorporate a high-value feature. The goal is simply a tool so that product managers can select which features should go in the next Program Increment, or PPM can decide how to approve and prioritize large epics in the Portfolio backlog.
        SAFe accounts for the reality that most organizations do not use CoD, nor do they understand “business value” very well. This is different than traditional Scrum-based approaches (inclusive of LeSS), which tend to assume the PO can do this aspect of the PO’s job well (my personal experience lead me to characterize that as flawed optimism). This is why the SAFe WSJF formula explicitly asks for the urgency and RR/OE perspectives, lest PMs and POs inadvertently lead their teams into mountains of technical debt or not account for the temporal aspects of adjudging value. It also loosens the WSJF calculation to be more easily applicable in scenarios where trade-offs are not purely economic, e.g. there are regulatory “mandatory non-value added” (to use some Lean parlance) concerns to address.

        Bottom line, I find your (and Jason Yip’s) concerns about SAFe WSJF unmoored from its intended context, and they ignore piles of empirical evidence that it’s working very well for its practitioners (not to mention, your waterfall post shows you understand how small batching is the “secret sauce” that drives the bulk of “Agile” benefits, so critiquing SAFe WSJF is in that light the epitome of being nit-picky).

        1. Post
          Author

          Thanks again for commenting, Andrew. It’s good to have some engagement on this topic and I’m hoping the discussion is helpful for the wider community who is interested in using Cost of Delay and WSJF.

          You say you’re not interested in answering the points that this post makes about SAFe’s interpretation of Cost of Delay and WSJF. Why is that? You go on to make a whole bunch of unrelated points and straw man arguments. If you’re not going to address the central points, then please don’t be surprised if I find your arguments unconvincing.

          The article you are commenting on is my assessment of the SAFe interpretation. I don’t expect everyone to agree. To be clear, I’m not trying to “inflict help” on you or anyone else. If SAI and SAFe consultants can’t (or don’t want to) see the problems I’ve raised then that’s completely up to you. It really doesn’t bother me either way. You’re welcome to sell and use whatever prioritisation/scheduling method you like.

          You’re also welcome to believe your approach is better than others based on whatever evidence you select to inform your opinion. For me, I’d suggest there is a big difference between subjective evidence (people’s opinion) and objective evidence (say, of the earth orbiting the Sun). It was people’s opinion for many years that the earth was the centre of the universe and did not move, despite objective proof to the contrary. I find it ironic when people cite the HiPPO as evidence that SAFe-style WSJF is better than the HiPPO. I’m sure we can find plenty of subjective evidence that MoSCoW or Value/Effort or some other formula is better than what they were using before. This is a very low bar, especially considering Hawthorne Effect is likely to be strong. Of course, customer feedback is important, but it is not terribly valuable evidence in my view.

          Why are there no SAFe consultants prepared to answer the points I’m making? I wonder how long before its customers start asking the same questions? Even if they don’t, why resist improving it? It wouldn’t be hard to improve the SAFe interpretation significantly with just a little effort – but if you’re going to insist on defending it as already “good enough” and assert (wrongly) that my points above are shooting for “perfect” then I don’t think we’re going to get very far.

          If however anyone is prepared to answer the points, I’m more than happy to make some simple suggestions for improving it. I would consider this a safe-to-fail experiment. Any takers?

    2. Hi Andrew

      I think you are missing something key. SAFE is not a community, it is a franchise operation. The SAFE franchise is not interested in feedback or critique other than from their customers who by their very nature do not know that much. You said it yourself, its meant to be a Shu level practice yet prioritising a portfolio is not a shu level task. In order to prioritise a portfolio you should be an expert and it is dangerous to treat complex problems as obvious. For example, using WSJF creates a “cognitive” bias in the type of work that you do. Check out Clayton Christenson’s talk on ROI at the Drucker Forum to see the impact of using the SAFe interpretation of WSJF.

      Using Fibonacci for business value is outright crazy and indicates a failure of understanding of business value and the wisdom of crowds. It would be much more sensible if the SAI said “Go and read Joshua’s blog” but that’s not how you run a successful franchise. ;-)

      1. Ya do know that for 18 months SAFe classes showed Joshua’s CoD video & said EXACTLY THAT?

        Got cut recently for reasons of time.

      2. I would say SAFe is a franchise like Scrum is a franchise. Scrum is a community and so is SAFe. Like Scrum, SAFe is open and freely revealed — everything you could possibly want to know about what it entails is right there on the website, accessible to you.

        SAFe is very interested in feedback; the Value Stream layer in SAFe 4.0 exists in large part from inputs from the likes of [a Scaled Agile Gold Partner consultancy] and the company I work for, who argued that the framework was great for very large-scale, complex (i.e. not purely software) systems as well.

        And yet while SAFe is prescriptive, it is also very flexible. I encourage you to read Al Shalloway’s blog post on “SAFe as a framework”.

        But now that we’re done treating Agile approaches like theological treatises to be either revered as dogma or denounced as heresy, let’s talk about Fibonacci. Why is Fibonacci “outright crazy” for calculating CoD? As Josh surely knows (he wrote the Qualitative CoD post after all), CoD is not always straightforward to denominate in, say, dollars or pounds or euros. However, many customers are able to weigh the value and urgency of items in relative terms against like items. Fibonacci is one way to do that in the WSJF numerator field, but it’s not going to break SAFe’s WSJF concept to use another quantitative framework. So long as the quantitative framework for the numerator and denominator are consistent across the items you’re measuring, you should be fine from a relative prioritization standpoint.

        Finally, maybe I’m dense, but I didn’t catch Christensen’s talk — maybe you could quickly summarize how SAFe WJSF leads to bad prioritization decisions according to Christensen?

  4. Since SAFe is a successful franchise it seems to me there is a market for it. Indeed their interpretation of WSJF is not correct but at least it is a step in the right direction. If the market finds their interpretation of WSJF to be inadequate then Don Reinertsen classes would probably be in demand.
    One thing I agree with Joshua is that the SAFe franchise should have the integrity to acknowledge the points he makes in this post.

    1. SAFE IS a successful franchise and they have demonstrated there is a demand for a “Scaled Agile” product. It does not mean that their product is good. Think about McDonalds at the time of “Supersize Me.”. We may well see similar outcomes from organisations adopting SAFE with companies becoming sick after a while.

    2. Again, I do not understand how it is inadequate. The arguments levied against it are that it is sub-optimal, but that argument is being made both without acknowledging just how successful SAFe has been for companies like John Deere, TomTom, and Lego, it’s also being proffered sans a more optimal approach for applying CoD to Epic, Capability, and Feature backlogs.

  5. Re: business value, I’ve been wondering if anyone is reading Doug Hubbard’s How to Measure Anything. It’s all about expected value of perfect information & the economics of ascertaining that. Seems like a good framework for understanding the merits of specific techniques like Fibonacci.

      1. Post
        Author

        Personally speaking, I’m a big fan of addressing the actual points made (rather than simply asserting what is already in place is “good enough”). What you’re really protecting appears to be dogma, since it has no scientific basis that I can tell.

        Douglas Hubbard’s “How to Measure Anything” actually makes a great case for improving our currently poor understanding of Cost of Delay by making efforts to reduce the uncertainty. As with most things, our lack of confidence when it comes to quantification most likely stems from a lack of experience of doing it. Indeed, Hubbard would say:

        • Your problem isn’t unique.
        • You already have more data than you know.
        • You need less additional data than you think.
        • The additional data you need is actually easier to find than you think.

        Using Fibonacci, on the other hand, without actually referencing it back to some sort of quantification, does little to reduce the uncertainty. In fact, I’d go so far as to say that “putting numbers” against the parameters may even give us false confidence that we are doing something “sciencey” when it is actually nothing of the sort.

  6. I really believe that the cost of delay is all about changing the type of conversations people have in the company. In my opinion, taking shortcuts is not desirable nor a healthy behaviour.

    I love seeing people angrily talking about how upset they are because a user story, epic, activity or attitude made a 3.000.000 $ lost in profit in the last month. It is a healthy conversation, otherwise you feed many cognitive biases.
    Remember the problem you are trying to solve… to have a way to represent almost everything in a unique “currency” instead of using proxy variables.

    Even if you have no idea how long something would take, you can use triangulation and get some numbers, and again… the kind of conversations you will have would depend on how clear you can express it, and for that, quantifying the cost of delay in real money is a must.

    My feelings are that if companies or the government believe it is difficult to quantify something in money, they would rather open an NGO.

  7. Interesting comment about cost of delay not being all that applicable to internal user preference. As an internal-facing UX designer, someone who works on tools for corporate employees, I would really like you to say more on that point.

    Am thinking of Jez Humble’s point (citing Doug Hubbard’s book) that user engagement (will users actually use what you build) is pretty much the only thing that’s actually predictive of ROI.

    The way I look at it is if providing something to a certain user group will generate business value, the delay of it’s introduction has a cost. This could be features, or possibly stories. Every user story in a backlog could have a cost of delay; and, when you’re working on the first user story you’re accruing the cost of delay for that plus all subsequent user stories. So the question is, what’s the prioritization order that will produce the lowest overall cost of delay? The challenge as you’d probably emphasize is monetizing this, which is something I admittedly struggle with. Here’s one possible scenario:

    Usability improvements are tied to a measurable variable which typically can be monetized (number of errors, task success, # of support calls, time-on-task etc.). What if a user experience expert says “Hey, if we made this change I’m confident we would could save our developers two hours a week!” Maybe the highest-paid person responds, “Yeah, but it would cost something like $500,000 to make that change, so we’re going to put it off.”

    UX expert: “Yeah but that’s just the cost of making the change.”

    HiPP: “Right…and I’m not going to fund that at this time. We could probably do it in six months.”

    UX expert: “But you need to consider the cost of not making the change. I’m confident we can save developers two hours a week. Say they make on average $90,000 a year and there are 2,088 work hours this year. That’s $43 an hour. That’s $86 per developer per week we’d be saving. There are 1,000 developers. That’s $86,000 a week and $344,000 a month. If you really don’t do this for six months you’re losing more than $2 million. Compare that to your $500k.”

    1. Post
      Author

      Hi Charles, thanks for reading and commenting. I think you might be misunderstanding what I am saying:

      I’m not so sure about the users preference question. If you are talking about internal users (a common case in SAFe’s Target Market) this is often a fairly weak indicator of value.

      What I mean by “weak indicator of value” is NOT to ignore it. I’m also NOT saying you shouldn’t attempt to quantify the value. Quite the opposite!

      What I AM saying is that I would normally encourage organisations to focus more on the pain of paying customers or external users than internal users. The relative “gut feel” value that internal people are likely to put on internal pain is often overstated compared to the Cost of Delay they are incurring by not delivering improvements that would attract new customers or retain existing customers.

      Either way, quantifying the Cost of Delay is a good way to avoid this trap. As such, I would propose that you do exactly what your scenario nicely illustrates: to express the Cost of Delay so you can make an Apples to Apples comparison with alternative options.

      My fear with SAFe’s relative scoring system (and treating value in two different buckets that are not normalised against one another) is that their formula skews the cost of delay. The very organisations who need to focus more on creating new customer and delivering enhancements that would retain customers end up prioritising improvements that benefit internal users instead.

      Does that make sense?

      1. Thank you for your reply. In many organizations though, internal and external users are supported by wholly different silos. In my case, for instance, I would only ever be looking at applying CoD to internal users (employees), and external users would never by definition be in the mix. Any information you have on the application of CoD to IT projects would be excellent. I’m familiar with the Maersk study, but what I’ve seen doesn’t tend to get into HOW CoD is actually derived. (From what I’ve heard, even Reinertsen’s book really doesn’t.) Thanks again. Lean Enterprise mentions using your OMTM (one metric that matters–a Lean Analytics concept) to think about CoD, which is an interesting idea, but doesn’t really go beyond that.

Leave a Reply to Troy Magennis Cancel reply

Your email address will not be published. Required fields are marked *