Number 2 in my series on criticisms of Benefit: Cost Analysis (BCA) addresses the concern that it is not feasible to do a BCA on projects for which we lack good information. Sometimes people are concerned that we are just too uncertain about the details for it to be sensible to do a BCA. (There is a video version of the post below, if you prefer.)

It’s true that there often is a high level of uncertainty about the numbers that would be needed to complete a BCA. For example, we might have weak evidence about how effective the project would be at generating the intended benefits, or we might not know how many people would be affected.

My response to people using this as a reason not to do a BCA depends on what they see as the alternative to doing a BCA. Maybe they are thinking that they should hold off on making a decision for now so that they can collect additional information to help them make a good decision later (potentially based on a BCA). They might commission new research or a review of similar past projects. This is fine and sensible. In the issues I work on, I’d like to see this happen more often. I think decision makers are often too impatient to make decisions, resulting in investment in projects that they would have rejected if they’d had more information.

If the alternative to doing a BCA is to make a qualitative, subjective decision about the project, unsupported by analysis, or to use a weaker less information-rich analysis, then I’m less convinced. All of the information required to complete a BCA is relevant to sound decision making, so if you decide to make the decision without doing a BCA, you aren’t solving the problem of lack of information – you are just burying it.

For this and other reasons, I would argue that high uncertainty about a project is an additional reason in favour of doing a BCA, not an argument against it. Reasons for doing a BCA despite the uncertainty include the following.

Even if the knowledge gaps are huge, doing a BCA provides a structure that helps you organise your thoughts about the project and requires you to ask the right questions about it.

Doing a BCA encourages you to seek out the best-available information. The process of conducting a BCA means that you have to work through the list of information that is needed, and this inevitably highlights what exactly is uncertain. If you don’t do a BCA, you might not try to find quantitative information about, say, the environmental impacts of a project. Deciding not to do a BCA probably means that there will be even more uncertainty affecting a decision than there needs to be because nobody is making an effort to get the best information.

Doing a BCA requires you to properly define the project in question. The benefits, costs and risks depend on the details of what the project will actually do and when; who else will contribute resources; how cooperation with or participation in the project will be achieved; who will bear costs; and so on. Making a decision without knowing these details means that there is more uncertainty in the decision than there needs to be, and you are liable to make poor judgements about the project.

Doing a BCA allows you to assess whether uncertainty matters. Using sensitivity analysis, you can make judgements about whether the project is highly likely to be a good one, highly likely to be a bad one, or somewhere in between. If the result is clear-cut and wouldn’t change even with large changes in the inputs, the uncertainty probably doesn’t matter.

Doing a BCA can help you identify which uncertainty matters. There might be a number of variables that are highly uncertain, but only one of them has a large impact on the overall results.

A BCA provides a great vehicle to support adaptive management, where a project is commenced with a strong focus on learning and filling the key knowledge gaps. As knowledge improves over time, the BCA can be revised and used to guide further decisions about changes to the project design, or whether to abandon the project.

A good benefit: cost analyst will use a variety of strategies to capture and record the relevant knowledge gaps and evaluate what they mean for decision making. This could include:

• Keeping a record of what knowledge gaps come to light and reporting them to decision makers
• Providing decision makers with an assessment (e.g. a rating) of the overall quality of the information used in the BCA
• Break-Even Analysis indicating how much an uncertain variable would need to change to change the BCA result. This contributes to judgements about whether uncertainty matters and for which variables it matters.
• Sensitivity Analysis (SA), ranging from single-variable SA, through multi-variable SA, to using Monte Carlo Analysis to generate a probability distribution of the results. The latter is tremendously informative and should be considered best practice.

So, considering all that, even if there are major knowledge gaps and uncertainties, it doesn’t follow that a BCA should not be done. In fact, it just makes the case for doing a BCA even more compelling.

Other BCA Criticisms

1. “Any result you want“
2. “Too much uncertainty“
3. “Discounting is bad“
4. “Not fair“
5. “Money isn’t everything“

Video version of this blog post

Playlist of videos for all five BCA Criticisms

Brief introductory course on BCA

If you are new to BCA, here is a very introductory course, in 10 videos totalling 1 hour 14 minutes.

Comprehensive set of courses on Applied BCA

I invite you to consider enrolling in my three 4-week online courses on Applied Benefit: Cost Analysis, covering The Essentials, Measuring Benefits and Practical Issues. Build your BCA expertise and gain the practical skills you need to undertake a complex Benefit: Cost Analysis. Online: high-quality video lectures and interviews, live workshops. No existing economics background required.

Discounts are available for bulk enrolments and for enrolments from selected countries. Bursaries (with a discount of 90%) are also available for enrolments from selected countries.

For information on content, pricing, and how to apply for discounts or a bursary download this flier:
Applied BCA Flier v8 . For details of the content of each course, see PD385.

The course runs starting in late February and late July each year. To enrol, click here or on the image at left, and go to the Business and Commerce section.

“A fantastic program and course. It is by far the best course I have been involved in. I learned so much and there is still lots to learn which you have shared with links so I can return to refresh/learn as needed.”

“Essential for new BCA users.”

“Taught in an engaging way with many real-world examples.”

Further reading

Pannell, D.J. (1997). Sensitivity analysis of normative economic models: Theoretical framework and practical strategies. Agricultural Economics 16: 139-152. Full paper (100 K) IDEAS page

Pannell, D.J. (2008). Sensitivity analysis with economic models, Pannell Discussions 126.

Pannell, D.J. (2021). BCA criticisms 1: “any result you want”, Pannell Discussions 354.