325. Ranking projects based on cost-effectiveness
Where organisations are unable or unwilling to quantify project benefits in monetary or monetary-equivalent terms, a common approach is to rank potential projects on the basis of cost-effectiveness. Just like ranking projects based on Benefit: Cost Ratio (BCR), this approach works in some cases but not others.
To rank projects based on cost-effectiveness, you choose the metric you will use to measure project benefits, estimate that metric for each project, estimate the cost of each project, and divide the benefit metric by the cost. You end up with a cost-effectiveness number for each potential project, and you use these numbers to rank the projects.
An advantage of this approach is that it sidesteps the challenges of having to measure all the benefits in monetary or monetary-equivalent terms, which is what you have to do calculate a BCR. A disadvantage is that it only works to compare projects that generate similar types of benefits, which can all be measured with the same metric.
Assuming that we are satisfied with your benefits metric and that the projects to be ranked are similar enough, the question is, in what circumstances is it appropriate to rank projects based on cost-effectiveness? (Assuming that the objective is to maximise the overall benefits across all the projects that get funded.) It is logical to ask this given that cost-effectiveness is closely related to the BCR (it has the same structure – it’s just that benefits are measured differently), and we’ve seen in PD322, PD323 and PD324 that ranking projects by BCR works in some situations but not others.
It turns out that the circumstances where it is logical to use cost-effectiveness to rank projects are equivalent to the circumstances where it is logical to rank projects using BCR.
(i) If you are ranking separate, unrelated projects, doing so on the basis of cost-effectiveness is appropriate. Ranking projects by cost-effectiveness implies that there is a limited budget available and you are aiming to allocate it to the best projects.
(ii) If you are ranking mutually exclusive projects (e.g. different versions of the same project), ranking on the basis of cost-effectiveness can be highly misleading. If there are increasing marginal costs and/or decreasing marginal benefits (which are normal), ranking by cost-effectiveness will bias you towards smaller project versions. In PD323, I said to rank such projects by NPV and choose the highest NPV you can afford with the available budget. If we are not monetising the benefits, there is no equivalent to the NPV — you cannot subtract the costs from a non-monetary version of the benefits. This means that, strictly speaking, you cannot rank projects in this situation (mutually exclusive projects) without monetising the benefits. If you absolutely will not or cannot monetise the benefits, what I suggest you do instead is identify the set of project versions that can be afforded with the available budget, and choose the project version from that set that has the highest value for the benefit metric. (Theoretically it should be the project version with the greatest net benefit (benefits – costs) but that is not an option here because in Cost-Effectiveness Analysis the benefits and costs are measured in different units.)
You don’t divide by the costs, but you do use the costs to determine which project versions you can afford. This is a fudge that only makes sense if you adopt the unrealistic assumption that any unspent money will not be available to spend on anything else, but it seems to me to be the best way to go, if monetising the benefits is not an option.
(iii) If you are ranking separate, unrelated projects, and there are multiple versions available for at least one of those projects, then cost-effectiveness does not work and the rule about choosing the highest-value benefit metric does not work either. Instead, you should build an integer programming model to simultaneously weigh up both problems: which project(s) and which project version(s). There is a brief video showing you how to do this in Excel in PD324. In the video, the benefits are measured in monetary terms, but the approach will work if you use non-monetary measures of the benefits.
There are a number of tools available for ranking projects based on cost-effectiveness (e.g. Joseph et al. 2009) but it is important to be clear that the approach only works in certain cases.
Even if you are using cost-effectiveness in the right circumstances (case (i) above), it has a couple of limitations relative to using BCR. One is that you cannot use it to rank projects with distinctly different types of benefits that cannot all be measured with the same metric. Another limitation is that cost-effectiveness provides no evidence about whether any of the projects would generate sufficient benefits to outweigh its costs.
Further reading
Joseph, L.N., Maloney, R.F. and Possingham, H.P. (2009). Optimal allocation of resources among threatened species: a project prioritization protocol. Conservation Biology, 23, 328-338. Journal web site
Pannell, D.J. (2015). Ranking environmental projects revisited. Pannell Discussions 281. Here * IDEAS page
I have, perhaps naively, assumed that a major role of cost-effectiveness analysis is to help determine the optimum means of achieving a certain, fixed outcome. That is, for whatever reason, the outcome is a given, what then is the most cost-effective way of achieving this outcome? For example, because of the importance of a particular environmental asset, it is accepted that waterway nutrient levels need to be reduced below a certain value. There are obviously many ways of achieving this, and the ‘cost’, in the broadest sense, of the different options needs to be evaluated. Thus the supposed disadvantage, ‘that it only works to compare projects that generate similar types of benefits’ is actually an advantage.
Thanks Brian. Yes, good point. “Cost effectiveness” is used in a couple of ways. Sometimes it used to mean exactly what you say. Under that definition, the idea is that you have a specific target or outcome that you want to achieve, and several ways of achieving it. In that case, the various options all have the same benefits, and the best option is the cheapest option. This approach also avoids the need to quantify the benefits in monetary or monetary-equivalent terms. Because all the options have the same benefits, there is no need to monetise them.
In reality, it is rarely the case that different project options actually deliver the same benefits. Even if they are expected to achieve the same level of a particular target, they may vary in their reliability, or their effects on other factors that people care about. In that case, the analyst needs to carefully consider whether it is reasonable to assume that the benefits are equivalent.
In this post, I use “cost effectiveness” in a slightly different way (the way it is defined in Wikipedia, for what that’s worth). I’m still assuming that the benefits won’t be monetised, but I am accepting that the benefits will probably be different for the various projects. As long as I’ve got an acceptable way of measuring those benefits, I can use that in my decision making about the projects, in the ways outlined in the blog post.
If you prefer, I could call this “cost efficiency” or something, but the points would be the same.
Thanks for the clarification. I am far from having any expertise in these areas, and is a subject I’ve always found confusing. The only thing I’d add is I don’t the costs are only monetary; different options/ projects may have different social ‘costs’ for different sections of the community. The decision may not always be straightforward.
That is an interesting point about the possibility of non-monetary costs (negative spin-offs) arising due to a project. If you were doing a Benefit: Cost Analysis, you would quantify those costs in monetary-equivalent terms and subtract them from the benefits. i.e. you would treat them as negative benefits. If you were calculating a Benefit: Cost Ratio, you would subtract them from the numerator, rather than adding them to the denominator, for reasons spelled out in PD322.
What if you are doing a cost-effectiveness analysis of the type I describe in this post? Suppose your measure of benefits is probability of non-extinction of a species, and a project has a negative spin-off effect on, say, water pollution. I think you are in a pretty deep bind. For the analysis to be logically sound, you’d need to subtract the negative spin-off from the measure of benefits, but you can’t do that because they are measured in completely different units. If the negative spin-offs are big enough that you need to account for them, you’d have to switch to doing a Benefit: Cost Analysis, rather than a cost-effectiveness analysis. If you stuck with ranking projects based on cost-effectiveness and ignored the negative spin-offs, your ranking could easily be wrong, in the sense that it would not lead to the best overall result.
The situation is the same as when you are trying to analysis a project that generates two quite different types of benefits. Unless you have a benefit metric that works for both benefit types, you can’t do cost-effectiveness analysis of the type I describe here. You’d have to do BCA.
Thank you, David for this valuable information on quantifying project benefits.
David – Very confusing. You dance around with non operational words that seem to be invented on he run. Whose discount rate would you use? That would be your subjective valuation as to what value these non-monetarised elements are assigned? Have you ever considered taking a rigorous course in capital theory? I have to disagree – this information is not valuable from our perspective. It is not economic theory as we understand such. S. Baxter
This piece is targeted at people who use cost-effectiveness analysis, which is not really a standard economic tool. Many of its users are not economists, and I am attempting to make the piece understandable to them, hence the non-standard terminology. For some of it (mutually exclusive projects), I’m not actually aware of any standard terminology in economics anyway. But the point about mutually exclusive projects is an important one that most people seem unaware of, including many economists. The question about discount rates is irrelevant to the points I’m making here.