Many different types of metrics are used to prioritise projects for funding. In some cases, the metrics used are not much better than completely uninformed random choices.

Imagine yourself in the following situation:

• You are an environmental manager with a fixed budget trying to decide which projects to fund. You want to choose the projects that will deliver the most valuable environmental outcomes that you can afford.
• Money is tight. There are hundreds of projects you could potentially fund, but you only have sufficient funds at your disposal to fund a small percentage of them.
• You have the information needed to evaluate the projects: the significance of the environmental assets affected, the degree of degradation they have suffered or are likely to suffer in future, the effectiveness of proposed works in preventing or turning around that degradation, the likely adoption of those works by landholders, the risks of project failure, and the costs of each project.

How should you combine this information to choose the best projects? Does it really matter how you combine it? Of course these questions are relevant to other sorts of projects as well, but I’ll talk about them in an environmental context, which is where I’ve had to deal with them recently.

In principle, the best strategy is to rank the projects according to their overall benefits divided by their costs. This is well known and easy to understand but, remarkably, many exercises in prioritising environmental projects ignore the project costs. In simulations I’ve done recently (Pannell, 2009), I found that if funds are tight (they usually are) the simple error of ignoring project cost would result in choosing projects with about 30% lower environmental values, even if you did everything else perfectly.

Another potential error is to ignore one or more crucial variables. For example, in the world of environmental management, it is remarkably common for people to fail to explicitly consider both the adoptability of the proposed on-ground works, and their technical feasibility in addressing the environmental problem. My simulations showed that leaving out a couple of crucial variables when choosing projects would mean that you would probably lose around 50% of the potential environmental benefits from the investment!

Next there is the question of how to calculate the overall project benefits. In particular, should the relevant variables be combined by multiplication or addition? This depends on how the variables are related to overall benefits. Commonly, overall benefits are proportional to the relevant variables. For example, benefits would be proportional to the measure used to score asset significance or value, the level of degradation that is avoided by the on-ground works, and the probability of project success, and they would be approximately proportional to the adoption of works by the community. For cases like these where benefits are proportional to the variables, the appropriate mathematical formula is clear-cut: the variables should be multiplied together.

In real-world environmental programs, estimating project benefits using a weighted additive metric is far more common than using a multiplicative metric, even when the constituent variables are likely to be proportional to benefits. Indeed, as far as I am aware, the only practically used tools in Australia that use multiplication are the Benefit: Cost Index in INFFER and the Project Prioritisation Protocol (Joseph et al., 2009), (although the latter could be more comprehensive in the factors it considers). Perhaps the preference for addition reflects the popularity of Multi-Criteria Analysis, in which weighted additive benefit scoring is by far the most common approach, and is often used for all variables in the equation.

My simulations showed that using a weighted additive benefits index when a multiplicative one should be used results in losses of up to 55% of the potential environmental benefits. This is quite a remarkable result. Even if you have perfect information about the projects, and you do everything else correctly, adding when you should multiply can lose you more than half of the potential benefits of the program investment.

I’m not saying that addition should never be used. For example, it would be appropriate to add together the benefits for different stakeholder groups. However, addition should not be used as the default for all variables. Within each stakeholder group, there will be variables that must be multiplied if the results are to be sensible.

It is striking how sensitive the benefits of investment are to the way that project rankings are calculated. If you do pretty much anything wrong, you are likely to lose around 30-50% of the potential benefits, and combinations of errors (which are the norm in practice) may push the losses up towards 60%.

That is not all that much better than you get from completely uninformed random selection of projects: you lose 70 to 80% of benefits under the dumbest possible strategy.

Clearly, environmental managers need to pay a lot more attention to the metrics they use to prioritise projects. The costs to the environment from getting it wrong are huge.

The simulations also show how important it is to prioritise environmental projects. Under major national programs like Landcare, the Natural Heritage Trust and the National Action Plan for Salinity and Water Quality, the philosophy was to try to engage with as many people as possible, without seriously examining whether they can make a difference to key environmental outcomes. Compared to a more systematic targeted approach, this inclusive strategy probably resulted in the programs achieving around 70 percent less valuable environmental outcomes than they could have generated, even assuming (bravely!) that the projects were well designed and well implemented.

The current program, Caring for our Country, is at least targeted to particular environmental assets, although there remains room for improvement in how the target assets and projects are chosen. This study shows that targeting itself is not enough. If you don’t do the targeting well, it hardly helps!

David Pannell, The University of Western Australia

Further Reading

Joseph, L.N., Maloney, R.F. and Possingham, H.P. (2009). Optimal allocation of resources among threatened species: a project prioritisation protocol, Conservation Biology 23(2), 328-338.

Pannell, D.J. (2009). The cost of errors in prioritising projects, INFFER Working Paper 0903, University of Western Australia. Full paper (350K)

Pannell, D.J., Roberts, A.M., Alexander, J., and Park, G. (2009). INFFER (Investment Framework For Environmental Resources), INFFER Working Paper 0901, University of Western Australia, Perth. Full paper (74K)