Episode 20, the last in this series on principles to follow when ranking environmental projects. It provides a brief summary of the whole series. 

You can obtain a PDF file with all 20 episodes of this series integrated into one document here.

Around the world, thousands of different quantitative systems have been used to rank environmental projects for funding. It seems that every environmental body creates anew or re-uses at least several such systems each year. Judging from the examples I have examined, most of the systems in use are very poor. The performance of many of them is not much better than choosing projects at random. If only people would be more logical and thorough in their approach to ranking environmental projects! The potential to reduce wastage and improve environmental outcomes is enormous. That’s why I wrote this series.

There are many ways that you can go wrong when putting together a formula to rank projects, and unfortunately the quality of the results is quite sensitive to some of the common errors. Common important mistakes include: weighting and adding variables that should be multiplied; messing up the comparison of outcomes with versus without the project; omitting key benefits variables; ignoring costs; and measuring activity instead of environmental outcomes.

Fortunately, though, it’s not hard to do a pretty good job of project ranking. A bit of theory, some simple logic and a dose of common sense and judgment lead to a set of specific guidelines that are presented in this series. The essential points are as follows.

1. The core criterion for ranking projects is value for money: a measure of project benefits divided by project-related costs. This is the criterion into which all the variables feed. It’s how you pull everything together to maximise environmental outcomes.
2. You should rank specific projects, rather than environmental assets. You cannot specify numbers for some of the key variables in the ranking formula without having in mind the particular interventions that will be used.
3. There are always many different ways of managing an environmental asset, and they can vary greatly in value for money. Therefore, it can be worth evaluating more than one project per asset, especially for large, important environmental assets.
4. Benefits of a project should be estimated as a difference: with versus without the project, not before versus after the project.
5. Weak thinking about the “without” scenario for environmental projects is a common failing, sometimes leading to exaggerated estimates of the benefits.
6. There are two parts to a project’s potential benefits: a change in the physical condition of the environment, and a resulting change in the values generated by the environment (in other words, the value of the change in environmental services).
7. Those potential benefits usually need to be scaled down to reflect: (a) less than 100% cooperation or compliance by private citizens or other organisations; (b) a variety of project risks; and (c) the time lag between implementing the project and benefits being generated, combined with the cumulative cost of interest on up-front costs (i.e. “discounting” to bring future benefits back to the present).
8. If in doubt, multiply. That’s a way of saying that benefits tend to be proportional to the variables we’ve talked about (or to one minus risk), and the way to reflect this in the formula is to multiply by the variables, rather than weighting and adding them. Don’t take this too literally, however. You can mess up by multiplying inappropriately too. 
9. Weighting and adding is relevant only to the values part of the benefits equation (when there are multiple benefits from a project), not to any other part.
10. Don’t include private benefits as a benefit or voluntary private costs as a cost, but do include involuntary private costs as a cost.
11. Other costs to include are project cash costs, project in-kind costs, and maintenance costs (after the project is finished). Costs get added up, rather than multiplied.
12. Uncertainty about project benefits is usually high and should not be ignored. The degree of uncertainty about each project should be considered, at least qualitatively, when projects are being ranked. Also, decisions about projects should not be set in stone, but modified over time as experience and better information is accumulated. Strategies to reduce uncertainty over time should be built into projects (e.g. feasibility assessments, active adaptive management).
13. Where the cost of all projects that are in contention greatly exceeds the total budget, it is wise and cost-effective to run a simple initial filter over projects to select a smaller number for more detailed assessment. It’s OK to eliminate some projects from contention based on a simple analysis provided that projects are not accepted for funding without being subjected to a more detailed analysis.

There are a number of simplifications in the above advice. Simplifications are essential to make the system workable, but care is needed when selecting which simplifications to use.

In summary, the content and structure of the ranking formula really matters. A lot. A logical and practical formula to use is:

where

BCR is the Benefit: Cost Ratio,

V(P’) is the value of the environmental asset at benchmark condition P’,

W is the difference in values between P₁ (physical condition with the project) and P₀ (physical condition without the project) as a proportion of V(P’),

A is the level of adoption/compliance as a proportion of the level needed to achieve the project’s goal,

R_t, R_s, R_f and R_m are the probabilities of the project failing due to technical risk, socio-political risks, financial risks and management risks, respectively,

L is the lag time in years until most benefits of the project are generated,

r is the annual discount rate,

C is the total project cash costs,

K is the total project in-kind costs,

E is total discounted compliance costs, and

M is total discounted maintenance costs.

V can be measured in dollars, or in some other unit that makes sense for the types of projects being ranked. The advantages of using dollars are that it allows you to (a) compare value for money for projects that address completely different types of environmental issues (e.g. river water quality versus threatened species) and (b) assess whether a project’s overall expected benefits exceed its total costs.

For some projects, it works better to calculate potential benefits in a different way: [V(P₁) – V(P₀)] rather than V(P’) × W. They are equivalent but involve different thought processes.

A simplification that might appeal is to combine all four risks into one overall risk, R. If you do that, also drop ‘× (1 – R_f)’ from the denominator (because you no longer have a separate number for R_f). This simplification makes the formula look a bit less daunting, but it probably doesn’t really save you any work, because you should still consider all four types of risk when coming up with values for R.

This formula works were there is a single type of benefit from a project, or where the V scores for multiple benefits have already been converted into a common currency, such as dollars, and added up. If a project has multiple benefits and you want to account for them individually, replace V(P’) by the weighted sum of the values for each benefit type. For example, if there are three types of benefits, use [z₁ × V₁(P’) + z₂ × V₂(P’) + z₃ × V₃(P’)], where the z’s are the weights. I’m assuming here that the other benefit variables (W, A, R and L) are the same for each benefit type. If that’s not approximately true, you need to adjust the formula further.

One reaction I get is that it all looks too complicated and surely isn’t worth the bother. My response is to ask, if you could double your budget for projects by putting a bit more effort into your project ranking process, would you do so? Of course you would. Doubling the environmental benefits generated from your environmental investments is rather like doubling your budget. If your current ranking system is of the usual questionable quality, doubling the benefits (or more) is readily achievable using the approaches advocated here.

That’s all! Thanks for reading and best of luck with your project ranking endeavours.

Further reading

Pannell, D.J., Roberts, A.M., Park, G. and Alexander, J. (2013). Designing a practical and rigorous framework for comprehensive evaluation and prioritisation of environmental projects, Wildlife Research (forthcoming). Journal web page ♦ Pre-publication version at IDEAS

Here is another version of this summary, as published in the Decision Point magazine, in case it is helpful to have a pdf.