Over the years of working on land and water conservation issues, I’ve become familiar with the decision-making processes used in a number of organisations to prioritise projects in this area. These include national, state and regional bodies. In almost all cases, there were serious problems with the processes used. Also, in many of the decision tools available to managers, some of the same problems are commonly present.

The main problems are as follows.

Missing variables. Simulations I’ve done indicate that leaving out important variables from the analysis of projects can have a a huge effect on the overall benefits from a portfolio of projects (see PD158). It can easily wipe out something like half of the potential net benefits from investment. However, some really important variables very commonly are omitted, including:

• the effectiveness of the proposed changes in land and water management in bringing about the intended improvements in the natural assets;
• the likely level of adoption by land and water managers of the proposed management changes;
• the timing of benefits;
• the ongoing maintenance costs needed to preserve the benefits generated by the project;
• the likelihood of the project succeeding, accounting for technical risks, political risks, long-term funding risks, etc.

Badly structured metrics. Most processes use some sort of metric (i.e., a formula to calculate a number) to represent the benefits from improvements to the natural assets. Most benefit metrics in actual use involve weighting each of the variables and adding them up. For some key variables, this is a disaster. It does not accurately reflect how project benefits would vary in response to changes in those variables. For example, if two projects are otherwise similar, but one has double the technical effectiveness of the other, then the environmental benefits are going to be roughly twice as big, but an additive metric cannot give you that result. The same is true, more or less, for the likelihood of project success, for asset value, and for adoption. These key variables need to enter the metric as multipliers, not as additions. Getting this wrong can again result in losses of up to 50% of the potential environmental benefits from investment (PD158), by leading you to projects that are not actually the best ones.

Problems with project cost. Some processes ignore project cost altogether. Others include it in a weighted additive metric. Almost all ignore ongoing maintenance costs. These are all serious problems. If the metric is to be used to rank projects, then it should consist of a measure of expected benefits divided by total costs, including both current project costs and ongoing maintenance costs.

Lack of project specificity. Many project proposals are not very clear and specific about one or more of the following: their goals; which natural assets will be affected; which changes in land and water management are hoped for; and which delivery mechanisms/policy tools will be used to try to generate the management changes. The decision process should require sufficient specificity on these things to allow proper evaluation of the projects (and to facilitate evaluation of them, later on).

Lack of quality assurance. In my view, there should always be some sort of expert review of the numbers provided by project proponents, but especially when there is a competitive funding process. Without that, the risk is that funding will flow to the projects that have been exaggerated the most, rather than to the best projects. A related problem is that there is very often a lack of logical consistency between the numbers presented in a proposal. For example, high levels of adoption are assumed, but the budget to promote adoption is too modest. The best system for building in expert review is likely to vary in different contexts. I have found that it can be useful to include expert review in the project development process, but it could come later. In the former case, decision makers should be asking for evidence about the expert review process that has been used, while in the latter they would receive advice directly from the expert reviewers.

Notably, the problems with the existing systems are not due to lack of effort. The organisations in question sometimes put considerable time and resources into assessing projects. It is just that the processes often miss the mark.

None of this would actually be all that hard to fix. Some of the issues don’t even require any more work — just changes in the way that the numbers are handled. Others do require more effort, to gather additional information, or to strengthen quality assurance. However, the additional effort is not onerous, and easily justified by the benefits that can result.

As far as I’m aware, INFFER is currently the only decision framework or tool for environmental or natural resource projects that deals correctly with all the above issues (except quality assurance, of course, as that requires a separate process). That’s not to say that other tools could not be adapted or extended appropriately — it would be quite feasible.

David Pannell, The University of Western Australia