263 – The EQ Index
Here is an idea: an index measuring the quality of the process used to evaluate environmental projects? It would provide a simple summary of the extent to which the evaluation process can be relied on to produce information that is useful for decision making. It would also provide a checklist of issues for people to think about when they are putting together an evaluation or prioritisation process.
Let’s call it the EQ Index, for Evaluation Quality.
I like the idea because many existing systems have weaknesses that lead to poor decisions about environmental projects. If people talked about and compared the EQ Index values for different processes, it might encourage a general improvement in their quality.
The EQ Index is a score out of 10, consisting of the following factors. Score 1 for each of these factors that is adequately accounted for in the evaluation process.
- Behaviour change. Does the process explicitly account for the likely degree of behaviour change resulting from the project? This could be voluntary behaviour change in response to information or payments, or compliance with a regulatory requirement. If relevant, it might include cooperation with the project by other organisations. (PD240)
Chain of links from management changes to outcomes. Does the evaluation quantitatively consider the chain of links between the changes in behaviour or management and the outcomes being sought? For example, a water pollution project might prompt farmers to reduce their use of fertilizers, the reduction in fertilizer use causes a reduction in export of nutrients from the farm into a water body, and that reduction in nutrients causes improvements in environmental conditions leading to improved ecology and ecosystem services. Are these links quantified?- With versus without. Are the benefits of the project estimated by comparing outcomes without the project and outcomes with the project (not outcomes before and after the project). Many existing systems fail to do this correctly (Maron et al., 2013). (PD237, PD93)
- Values. Different outcomes have different values to the community. For example, not all threatened species are seen as equally important by the community. The Great Barrier Reef is more valuable than an urban creek. Score a point if the process accounts for differences in value or importance for the outcomes of different projects. (PD239, PD218)
- Risks. The process should account for the fact that different projects have different probabilities of success or failure, depending on issues such as community or political opposition, technical feasibility, financial risks, and the capabilities of the organisation implementing the project. The risk of project failure should be explicitly recognised and factored in. (PD241)
- Time lags. Some projects deliver benefits quickly and others may take a century or more. This matters because interest payments on project costs accumulate over time and benefits in the distant future need to be large enough to outweigh the accumulated costs. Therefore the process should estimate time lags and should discount benefits and costs accordingly. (PD242)
- Project costs. The process needs to account for project costs because projects with the biggest benefits don’t necessarily provide the best value for money. For example, if a project is very expensive, funding it may may mean that you have to give up many other smaller projects that would have been better in aggregate. (PD246)
- Maintenance costs. Many projects need ongoing funding, beyond the initial project, in order for the environmental benefits to be continued. Ignoring this is common but can lead to poor selection of projects. (PD249)
- The maths. One requirement is for the evaluation process to focus on value for money. That means dividing overall benefits by costs. (PD236) An optimisation model achieves the same thing, but is more work. The second requirement is for the benefits-related factors (numbers 1 to 6 above) to be combined appropriately. In particular, they should often be multiplied together (see Pannell (2013) for details). Many systems estimate weights for each of these factors and add them up to provide an indication of project benefits, but this is often a serious error. (PD253)
- Data quality. If the information used in the evaluation comes from reputable models and published research, or the information is reviewed by independent experts, score 1. If it consists of estimates provided by proponents of the project, score 0. (PD213)
If a factor is not relevant to a particular project, the project scores 1 for that factor. For example, a project that consists of building physical infrastructure might not require behaviour change, so it automatically scores 1 for factor 1.
What scores should be considered good, bad and indifferent? Here’s my suggested system.
8.5 to 10: Good
6.5 to 8: Room for improvement
0 to 6: Bad
I think 7.5 to 8 should be the minimum acceptable score. Below that, there is a high probability of serious errors in decision making.
To illustrate the system, here are a few assessments of EQ Indexes for different processes I’ve looked at. The scoring requires some judgement, but I think it’s not too difficult.
EQ Index examples
Example Factors included EQ Index score
Caring for our Country 2009 competitive round 5, 7 2
Rodruquiez et al. (2004) 4, 10 2
Project Prioritisation Protocol (Joseph et al. 2009) 2, 3, 4, 5, 7, 8?, 9, 10? 6-8
INFFER (Pannell et al., 2012) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10? 9-10
It would be nice to see people referring to the EQ Index for their project evaluation process. How does your process score? (I will give a prize to the first person to comment below with a convincing estimate of the EQ Index for a project prioritisation/evaluation process that they use.)
Further reading
Joseph, L.N., Maloney, R.F. and Possingham, H.P. (2009). Optimal allocation of resources among threatened species: a project prioritisation protocol, Conservation Biology 23(2), 328-338.
Maron, M., Rhodes, J.R. and Gibbons, P. (2013). Calculating the benefits of conservation actions, Conservation Letters 6(5), 359-367.
Pannell, D.J. (2013). Ranking environmental projects, Working Paper 1312, School of Agricultural and Resource Economics, University of Western Australia. Full paper ♦ IDEAS page for this paper
Pannell, D.J., Roberts, A.M., Park, G., Alexander, J., Curatolo, A. and Marsh, S. (2012). Integrated assessment of public investment in land-use change to protect environmental assets in Australia, Land Use Policy 29(2): 377-387. Journal web site ♦ IDEAS page for this paper
Rodríguez, J.P., Rojas-Suárez, F. and Sharpe, C.J. (2004). Setting priorities for the conservation of Venezuela’s threatened birds, Oryx 38(4), 373-382.
It is a pity David to be so constrianed to the funded project model.
I use the term “project” broadly to mean any kind of investment that is intended to improve the environment. I don’t see that there is a constraint. Perhaps I’m missing your point?
a touch of bias on the INFFER score i think..
prehaps the low scores would be improved if you included as many factors as you did for INFFER
No bias at all. Look at the criteria and look at what’s in INFFER. INFFER was designed with these criteria in mind, so it’s hardly surprising.
Yes, the low scores would increase if they included the additional factors that they currently ignore. That’s the point.
David, what would your view be on the use of half-marks? In the intro, you say “Score 1 for each of these factors…”. In the wrap-up, you make reference to total scores of 8.5,7.5,6.5, implying that half-marks are accepatable.
Which of these factors could receive a half-mark? Can an evaluation system partly consider some of these factors? My blunt understanding is that it either does or it doesn’t. (i.e. you can’t be half-pregnant). Or can we sit on the fence on some of the factors?
Enjoying your work
Michael
Thanks Michael. I see half marks being possible for several of the criteria – ones that could be included but not necessarily in the ideal way. Might be relevant to numbers 1, 2, 4, 5, 7, 9, 10.