By Matt Baumann (Matthew Baumann Associates) and Caroline Hattam (ICF)
15th November 2021
Introduction:
This blog provides a discussion of the potential role for evaluation in supporting and assessing progress on action for nature and biodiversity. It highlights some of the challenges for evaluation in this policy space and introduces an evaluation ‘route map’ to underpin further scoping and evaluation planning. The blog is based on an assignment undertaken by an ICF-led[1] team for Defra during winter 2020-21.
To the untrained eye, nature might appear to be doing well. However, the UK has been recognised as one of the most nature depleted countries globally. Of the 20 biodiversity indicators reported to the Convention on Biological Diversity (CBD), only 3 have shown a positive trend in recent years.
Over the past 50 years, changes in habitat management (and, in turn, habitat quality for the species concerned) have been the most common cause of biodiversity loss but there are also many other causes (Burns et al 2020[2]; Wilson et al 2019[3]; Roy et al 2012[4]; Ockenden et al 2014[5]).
Biodiversity is complex. Biodiversity has multiple dimensions (e.g., genes, species and ecosystems) and multiple scales (e.g., local, regional and national). The behaviours and states of all entities in nature, and the pressures on nature from humans are constantly evolving, adapting and interacting.
This makes policy for biodiversity recovery challenging. In Defra, effective policy making to address drivers of biodiversity loss relies on evidence and analysis. However, measuring and assessing the status of and pressures on nature is itself both a significant endeavour (relying on a vast system of data flows and data capture through formal and informal systems, across these same domains and spatial levels) and an imperfect science (evaluating cause and effect relationships across this open adaptive system in which so much is interdependent). It is not surprising that outcomes of different actions tend to be complex and hence difficult to predict.
Programmes designed and delivered in response to pressures on nature and biodiversity loss tend to be complex and affect outcomes across multiple domains, and strategies even more so because they are a combination of multiple programmes and funding streams and rely on actions across multiple government departments.
Defra is leading work to define England’s approach to the recovery of nature. The main objectives, outcomes and targets for England’s action on nature (referred to as ‘the strategy for nature’ from here on) are still under discussion but are likely to focus on the 25 Year Environment Plan targets to create or restore 500k ha of wildlife rich habitats by 2042, restore 75% of protected sites to favourable condition, and protect 30% of England’s land by 2030.
Scoping an evaluation:
Defra commissioned an ICF-led team to develop a ‘route map’ for the planning of an evaluation of the strategy. This work took place over four months as the strategy was being developed. The assignment showed that evaluation can play an important role in guiding action for nature:
- Ex ante evaluative thinking can inform strategy development. Systems thinking in the context of the development of a coherent theory of change can help clarify key causal pathways and interdependencies between actions and system responses.
- As the strategy is being developed evaluation can be used to guide an adaptive approach to delivery. For example, evidence from evaluation can be used to support policy-makers to make evidence-based decisions about whether the interventions, emphasis, and funding in place are sufficient to achieve the strategy’s goals.
- Finally, learning and evidence from strategy evaluation can inform the design of future action on nature. Tracking implementation and outcomes beyond the lifetime of the strategy will be important to identify the emergence of longer-term effects. This will be necessary for continued tracking of progress towards the Environment Bill targets and those of the 25 Year Environment Plan.
Whilst there are numerous challenges to achieving the vision for evaluation outlined above, here we focus on three: strategy emergence, measuring and understanding change and evaluation management and governance.
- Strategy Emergence. Given that the strategy was at an early stage of development, we used Theory of Change (ToC) to facilitate deep thinking about how the strategy activities could be expected to achieve key goals. Specifically, we focused on how these activities could be used collectively to overcome a range of barriers to achieving the strategies’ key habitats and species objectives and contribute to a wider range of benefits and outcomes. The approach to ToC we applied is based on causal mapping, which fuses a complexity and systems framing with the logical steps and formats used in ToC. It represented a provisional ‘ex-ante’ evaluation of the strategy. A very high-level summary of the detailed ToC is shown in Figure 1 below.
Figure 1: High level ToC for the Strategy for Nature
Because it is likely that there will be further evolution of the strategy both in the short and longer term, and the relatively high level of our ToC, we recommended further work to develop the strategy ToC and related ‘programme theory’ to incorporate the contribution of emerging polices and funding streams and shifts in the framing and narratives proposed in the strategy as well as the development of ‘nested’ ToCs for individual policies and programmes or for clusters of policies and programmes. Furthermore, ToCs will need to be regularly reviewed in response to strategy evolution – or to guide such evolution.
- Measuring and understanding change. Whilst measuring top level indicators across the range of domains is feasible at the multiple scales required, understanding causal relationships between entities (including the causal effects of biodiversity policy) is challenging for the reasons outlined above. An additional challenge for evaluation is that underlying, and as yet uncertain, features of the strategy, such as adaptive management, governance, funding, improvements in data, monitoring and evaluation could create synergistic, additive and catalytic effects. Understanding these effects will be crucial if recommended efforts to actively coordinate and manage multiple programmes is to be understood but doing so will be difficult.
The ToC work outlined above and the proposal for further work represents an initial step towards getting a grip on hypothetical causal relationships. A second major proposal is for substantive investment in methodological R & D. As noted above, the behaviours and states of all entities in nature and the pressures from humans are constantly evolving, adapting and interacting – thus understanding, measuring and evidencing change and the interactions between policies, programmes and activities is highly complex. Feasible approaches will need to be developed and tested for measuring and understanding the complex multi-dimensional cause-effect relationships over time.
- Evaluation management and governance. There are numerous challenges for evaluation management and governance but limited space here to discuss them. Our main recommendation is that evaluation of the strategy itself needs to be treated as a major ‘evidence system’. This will require a high level of resourcing, coordination, management and effective complexity sensitive procurement systems if the many benefits of evaluation are to be realised and the many risks of duplication, overlap, and complexity are to be overcome.
A final important reflection on the role of evaluation in the strategy for nature is that for evaluation to be able to make the kinds of contributions to the strategy’s delivery outlined above, it would be advisable for Defra to adopt a more adaptive approach to management of the strategy than has been feasible in the past. However, a major challenge of adaptive management for large scale complex systems such as a national biodiversity strategy is the sheer scale of what is included – Defra’s role would become more akin to ‘system stewardship’[6].
Conclusion:
Complex missions such as those embodied in the strategy for nature and in the wider 25 Year Environment Plan require a new way of thinking about evaluation. Rather than relying solely on piecemeal evaluation on a programme-by-programme basis, we need also to start thinking about how to evaluate clusters of programmes embodied in strategies and to develop evaluation (and learning) systems that support multiple actors at multiple levels to understand the effectiveness of their work over time. This kind of evaluation will not be easy to commission or undertake but it is clear from this assignment that there is increasing recognition of the need for a new approach to meet the substantial policy challenges of today’s complex policy environment. Changes to the operating context (adaptive management) may also be necessary if evaluation is to fulfil its potential for learning and help steer action towards strategy targets and outcomes.
[1] The ICF-led team included Caroline Hattam, Andrew Jarvis, David Elemna and Aisha Ahmad (ICF) Miranda Bane and Michael Pockock (UK Centre for Ecology and Hydrology), Ben Shaw (Centre for Evaluation of Complexity at the Nexus (CECAN Ltd)), and ICF associates Matthew Baumann, Matt Rayment, and Graham Tucker.
[2] Burns, F., Eaton, M.A., Barlow, K.E. et al. (2016) Agricultural Management and Climatic Change Are the Major Drivers of Biodiversity Change in the UK. PLOS ONE, 11, e0151595.
[3] Wilson, J.F., Baker, D., Cheney, J. et al. (2018) A role for artificial night-time lighting in long-term changes in populations of 100 widespread macro-moths in UK and Ireland: a citizen-science study. J. Insect Conserv. 22, 189–196
[4] Roy, H.E., Bacon, J., Beckmann, B. et al. (2012). Non-native species in Great Britain: establishment, detection and reporting to inform effective decision making. Wallingford, UK: NERC Centre for Ecology & Hydrology
[5] Ockendon, N., Baker, D.J., Carr, J.A. et al. (2014) Mechanisms underpinning climatic impacts on natural populations: altered species interactions are more important than direct effects. Global Change Biology, 20, 2221-2229.
[6] For more information on adaptive management: The journal of Environmental Management ran a special issue on adaptive management in May 2011: https://www.sciencedirect.com/journal/journal-of-environmental-management/vol/92/issue/5 and Science Direct provides a resource summarising key texts defining aspects of adaptive management in earth and planetary sciences: https://www.sciencedirect.com/topics/earth-and-planetary-sciences/adaptive-management. System Stewardship has been defined and articulated well by the Institute of Government in 2011: https://www.instituteforgovernment.org.uk/sites/default/files/publications/System%20Stewardship.pdf