By Helen Wilkinson and Ben Shaw (April 2025)
With the CECAN annual conference on the horizon and the process to update the Magenta Book – the Government’s formal guidance on policy evaluation – underway, this blog revisits two questions explored at last year’s CECAN conference Is evaluation a fad? and Is complexity in evaluation just a fad?
As might be expected from a group with a professional interest in evaluation, the discussion at the conference argued that evaluation is not a fad, but rather a crucial practice to generate and share learning, especially where there is inherent complexity. While there are significant challenges, there are ways of working with these to generate useful and usable outputs.
At this time of pressure on national finances, good quality and timely evaluation evidence is critical to ensure effective public spending and delivery of government objectives. The ongoing embedding of evaluation practice across government can be seen in the work of the Evaluation Task Force, which has included requesting and publishing Department evaluation strategies, providing resources on conducting evaluation and developing an Evaluation Registry – a database on evaluations across government. It also has a live call open for its Evaluation Accelerator Fund (EAF), which supports evaluation across government to transform understanding of the impact of activity in priority policy areas.
However, in relation to complexity in evaluation, there is more to do to build on the foundations developed in recent years. There is a need for a nuanced approach to evaluation that acknowledges both the challenges and opportunities presented by the complexities associated with delivering long-term environmental and social impact.
Where there is deep complexity, policy design, delivery and evaluation are hard. It is difficult to know what will work at this time and in these circumstances. Responses to policy implementation may be quite different to those anticipated, requiring early detection and ‘in-flight’ course correction. Complex pathways to impact, enacted over long timescales make detecting and attributing change challenging. Communicating complexity and its implications is difficult, and people can struggle to acknowledge and work with the uncertainty complexity brings. The resources needed to work through this complexity, including both funding and people’s time, can be limited, with stakeholders often focused on the short term – something that the rise of popularism can only exacerbate. However, it is important that early lessons are learnt and turned into actionable, timely recommendations.
Some ways of tackling the challenges include:
- Early evaluator involvement, integrating evaluation with policy development,enables learning from the earliest stages. Adaptive and developmental evaluation approaches provide an effective and practical approach to dealing with complex issues, building-in mechanisms for involving and sharing emerging findings with stakeholders from the earliest stages.
- A pragmatic mindsetshould be adopted – shifting focus to methods that balance complexity considerations with practicality and adopting actionable metrics that translate into concrete actions. National frameworks such as Scotland’s Performance Framework may be helpful, helping ensure evaluations align with broader policy goals.
- A need to build evaluation capacity – initiatives like evaluation task forces and communities of practice within government can foster a culture of evaluation, but more training is required. Equipping stakeholders and evaluators with a strong understanding of complexity in evaluation will lead to more effective practices.
The CECAN Annual Conference, due to be held on 30th June 2025, will pick up on these issues and it is to be hoped that any revisions of the Magenta Book can integrate the challenges of handling complexity in policy into the main body of the Guidance. There is also the issue of whether it is time for a closer integration of Green Book and Magenta Book guidance, but that’s a topic for another blog.