On 11th July nearly 200 delegates attended CECAN’s annual conference in the elegant and central London venue of One Birdcage Walk. Just across the road is HM Treasury, so as we enjoyed our tea and lunch, we were able to peer through the windows of the UK government policy-making machine and ponder how it is addressing the complexities of the policy and evaluation challenges now facing the UK, and also how it might benefit from the rich and insightful discussions taking place at the conference.
The Challenges of Complexity for Government
Drawing on his experiences at the centre of government Matthew Taylor, Chief Executive of the RSA, set the scene for the day in his keynote speech. He highlighted the major problems of the declining legitimacy of the institutions of government and the effectiveness of government in delivering for their citizens. There is an urgent need to address these challenges and better link the communities of policy innovation and politics. This short blog can’t do justice to the full content of Matthew Taylor’s comments but he summarised his response to the challenge of complexity in the pithy statement ‘Think like a system. Act like an entrepreneur’ and concluded that his speech had been an extended advertisement for all the work that CECAN had been doing – an endorsement we were of course delighted to hear.
The following panel discussion with Gary Kass (Natural England), Charlotte Allen (Defra), and Penny Hawkins (independent evaluation consultant) picked up on the themes of Matthew’s speech. Criticisms were raised of ‘traditional’ models of evaluation run for accountability rather than learning purposes. Accountability obviously shouldn’t be ignored, but the context of implementation and need to be responsive and adaptive in policy-making, implementation and evaluation to create learning is vital. Point in time evaluations, reporting many years after the event are of limited use when evidence that informs emerging practice in fast moving and fluid situations is often needed. The need to create spaces for experimentation for better ways of working which involve the people affected by interventions – work with, not on stakeholders – was agreed across the panel.
Complexity, policy and evaluation in practice
With these challenges in mind delegates moved into two sets of four parallel sessions. These were a showcase for the diversity of work the CECAN team and its wide network have been doing on many of the challenges highlighted in the opening plenary. They were intended to provide something for everyone – primers on complexity and evaluation for those new to the area, more detailed sessions on the latest methodological developments for seasoned evaluators, and a selection of CECAN’s growing body of real world case studies. While CECAN is an academic research centre our work has intended to be of direct value to policy-makers and analysts and has involved novel ways of working with our co-funders. The co-produced case studies have been getting to grips with complexity and evaluation in real world policy settings with our government partners. The presentations from these sessions are available and much of the material is also summarised in existing or forthcoming CECAN outputs, notably our Evaluation Policy and Practice Note series, the CECAN Syllabus and the CECAN Manifesto.
Building government capacity to deal with complexity
The final plenary moved on to a forward-looking discussion of how capacity to deal with complexity can be built across government. With contributions from Siobhan Campbell (Department for
Transport and outgoing chair of the Cross Government Evaluation Group), Jonas Schönefeld (Technische Universität Darmstadt) and Dione Hills (Tavistock Institute) and concluding comments from Elliot Stern (Chair of CECAN’s Advisory Board) the value of work currently being done in this area is clear to those currently involved. However, the challenge is to scale it up more widely to develop awareness and practice of complexity sensitive policy-making and evaluation across government. This needs action on multiple fronts – ongoing joint working and co-production of policy and evaluation challenges between policy-makers, analysts and evaluators both inside and outside of government; a shared plain English language to communicate with on these issues; creating spaces for experimentation and testing of methods; consideration of evaluation governance and the nature of institutional support for new approaches.
The forthcoming revision of the Magenta Book and Annex on dealing with complexity in evaluation will be an important step forward. However, embedding complexity-sensitive approaches into government policy-making an evaluation will require ongoing work and resources. The prize, and its significant one, is better and more effective policy and government and a contribution to the rebuilding of legitimacy of government. We look forward to continuing to contribute to this agenda through CECAN.
Ben Shaw, Deputy Director of CECAN
Details of the conference and presentation slides are available below:
Search #EvaluatingComplexity on twitter for comment and photos on the event.
Presentations:
- 1.1 Evaluation of Complex Policy and Programmes: A CECAN Module for Future Policy Analysts and Evaluators
- 1.2 Developing Guidance for Government on Dealing with Complexity in Evaluation– an Annex to the Magenta Book?
- 1.3 Making Evaluation Real: Qualitative Comparative Analysis and environmental policy (Case based methods Part 1):
- 1.4 Participatory Systems Mapping: Exploring and Negotiating Complexity in Evaluation with BEIS and DEFRA
- 2.1 Where do I start in Navigating Complex Evaluation Methods? Developing and using the Choosing Appropriate Evaluation Methods tool
- 2.2 Two New Advances in Finding Dynamic Patterns in Complex Systems (Case based methods – Part 2):
- 2.3 Complexity, Policy and Evaluation in Practice: Improving Theories of Change to Address Complexity
- 2.4 Getting value, values and valuation into evaluation:
Posters:
- Anna Kaxira – Complex Policy Interlinkages Between Anaerobic Digestion and Food Waste in the UK
- Betheney Wills – Case Study of the Marine Pioneer
- Brian Castellani – Case Based Methods and Agent Based Modelling
- Nick Kirsop-Taylor – The Criticality of Culture in Delivering and Evaluating Public Institutional Approaches to Managing Complexity Across the Nexus: The Case of Natural Resources Wales
- Richard Gunton – The Pluralistic Evaluation Framework
- Thomas Dixon – Democratising Complex Evaluation
- Henry Leveson-Gower – Trust in Water
- Joanna Boehnert – The Visual Representation of Complexity