Programme of work and research themes
In March 2019, the Centre for the Evaluation of Complexity Across the Nexus (CECAN) transitioned into its second phase with the aim of not only continuing and extending the work begun in 2016, but also of deepening and accelerating the transformation of complex policy evaluation. The CECAN Executive Group devised and refined a programme of work and mapped it to our Research Themes, as indicated in the table below. As of the 2020-21 year, CECAN’s primary focus for the remainder of our funding period will be on securing our legacy by consolidating and embedding our work in practice.
Research Goals
- To explore the contribution that complexity-appropriate evaluation methods can make across the policy cycle.
- To identify options for innovation in complexity-appropriate evaluation methods.
- Where possible, to conduct real world testing and refining of complexity-appropriate evaluation methods and co-produce solutions to the evaluation challenges posed by complexity in policy making and analysis.
- To understand the processes of building individual and institutional capacity to deliver complexity-appropriate policy analysis.
Knowledge Exchange Goals
- To build on the release of CECAN’s annex to the revised HM Treasury Magenta Book, “Handling Complexity in Policy Evaluation” by developing and delivering a series of engagement and training events to promote the Guidance and strengthen the case for using it in practice across Government.
- To consolidate and extend CECAN’s network of policy-makers, analysts, evaluators inside and outside of Government.
- To continue our programme of capacity building events (seminars, webinars, workshops) and outputs (Evaluation Policy and Practice Notes (EPPNs), reports and journal articles) using them to meet the knowledge needs of existing and new audiences in the CECAN network.
- To create a CECAN Evaluation Toolbox to enhance the embedding of CECAN among stakeholders.
Delivering Impact Goals
- To further develop the reach and practical implementation of CECAN’s work by exploiting opportunities through CECAN Ltd. In particular by positioning CECAN Ltd as a provider of ‘enhanced consultancy’ through our established reputation to act as an embedded ‘critical friend’ (i.e., not just another consultancy).
- To capitalise on the significant further opportunities for impact that the publication of CECAN’s guide to Handling Complexity in Policy Evaluation in HM Treasury’s Magenta Book and the Defra Complexity Evaluation Framework presents to translating the guidance into practice (e.g. training on methods or advice to commissioners of evaluation) and to seek opportunities (possibly via CECAN Ltd) to draft more departmental specific guidance, supported by the publication of our own CECAN Evaluation Toolbox.
- To use this work to underpin the possibility of establishing CECAN as a ‘standard’ for evaluation.
Notable highlights and reflections on progress (2019 – present)
Workstream 1: Exploring the contribution that complexity-appropriate evaluation methods can make across the policy cycle
One of the key insights from CECAN’s first phase was that complexity appropriate methods have value and should be used across the policy cycle in policy development and appraisal, not just in evaluation. Complexity also requires evaluation to be conducted earlier in the policy process with more frequent cycles of ongoing data collection and action being used to inform policy implementation.
Centre for Connected and Autonomous Vehicles (CCAV) case study: a systems map of ‘Mobility as a Service’ across the policy cycle, covering policy design and evaluation
We have raised awareness of CECAN and complexity appropriate methods of analysis across Government and beyond our original partners (i.e., the Department for Environment, Food and Rural Affairs (Defra) and the Department for Business, Energy and Industrial Strategy (BEIS)). We receive regular requests for assistance in policy development and evaluation. With the BEIS / Department for Transport (DfT) Centre for Connected and Autonomous Vehicles (CCAV), we have developed a new case study to conduct a systems mapping of ‘Mobility as a Service’ across the policy cycle, covering policy design and evaluation. The CCAV case study outputs will feed directly into a regulatory review and consultation on the Future of Mobility. This work is part of the wider Industrial Strategy: a key area for CECAN to build expertise and awareness of its work, and is further described in CECAN Evaluation Policy and Practice Note (EPPN) No. 18.
CECAN’s Complexity Evaluation Framework for Defra, January 2020
In the first half of 2019, CECAN was commissioned (via its spin out company, CECAN Ltd) to develop a Complexity Evaluation Framework (CEF) for Defra. The framework is intended to guide analysts and policy-makers across Defra in ensuring complexity thinking is appropriately embedded into evaluation with the aim of producing robust evaluation evidence to inform policy. In January 2020, following a process of a peer review, Defra published the CEF and subsequently commissioned an independent evaluation of it. This evaluation was undertaken by Steer-ED Consultancy, with positive findings.
Publication of CECAN’s annex on complexity in HM Treasury’s Magenta Book, March 2020
CECAN’s guidance, Handling Complexity in Policy Evaluation, in HM Treasury’s Magenta Book (the central guidance for all UK government departments on evaluation) was published in March 2020. More than 200 policy analysts, evaluation professionals and other stakeholders participated in the interactive launch events that CECAN ran in London, Birmingham and online through March and April 2020. CECAN has plans for a series of future engagement activities that will further promote and support the use of the guidance.
CECAN Special Issue for Evaluation journal, January 2021
CECAN was invited to compile a Special Issue for the journal Evaluation (Sage) and this was published in January 2021. The CECAN Special Issue, ‘Policy Evaluation for a Complex World: practical methods and reflections from the UK Centre for the Evaluation of Complexity Across the Nexus’ consists of six papers plus an editorial, which draw not only on our case study work but also on our broader learnings, e.g. reflections on developing our guidance for HM Treasury’s Magenta Book, our work on Systems-based Theory of Change, and the application of our COMPLEX-IT app.
Workstream 2: CECAN Methods Innovation Lab: delivering innovation in complexity-appropriate evaluation methods by exploration of the full range of new and emerging methods
Participatory Systems Mapping
Our Participatory Systems Mapping approach – developed and used in the first phase of CECAN case studies – has continued to grow in popularity, leaving a lasting impression on those who engage with it. Through CECAN Ltd (CECAN’s spin-out company that was launched in Summer 2018) we have worked with Mott Macdonald on a Systems Analysis of Water Resources project funded by Defra. The project used CECAN’s Participatory Systems Mapping approach to explore how whole systems and complexity-appropriate approaches can be used to provide insight on, and inform management of, river catchments. Defra described CECAN as “pivotal to the success of the project” and Mott Macdonald requested repeat engagement with CECAN when they were commissioned to work on a different project with Water Resources South East. Through ongoing engagement with government and industry partners, CECAN is becoming established as ‘the place to go’ for providing technical advice in complex evaluation.
CECAN has also been addressing the broader challenge of turning systems maps into something actionable for evaluation purposes. A team from CECAN – including complexity and system mapping experts and evaluation practitioners – has developed a methodology that plots a path through from systems mapping to a Theory of Change (ToC), which evaluators can use to frame their evaluations. This allows evaluators to develop practical ToC maps that take into account feedbacks, wider context, and potential negative or unexpected outcomes. A paper has been included in the Special Edition of the journal Evaluation. The approach has generated particular interest with the team at the Medical Research Council and Scottish Government Chief Scientist Office, Social and Public Health Sciences Unit (SPHSU)at the University of Glasgow. CECAN is developing a collaboration with SPHSU and we are exploring a range of topics of mutual interest, including system based ToCs and innovative tools designed to promote collaboration on mapping, understanding and communicating complex systems. Via CECAN Ltd, we are also working with consultancy partner Technopolis to develop a Theory of Change tool for Defra.
A book about Participatory Systems Mapping will be forthcoming in 2021, authored by CECAN researchers, Dr Alex Penn and Dr Pete Barbrook-Johnson. The book will explore a range of new and old systems mapping methods, focussing on practical insights for mapping in real world contexts. Each method covered in the book will be accompanied by a podcast and a detailed guide on the realities and challenges of building and using systems maps.
New and emerging methods
CECAN is looking to bring a wider range of methods into our work with a focus on further innovations in systems mapping approaches, exploring ‘rapid methods’ suitable for addressing complexity quickly and for non-experts. In addition, CECAN has been developing a relationship with the Alan Turing Institute, looking at work on Digital Twins. Work in this area is also underway with CECAN researchers based at UCL.
Tools
CECAN has developed a number of practical tools to facilitate uptake of our methods. They are all open source and free to run online or download. They have all been successfully used in real-world situations.
COMPLEX-IT
In November 2019, we launched our COMPLEX-IT software. Developed by CECAN researchers at Durham and Surrey, COMPLEX-IT is designed to increase access to the tools of computational social science (e.g., artificial intelligence, micro-simulation, predictive analytics) through its visually intuitive learning environment. The Covid-19 Community Health and Social Care Modelling Team at Durham University were able to use the COMPLEX-IT software to create a model that provided weekly reports on Covid-19 for the local authorities in the North East of England to support decisions and planning accordingly.
PRSM
We have also developed a Participatory Systems Mapping app, PRSM, which enables groups of people, each using their own computer (or tablet) to collaborate in the drawing of a system map (a network of factor nodes and presumed causal links). The app includes a built-in chat feature and every edit is broadcast to all the other participants in real time. Development of the PRSM app has drawn on CECAN work at Warwick on trophic analysis (quantifying how far up or downstream each node is to indicate the extent to which the network fails to line up). The PRSM app is being used in our work with the Health and Safety Executive developing a map, with other stakeholders, of how a wide range of workplace and other factors influence mental health. The great strength of the app for this application is the ability it provides for groups to work collaboratively on maps in virtual workshops.
“Choosing Appropriate Evaluation Methods” tool
CECAN Researcher, Dr Barbara Befani, has updated the “Choosing Appropriate Evaluation Methods” tool that she originally developed in 2016 with Bond Consultancy. The tool is an accessible aid to help users to understand evaluation methods and choose the right ones for their purposes. It has already proven popular in CECAN’s case study work.
Workstream 3: Real world testing and refining of complexity-appropriate evaluation methods to co-produce solutions to the evaluation challenges posed by complexity in policy making and analysis
Case studies with Defra
CECAN has been engaged at both a local and national level in Defra’s Tests and Trials (T&T) projects to develop and inform policy under the Environmental Land Management Scheme (ELMS). CECAN researchers at Newcastle’s Centre for Rural Economy are involved in the Curlew Contracts T&T with Northumberland National Park, acting as a ‘critical friend’, evaluating activity and processes, offering internal challenge, drawing together lessons, and reporting to Defra. Through CECAN Ltd, CECAN is working in collaboration with ICF Consultancy to advise on complexity appropriate methods that could be applied in the national evaluation. The impact of our work with Defra through our involvement with the ELMS programme is to bolster CECAN’s role as a ‘critical friend’ rather than a consultancy partner. We have strengthened our identity as an advocate for complexity thinking and approaches in policy making. We have also learned about the importance of building not only capacity but complexity competence within stakeholder groups.
The use of the CECAN approach to Participatory Systems Mapping has spread well beyond the initial future farming / rural development case study group at Defra. A joint CECAN/Defra report capturing lessons and impacts from this work was published internally at Defra in December 2019 (Penn, A.S, Jones, C. et al (2019) Understanding Complexity in the Rural System in England: Analysis and Insights from the FF&C Systems Mapping Project, CECAN & Department for Environment, Food and Rural Affairs. UK Government Internal Report) and this has been widely circulated across Defra by the Future Farming team. Nonetheless, converting this into academic publications is difficult as the report contains programme information that is highly classified.
Case studies with BEIS
CECAN was authorised to publish a report about our Energy Trilemma case study with BEIS (2018) and a subsequent paper based on our two BEIS case studies – ‘Participatory systems mapping in energy evaluation’ – is included in the CECAN Special Issue in Evaluation. Our most recent case study culminated in a final report that was submitted in August 2020 to the Centre for Connected and Autonomous Vehicles (CCAV) and a CECAN Evaluation Policy and Practice Note (EPPN).
Other case studies / new areas of work
In developing new case studies, we have wanted to ensure that they use approaches that we have not used to date or where there is a clear opportunity for development of an existing approach. For example, there is a clear demand for further case studies based on systems mapping approaches. However, we have resisted developing these unless there is an opportunity for methodological development of contextual novelty, as with the CCAV case study.
CECAN’s case study work, particularly with Defra, continues to be extremely well valued by stakeholders. Stemming from our systems mapping work with Defra’s rural development team, CECAN researcher Dr Alex Penn was invited by the head of evidence at the Marine Management Organisation to give a keynote presentation at the Defra MOOSC conference (an internal conference for Defra and all arm’s length bodies involved in Marine policy) in early November 2020. Her presentation covered our use of complexity approaches in a large, complex, rapidly changing policy area to inspire those working in the Marine policy area, who are keen to use our approaches as they tackle the challenges presented by Brexit. This is a great endorsement of CECAN’s work and presents a strategic opportunity for us to open up work in Defra Marine and to do something of a similar scale as happened in Future Farming. CECAN is eager to capitalise on the high-level endorsement of and clear appetite for our work that we have been creating but the lack of funding available for our future will present a challenging obstacle.
CECAN Ltd consultancy
CECAN continues to engage heavily in real-world testing and innovation, aided by its ability to bid for government contracts through CECAN Ltd. Examples of this include our Water Systems Mapping work at Defra mentioned previously; our work with Technopolis Ltd to evaluate UKRI’s Strategic Priorities Fund; our evaluation of the Forces Connect South East Veterans’ Hub Project for Surrey County Council; our work with the Food Standards Agency to design and apply a methodology to compare and evaluate the impact of their research outputs; our work with the Climate Change Committee on the interacting risks of climate change; and our work with the Health and Safety Executive to explore how stakeholders can work together to improve mental health in the workplace. Further information about CECAN Ltd’s work is available here.
Workstream 4: Building individual and institutional capacity to deliver complexity-appropriate policy analysis
Workshops and seminars
Our programme of workshops, seminars and webinars is intended to promote the work and outputs from CECAN along with the work of related researchers and evaluators, and build capacity to use complexity appropriate approaches. Our webinars regularly reach audiences of 100 or more with many participants joining from outside the UK. In May 2019, CECAN researchers were filmed by SAGE discussing their research on complex systems using participatory systems mapping and the film is included in the SAGE Research Methods website. We held a CECAN showcase for BEIS analysts and researchers in October 2019 to raise awareness of complexity concepts, methods and practices. Feedback from our seminars, webinars and workshops demonstrates that these are received positively. This is further illustrated by requests for repeat engagement, for example the Department for Education (DfE) requested CECAN to provide another bespoke training course, on a new subject, in November 2019. CECAN also provided a one-day bespoke training session for the £5m SIPHER project team: an ambitious systems science project focussed on public health issues at different geographical scales in the UK, funded by the UK Prevention Research Partnership.
CECAN Ltd training provision
CECAN Ltd has facilitated the further development of CECAN’s bespoke Continuing Professional Development (CPD) training provision, enabling CECAN to offer organisations a more cost effective and focused short course delivery method. For example, in 2019 the company provided a two-day course, ‘Advanced Evaluation’, to over 20 Department for Education (DfE) delegates at a location of their choice. Based on CECAN’s Syllabus (2017), the course was tailored to the requirements of the DfE and received extremely positive feedback. CECAN Ltd was later commissioned to deliver bespoke training in complex evaluation to Public Health England between November 2019 and March 2020. CECAN has also continued working with the UK Evaluation Society – via CECAN Ltd – to develop and deliver a CPD Masterclass series. Due to its sell-out success, the series was extended beyond its initial scope.
CECAN PhD studentships and Fellowships
CECAN’s PhD students continue their projects building their own and wider partners’ knowledge and capacity to deal with complexity. Current projects include:
- “Interfacing scientific data and complexity policy” (Lisa Stoltz, Warwick)
- “Building policy-makers’ capacity to evaluate complexity” (Anna Kaxira, Westminster)
Completed projects include:
- “A case study of the Marine Pioneer: investigating local management and natural capital valuation” (Betheney Wills, Surrey): Betheney was awarded a PhD in November 2020 following a viva in which her thesis was approved without corrections. Her doctoral research was a case study of the Marine Pioneer, a UK government pilot testing natural resource governance and management principles from the 25-Year Environment Plan. The aim of the research was to investigate the effects of testing local scale governance alongside the experimental implementation of the Natural Capital Approach in the Marine Pioneer. Betheney has since been successful in securing the position of Marine and Fisheries Senior Research Officer in the Social Science Team at Defra.
Following on from the funded Fellowship programme in the first phase of CECAN, we have been able to support three ‘no cost’ or alternatively funded Fellowships, and these are:
- “Evaluating the impact of adaptation measures and resilience building by examining the complexity nexus using a systems-thinking complexity approach” (November 2018 – January 2019, funded by DFID, via IOD Parc)
- “Evaluating the Potential for Natural Capital Investment to Reverse Soil Degradation: A Dynamic Simulation Approach Exploring Connections between Soil Health and Money in England” (from February 2019, No-cost Fellowship)
- “How do organizations learn from evaluations? Highlighting the role of social complexity” (from February 2019, No-cost Fellowship)
Although the Fellowship programme is now closed, outputs from the work continue to be produced. An example of this is CECAN Fellow, Dr Richard Gunton’s Pluralistic Evaluation Framework, which was presented at a joint meeting of the British Ecological Society journal, People and Nature, and the Valuing Nature Programme, in March 2020. Jonathan Nichols’ fellowship led to a presentation to Defra and a CECAN webinar, with the results from his work being written up in his Masters’ thesis: “Evaluating the potential for natural capital investment to reverse soil degradation: a dynamic simulation approach exploring connections between soil health and money in England”. Other publications arising from CECAN Fellowship projects include:
- Barbrook-Johnson, P. et al., (2020), How do policy evaluators understand complexity?, Evaluation, Vol. 26(3) 315–332. Sage. DOI: 1177/1356389020930053
- Kirsop-Taylor, N., Hejowicz, A. P., Scott, K., (2020), Four cultural narratives for managing social-ecological complexity in Public Natural Resource Management. Environmental Management 66, 419–434. Springer. DOI: 1007/s00267-020-01320-6
- Kirsop-Taylor, N., Hejowicz, A. P., (2020), Designing public agencies for 21st century water-energy-food nexus complexity: the case of Natural Resources Wales. Public Policy and Administration, 0(0) 1–21. Sage. DOI: 1177/0952076720921444
Formal guidance documents
Concepts, methods and frameworks for complexity appropriate evaluation are being incorporated into government guidance and practice, with CECAN taking a leading role in this. As previously mentioned, CECAN developed a Complexity Evaluation Framework (CEF) for Defra that was published in January 2020. The CEF has provided Defra with a clear and accessible description of what complexity is and why it matters to them, along with the rationale for why complexity-appropriate is crucial to help, and what this looks like. Furthermore, through interviews, workshops and presentations over the course of the CEF development project, the CEF work has contributed to capacity building in Defra and the Defra Group by raising awareness of and fluency in complexity concepts and complexity-appropriate evaluation. The CEF was used by Defra almost straight away to inform the early stages of an evaluation strategy for forthcoming evaluations in waste and resources. Other government departments have expressed interest in similar work tailored to their specific policy challenges.
CECAN’s work on producing guidance documents has been extremely well received. Anecdotal feedback from participants at our Magenta Book Guidance events confirms that users can really see the value of applying the guidance to their own work. With regard to Defra’s Complexity Evaluation Framework, the findings of Steer-ED Consultancy’s independent impact evaluation were published in October 2020 and conclude that: “Overall, there is no question that the main content of the CEF, in terms of the key considerations and questions, is proving useful and is already contributing to adoption of some complexity-aware evaluation practices in Defra.” and “Although still in early stages of use, there is already some evidence that the CEF is changing behaviour – for example by encouraging a more holistic or nuanced approach to evaluation, the adoption of complexity-appropriate methods, or more stakeholder-driven approaches.”
New areas of work
The diversity of requests made to CECAN to conduct work and the inclusion of complexity appropriate methods in tender specifications indicates that CECAN is having an impact on the practice of policy evaluation amongst commissioners, providers and users of evaluation across Government and beyond. New work funded by the charitable foundation Lankelly Chase is exploring the third sector’s ability and needs in conducting complexity appropriate and systems-based evaluation in the area of severe and multiple deprivation. This is broadening the remit of CECAN from Westminster and Whitehall policy making to the local delivery of services and the relationship between central policy and local service delivery.
Workstream 5: Securing CECAN’s future and its legacy
CECAN researchers are engaged in a number of projects that draw on the expertise we have developed. Some examples of these are:
- ANTICIPATE: Actively anticipating the unintended consequences on air quality of future public policies (NERC, February 2019 – August 2022), Prof. Nigel Gilbert
- LIPSIT: Local Institutions, Productivity, Sustainability and Inclusivity Trade-offs (ESRC, October 2019 – September 2021), Prof. Nigel Gilbert
- “Implementing innovative legal and organisational structures for a just transition: a catchment management beacon project” (University of Surrey ESRC Impact Accelerator Account), Dr Pete Barbrook-Johnson
- PURE: Peri-Urban Resilient Ecosytems (GCRF Pump Priming Grant, January – July 2020, extended due to Covid-19 pandemic), Dr Adam Hejnowicz and Dr Richard Gunton
- SysRisk: Systemic environmental risk analysis for threats to UK recovery from COVID-19 (NERC, January -December 2021), Prof. Nigel Gilbert
Reflecting on objectives
In its second phase, CECAN will deepen and accelerate the transformation begun in 2016, by:
- Extending complexity appropriate approaches and methods across the policy cycle, providing a coherent and agile approach to the development, monitoring, implementation and evaluation of policy;
- Broadening the reach of CECAN innovations from the Nexus to other policy areas;
- Promoting joined-up governance and agile policy making across government and non-governmental organisations;
Ensuring the long-term sustainability of the Centre.
In its first phase, CECAN explored the domain of policy evaluation and complexity developing a novel co-production approach for a research centre that was challenge-led, user-focused and interdisciplinary. In the second phase we have been consolidating this approach to develop the considerable research agenda that lies at the intersection of the domains of complexity, policy, evaluation and nexus. The scale of these issues present challenges for CECAN – transforming the practice of policy evaluation across Government is a huge and long-term challenge that goes beyond what a centre of CECAN’s scale can achieve.
1. Leadership: Provide practical and intellectual leadership and an international focus for the development of innovative approaches to policy evaluation and complexity.
2. Methodological innovation: Develop concepts and methods for complex policy evaluation based on an understanding of user needs.
3. Methodological application: Explore the application of complexity appropriate methods with policy-makers and analysts in departmental and agency settings.
4. Capacity building: Build capacity to use these methods among policy-makers, evaluation practitioners and researchers.
An important insight from CECAN is that the uptake of complexity appropriate evaluation and analysis requires more than the existence of methods that can address complexity. Complexity appropriate methods require complexity capable institutions. The problems of interest, the institutional settings, structures, processes and cultures they occur in, the knowledge, expertise and capacity of individuals in those institutions and resources available will all affect the uptake of methods. In its second phase, CECAN has placed a more explicit focus on understanding how institutional capacity can be built to adopt new methods of complexity analysis. We are also having to think about the models of innovation applicable to methods development. It is also clear that the scale of the challenge requires us to consider the means and mechanisms that insights from CECAN can be scaled more widely across Government without CECAN having to deliver across all of Government.
CECAN’s Legacy: what will our story be?
Since its launch, CECAN has worked directly with government departments to:
1. Engender a culture shift within Government such that acceptance and awareness of complexity-appropriate policy evaluation methods is increasing.
-
- CECAN’s co-produced suite of pan-governmental case studies spanning a multi £Billion policy portfolio have involved sustained dialogue and long-term relationship building with stakeholders.
- References to CECAN and to our work have appeared with increasing frequency in government Invitations to Tender (ITT) and Areas of Research Interest (ARI) documents (e.g., Defra’s ITT for CEF and the FSA’s ARI 2017-18).
2. Build capacity within Government to ensure that such methods may be used appropriately.
-
- CECAN has built a network of c. 1000 government representatives, practicing evaluators and researchers, academics, evaluators, policy analysts, method and domain specialists, professional organisations (e.g. UK Evaluation Society and the Cross Government Evaluation Group (CGEG)), think tanks and the media. Members of this network actively engage with CECAN activities such as workshops, CPD seminars, webinars and larger conferences.
- Feedback from CECAN’s network confirms that CECAN’s work has contributed to a civil service that is better equipped to tackle their complex evaluation challenges.
- CECAN continues to deliver bespoke training to departments outside of those working in ‘Nexus’ areas, for instance to the Department for Education and to Public Health England. CECAN Ltd continues to facilitate and increase CECAN’s ability to offer bespoke training and consultancy.
3. Ensure the wide-scale operationalisation of these methods by contributing directly to official government guidance on evaluation.
-
- CECAN’s guidance in the revised HM Treasury Magenta Book, “Handling Complexity in Policy Evaluation”, March 2020.
- CECAN’s Complexity Evaluation Framework for Defra, January 2020.