By Ben Shaw, Principal Consultant, CECAN Ltd (December 2024)
The CECAN 2024 conference ‘Looking Back and Looking Forward: Closing the loop from evaluation to policy making’ considered the questions ‘What is the role of systems thinking in evaluation? Is it the answer to wicked problems?’ in one of its breakout sessions. In this blog Ben Shaw considers these questions, building on some of the themes raised at that session and his recent experience of systems thinking in CECAN Ltd’s evaluation and research projects.
In December 2024 I attended an excellent Systems Thinking Masterclass at the Oxford Martin School. Convened by Dr Pete Barbrook-Johnson (Oxford), Prof Jaime Toney (Glasgow), Dr Elaine Heslop and Mike Heslop (Glasgow + Lucidity), it brought together representatives of government departments and agencies and other stakeholders to discuss the current state of the art of systems thinking in policy making. The event highlighted the appetite for systems thinking in policy making and delivery, the diversity of approaches in use and being developed, and some challenges and opportunities for future work. Among the challenges raised was that of embedding a broad ranging and multi-faceted set of concepts, methods and processes into current policy making practice and delivery.
At the CECAN conference held in July 2024 there was similar appetite for systems thinking in evaluation and acknowledgement of the role it could play in resolving long-standing policy and evaluation challenges. The strengths of systems thinking in evaluation were highlighted such as:
- Its value in characterising the whole systems of interest to ‘see the big picture’, locating the evaluated activity fully in its context and linking causes and symptoms,
- The ability to provide a framework for wider evaluation activity,
- How systems thinking can complement, support and strengthen other methods, e.g. developing complexity-sensitive theories of change,
- The contribution systems thinking can make to understanding ‘why’ questions and the pathways to outcomes and impact,
Against these strengths, weaknesses were highlighted such as:
- The resource implications of doing systems thinking well, especially the requirements of participative processes,
- The generation of unmanageable and paralysing complexity through too many actors and issues being considered together — the everything, everywhere all at once problem,
- Systems thinking can question the feasibility of an evaluation and be hard to turn into a tangible methodology.
These weaknesses are not inherent to systems thinking, but rather are attributes of systems thinking done badly. They can be resolved by setting clear boundaries and scope for the evaluation. Clarity on the evaluation purpose, its use and user needs is critical to any successful evaluation, whether using systems thinking or not.
A striking thread in these two conversations is that systems thinking blurs the boundaries between policy making and evaluation. This has also been an important insight from CECAN’s work — that systems thinking and the presence of complexity compresses and blurs the traditional distinction between ex ante appraisal and ex post evaluation. This is particularly the case where there is the need for ongoing and/or rapid learning and adaptation to understand, manage and resolve the delivery of a policy challenge.
The blurring of ex ante and ex post points to the need to embed systems thinking and consideration of its implications for evaluation strategies early in policy development. CECAN’s experience of systems mapping, as an important tool of systems thinking, is that it can be valuable in all stages of policy development, delivery and evolution, contributing to understanding and problem formulation, response development, stakeholder engagement, planning and delivery, and monitoring and evaluation.
We also noted that systems thinking needs to be fully embedded in an evaluation and that the insights generated should inform all the methods used in the evaluation. We have observed some commissioning processes seeking a systems thinking element as a discrete or ‘bolt-on’ element of an evaluation. Systems mapping, for example, can be used at early and scoping stages while the rest of the evaluation reverts to the use of conventional methods. This adds cost to evaluation without getting the full benefits of systems thinking.
The second half of the breakout session considered whether systems thinking is the answer to Wicked Problems. The original definition of a wicked problem was made in the 1970s by Rittel and Webber, who developed ten definitional propositions to a wicked problem. Space precludes detailing these, but essentially wicked problems are:
“that class of social system problems which are ill-formulated, where the information is confusing, where there are many clients and decision makers with conflicting values, and where the ramifications in the whole system are thoroughly confusing”[1]
Critically, Rittel and Webber highlighted that wicked problems confound rational approaches to problem solving. The US could put a rocket on the moon – a technical challenge, but could not solve social challenges in its cities – a wicked problem – with conflicting formulations of the problem, responses and what success looks like.
So can systems thinking be the answer to Wicked Problems? If by ‘answer’ we mean a single conclusive resolution, then the answer is no. However, what systems thinking can provide is concepts, tools and importantly processes that can explore, respond and improve wicked problems. With this approach, then the answer is yes.
A final observation from these events and recent CECAN work is that there is encouraging increase in systems thinking for policy and evaluation, the development of methods, tools and processes and the delivery of significant value. However, while there is a growing number of individuals using systems thinking and a community of people who ‘get it’, there is need to think more about the institutional enablers of system thinking and about the governance and structures this may require in policy and delivery to capture its full benefit. Can we move from individuals capable of systems thinking to developing systems thinking-capable organisations?
CECAN is currently working on a new ‘step-by-step guide’ on How to go about doing a complex evaluation’ which will be released in early 2025. This will explore these and wider issues more fully. To receive notification of when this is available, please subscribe to the CECAN mailing list here.
[1] Excerpt from Brian W. Head, 2022, Wicked Problems in Public Policy, Palgrave Macmillan, Cham, Switzerland.