27th July 2021
Authors:
Jaideep Visave, University School for Advanced Studies (IUSS) – Pavia, Italy,
Scott Chaplowe, Evaluation and Strategy Specialist, and
Dr Adam P Hejnowicz, School of Engineering, Newcastle University
In June 2021, the Sustainability Research & Innovation Congress 2021 (SRI2021) held a series of online workshops as part of its “Transformations Conference 2021: Enabling Positive Tipping Points”. This post shares the learning from one workshop, “Evaluation as a Pathway to Transformation for a Sustainable Future.”
The workshop responded to the growing focus on transformation in the UN 2030 Global Agenda, and the need to address urgently today’s globally interconnected social and environmental challenges. It framed transformation as call for radical and rapid innovation, and a shift away from business-as-usual approaches across multiple scales and sectors to safeguard our planet and ensure related human prosperity in the years ahead.
As a field that straddles both theory and practice, evaluation is uniquely positioned to support transformational learning and change. Indeed, the growing focus on transformation within evaluation is reflective of this wider narrative as well as the increasing uptake and influence of complex systems analysis.
However, evaluation’s capacity to contribute to transformational change depends on its ability to transform within. This workshop, “Evaluation as a Pathway to Transformation for a Sustainable Future,” brought together different perspectives and considerations to help elaborate the bigger picture – sustainable development.
We asked how can evaluation, a profession in the business of assessment and advising, inform and hasten transformation? This was explored through three lenses: i) methodological cautions and recommendations for evaluation’s potential to contribute to transformational change; ii) the critical role of new data science and related technologies for enhancing evaluation’s transformational potential; and iii) the potential role of evaluation for delivering the sustainable development goals (SDGs) through national voluntary reviews.
Sustainability Revisited: An illustration from evaluation[1]
Evaluation’s potential role in affecting transformational change will largely depend on its ability to transform itself. Drawing upon the article Scott Chaplowe published with Adam Hejnowicz, Evaluating Outside the Box: Evaluation’s Transformational Potential, this first presentation by Scott summarized four interrelated “boxes” or bad habits that constrain evaluation’s transformational potential: a project fixation, a short-term temporal fixation, a quantitative fixation, and an accountability fixation. He then illustrated the how evaluation’s transformational potential is limited by the “Sustainability Criteria” in the DAC Evaluation Criteria, which restricts sustainability to the continuity of the planned impact in an intervention rather than the original intention of sustainability: to look at the holistic impact on interrelated natural and human systems.
Data Science and Evaluation for SDGs[2]
This presentation argued that data science (in our case Big Data, Machine Learning, Artificial Intelligence and Digital Twins) represents a new frontier of innovative technologies that can enhance evaluation’s capacity to understand development problems, assess project and programme impacts, and contribute to better informed decision-making and improved policymaking (IATT, 2019; Rathinam et al., 2020; USAID, 2018; York and Bamberger, 2020). Across all three pillars of sustainable development (Vinuesa et al., 2020), data science can contribute to understanding poverty traps, healthcare service provision and disease management, planning for disaster risk reduction and future urban sustainability, to name just some examples.
Harnessing new data streams at higher degrees of granularity and coverage, predictive and learning analytics, and prototyping tools provide clear opportunities to innovate evaluative practice, enabling evaluation to become more adaptive and reflexive. That said, uptake and use of these technologies for evaluation are still at the early stage, but the opportunity to change this is now.
Equally, the use of data science within evaluation and its utilisation for transformational change is not a silver bullet, as many challenges still exist (Picciotto, 2020), especially around data security, accessibility, and bias. Here, evaluation has the potential to aid data science by helping to address: (i) data quality and validity; (ii) social exclusion and sample bias; (iii) engagement with a broad constituency of actors to establish key evaluation questions; (iv) the value of theory in data gathering, curation and analysis; (v) mixed method approaches; and (vi) the importance of validating data streams that will likely not be explicitly captured for evaluation purposes.
Moving forwards, bringing evaluation and data science together should be a priority and will be essential to both achieving longer-term positive impacts in the sustainable development evaluation space.
Voluntary National Review: a missed opportunity in evaluating SDGs [3]
The third presentation by Jaideep Visave focused on Voluntary National Review (VNR) as a process through which countries assess their performance towards achieving the SDGs goals and the pledge to leave no one behind. The SDGs are deeply interrelated in complex ways, which countries need to consider in the design and measurement of interventions seeking to achieve the SDGs. This includes not just the links between the SDGs and targets, but also across sectors and between actors working towards these goals. Governments need to incorporate the SDGs into national and local budgetary processes to improve policy coherence; link SDGs monitoring and evaluation processes to existing local and national monitoring mechanisms, and build capacity for evaluation.
In 2020, 45 countries presented VNRs. However, a study of 43 VNR reports indicates that of those countries conducting a VNR, there is more emphasis on monitoring than evaluation. Without evaluation, there is no formal mechanism to assess whether the right things are done in the right ways, and no means of accountability to prevent repeating past mistakes, wasting scarce funds, and potentially “doing harm.” One of the bigger challenges for VNRs is the limited national VNR evaluation capacity, experience, and resourcing. This is compounded by stark inequalities in global policy systems, highlighting uneven burden sharing.
Moving forwards, embedding evaluation evidence in VNR reporting will be crucial for (i) diagnosing complex development problems; (ii) design and planning; (iii) implementation, monitoring and formative (real-time) evaluation; (iv) assessing intended and unintended outcomes and impacts; and (v) iterative learning and adaptation.
Final thoughts
In this blogpost, we have summarized key considerations from the workshop to enable evaluation to make a transformational contribution to sustainable development, whilst at the same time also potentially transforming evaluation practice.
Looking to the future, evaluation should and must go beyond the ‘four boxes’ that constrain it and follow a strongly systemic and complex systems thinking approach. Drawing upon ongoing technological innovation, data science can contribute to complex systems analysis in evaluation to inform decision-making and the potential for transformational change. Evaluation informed VNRs can enable governments to address cross-cutting and complex sustainability issues, build the capacity of evaluators and organizations, and ensure ‘inclusiveness’ and ‘coherence’ across scales.
A personal note
We hope this blog opens up and stimulates a space for debate to all interested. On a more personal level, for me (Jaideep), I found this experience – preparing for, delivering, and now writing this blog – immensely helpful: giving me freedom to look at developmental problems and their solutions through my own eyes and expressing those on an international platform. This experience gave me a profound realization about my sense of ‘personal ownership’ and professional engagement in forums about development problems and compelled me to act, such as lead writing this blogpost. Preparing this blog has been like getting light (in terms of knowledge, experience) from sharing with others to navigate ‘my own’ path, thus lighting candle to candle. Such engagement and collegial collaboration and guidance along with useful resources has helped me, and I strongly recommend such engagement to other young and emerging evaluators (YEE) to help navigate their career paths towards solving developmental problems in in a meaningful way.
Keywords
Evaluation, Transformational change, Data Science, Sustainable Development Goals, SDGs, Voluntary National Reviews (VNRs)
Web resources
The following resources are also relevant for ‘Evaluation as a Pathway to Transformation for a Sustainable Future’:
Approaches to Evaluation
Methods and Tools for Evaluation
Evaluation and Sustainable Development
About Authors:
Jaideep Visave, University School for Advanced Studies (IUSS) – Pavia, Italy
Jaideep Visave has the over 9 years of work experience in designing and implementing and evaluating Disaster Risk Reduction (DRR) programs in across Europe and global south. Jaideep has two Masters: an M.Sc. in Disaster Management from India and a Masters in Risk and Emergency Management. Contact: Email and LinkedIn
Scott Chaplowe, Evaluation and Strategy Specialist
Scott Chaplowe is an evaluation and strategy specialist with over 20 years’ experience working with organizations seeking sustainable solutions for social and environmental challenges. Recent jobs include the Director of Evidence, Measurement and Evaluation for climate change at the Children’s Investment Fund Foundation, and Senior M&E Advisor with the International Federation of Red Cross and Red Crescent Societies (IFRC). Scott has authored publication and engages in forums at the intersection of evaluation, sustainable development and transformation, including the International Evaluation Academy, EVALSDGs, and Blue Marble Evaluation. Contact: Email and LinkedIn.
Dr Adam P Hejnowicz, School of Engineering, Newcastle University
Adam is a Research Associate in the Living Deltas Hub, Newcastle University, a Visiting Researcher in the Department of Biology at the University of York, and a member of Centre for the Evaluation of Complexity Across the Nexus. With a background in natural and social sciences, his applied, policy-facing, transdisciplinary and participatory research focuses on the sustainability and governance of complex social-ecological systems. Contact: Email and LinkedIn.
[1] presentation given by Scott Chaplowe, at Transformation Conference on June 17, 2021.
[2] presentation given by Dr Adam P Hejnowicz at Transformation Conference on June 17, 2021
[3] presentation given by Jaideep Visave at Transformation Conference on June 17, 2021