23rd June will go down in history as a very difficult day in Westminster – Brexit Day. Nonetheless, Andrew Darnton and Andrew Harrison boldly produced an engaging and thought provoking complexity seminar in Whitehall and we thank them wholeheartedly for a superb event. The seminar topic was ‘Revaluation: Measuring Paradigm Shift’. ‘Revaluation’, a new innovative method for measuring change in complex systems, was presented and discussed. Revaluation comes out of a year’s work evaluating ‘bottom-up’ change within the NHS. It is an approach based on participative methods, in which evaluators and evaluated are equal actors in a system of measurement, designed to reveal the ‘full value’ of the change intervention or movement. Since its evolution in the ‘messy’ system of the NHS, Revaluation has been applied to the challenge of tackling biodiversity, and to repositioning the public arts. Notes from this seminar are listed below. Notes and presentation slides are available to download in PDF here.
CECAN Lunchtime Seminar #1
Andrew Darnton and Andrew Harrison
Introducing Revaluation: Measuring Paradigm Shift
DECC, 24/06/2016 (‘Brexit Day’)
[Note: to be read in conjunction with Two Andrews’ powerpoint presentation]
Following AD’s Intro, participants discussed their initial responses in pairs (buzz groups), including… [participants named with initials, where known]:
- The problem [with evaluating complexity /valuing change] is still natural language. You can’t use this for reasoning. So you turn to models, which then lose the complexity… [JK]
- Second loop evaluation yes; but third loop? Not well articulated [BFW]
- Doesn’t conventional evaluation make the invisible visible also? [BFW]
- This is so far from conventional evaluation [AP/NG]
- What contexts can Revaluation best be used in? [AP/NG]
- Is Revaluation complementary or alternative to other complex methods? [ES]
- This looks more like a change method, not measurement [ES]
- A good method, but how to balance practicality with complexity? [x]
- I like the third dimension (capacitate): we need a focus on networks and relationships [y]
After AH’s NHS Change Day case study, two questions were asked:
‐ You speak of measuring in multiple currencies. When do they emerge – because you wouldn’t want to prescribe them from the start? [AR]
AH: Start with iterating people’s/subsystems’ own stories: what currencies do they use? Then consider their audiences: what language do they need to speak to connect with those they wish to reach.
‐ You speak of irresolvable tensions. This sounds good to me: say more… [z]
AH: Let me give an eg. from Change Day. Making change requires negotiating where the vertical system (hierarchy) meets the horizontal (activists). This is a dilemma: it won’t go away, and is a source of creative tension. Naming it makes clearer the tacit theory of change for the system (here, the NHS).
In closing, AD responded to the earlier ‘considerations’ about which contexts, and whether Revaluation is a change or a measurement method.
- Revaluation is currently being applied to Welsh Government’s Nature Fund: 20 landscape‐ based partnerships of diverse actors. Centrally funded but with broad outcomes loosely specified. No objectives. And this is a paradigm shift: in governance/delivery, and in conservation approaches.
- It’s also hard to see the whole system, in these ‘more than programmes’ with distributed interventions. Compare this to the NUS Students’ Green Fund, funded with £5m from HEFCE, distributed to 25 bidding universities, each with nested student grants programmes. Even knowing/seeing the system, and what actions have been undertaken within it, is a challenge. Note also timelags throughout as monies are bid for, distributed, then spent. The research question here is what did I get for my money: ‘WIGO’ (What is Going On)?
- These interventions go beyond conventional policies: whole system approaches.
- Revaluation points to a relationship between the intervention and the measurement system: evaluation should ‘follow the contours’ of the system under enquiry. So in answer to Elliot: Revaluation is both – closing up the loop between change activity and measurement (action and reflection).
- Revaluation is a paradigm shift in evaluation, supporting paradigm shifts in intervention (policy, and governance systems). Today [Brexit} is a good day for paradigm shifts – although I hope Revaluation is not for this one, but the next one.