Anne Liddon, Science Communications Manager / Fran Rowe, Research Assistant / Amy Proctor, Research Associate, Centre for Rural Economy, Newcastle University
As we all know CECAN is pioneering, testing and promoting innovative policy evaluation approaches and methods across Nexus domains through a series of “real-life” case study projects – it’s about measuring best value in complex, inter-connecting systems. And what could be more complex and “real-life” than Christmas? We are all busy people and we want to get the best out of this annual festivity but what are the best methods for evaluation that can help us to do that? We have been consulting with the most important actors involved (elves, reindeer, over-excited children, and, of course, Santa Claus himself) to consider the challenges and opportunities.
It’s clear, of course, that Christmas has very many characteristics common to Nexus problems. The complexity of bringing together a number of family members – who may vary in their expectation of outcomes and, indeed in how pleased they are to see one another – is a challenge in itself. What can we begin to measure in order to come up with some indicators for a successful Christmas? Possibilities could include: how much money is spent; how many unwanted gifts are given and received; how much weight is gained; how many hangovers suffered; how much rubbishy television is watched. On the other hand, indicators for a more potentially successful Christmas might be: how many Christmas cards are received; how many family members are still speaking to one another by Boxing Day; how much chocolate is consumed before breakfast without anyone actually being sick; whether the Christmas pudding is successfully flamed without setting fire to any soft furnishings.
What methods could best be employed for this key seasonal task? A successful Christmas can be subjective and not everyone may emerge at New Year feeling the same way about it, so we recommend bringing together both qualitative and quantitative methods in any evaluation. Qualitative Comparative Analysis seems an obvious choice if we are to pin down the combination of factors that will make Christmas successful. This method also offers exciting opportunities for understanding what makes one person’s Christmas more successful than another’s, and so stimulating some healthy competition, possibly involving whole streets of life-size reindeer-drawn sleighs, snowmen with fearsome, glaring eyes on house roofs, giant Santas disappearing down chimneys, and house fronts festooned with flashing lights. Unintended consequences can, however include overloaded electricity substations and unaffordable electricity bills in January, so these will also need to be factored in. Small children, and even adults of a nervous disposition, have also reported nightmares arising from such scenarios.
Agent Based Modelling is another possibility and here the elves would of course be key. Is it possible to create a model of their complex activity in Santa’s workshop from year to year and so predict outcomes for the future? This does seem to be an area worth exploring, although elves are notoriously fickle in their behaviour, often taking malicious delight in subverting any focus group approaches, even making paper aeroplanes out of questionnaires, and thus the challenge would be considerable.
What about Fuzzy Cognitive Mapping? It’s an obvious candidate for measuring hangovers and might even be adapted into a party game format for Boxing Day, though more work would be needed to see whether it has more measurable impact than a dose of paracetamol and a nice lie down. We wouldn’t recommend attempting to apply Bayesian Belief Networks as beliefs about Christmas are, of course, a personal matter and not something we would attempt to evaluate.
Realist Evaluation could prove helpful. However, past experience does suggest that expecting to gain insights into what works to make Christmas successful is probably unrealistic and that Christmas will proceed to much the same pattern as it did last year, with too much food and drink, weird gifts that were kindly meant, and families dragged off for a walk on Boxing Day just to get them out of the house. We would, however, encourage any volunteers who would like to undertake a case study to come forward. Santa has agreed to be scrum master on this one and work will, of course, have to begin in April as Christmas gets earlier every year.
In the meantime, have a good one – and you might even want to ignore evaluation for a day or two!