By Richard Gunton, CECAN Research Fellow
Policy evaluation is about assessing the value of policies, but too little attention, it seems to me, is paid to the meaning of “value” in all this. The English word “value”, of course, has multiple meanings that include numerical (e.g. “a parameter value”), economic (e.g. “good value”) and ethical (e.g. “a value judgment”). And herein lies an interesting problem, because this multiplicity of meanings makes it easy to present questions having an ethical component as if they were objective problems that might be solved in purely mathematical or economic terms. Evaluating the National Curriculum’s mathematics programme is not at all like evaluating x-squared when x equals a half! The ultimate question that evaluators ask of policies is: how good is this policy? And that is primarily a question of value in the ethical sense, because goodness is a normative concept.
Policy evaluators have a range of tasks. Commonly they are commissioned to assess how far a policy has met its own stated objectives, and to some extent they are also expected to evaluate the policy in a broad sense: how good is it? The former task may appear to concern only quantitative matters of fact, easily verifiable – how many families, or farms, or schools received support, or what financial value was added to some kind of business ventures, or how many jobs were created? But most policies will have multiple objectives and will not fully meet all of them. This means that any overall evaluation still has the task of evaluating the relative importance of the different objectives, and the relative seriousness of falling short in some or all of them. And a procedure for doing that is unlikely to be specified by the policy-maker. But as soon as we bring in the broader task of evaluation – the question of goodness – the objectives themselves will be open to evaluation. The “goodness” question will typically be expressed in terms of (good) value for money, and perhaps also the speed of impact (goodness per unit time) or with other denominators such as personnel required. In other words, no matter how evaluators’ task is circumscribed, there’s no value-free evaluation.
This is the theme of my fellowship with CECAN. But what does the question of value have to do with complexity? This is where I plan to bring a distinctive insight. Norms are diverse – think of the classical platonic triad of the true, the beautiful and the good. But the effects of a “good” policy should be much more than morally good (e.g. fair), or indeed beautiful or truthful (e.g. realistic). They should also represent good value, and typically should be progressive and socially appropriate too. There are clearly many distinct ways in which a policy can be evaluated as good – even before we consider its specific objectives. And understanding the views of different stakeholders is a crucial part of this, since there is no a-priori objective way of balancing the different aspects of goodness. Value is thus a complex notion, not a simple one, calling for nuanced analysis. Some kind of framework or tool, therefore, is needed for recognising and combining diverse axes of goodness to help evaluators take account of this complexity in ways that are democratically transparent.
I came to CECAN from a project looking at how objectives and motives for nature conservation are classified, multiplied and quantified through the “ecosystem services” framework. Conservation policies (such as prioritising areas or species for protection or management) are increasingly defined and evaluated with reference to somewhat arbitrary lists of ecosystem services that are supposed to be universal benefits or opportunities provided by natural and semi-natural environments to humanity at large, on the model of commodities markets. In putting it like that I may already have given away my wariness of the ecosystem services framework – but it has yielded too many benefits for the whole idea to be thrown out! So our project sought a framework that would share its main strength of finding multiple distinct axes of valuation without some of its drawbacks – and we called our proposal the “ecosystem valuing framework” [http://www.sciencedirect.com/science/article/pii/S0169534717300150]. And the framework is actually so general that it can readily be applied to policy more generally, and its evaluation.
My challenge in the coming months is to understand better how policy objectives tend to be designed and evaluated at present, and what policymakers and evaluators especially need in order to demonstrate democratic accountability in policy appraisal and evaluation. My initial focus will be on Defra’s Rural Development Plan for England as a case study – a policy area where making connections from the ecosystem valuing framework should be relatively straightforward. I shall report on progress in a few months’ time.