the problem of 'evidence centres'

Reading the news this week that the government is to set up new 'evidence centres' to inform, criticise and influence £200bn-worth of public policy, my immediate thought was - why now? New Labour came into power singing the praises of 'evidence-based policy' in 1997, and became widely celebrated for its ability to bring new insights from economics, psychology and sociology straight into Downing Street, via the Strategy Unit and academic-wonks such as Geoff Mulgan and David Halpern. Even before then, the relentless auditing and measuring of public services was introduced under the guise of 'new public management' (though new research by Christopher Hood and Ruth Dixon shows that this never achieved great savings). And the Treasury's Green Book has offered a set of techniques for policy evaluation and appraisal for decades.

It seems now that policy is undergoing quite a rapid shift from treating economics as the standard for policy knowledge to treating medical research as the standard. This is clear from the fact that the evidence centre idea arose initially from the proposal of creating a 'NICE for social policy', to find out 'what works'. Economics itself is undergoing a much slower mutation, from aspiring to the status of physics to aspiring to that of biology. The infrastructucture of government that is needed to facilitate these shifts is quite different and far more extensive.

The key thing here, epistemologically, is that the state is trying to relinquish an a priori account of causality. The methodology of neo-classical economics has a very particular theory of causality, which it packages up within its assumed psychological theory of preference satisfaction. This theory then structures what types of evidence are collected, how they are interpreted, the political uses to which they are put. Inevitably, like any useful scientific framework, it also limits what types of conclusions might be drawn and the types of knowledge that might arise. A scientific framework that presupposed nothing about the world - or in this case, society - wouldn't be able to make any sense of the world.

Yet the spread of medical epistemology into public policy is strangely anti-theoretical, thanks to a somewhat naively optimistic view of a single technique: the randomised controlled trial (RCT). RCTs operate according to induction. The facts are meant to speak for themselves; the data and the theory are kept neatly and self-consciously separate from each other. A medic, Ben Goldacre, has co-authored a paper on the policy applications of RCTs for the British government, which opens with the line 'RCTs are the best way of determining whether a policy is working'. Elsewhere, RCTs are often referred to as the 'gold standard' for scientific testing, a term that confirms the dangers of metaphorical tourism, given that, while economists are happy to speak biologically of 'toxicity' and 'contagion' in the financial system, only the crankiest libertarians amongst them would countenance a return to the actual gold standard.

This is supplemented epistemologically by the rise of Big Data, which no doubt is already on the minds of forward-thinking policy experts, especially in the domain of health behaviours. The very character of Big Data is that it is collected with no particular purpose or theory in mind; it arises as a side-effect of other transactions and activities. It is, supposedly, 'theory neutral', like RCTs, indeed the techno-utopians have already argued that it can ultimately replace scientific theory altogether. Hence the facts that arise from big data are somehow self-validating; they simply emerge, thanks to mathematical analysis of patterns, meaning that there is no limit to the number of type of facts that can emerge. There's almost no need for human cognition at all!

The problem with all of this, politically, is that causal theories and methodologies aren't simply clumsy ways of interfering with empirical data, but also provide (wittingly or otherwise) standards of valuation. The reason economics has proved so powerful as a governmental tool is not because it is empirically correct about the drivers of human behaviour (no economist since Jevons has really claimed this; sociologists are barking up the wrong tree in this particular indictment) but because it provides a very clear idea of what a 'good' or 'bad' outcome would look like, in a particular situation. Pragmatically speaking, it saves decision-makers from having to have moral arguments about things, by placing numerical values on them instead.

Health policy is one area where this type of simplification is hardest to achieve, because moral trade-offs are often impossible to keep at bay. From the little I know of NICE and health economics, a great deal of what they wrestle with is not establishing 'facts' or 'evidence', but coming up with incredibly convoluted ways of agreeing what the standards of evaluation should be in the first place. QALYs are the most obvious example of this, but philosophical arguments about 'experienced utility' versus 'reported utility' and so on rage on in health economics, because pain, suffering and life itself are at stake, which can't be easily subjected to an efficiency analysis. As a goal, 'health' or 'wellness' lacks the finality of 'efficiency' or 'consumer welfare', meaning that there is a risk that health and wellbeing policies can never truly 'work'.

It is therefore a little ironic that NICE has become the model for social policy evaluation, given that social policy-makers already have a very clear toolkit for how to evaluate policies for over 40 years (the Green Book), whereas health policy evaluation is in a permanent - and necessary - state of philosophical self-doubt. By adopting the inductivist epistemology associated with RCTs and Big Data, social policy-makers may learn a great deal more about the world, but may also become commensurately less sure of what it even means for a policy to work in the first place.

For those of us with an innate suspicion of government positivists, this should be a good thing. Government might become more humble in its ambitions, cancel policies more readily, recognise the complexity of society. On the other hand, there is a risk that, as with RCTs in psycho-pharmaceuticals, diagnoses of social pathologies might start to spiral. Whole new problematic demographic sub-groups will start to appear to the gaze of the data analyst; new correlations of behavioural problems will be spotted; the perceived sources of our social, psychological and neurological malaises will simply multiply, and we'll long for an age when it was all just a problem of the wrong 'incentives'. Tesco's Club Card is rumoured to produce 18,000 sub-groups of customer; the equivalent for the state would be 18,000 sub-groups of pathological behaviour to be nudged back into line. Without the extreme simplifications of rationalist theories, society would appear too complex to be governed at all. The empiricist response to the government's paper title, 'What Works', might end up being 'very little', unless government becomes frighteningly 'smart'. Alternatively, if theory no longer provides the procedures of evaluation, there is a risk that private backroom politics will do so instead.

I exaggerate, of course. But the political issue is really this. Where clunky-old economics, with its unrealistic models, is used to deliver 'evidence-based policy', political and moral debate can be sidelined; yet the standard by which things are being judged is not difficult to discern. There is some publicness about this (I did some interviews with government economists a couple of years ago, and they couldn't understand why journalists never dipped into departmental policy evaluations, which are all published, and many are quite politically problematic). But where the state becomes a theory-less, inductivist, RCT-ing, data-analytical state, accumulating more and more data to find out 'what works', we are entitled to ask what working might actually mean. A clear and transparent utilitarianism, oriented around efficency, may be preferable to a vague and opaque utilitarianism, oriented around some metaphor of systemic 'wellness'.

Nothing simply works unambiguously in social policy, gold standard or no gold standard. No policy delivers benefits without any 'side-effects' (to play along with the game of policy doctors and nurses). A policy might 'work' in terms of reducing unemployment, but lead to an increase in family break-down. The inductivists response would be - yes, and that's precisely the type of pattern that our new evidence centres will detect! So why use the rhetoric of 'what works', when it is plain that nothing unambiguously works, at least without also offering the standard (the QALY for social policy, if you like) through which ethical dilemmas and trade-offs will be addressed? If all of this opens up space for non-utilitarian political debate about the multiple, competing purposes of social policies, and the types of procedures and authority that might be used to navigate between them, then that would be welcome. But the alternative is a Tesco Clubcard-state, which gets smarter and smarter, and more and more opaque as to whom its interventions are targeted at and why.

Comments

[Dave Timoney]

On Exactitude in Science

In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.

Jorge Luis Borges

[Will Davies]

well quite!

[Ian C]

A great post - thanks.

But the state commissioning all this 'what works' analysis is anything but 'theory-less', and that makes a big difference to the kinds of utilitarianism on offer. At least three ruling presuppositions can be identified, and arguably they inform everything that happens in the development of supposedly evidence-based policy. First, no evidence or analytical framework can be admitted that does not assume the desirability and infinite capacity for economic growth. Second, a crude utilitarianism applies to working out whether a body of evidence provides value for money or not, since state spending is inherently prone to be wasteful or pointless. Third, policy is aimed at influencing individuals, with the structural features of economy and society seen as a taken-for-granted backdrop except where these involve state ownership and service provision. All sorts of other prejudices apply about the motivations of the poor/rich and the kinds of incentives we respond to.

On the point about growth, note that almost any agency in Whitehall and beyond is now being designated a Growth Delivery Body. This has just happened to - of all agencies - Natural England, the state's environmental conservation advisor. Presumably the next step will be to impose performance targets and 'what works' evaluation systems on wildlife. That'll teach the bloody badgers.

[Helen]

Thanks for this great post.

I have been trying to think through this return to the rhetoric of 'evidence-based' policy & its new institutional forms myself, but wasn't able to articulate the problem as you have here. Related to your point about the centres creating a supposedly theory-less arena for policy, it strikes me that this plays into a deep-seated, and perhaps quite British, desire for neutral, unmediated empiricism, i.e. letting the 'facts' speak for themselves - though this is ultimately unobtainable. The centres represent, to me, the height of technoscientific optimism, assuming that we have the capacity to know unproblematically, universally and with certainty 'what works', never mind be able establish standards of value which would be able to assess this. [Here is my attempt to examine this: http://thetopograph.blogspot.co.uk/2013/02/evidence-based-policy-and-problem-of.html ]

I find it troubling that though the inherently political nature of all of these policy decisions is openly acknowledged by 'evidence-based policy' proponents, there is still an underlying assumption that these facts and values can be separated.

As somebody who works in public participation in science policy, I also whole-heartedly agree with Ian C's point that the working definitions of 'evidence' in these centres are likely to remain fairy narrow and exclusive.

[Bengoldacre]

This praise for economics over EBM - because economics posits theories, while EBM merely measures whether something works - reminds me of the old joke about the economist, who says:

"Yes, it works in practice. But does it work in theory?"

[William]

I always thought that joke was about French politicians...

It's scarcely praise for econonomics. It's just an attempt to tease out what is going on here pragmatically. From a sociological perspective, the problem the state (and its various officials and experts) faces day-to-day is not 'is X true?' but 'what should we do next?'. This is conditioned by other questions such as 'what can we afford to do?' and 'what can we get away with doing?'. The extent to which empirical learning becomes adopted by the state depends heavily on how it assists with answering those questions. So of course, new methods and data are useful, but we need to be realistic about how much autonomy scientists can have in such a situation.