Kathryn Oliver; Theo Lorenc; Simon Innvær

“ Develop conceptual clarity around and metrics to evaluate ‘impact’ of research on policy and populations. Without clear methods to understand how policy works and how it changes in response to information, it will be impossible for researchers to know whether they have had an effect on policy. Research usage could take many forms, from agenda-setting, to provision of policy options, to challenging debates, or refuting arguments. Impact could therefore be a change in policy, consistency in policy, changes in population level outcomes. ”

“ Broadly, practitioners and academics have focused on facets of the use of evidence by policymakers and practitioners, have written polemics encouraging colleagues to do so [18] , [20] - [21] , identified barriers and facilitators into evidence use [22] , [23] , and designed interventions to increase the use of evidence by policymakers [24] . Evidence-based policy and practice, knowledge translation, and related concepts have become touchstones across a vast range of disciplines – almost sub-disciplines in their own right, with canons and conceptual toolkits of their own. ”

“ We are not advocating against EBP. On the contrary, we believe that better policy decisions would be a desirable outcome, and that evidence ought to play a role in those decisions. Rather, our conclusions from this critical review of the literature are about EBP research, rather than EBP itself. Researchers have directed their attention at how to increase the impact of their own outputs, rather than on understanding the processes behind policy change. ”

“ With these exceptions, very few studies have, as yet, taken anthropological or historical approaches to understanding the role of evidence or research in policy.
Research in the area thus focuses primarily on how to increase uptake of research, on designing and evaluating interventions aiming to increase research use, and on identifying barriers and facilitators of research use by policymakers [58] - [59] . This pattern of research on evidence use skews the debate by focusing on exceptional cases of research use in policy-making, rather than the normal discharging of statutory business. ”

“ Asserting the existence of and describing and prescribing interventions to close the ‘research-policy gap’ is a stance which, we argue, is likely to perpetuate and even create gaps between the professions. ”

“ This may be due to the methods used in the studies: most use only interviews or surveys to ask researchers and policy-makers about their perceptions about evidence use; very few use methods such as participant observation to observe how evidence is actually used in practice, or attempt to find documentary proof of research use (with exceptions; [59] ) . These lists, while important, cannot on their own lead us to an improved understanding of the role of evidence in the jigsaw of the policy process. ”

“ Policy and academic actors were conceptualised as opposing sets of actors, with different priorities, languages, practices, and priorities. For proponents of this perspective, ‘bridging’ this gap becomes a priority. It seems likely, however, that insisting on the existence of this ‘gap’ may polarize previously neutral actors; and indeed this debate fails to recognise that this may be a UK-specific problem (cf. to Dutch studies) .
Models of the research and policy processes are, as noted by previous theorists of EBP, rarely made explicit in evaluations of applied research. ”

“ Much of the research in this area is theoretically naive, focusing primarily on the uptake of research evidence as opposed to evidence defined more broadly, and privileging academics’ research priorities over those of policymakers. Little empirical data analysing the processes or impact of evidence use in policy is available to inform researchers or decision-makers. EBP research often assumes that policymakers do not use evidence and that more evidence – meaning research evidence – use would benefit policymakers and populations. ”

“ The assumptions governing the design and outcomes of research into policy will probably be significantly different in different institutional areas. It is often not clear what constitutes ‘a decision’, nor who is involved in it, or whether research evidence is relevant or timely. ”

“ We are not the first to note that “ [t] he assumption that the use of evidence would improve the outcome of the policy process remains relatively untested by any form of empirical analysis” [81] - [82] . The absence of robust evaluation evidence showing that evidence utilisation actually leads to better outcomes is widely admitted. The bulk of the intervention evidence in EBP uses only research utilisation or uptake as an outcome (or, in some cases, merely attitudes and intentions regarding research use) . ”

“ We set these data in the context of the wider theoretical literature about evidence and policy, drawing insights from policy sciences. Secondly, we aim to identify and challenge some of the more normative assumptions which are widely prevalent (if often implicit) in the EBP literature, particularly the following: that the policy-evidence ‘gap’ needs ‘bridging’; that policy is usually not based on any data; that policy requires research evidence, preferably evaluative intervention research evidence; and that if more evidence were used in policy making, policy would be better. ”

“ This stance is both unhelpful and divisive, blinding researchers to the important questions raised above regarding the types of information used, by whom, for what purpose, and under which circumstances. Rather, our role as scientists ought to be to investigate the processes surrounding the use of evidence and policy activities more widely and to disseminate findings in order to help others make informed decisions, of all kinds. ”

“ Barriers to use of research are equated with barriers to direct use of research, while the broader concept of enlightening (indirect) use is rarely seen as equally relevant and useful.
While this is still widespread, theoretical learning from other fields is filtering into the debate. Researchers in policy studies have long seen the policy process itself as a contested arena of negotiation [35] , [36] . ”

“ If universities were to provide assistance for local policymakers in the analysis of existing data, a relationship of mutual benefit could start to develop – an end in itself, according to reviews of barriers and facilitators of evidence use [22] , [26] . Moreover, such organizations would provide natural laboratories for studying the role of institutional and organizational factors on the practices of policy formation and implementation, noted in our review as likely to affect evidence use. ”

“ The majority of academic studies in EBP research are, unsurprisingly, written by and for academics, with little involvement of policy-makers as co-authors; indicating that policy-makers are not involved in developing or carrying out relevant research. ”

“ Furthermore, attention to context is vital; pressures faced by researchers and policymakers in low- and middle-income countries may be very different from those in Western settings. Finally, as argued above, although similar barriers and facilitators are often identified across policy settings [27] , there is variation of practices and processes across policy areas. Analysis of these variations would be a whole research agenda in itself. ”

“ Nonetheless, a large proportion of the available research still rests on an uncritical acceptance of these assumptions. Below, we describe the effect of these assumptions, justify our rejection of them, and discuss the implications of taking a more critical approach.
Assumption 1: that a policy-evidence ‘gap’ exists
The ‘evidence-policy gap’ is a widely-acknowledged construction in policy-related research, asserting the existence of two separate communities with their own ecosystems and languages [48] , [67] - [68] . ”

“ It seems likely that these sources of information have been undervalued by evaluation methodologists, who often value trial data above other types. A more naturalistic approach using empirical methods to study policymakers in vivo, would conceive of evidence as one of many influences on a decision. ”

“ There are examples of institutions which specifically allow researchers and policymakers to learn about each other’s priorities and ways of working [43] , [102] . In general, however, the literature above suggests that there are insufficient opportunities and incentives to form links with policymakers directly. ”

“ How to Further Strengthen Open Policy Making – A Guide For Policy Makers, BIS/13/581, Government Office for Science, 2013, 1 2 Martin, BR (2011) , “The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster?”, Res Eval 12: 247–254 ↑ Joint Statement on Impact by HEFCE, RCUK and UUK, 2011, ↑ Armstrong, R ”

“ MIT press 1 2 3 4 Lomas, J & Brown, A (2009) , “Research and advice giving: a functional view of evidence-informed policy advice in a Canadian ministry of health”, Milbank Q 12: 903–926, 20021590 LomasJBrownA, pmid:20021590 ↑ Cairney, P (2014) , “How can policy theory have an impact on policy making? The role of theory-led academic-practitioner discussions”, Teach Pub Adminis, DOI 10.1177/0144739414532284 ↑ Buse, K ”