Foot in Mouth

Finding Better Uses for Policy Research

By Jonathan Grant, Mirjam van het Loo, Sally Ann Law, Silvia Anton, and Jonathan Cave

Jonathan Grant and Silvia Anton are analysts in the Cambridge office of RAND Europe. Mirjam van het Loo and Jonathan Cave are analysts in the Leiden office of RAND Europe. Sally Ann Law is a psychology consultant to RAND Europe.

Three years ago, foot-and-mouth disease ravaged the United Kingdom, precipitating the slaughter of more than six million animals, the decimation of livestock farming, and a steep decline in tourism. The epidemic is estimated to have cost the country as much as $6 billion.

Managing the crisis placed heavy demands on research to discern the causes and likely progress of the epidemic, the relative efficacy and costs of slaughter or vaccination policies, the probable public responses, and the resulting distribution of economic consequences. In short, the epidemic was a very good example both of how governments depend on research in times of crisis and of why governments should plan for future research needs over the long term, if only to be prepared for these kinds of crises.

The European Union has set a target for each government to spend a full percent of national GDP on research by 2010.

In cases such as foot-and-mouth disease, crisis management also depends on the informed reactions of others (farmers, supermarkets, slaughterhouses, consumers, and trade partners). Therefore, such situations cry out for innovative ways to communicate the results of scientific research and to explore the implications collectively.

Closing the Gap Between What Researchers Produce and What Policymakers Need Reasons for the Gap Possible Way to Close the Gap Research results are not easily accessible Technological solutions (web sites and email newsletters) Use of communications analysis and science writers Dissemination strategies mapping products to audiences Poor understanding of policy questions by researchers New forms of sustained engagement by researchers in policy consultations with umbrella research groups Poor communication of research results by researchers Making translation of research an intrinsic part of the research project cycle Requiring (and evaluating) a range of high-quality communications products tied to the research results Adding incentives for future work based on the clear communication of current work Poor understanding of research results by policymakers Requiring policies to be based explicitly and demonstrably on relevant knowledge (commissioned and extant) Careful training of policymakers to ensure the “intelligent customer” role More bilateral research partnerships between government and scientists No direct, short-term relevance of research results for policy Early involvement of potential research users in research projects Research portfolio management that aligns projects more closely with policy goals Lack of resources for dissemination activity Contracts that include funding for dissemination Contracts that include spin-off options, such as intellectual property rights, on research products Career incentives, such as the consideration of dissemination activities in researcher salary reviews

How RAND Europe Acts as a Partner in Search of “Value for Money” The National Audit Office (NAO) is the UK’s supreme audit institution. Like the U.S. General Accounting Office, the NAO is responsible for certifying the accounts of all government departments and for reporting to Parliament on the economy, efficiency, and effectiveness with which departments and other public bodies have used their resources. In 2003, the NAO invited RAND Europe to become a strategic partner, undertaking full "value-for-money" studies on its behalf and supporting the NAO in its own studies of this nature. To date, RAND Europe has been involved in the following NAO studies: Tax collections. The study is analyzing the feasibility of developing mathematical models to calculate likely tax revenues and costs associated with alternative taxation methods. International tax returns. The study is comparing the filing of corporate and income taxes in five countries (Germany, the Netherlands, Sweden, the United Kingdom, and the United States). Joint targets. The study is developing and applying ways to evaluate joint working arrangements among departments with common policy targets (e.g., trade, poverty, conflict, and development). Benefit regulations. The study is charting the apparently increasing and unnecessary complexity of public entitlement programs and examining ways to manage or to reduce it. Diversity in public services. The study is assessing how departments deliver services to diverse populations—and the barriers and conduits to reaching them. Regional development. The study is examining the overlap between the target populations served by regional development agencies and those served by other departments. Defense. The study examined the prioritization and management of science and technology research within the UK Ministry of Defence. Information technology (IT). The study critiqued the procurement and management of large IT research projects commissioned for the purpose of improving service delivery. Better public services. The study compared the approaches used by departments in Germany, the Netherlands, and the United States to convert their resources into improved services.

The challenges of managing, using, and communicating research pertain to all areas of policymaking. In June 2002, the United Kingdom’s National Audit Office (NAO) asked RAND Europe to conduct a “value-for-money” study of how government departments procure research and how well that research is used to develop policies and to improve service delivery. Following the successful completion of this work, last year the NAO invited RAND Europe to become one of its strategic partners and to support its work through various projects (see sidebar).

Our initial report for the NAO, Getting the Evidence, concluded that while UK departments have been modernizing research procurement, more could be done to consolidate and to exploit the improvements. In general, we concluded, the departments needed

• more-strategic management and use of research
• more-proactive and more-innovative dissemination of research findings
• more sharing of research results across departments and over time.

These guidelines could help departments increase the value of research expenditures, not just in Britain but around the world. Typically, governments in developed countries spend between a half percent and a full percent of national gross domestic product (GDP) on research.

The European Union has set a target for each government to spend a full percent of national GDP on research by 2010, and so the importance of obtaining maximum value for research will only grow. In the sections below, we cite examples of some of the best practices that we have already identified in following our three overall guidelines.

More-Strategic Focus

Research goals help determine how research is commissioned and evaluated. All government departments in Britain are required to publish their research goals and strategies. However, the departments could generally do a better job of distinguishing between short-term and long-term goals. With respect to long-term goals in particular, departments should strive to balance long-term demand (the emergent research agenda) with long-term supply (research capacity and capabilities).

For help in clarifying long-term goals and developing and maintaining long-term capabilities, departments should regularly consult with research users and providers. Users include policymakers, nongovernmental organizations, private companies, the media, other researchers, and concerned segments of the general public. Providers include not just existing suppliers inside or outside government but potential suppliers as well.

One example of good practice that we identified for clarifying long-term goals is the “horizon scanning” program of the UK Department for Environment, Food, and Rural Affairs. The program identifies emerging scientific opportunities and risks, such as an increasing prevalence of tuberculosis in cattle, and explores novel ways of framing long-term research problems. The department identified new priorities through an Internet-based consultation that elicited 282 research ideas from more than 400 individuals working in research, government, industry, and nonprofit organizations.

Another example of good practice in nurturing long-term capabilities is a fellowship program funded jointly by the UK Office of the Deputy Prime Minister and the UK Economic and Social Research Council. The program is designed explicitly to build long-term research capacity in the areas of technical excellence and strategic thinking.

If research goals are clearly defined, then research questions can be clearly posed and research contracts can be prudently awarded. The first step in awarding the contracts is to ensure coherence between the competitive mechanisms and the intended uses of the research.

Research specifications can be direct (based on detailed requirements) or indirect (based on a general call for proposals on a broad topic, such as poverty elimination). Competition among bidders can be open (advertised to all interested parties) or closed (limited by invitation). Finally, evaluations of bids can be formal (relying on predetermined objective criteria for evaluating bids) or informal (relying on professional judgment and expertise).

Two examples illustrate the range of good practices we found. The UK Office of the Deputy Prime Minister regularly needs to answer specific questions about housing, homelessness, and urban issues, necessitating frequent competitions. To keep the process fairly open while managing costs, the office invites all bidders to submit preliminary letters of interest. Based on the letters, the office narrows down each competition to 3–6 contenders for a further, more-intensive, more-expensive, and more-objective review of detailed proposals.

In contrast, the policy mandate of the UK Department for International Development requires more general knowledge about social science. Therefore, competitions are indirect (broad in scope), open (to encourage new ideas), and informal (without prespecified research questions or methodologies). Technical experts (academics) judge the proposals. Research users (internal policy staff) review the relevance of proposals to departmental goals. This combination of technical and relevance review helps to guarantee the utility of the research and its translation into practice.

Quality assurance should be an integral part of research. Departments should go beyond the standard evaluation protocols of identifying appropriate statistical measures of quality, which do not necessarily ensure the validity of the research. Beyond the use of standard quantitative measures, departments should seek qualitative guidance from research users throughout the duration of the research.

Qualitative guidance could be gleaned from user surveys and from peer reviews. Peer reviewers could also synthesize the quantitative data with the qualitative surveys. The early involvement of potential users of different types of research will greatly increase the likelihood and value of its ultimate use.

We found an exemplary approach already put into practice by the Canadian Health Services Research Foundation. Called “Linkage and Exchange,” the program stipulates the intimate involvement of potential research users in each stage of the research process: setting priorities, evaluating applications, funding projects, conducting research, and communicating results.

In setting priorities, the Canadian foundation hosts nationwide workshops every three years so that policymakers can establish research priorities for the ensuing five years. In evaluating applications, the foundation uses peer review panels, with equal representation of researchers and policymakers, to assess scientific quality and policy relevance. In funding projects, the foundation funds only half of the research costs, requiring joint sponsorship from other partners. In conducting research, the foundation requires each research team to include at least one policymaker actively engaged in the area under study. Finally, in communicating results, the foundation supports the dissemination of research evidence that pertains to a topic identified by the policymakers.

More and Better Dissemination

Policymakers and other research users often complained to us about the poor communication of research results. In fact, there was a widely perceived gap between what researchers produce and what policymakers need. We identified several reasons for the perceived gap and have begun to suggest ways to close it (see the table).

On the whole, departments are more likely to carry out “passive” dissemination of research (upon request), probably because the responsibility for dissemination has traditionally fallen to researchers themselves who, in turn, do not usually consider dissemination to be an intrinsic part of the research. Typically, researchers have no career incentive to disseminate their findings widely to policy audiences. Moreover, researchers often do not have access to relevant policy audiences beyond the departments commissioning the research.

But passive dissemination is insufficient. It breeds inefficiency and inconsistency. It breeds inefficiency because it encourages departments to duplicate the research efforts of one another. It breeds inconsistency because it drives departments to draw on separate pools of validated scientific knowledge, which not only hinders collaboration but could even lead departments to work at cross-purposes. Research managers should promote targeted and innovative communications strategies for several reasons: to increase the use of solid research, to inform policy, to improve service delivery, and to prevent inefficiency and inconsistency across a nation’s entire research agenda.

An exemplary dissemination practice is an Internet-based service established in 1997 by the UK Department for International Development. The aim of the service is to inform the policy debate by presenting information in a user-friendly and accessible manner. A team of in-house development researchers and freelance professional journalists works together to summarize the research reports into short “Research Highlights” that focus on the policy-relevant aspects of the research for a worldwide audience of policymakers and development practitioners.

More Sharing of Results

Following publication of Getting the Evidence, the UK Office of Science and Technology asked RAND Europe to run an interdepartmental workshop geared toward the possibility of initiating a network of UK government research managers. Science Minister Lord David Sainsbury opened the workshop. The Prime Minister’s Chief Scientific Adviser, Sir David King, closed the workshop. Participants concluded that they could identify and share good practices and help one another enhance the overall effectiveness of managing, commissioning, evaluating, using, and disseminating research.

The managers clearly want to learn from one another but have yet to identify appropriate interdepartmental mechanisms to promote such learning. Further discussions will be necessary to confirm whether a formal network is indeed the best way to achieve the goal.

Meanwhile, the tough questions facing researchers today will prove difficult to answer without closer coordination and clearer communication among researchers and policymakers. Many daunting scientific challenges — from foot-and-mouth disease to mad cow disease to genetically modified organisms — also involve entrenched political and economic interests.

Today’s questions often transcend scientific disciplines while opening the door to the dangers of parochialism, populism, and scientism. In these cases, research managers bear a tremendous responsibility to focus, to disseminate, and to share research so that the results can be the most relevant, the most understandable, and therefore the most useful to all.

Related Reading

Getting the Evidence: Using Research in Policymaking, Report by the Comptroller and Auditor General, HC 586-I, Session 2002–2003: 16 April 2003, London: The Stationery Office, 44 pp. Available on the National Audit Office web site.

586-I (PDF)

An International Review on Governments’ Research Procurement Strategies: A Paper in Support of Getting the Evidence: Using Research in Policy Making, Report by the Comptroller and Auditor General, HC 586-II, Session 2002–2003: 16 April 2003, London: The Stationery Office, 47 pp. Available on the National Audit Office web site.

586-II (PDF)