Standards for High-Quality Research and Analysis
RAND's Standards for High-Quality Research and Analysis
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND focuses on the issues that matter most: health, education, national security, international affairs, law and business, the environment, and more. The quality of RAND's research and the importance of its contributions to its clients and sponsors as well as to the public good have earned RAND an enviable reputation.
In the seven decades since RAND's establishment, its work has changed in many respects. Policy issues have become more complex and dynamic. The scope of RAND's research agenda has expanded. Its clients and sponsors have diversified, as have the research products and services they require. Research methodologies and technologies have advanced. Data and information have multiplied.
Nevertheless, throughout seven decades of change, one aspect of RAND's research has never changed—its commitment to high quality and objectivity. Every RAND report, article, database, and presentation is carefully peer-reviewed before its public release. RAND also conducts periodic external and internal reviews of its body of research.
RAND's standards for high-quality research and analysis articulate longstanding RAND concepts and values regarding the characteristics of high-quality studies. They are important tools for everyone involved in conducting, managing, and evaluating RAND's research—the research teams, research managers, and peer reviewers. They are also important to the users of RAND's research, in both the policymaking and research communities, who naturally have an interest in the quality of the research and how that quality is assured.
The standards described below help to ensure that RAND accomplishes its mission of improving policy and decisionmaking through objective, high-quality research and analysis.
Problem formulation is a strength of RAND's research style. A study addressing a practical problem must formulate it within as full a context as possible, to avoid coming up with answers that are suboptimal, naïve, or infeasible. So long as the problem remains tractable, a more broadly conceived study is potentially more useful, more valuable, and, perhaps, enduring. An improved formulation of the problem can become a major research output.
For a study to be successful, a problem must be formulated in a way that ensures it can be, if not solved, at least addressed effectively, with a substantial measure of progress. In order to judge the success of a study and to apply its results, it is important to have a clear understanding of what it is attempting to accomplish—that is, its purpose.
The selection of the study approach must be appropriate to the problem and the purpose. For some types of research, established formal methodologies are appropriate—for example, regression analyses, validated models, or survey protocols. Where no obvious formal methodologies apply, the analytic approach should always be described, and it should be defended if it is at all novel or controversial. The introduction of a new method, the combination of multiple methods, and the application of an existing method to a new kind of problem are situations that obligate the study team to explain the rationale for the selected approach and its advantages over alternatives.
Obviously, even a brilliant design does not guarantee a high-quality study. The research and analysis must also be executed with rigor, technical prowess, and creativity.
A high-quality study cannot be done in intellectual isolation: It necessarily builds on and contributes to a body of research and analysis. The relationships between a given study and its predecessors should be rich and explicit. The study team's understanding of past research should be evident in many aspects of its work, from the way in which the problem is formulated and approached to the discussion of the findings and their implications. The team should take particular care to explain the ways in which its study agrees, disagrees, or otherwise differs importantly from previous studies. Failure to demonstrate an understanding of previous research lowers the perceived quality of a study, despite any other good characteristics it may possess.
Data and other information are key inputs to research and analysis. Data-generation methods and database fields should be clearly specified, and the data should be properly screened and manipulated. The research team should indicate limitations in the quality of available data. In addition, information presented as factual should be correct and verifiable.
Assumptions can mask uncertainties that affect the validity of findings and the expedience of recommendations. Major assumptions underlying a study must be explicitly identified and defended. A high-quality study usually enhances the robustness of its results by varying assumptions in order to analyze alternative scenarios.
A study's findings represent new knowledge that must be integrated with existing knowledge.
For analysts, findings have scientific value and augment what was previously known. They may reveal limitations in research methods or suggest extensions. They may also suggest that a theory can be held with increased confidence, needs refinement, or must be rejected.
For decisionmakers, high-quality findings may bear importantly on a particular decision or deliberation. Or they may contribute to an ongoing debate, helping to frame or inform the discussions in a policy area.
The implications and recommendations should be logical, warranted by the findings, and explained thoroughly, with appropriate caveats.
Based on its findings, a study team derives implications and may develop recommendations.
A high-quality study thoroughly explores the implications of its findings. It examines where new knowledge and old knowledge are congruent and where they are not; it examines whether existing theories and conceptual frameworks have been strengthened or must be modified.
To recommend a course of action is a highly accountable step for researchers. Because of the nature of RAND's mission, the actions of decisionmakers relying on our studies may affect the resources or well-being of many millions of individuals—even entire nations or regions. For this reason, recommendations must follow logically from a study's findings and implications and be strongly supported by them.
Particularly when recommending courses of action, a high-quality study must include caveats to help ensure that it is not applied to inappropriate cases or with unrealistic expectations.
Quality standards also apply to the way in which research is documented. Documentation includes paper-based documents, oral presentations, and web-based formats. Documentation standards apply to both textual and graphical elements.
Accuracy is particularly salient as a prima facie indicator of the quality and credibility of research. In instances where there is significant variability or uncertainty, it is important for the study to indicate the confidence with which one should regard the accuracy of what is presented.
High-quality documentation should make a study understandable to its intended audiences. Textual exposition should be straightforward and precise. Necessary technical terms should be defined and explained. To help explain complex and novel ideas, the documentation should augment textual exposition with graphical or pictorial elements.
High-quality research documentation should be temperate in tone. It should sound neither so flat as to appear unengaged nor so emotional as to appear partisan. Almost all RAND research is relevant to two intersecting sets of discussions: one among analysts, and another among decisionmakers. A temperate tone is best suited to both communities, as well as to wider audiences, such as the general public.
A high-quality study makes an impression and compels respectful attention; it cannot be ignored or dismissed by those working in the area it addresses.
RAND conducts policy research that focuses on real-world problems and practical issues: Therefore, a high-quality study must be not only interesting but also useful. It must contribute to RAND's mission of helping to improve policy and decisionmaking through research and analysis.
As a nonpartisan organization, RAND is widely respected for operating independently of political and commercial pressures. RAND's research style involves frequent interactions with its clients and sponsors, who are recognized as an important source of expertise; but because RAND conducts its research in the public interest, a high-quality study must also be relevant to other stakeholders in the policy domain. Interaction with a broad set of stakeholders can help to ensure the relevance of the research and the practicality of its recommendations.
RAND's clients and sponsors, as well as the general public, must trust RAND's work. RAND is trusted in part because its research is respected for its objectivity. Objectivity is valued in science as a contributor to analytic success, and it often helps to improve the likelihood that research will be used. Frequently it is RAND's reputation for objectivity, in combination with other aspects of quality, that results in its research becoming influential.
Independence refers to intellectual independence, not financial independence. In part to obviate any inference that its research may be biased by its relationship to research clients or sponsors, RAND has instituted strong policies and mechanisms to ensure intellectual independence. RAND has a rigorous research quality assurance process, its researchers commit themselves to seeking and using critical assessments of their work in all phases, and RAND routinely places its peer-reviewed research in the public domain.
All RAND research faces the challenge of addressing both scientific perspectives and policy perspectives. RAND research seeks balance among competing perspectives by treating them fairly, portraying them accurately, and weighting them according to merit.