4. A Framework for Assessing Benefits
An assessment of outcomes that is designed to provide feedback in real time
requires that government policymakers and program managers develop and apply
meaningful and dependable measures. In the past, R&D has been assessed
ex ante, using peer review or cost/benefits analysis, or ex post,
using bibliometric measures or technical review. These measures were valid at
the two ends of the spectrum, but tools to assess ongoing activities and
performance (as required by GPRA) have not been well developed, in part because
there has not been much demand for them and in part because scientific activity
is so difficult to measure. Nevertheless, R&D activities by law now must
attempt to apply quantitative measures, in part to defend the use of scarce
federal funds. Providing a framework within which to apply available measures
to real-time assessment is the challenge addressed here.
This study tests two important assumptions: first, that ex ante and
ex post measures of scientific activity can be adapted to ongoing
research and, second, that to apply these measures, the character and the goals
of the research must first be made explicit. This study further seeks to test
whether the character of the research reveals the goals of the activity.
Benefits expected from ICRD are often not stated up front but may be implicit
in the character of the research and the type of research mechanism (i.e.,
collaborative technical support and so on) chosen to fund and conduct the
research (i.e., contract, grant, U.S. government laboratory). When goals are
made explicit, the reasons and types of cooperation can help enumerate the
expected benefits. Once it is known what benefits are expected, it may be
possible to craft measures to determine whether the nation is reaching its
goals and receiving the expected benefits.
Assessing the benefits of ongoing ICRD requires matching the reasons and types
of ICRD with appropriate measures. Available measures, generally used to
report the outcomes of R&D--when placed in the proper context--can
provide real-time, usable information about the benefits of ICRD. This chapter
describes a framework for identifying specific measures that can assess the
benefits of ICRD and describes the results of a case study testing this
premise.
This study has identified a number of characteristics of ICRD that can help
shed light on expected, measurable benefits. These include (1) the mission of
the funding agency; (2) the character of the research (collaboration,
conferencing, etc.); (3) the reasons for choosing ICRD (large scale, global
nature, etc.); and (4) funding. The funding mechanism chosen by an
agency--grant, contract, cooperative agreement, or program-based
activity--reflects the benefits that government expects to derive from the
activity:
- Grants and technical support are intended to benefit the receiving party; no
direct product or service is expected by the funding agency. NSF grants to
researchers promote the conduct of science, not to meet technical
specifications of an NSF mission, but to advance excellence in science. USDA
grants or aid to Brazilian soybean farmers directly benefits Brazil, with an
indirect benefit (lower cost products, perhaps) accruing to the United States.
- Program-based, contract-supported activities are done for the direct benefit
of a government's mission-oriented program. Activities such as the manufacture
of satellites is often conducted under contract for DoD or NASA to meet their
specific technical needs.
- Under a cooperative agreement, both the government and the cooperating party
expect to benefit from the activity. University-based efforts under a
cooperative agreement to create an international science database for the
government is an example of a case where both sides benefit from government
R&D funds dedicated to a specific project.
In general, ICRD is funded, not to promote international cooperation, but to
meet direct and indirect and long- or short-term goals of the funding agency.
In Figure 4.1, the agencies funding ICRD are notionally arrayed along an axis
of benefits and goals on one hand and time on the other. Agencies are placed
on a discrete point along the axis based on a subjective judgment of the
character of that agency's ICRD. Pointers suggest a trend in either the
indirect or the long-term nature of that agency's activities. This
illustration suggests ways to think about which agencies have programs that
would benefit from enhanced quantitative assessment of ICRD. Clearly,
different assumptions of how agencies, or different programs within agencies,
conduct ICRD could change that agency's place on the axis.
Figure 4.1--Goals/Benefits Arrayed Against Time
- In the first third of the benefits/time axis, where ICRD projects have a
short-term, direct benefit, government agencies often use contracts or
intramural research as the method of funding ICRD. In these cases, ICRD is
conducted to meet a direct need of the government, for example, the development
of a satellite, a prosthetic device or a component for a flight simulator.
Government expectations, and notably, the ways to measure the results of these
activities, are usually explicitly stated in the contract and quickly measured
and evaluated on the basis of the product received.
- In the distant third of the benefits/time axis are technical support
activities or grants that may pay off either far into the future, or may never
pay off in the sense of creating a measurable outcome. These activities
include AID or USDA grants that support foreign scientific, technological,
health, or agricultural research or development, which have little or no direct
benefit accruing to U.S. citizens or a government agency, or in some cases,
even to the United States, except perhaps in the form of political good
will. Because of the indirect nature of the benefits and the generally
long time period it takes to produce results, benefits in this far area are
extremely difficult to measure. Governments usually undertake this type of
activity for reasons of principle, for strategic political reasons, and to
support human rights. Defining measures for these activities may not be
possible or realistic.
- In the middle third of the benefits/time axis are activities designed to
produce a benefit to the U.S. government or U.S. citizens. This is the field
we explore in this chapter of the report--where measures can provide some
insight, but where benefits have rarely been measured in the past. It is in
this middle third that many of the questions of accountability will be
discussed in the public forum, and where agencies will need to present new data
under emerging government requirements for outcome-based management.
Activities in this tier are dominated by grants but can also include
intramural, extramural, and contract-supported research.
In the process of conducting research and development, researchers create
products. The products can be new knowledge published in conference journals,
a scientific or technical product such as a new chemical catalyst, or a
commercial product such as an electronic sensor. The creation of new knowledge
and its use by others create "footprints" that provide a way to track the
benefits of research and development to
science, the economy, the nation, and international relations. The
footprints--article citations, patents, product sales, international
conferences--can be documented to varying degrees, depending upon the nature of
the "product." When compared with expectations or the performance of similar
activities in the same or other fields, a measure of the footprints can provide
feedback to policymakers and program managers.
Measures of the results and outcomes of research and development can paint a
picture of how well a program or project is meeting its goals, and thereby
producing benefits for the United States. The "proper context" for using these
measures almost always requires a quantitative report of the quality and
benefits of research, along with the quantitative measure of footprints.
Measures of the output and outcome of research and development are difficult to
apply across the board: not all measures will apply in all situations.
Measures must be matched to the nature of the activity being assessed.
In an effort to describe measures for ICRD, we have developed a framework to
identify first the benefits and then the measures that may shed light on the
extent to which the United States is receiving these benefits. The framework
developed for measuring the benefits starts with the four broad reasons
identified earlier for conducting ICRD: (1) the very large scale of the
equipment or investment required to conduct a project; (2) the global nature of
the subject; (3) the location of unique expertise or natural resource; and (4)
a miscellaneous category, where it is the agency's mission to support
international cooperation. For each of these reasons for funding ICRD,
projects are crafted to achieve the government's mission. To simplify the list
of cooperative projects, we boil these reasons down to four basic types: (1)
collaborative research, (2) technical support for a U.S. project or for a
foreign S&T project, (3) operational support of a facility to conduct
international cooperation, and (4) standards and database development.
At the intersection of the reasons for conducting international cooperation in
R&D and the types of cooperation, a list of expected benefits can be
enumerated. Arrayed in a matrix, these reasons and types of projects provide
the opportunity to identify the nature of the benefit that the government
expects from funding ICRD. Once we identify benefits, we can more easily match
measures to the nature of the activity being measured. Tables 4.1 and 4.2
present the benefits implied by the government funding at the intersection of
the reasons and types of cooperation.
Table 4.1
Benefits Expected from and Measures Suggested for Collaborative Research and
Technical Support
|
Collaborative Research |
Technical Support |
|
| Reasons for Cooperation |
Expected Benefits |
Suggested Measures |
Expected Benefits |
Suggested Measures |
| Very large scale equipment |
Enable conduct of large-scale project beyond reach of any one country; gain
access to foreign equipment; lower cost of research (e.g., space station; ITER; CERNa) |
Agency report on meeting milestones; survey participants for foreign $ and/or
in-kind contribution; expert judgment of technical feasibility/excellence; survey records; U.S. percentage
of time accessing equipment |
Improve opportunities for U.S. scientists, improve efficiency of research
equipment, data (e.g., aid in building/maintaining a tropical research lab in Costa Rica) |
Extent of usage per sq ft of facility compared with use of a similar domestic
facility; extent to which research conducted is published/cited; development of data used in published
reports |
| Global nature of subject |
Access to subject of study; leverage scarce funds; improve environment or
reduce hazards (e.g., CGC,b Ocean drilling, earthquake research) |
Survey scientists: gaining access to subject/data. Survey
agencies/investigators: leveraging foreign funds. Citation search: no. of co-authored papers increasing.
U.S. report: reducing specific hazards in U.S./the rest of the world |
Improve capability of U.S. data collection; reduce environmental hazards;
improve agricultural efficiency; coordinate data collection (e.g., helping Brazil reduce pollution
emissions) |
Agency report on usefulness of data; increased crop production; reduced
emissions; agency report on coordination of data collection/sharing |
| Unique foreign expertise |
Enable excellent science; share data; improve productivity of research;
improve U.S. science base (e.g., French excellence in materials science) |
Citation counts of jointly published articles compared with nat'l counts; biblio
counts of increased improve productivity of unit of knowledge produced; joint patent counts |
Build foreign science capabilities to improve science overall (e.g., working
with French scientists in Africa to contain disease) |
Agency report on ability of national scientists to manage labs; increased
foreign publications/citations; increased joint projects/publications |
| Government/agency mission |
Either meet direct need of agency for new knowledge (e.g., energy) or improve
foreign security or living conditions (e.g., DoE's energy security, AID/CDC's aid for infectious disease
control) |
Use biblio/cite counts to show accessing best knowledge in world; expert
judgment or reduced risk; U.S,/other int'l org indicators of improved living conditions |
Improve foreign standards of living; reduce infectious disease; improve
foreign agriculture (e.g., AID's support for clean water) |
U.S./UN data on indicators of standards of living; expert judgment of improved
containment of nuclear waste in states of the CIS |
aOriginally, Conseil Européen pour la Recherche Nucleaire. Now known as the European
Laboratory for Particle Physics.
bCucurbit Genetics Cooperative.
Table 4.2
Benefits Expected from and Measures Suggested for Operational Support and
Standards/Database Development
|
Operational Support |
Standards/Database Development |
| Reasons for Cooperation |
Expected Benefits |
Suggested Measures |
Expected Benefits |
Suggested Measures |
| Very large scale equipment |
Provide place for int'l collaboration or cooperation; enable sharing of data (e.g., NSF's
astronomical lab) |
Extent of usage of facility compared with use of a similar domestic facility; extent to
which research conducted is published/cited; development of data used in published reports |
Establish global standards that allow cooperative research or commerce; enable access to
data to increase research efficiency (e.g., standards for satellite production and signaling) |
Survey scientists, trade officials, and foreign standards organizations on
creation/acceptance |
| Global nature of subject |
Provide place for int'l collaboration or cooperation; enable sharing of data (e.g., NOAA's
hurricane database) |
Extent of usage of facility compared with use of a similar domestic facility; extent to
which research conducted is published/cited; development of data used in published reports |
Establish global standards that allow cooperative research or commerce; enable access to
data to increase research efficiency (e.g., standard for common pollution monitoring equipment) |
Survey scientists, trade officials, and foreign standards organizations on
creation/acceptance |
| Unique foreign expertise |
Build labs where expertise/subject of study is located to encourage cooperation and
excellence (e.g., polar research labs in Antarctica) |
Extent of publication compared with that of similar projects ($ per subject); citation
counts of jointly authored papers vice U.S.-authored papers; cost of facility per unit of knowledge
compared with similar or ideal facility |
Gain access to unique foreign data that will enable U.S. research (USGSa access to Chilean
seismology data set) |
Survey scientists or agency officials; report on ability to access and share data, and on
usefulness of data for research; publication counts |
| Government/agency mission |
Provide technical expertise to another country in need of improving national science for
int'l benefit (e.g., CDC/USGSa work in Russia on mine safety) |
Expert judgment of improvement of foreign science; reduce measurable hazards; increase
in foreign nationals' publication of int'lly recognized work |
Establish global standards that allow cooperative research or commerce; enable access to
data to increase research efficiency (e.g., NIST project on silicon density measures) |
Survey agency and/or cooperating partner on extent of national or global standards
creation/acceptance |
aU.S. Geological Survey.
Tools available to track and monitor research and development projects, and by
extension, ICRD activities, fall into four broad categories: (1) bibliometrics,
(2) milestones, (3) surveying, and (4) technical judgment.[1]
- Bibliometrics is the technical name for a range of analytical methods
using published materials (books, reports, patents, software, designs,
prototypes, and blueprints) to develop descriptive statistics, multidimensional
analyses, and graphical representations of the output of science.
Bibliometrics can take a number of technical data sources for analysis,
including the following:
- Publication counts
- Citation counts
- Co-citation analysis
- Co-word analysis
- Scientific mapping.
- Milestones. Scientific or technical projects often establish
"milestones," or achievements, expected over the course of a project. The
project team can map these milestones against actual achievements, thereby
providing useful information about the outputs of an ICRD project.
- Technical Review. Technical or expert review is the most widely used
approach in research evaluation, both in the United States and around the
world. In the United States, technical review varies among agencies, from very
informal assessment processes to highly structured retrospective quality
control mechanisms. Within one review, then, the process transforms the
descriptive judgments of peers into quantitative ratings, which can be compared
across projects to identify those that need improvement.
- Survey Methods. Traditional survey methods--either on paper, in
person, or by phone, where a group of participants or stakeholders are asked to
provide responses to a set of questions--are often used to assess the benefits
or outcomes of research.
The differences among the agencies and the range of the nature and character of
research and development in the various fields of science and technology make
it difficult to craft and recommend measures that will apply equally to all
areas of ICRD. A range of measures must be considered. Under NASA programs,
for example, programs tend to fall within two broad categories: scientific
data exchange and technical cooperation. Neither of these types of activities
require a great deal of scientist-to-scientist collaboration. NSF projects, however, tend to involve close
collaboration among researchers. Each type of research and development
activity, as well as the reasons for the research and the expected benefits,
must be considered in crafting measures.
In addition to helping the program officer identify measures that might apply
to a specific program, this framework may also be useful to the executive-level
policymaker negotiating international agreements. By referring to the
framework, the policymaker can identify measures that he or she may wish to
"build in" to international agreements to help monitor how well projects under
these agreements are meeting the goals laid out by the governments and
cooperating parties.
To test the ability of the framework approach to provide measures to track
activities and provide feedback on benefits, RAND conducted a case study of
international cooperative research and development in earthquake sciences and
seismology. This subject was chosen because of the active nature of U.S.
cooperation with other countries in this field, because of the growing
importance of disaster preparedness and relief, and because activities in this
area range from collaborative research projects, to technical support, to
sharing data and equipment. In the process, RAND developed useful data about
earthquake sciences and seismology, but the principal goal of this case study
was to see if it is possible to use the framework to identify measures that
provide feedback to policymakers on the benefits of ICRD.
Focusing on earthquake sciences and seismology, RAND used the RaDiUS inventory
and other sources to first identify the full range of international cooperative
activities being pursued by U.S. government-funded scientists. Second, we
examined goals established at the interagency level for cooperative research
and development activities. We analyzed these data to identify the type of
cooperation being pursued, the countries involved, and the nature of the
activities.
Based on the framework and RAND's review of the nature of the research,
expected benefits and possible measures for this activity were identified.
Expected benefits can be classified as access to subject of study, leveraging
scarce funds, reducing global hazards, and access to foreign data and equipment
that will allow research to proceed that might not otherwise have been
possible. Based on what we viewed as workable measures, we decided to test
three:
- Bibliometrics: A citation survey of papers authored jointly by U.S.
and foreign researchers in 1985 and 1995 on the assumption that, since
cooperative research has increased, jointly authored papers would increase.
- Survey: A participant survey, asking researchers to what extent
foreign partners contributed resources, either financial or in-kind, to the
project.
- Expert judgment: Experts reporting on the technology standards that
dominate equipment used in joint research to determine the extent to which the
United States is leading other countries in technology development.
Then, using this framework, we identified two reasons for ICRD cooperation:
the global nature of the subject and the very large scale of equipment or
nature of investment. Earthquakes and tremors occur every day all around the
world. In a number of countries, seismographic equipment measures these faults
and tremors and produces scientific data. Collection of this data analysis of
trends provides the basis for international cooperation. In addition, large,
expensive "shake tables" provide an experimental field to generate different
types of earth shaking at varying intensities so that scientists can measure
the effect of tremors on building structures. Japan and the United States have
the two most advanced shake tables in the world. The two tables have
complementary research capabilities and so are shared by researchers from both
the United States and Japan as well as researchers from around the world.
Measures we identified as appropriate for collaborative research on global
subjects and for large-scale research were considered. In this case study, we
focused our inquiry on measuring the benefits of collaborative research and
standards setting because these activities dominate U.S. government-funded ICRD
in this area.
Bibliometrics
A citation survey conducted for this study used two scientific bibliographic
services to identify papers on subjects that reflected one or more terms on a
list developed for this part of the study. The research contained in these
bibliographies was limited to basic research. The survey showed that papers
jointly authored by a U.S. scientist and a foreign scientist rose to 585 papers
in 1995 from 379 in 1985, even while funding remained constant in real terms.
More joint papers were multinational in authorship in 1995 than in 1985. When
papers were written by only two authors, Japanese researchers were the most
likely collaborators, followed by Russian and Chinese, respectively. Data does
not exist to compare the amount of R&D funds committed to earthquake
sciences and seismology in 1985. Anecdotal reports from scientists indicate
that total U.S. dollar funding for this research may have been higher in real
terms in 1985, although ICRD may have claimed a smaller share of the funding
than it did in 1995.
Survey
In a survey of one-fourth of the principal investigators associated with ICRD
projects researching earthquakes or seismology, RAND found that, on average,
the foreign financial contribution equals the U.S. contribution. In
three-fourths of the projects surveyed, investigators reported either financial
or in-kind contributions to the cooperative project. Of these projects
reporting a foreign contribution, 47 percent had a foreign contribution that
exceeded the U.S. contribution; 35 percent had an equal contribution from both
sides; and 28 percent had a foreign contribution that was less than the U.S.
side. The highest leverage of funds was for research projects with Japan.
Expert Judgment: Standards
In an informal survey of experts on the standards that guide the development of
seismology equipment, U.S. companies and research labs are setting the standard
for 80 percent of the essential research equipment used in this field. In the
area of building codes and local safety standards, while U.S. building and
safety standards are often studied by foreign officials and researchers, the
United States is not setting the world standard--in large part because building
and safety codes are locally determined based on specific terrain and urban
design. These standards are difficult to export to other parts of the
world.
Qualitative Findings
During the course of the survey, researchers reported that, as a result of
government funding and the existence of S&T agreements to encourage
earthquake research, they had been able to establish excellent ongoing
relationships with foreign scientists. During their research, U.S. scientists
and engineers report meeting key foreign researchers, leading to opportunities
to share data and conduct additional joint research. Joint papers often
resulted from these activities. Moreover, both U.S. and foreign students were
trained as a result of these projects. These activities have helped U.S.
researchers stay at the state of the art in earthquake and seismological
research, according to several researchers.
The tools used and the data collected for this case study give a good picture
of benefits accruing to the United States as a result of participation in ICRD.
The case study did not succeed in identifying these benefits with existing
data--new data collection through survey methods was required to get a full
picture. Moreover, quantitative measures appear to be enriched by the
qualitative reports received during the course of the survey, providing a
fuller picture of the benefits of research.
One of the goals of this case study was to test whether unobtrusive methods are
available to gather data on the benefits of ICRD. Indicators or output
measures that are already collected or readily available would reduce the need
for special studies to estimate how well ICRD is meeting goals and could
provide a continual monitoring mechanism. Unobtrusive measures are important
because, once investigators are asked to report on specific aspects of
research, these factors tend to become the "goals" toward which researchers
strive. This case study did not find that unobtrusive measures will provide a
full picture of benefits, at least not at this time. A survey of principal
investigators was necessary to identify the extent to which projects were
leveraging foreign financial contributions. Additional data collection may
help to reduce the extent to which direct surveys are required.
[1] We do not include a discussion here of
the social returns to research and development because this method is used to
assess the benefits of R&D on a broad scale, not at the program or project
level. For a full description of social rates of return analysis, see Steven
W. Popper, Economic Approaches to Measuring the Performance and Benefits of
Fundamental Science, Santa Monica, Calif.: RAND, MR-708.0-OSTP, 1995, and
Caroline S. Wagner, Techniques and Methods for Assessing the International
Standing of U.S. Science, Santa Monica, Calif.: RAND, MR-706.0-OSTP, 1995.
Contents
Previous section
Next section