Inventory Methodology

2. Inventory Methodology

No one set of government activities easily comprises the category "international R&D cooperation." Activities vary by agency mission, by country, by topic, and by many other variables. Conducting an inventory of ICRD requires significant detective work that includes reading thousands of individual program, project, and award data contained within RaDiUS or obtained from agency sources. The process for collecting data and the criteria for inclusion are presented here. Our findings are presented in Chapter Three.

Source of the Data

The vast majority of data on government R&D spending is electronically available and fully searchable through RAND's RaDiUS database, the first comprehensive, fully searchable data system that contains information on the approximately $70 billion of annual spending classified by the federal government as "research and development," as defined by OMB Circular A-11.

Identifying the Data Set

Figure 2.1 shows the five steps taken to create this inventory. Part one involved collecting data from official and primary data sources. The RaDiUS database and supporting data tables were searched using an iterative search strategy. Searches were conducted on words (such as "international" in conjunction with "collaboration"), on units of government (such as NASA), and on countries and continents (such as "Japan" or "Asia"). Hundreds of searches were run to capture all relevant programs, projects, and awards. Part two of the process generated abstracts of candidate programs, projects, and awards.

Figure 2.1--Method Used to Compile Data

Part three involved examining and sorting the data and running additional searches where needed. Once the full set of relevant activities was identified, we read the project descriptions and award abstracts, and, with reference to the criteria established for this study, rejected or counted the project in the inventory and classified the activity according to a range of characteristics. Over 9,000 program and project descriptions and award abstracts were read. Many of the 9,000 were not relevant to the study but were captured in the searches because they had terms such as "international implications" or "international reputation." In addition, many terms, such as "Japanese maple," Chinese hamster," and "New Zealand rabbit," were inadvertently captured. Once these awards and projects were eliminated from the data set, the inventory data set numbered approximately 3,000 projects and awards.

Part four of the process involved consultations with federal funding experts and with staff at the Office of Science and Technology Policy to identify where additional data were needed. We then contacted government officials to ask for assistance in validating data obtained from RaDiUS, and if necessary, in identifying additional budget data. In some cases, supplementary data were not available from the agency. (These cases are noted in the agency descriptions in Chapter Three.) Part five of the process involved compiling all the data collected from all sources, placing the data in spreadsheets and examining the data for duplications and obvious errors, then coding and analyzing the data set.

Refining the Data Set

Included in this inventory are any type of program-based activity--projects or awards (contract, grant, or cooperative agreement)--that have, as one of the principal purposes, the sponsorship of international cooperation. Clearly, there is much international activity, coordination, and sharing that is not captured by this inventory since we limited the study to activities where cooperation is a specific project goal. If a project or award description reported cooperation as a principal purpose, the full average annual FY95 budget authority[1] for that activity was included in the inventory. While this method may have led to overcounting in a limited number of cases, the alternatives were unworkable. Alternatives could have included (1) asking agency officials to report on the share of a project dedicated to R&D--a datapoint they usually do not have available, (2) contacting principal investigators directly and asking them to report on the extent of funding dedicated to ICRD--a Herculean task given the final data set of nearly 3,000 projects, or (3) having RAND staff make a judgment--an impossible task without additional information.

Cooperation is defined for the purposes of this study as federally supported activities where a U.S. government-funded researcher is involved in a project with a foreign researcher, a foreign research institution, a multinational institution, or a multinational research project. Projects and awards that fell within this definition encompassed scientist-to-scientist collaboration as well as cases of field research where a scientist worked with a collaborator to gain access to a natural resource, research for a Ph.D. dissertation when that activity was classified by the agency as "R&D," and cases where government agencies support the conduct of research through operational and technical support, again, where that activity is counted as R&D. The definition did not include activities for which a U.S. government official met for a brief time or intermittently shared data with counterparts from other countries.

Agencies that use contracts, grants, and cooperative agreements to conduct most or all of their research and development are the most fully represented in the RaDiUS database and therefore are the most fully represented in this inventory. When government money changes hands, records are made of those transactions, and the grant or contract recipient often provides a full description of the activities. This is often referred to as extramural research. Agencies that primarily sponsor extramural research include the National Science Foundation (NSF),[2] Health and Human Services (HHS),[3] the U.S. Department of Agriculture, and the non-lab-based activities of the Departments of Defense and Energy (DoD and DoE). If international cooperation was established after the grant or contract was awarded, the activity will not be captured by this search methodology.

When the R&D is conducted within government laboratories--intramural research--spending is more difficult to track. While we made an effort to identify and characterize these activities, ICRD activities in these parts of the government may not be fully represented in this study. Identifying and collecting information on intramural research involved first using RaDiUS to locate the likely federal agencies that contain these activities, and second contacting the agencies to seek the information directly. Even though we made extensive efforts to contact agencies with program or lab-based activities, at times it was difficult to decouple the international activities from other activities going on in these agencies or laboratories. (In two cases--NASA and the Smithsonian--we worked with the agencies to identify the international components of programs.) Agencies sponsoring this intramural activity include parts of NASA, the EPA, the Agency for International Development (AID), the DoD, the DoE, and the Smithsonian Institution.

Coding the Data Set

As the full data set was being compiled, the data were coded four ways:

By country, or, in cases where researchers from more than two nations are involved or where a U.S.-funded researcher reported working with a multinational research organization, as a "multinational" activity.
By type of cooperation, in categories developed by RAND, for identifying the character of the ICRD projects or programs funded by the U.S. federal government, as shown in Table 2.1.
By fields of science or technology, using a list, shown in Table 2.2, adapted by RAND from the National Science Board list of areas of science and technology.
By sponsoring agency.

Table 2.1
Types of Cooperative Activity Identified in the Course of the Study

Collaboration

Research activities where a principal purpose of the activity is to sponsor international collaboration of the following types: between a researcher funded by the U.S. government in a joint project with a collaborator from another country, where a researcher funded by the U.S. government is conducting a research program that involves actively sharing information with another researcher conducting the experimental or observational research, or where a researcher is contributing to an international cooperative project. (Not where a U.S. researcher is using international data or data from another country, not where a U.S. researcher is training foreign students in the United States or another country, and not where U.S. graduate students are studying in another country.)

Conference

Either foreign or domestic--and including symposia, workshops, or other official meetings where scientists from around the world participate in a scientific or technical meeting to describe and share ongoing research.

Contracts

Where the U.S. government contracted with a foreign source for the purpose of conducting research and development.

Database development

Where the U.S. government is sponsoring the creation of an international database of information being collected from sources worldwide, and which will be available to researchers from around the world.

Operational support

Where the U.S. government is funding the building, maintenance and/or operation of an international research center, designed specifically for the purposes of international collaboration, in the United States or in a foreign country.

Standards development

Where the U.S. government is sponsoring the development of a technical or scientific standard that will serve as the basis for future research, development, or production for practitioners around the world.

Technology transfer

Where the U.S. government is actively seeking to transfer technology from a foreign country to the United States.

Technical support

Where a U.S. government laboratory or a U.S. government-sponsored researcher is providing research and development results or other support to a foreign researcher or laboratory.

Table 2.2
Fields of Science Used to Identify the Nature of ICRD

Physical Sciences

Engineering Sciences

Life Sciences

Social Sciences

Mathematics

Chemical Engineering

Plant Biology

Economics

Physics

Computer Engineering

Agricultural Sciences

Anthropology

Chemistry

Communications Engineering

Biotechnology

Demographics

Earth Sciences

Materials Technology

Biomedical Technology

Other Social Sciences

Geology

Aerospace and Aeronautics

Environmental Sciences

Other Physical Sciences

Other Engineering Sciences

Other Life Sciences

Strengths and Limitations of This Approach

The data collection technique used in this study has significant strengths. First, the data have been gathered from the "bottom up" by identifying activities at the lowest level and aggregating up into programs, bureaus, and agencies. Second, this approach enabled consistent screening of the data using a single filter. This helped us ensure the comparability of data across agencies. Third, this approach has the advantage of identifying ICRD activities in actual operation as opposed to cooperation merely proposed in international bilateral and multilateral cooperative agreements. Fourth, the method we used is transparent and reproducible. This allows trend analysis over time and across agencies.

The approach used to conduct this inventory also has limitations. Some agencies do not compile or report data on activities at the project or award level. In these cases, the inventory includes only program-based activities at highly aggregated budget line items. AID, for example, reports data only at the budget line item, so no additional analysis or comparison of AID activities is possible. The AID budget line item data are delineated by region, but that is the most detailed data we could find for AID activities. When this inventory was performed, AID could not provide additional information on the types of R&D activities sponsored in these regions. The EPA also does not report detailed project-level activities. A small amount of DoE and DoD lab-based activities may also be unreported.

[1]In many cases, the activities identified in this inventory were funded on a multi-year basis. In these cases, RaDiUS reports, and the project team counted, the average annual funding figure.

[2]Close to 95 percent of NSF R&D funds leave the agency in the form of grants or contracts.

[3]Close to 80 percent of HHS RD funds leave the agency in the form of grants or contracts.

Contents
Previous section
Next section

Collaboration	Research activities where a principal purpose of the activity is to sponsor international collaboration of the following types: between a researcher funded by the U.S. government in a joint project with a collaborator from another country, where a researcher funded by the U.S. government is conducting a research program that involves actively sharing information with another researcher conducting the experimental or observational research, or where a researcher is contributing to an international cooperative project. (Not where a U.S. researcher is using international data or data from another country, not where a U.S. researcher is training foreign students in the United States or another country, and not where U.S. graduate students are studying in another country.)
Conference	Either foreign or domestic--and including symposia, workshops, or other official meetings where scientists from around the world participate in a scientific or technical meeting to describe and share ongoing research.
Contracts	Where the U.S. government contracted with a foreign source for the purpose of conducting research and development.
Database development	Where the U.S. government is sponsoring the creation of an international database of information being collected from sources worldwide, and which will be available to researchers from around the world.
Operational support	Where the U.S. government is funding the building, maintenance and/or operation of an international research center, designed specifically for the purposes of international collaboration, in the United States or in a foreign country.
Standards development	Where the U.S. government is sponsoring the development of a technical or scientific standard that will serve as the basis for future research, development, or production for practitioners around the world.
Technology transfer	Where the U.S. government is actively seeking to transfer technology from a foreign country to the United States.
Technical support	Where a U.S. government laboratory or a U.S. government-sponsored researcher is providing research and development results or other support to a foreign researcher or laboratory.

Physical Sciences	Engineering Sciences	Life Sciences	Social Sciences
Mathematics	Chemical Engineering	Plant Biology	Economics
Physics	Computer Engineering	Agricultural Sciences	Anthropology
Chemistry	Communications Engineering	Biotechnology	Demographics
Earth Sciences	Materials Technology	Biomedical Technology	Other Social Sciences
Geology	Aerospace and Aeronautics	Environmental Sciences
Other Physical Sciences	Other Engineering Sciences	Other Life Sciences