Mar 11, 2008
The Brain Korea 21 program (BK21), an effort to improve Korean universities and research, has attracted a great deal of attention in Korea, producing the need to understand how well the program is meeting its goals. RAND developed a logic model for identifying program goals and dynamics, suggested quantitative and qualitative evaluation methods, and recommended a prototype database of evaluation measures. Because formal evaluation will take some time to accomplish, RAND also offered suggestions for ongoing program management and the development of the Korean university research sector.
In the late 1990s, the Korea Ministry of Education and Human Resource Development (MoE), in response to concern over the relatively low standing of the nation's universities and researchers, launched the Brain Korea 21 program (BK21). BK21 provides fellowship funding to graduate students, postdoctoral researchers, and contract-based research professors who belong to research groups (sa-up-dan) at top universities. Recipients are selected on the merit of the research groups and universities to which they belong, not on individual merit.
The program has had two phases so far. In Phase I, which ran from 1999 to 2005, BK21 awarded about US$1.4 billion. In Phase II, which began in 2006 and is scheduled to run through 2012, BK21 will award an additional US$2.1 billion. Whereas Phase I emphasized university-level excellence, Phase II emphasizes department-level excellence and university-industry links.
BK21 has attracted a great deal of attention, in part because of its strategy of concentration, which results in relatively large awards and considerable prestige for recipients. Given the program's high profile, there is great interest in determining its effects on universities and on human resource development. The MoE and the Korea Research Foundation therefore asked RAND for advice on the best possible means for evaluating the program.
The RAND team began by developing a logic model to specify the goals and missions, inputs, activities, incentives, outputs, and outcomes of BK21 and the linkages among them. This model provided a conceptual framework for a quantitative evaluation model, as well as for evaluation metrics. Model components and variables for evaluation were derived by reviewing relevant literature and interviewing key BK21 informants. The team identified two main BK21 goals: developing research manpower and fostering globally competitive graduate departments and research universities. Inputs include program resources and rules. Activities consist of the research and teaching funded by the program. The program's incentives include lower costs for students to obtain degrees, income stability for young researchers, and subsidization of research and development for universities. Outputs and outcomes are research quantity and quality, links between universities and industry, and university prestige and ranking.
The team developed a fixed-effects estimation model to examine the quantitative effects of BK21. This type of model is advantageous because it allows researchers to ignore preexisting differences in outcome variables. It is appropriate for measuring program effects in such areas as number of papers published, faculty composition, number of PhD graduates, and number of patents issued. The team validated the model using Phase I data and found that it could identify program effects among schools that had received relatively low amounts of funding.
The report details in an extensive list of relevant metrics—i.e., criteria for evaluation—and measures—i.e., practical means to estimate the value of a metric—to allow for assessing quantitative and qualitative program effects. For example, the program goal of fostering globally competitive research universities can be assessed by means of metrics on graduate school infrastructure and prestige, which in turn can be measured by university ranking, number of faculty publications, and the amount of infrastructure funding. No measure will be perfect, but multiple measures can help evaluators check the accuracy of any one of them and determine the level of detail at which they wish to make measurements.
Not all measures are equally important. Evaluators should prioritize measures, taking into account the effort needed to assess them, the relative importance of the measures, and the ease of gathering data. Greater availability of data from applicants and recipients, relative to nonrecipient and nonapplicants, could result in structural bias in the evaluation. Thus, evaluators will need to choose measures carefully and work to obtain comparative data from nonrecipients and nonapplicants.
The report recommends using four groups of measures: (1) educational activities, (2) noninstructional activities related to scientific and academic inquiry, (3) research and development activities with industry, and (4) other "general" measures. In determining which data to gather, evaluators should consider the difficulty of gathering data; the ideal time between observations; qualitative differences, such as those between recipients and nonrecipients; the level at which to gather data; and possible data sources. The report also suggests a format for gathering this information, including a format for the data series necessary to support the measures we suggest.
Final analysis of the BK21 program's effectiveness will have to await a full data-gathering and analysis effort. In the meantime, BK21 program administrators must make ongoing decisions about allocating program resources. In addressing that topic, the report went beyond the findings of the evaluation model. It suggests that the sa-up-dan is not an optimal unit for support, even for graduate fellowships, because it does not promote full competition. Instead, Korea might consider increasing the role of merit-based competition in funding and awarding funds on an individual project basis. Project funding mechanisms, including size and length of funding, should be adapted to different fields of science. Korea may also wish to promote the development of market financing so that universities can borrow funds for construction of infrastructure and facilities. Because it may take a number of years to implement these changes, transition arrangements are needed so that the university research system can become more responsive yet avoid abrupt shocks and dislocations.