The Role of Judgment in Admissions

by James R. Vernon

Full Document

FormatFile SizeNotes
PDF file 6.1 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

This report explores issues involved in higher education admissions processes. It analyzes admissions and subsequent student performance at the RAND Graduate School. The academic literature on this problem informs the presentation of statistical relationships among admissions criteria, admissions committee ratings and performance measures. Transformation techniques provide the foundation for a discussion of the merits of alternative ways of thinking about selection, performance measurement and prediction. This research supports several conclusions: 1) Admissions committees confront conflicting objectives. Committee decisions may show inconsistency across applicants, partly because selection combines individual judgments through persuasion and consensus-building. Furthermore, quantified measures of committee members' ratings may not fully reflect their true judgments about the desirability of applicants; 2) Accepted applicants typically receive more ratings than rejected applicants. Moreover, committee members sometimes discuss applicants with each other and see other members' ratings before making their own judgments. Nevertheless, including the evaluation of an additional judge in average ratings increases the correlation between average ratings and both the admission decision and some performance measures; 3) While the RGS admissions committee implicitly gives great importance to GRE scores, different performance measures correlate most strongly with different selection criteria. This complicates the establishment of screening rules; 4) Transforming ratings and grades increases the correlation between them, and it increases the explained variance in admissions decisions. Thus, both admissions and later performance evaluation depend at least partially on a relative rating of students. Such relativism implies that admissions decisions, which depend at least in part on prediction of future performance, affect how that future performance will be measured; 5) The empirical results show the statistical significance of several selection criteria in predicting a variety of measures of student performance. They also compare the significance of these criteria to the significance of quantified committee-member ratings in predicting the same performance measures. Furthermore, this report shows several techniques for transforming measures of selection criteria, ratings and performance, in order to discuss the appropriateness of relative and absolute measures. Although this dissertation focuses on data from RGS, it presents a framework for thinking about problems associated with admissions that general audiences may find useful or enlightening.

Research conducted by

This report is part of the RAND Corporation Dissertation series. Pardee RAND dissertations are produced by graduate fellows of the Pardee RAND Graduate School, the world's leading producer of Ph.D.'s in policy analysis. The dissertations are supervised, reviewed, and approved by a Pardee RAND faculty committee overseeing each dissertation.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.