Developing Institutional Standards for Critical Thinking Using the Collegiate Learning Assessment

In recent years, critical thinking has emerged as an important component of college and university curricula. In 2000, the Council for Aid to Education (CAE) began developing an exam to help institutions measure their value-added to students' critical thinking skills—in other words, how much students improved their skills at solving problems, using analytic reasoning, and communicating their ideas clearly while attending college. This exam, the Collegiate Learning Assessment (CLA), launched in fall of 2004, asks entering freshmen and graduating seniors to draw inferences and conclusions from graphic and written documents and to respond to a variety of open-ended questions. More than 200 colleges and universities currently use the CLA to assess how well their students have learned to think critically over the course of their undergraduate education.

While the CLA fulfills an important need, many school administrators are still grappling with how to use the results from this new assessment tool. CAE returns numerical scores for students who take the exam but intentionally refrains from providing a standard for schools or students to compare themselves against, and instead suggests that standards should be tailored to each institution uniquely and that the primary focus should be on showing that students make meaningful improvements from the freshman to senior years. As a result, many administrators remain uncertain how to judge what is a satisfactory score.

To assist schools in understanding how to interpret CLA results, CAE asked RAND researchers to develop and test a method that would allow individual institutions to establish their own standards for student performance on the CLA. The RAND team reviewed the literature pertaining to setting such standards, especially for exams with open-ended test questions, also known as constructed-response exams. They then developed and tested a method that administrators can follow to establish and validate standards for interpreting CLA scores.

A Standard-Setting Process Should Include Several Key Elements

The RAND team identified a number of guidelines for standard setting from the literature they surveyed, including the following:

• Articulate the purpose of establishing standard scores.
• Select a panel that represents relevant stakeholders in an institution or department.
• Give clear instructions throughout the standard-setting process.
• Establish a system of checks and balances to ensure the consistency, reliability, and validity of the standards.
• Collect feedback from those who participate in the standard-setting process to confirm that they have confidence in the standards they set.

The Standard-Setting Method Developed for the CLA Involves Three Steps

Using the guidelines for standard setting suggested by the literature, the researchers designed this three-step method:

Step 1: Develop an Initial Set of Standards Using Individual Faculty

In this step, individual faculty members are asked to indicate their expectations for both freshman and senior performance, without discussing their opinions with other faculty. Faculty review sample CLA test prompts and student responses (prearranged in order by score from low to high) and indicate the range of scores that best represents each of the following categories of performance: Unsatisfactory/Unacceptable, Adequate/Barely Acceptable, Proficient/Clearly Acceptable, or Exemplary/Outstanding.

Step 2: Discuss in Groups and Arrive at Consensus on Institutional Standards

In the second step, faculty are asked to come to consensus regarding the cutoff points for each standard of performance. The faculty members work in small groups to review the same set of ordered sample student responses, compare the standards they set individually, and develop new standards from that discussion.

Step 3: Confirm the Standards Established in Step 2

In this step, faculty participants validate the performance cutoff points by sorting randomly ordered student test responses, with the test scores concealed, into the four categories of performance. If the process of setting standards has been successful, the average scores of the essays sorted into each category of performance should fall between the cutoff points established in the consensus step.

A System of Checks Is Key to Validating the Resulting Standards

To confirm the standards identified through this process, administrators should implement a system of checks and analyze the outcomes from each one carefully. Such checks should include looking at the consistency among individual judgments of the cutoff points for each of the scoring categories, analyzing whether the faculty members could apply their group consensus standards to a new batch of answers, comparing the results using two separate groups of faculty and different CLA constructed-response items, and examining feedback from the faculty members about their level of confidence in each step of the process. The results of these checks and others are important to confirm the standards' consistency, validity, and reliability.

This Method Enables Institutions to More Fully Utilize the CLA

This method will enable colleges and universities to more easily use the CLA to assess students' gains in critical thinking skills and provide feedback that leads to improvements in teaching and learning. It also facilitates the identification of CLA standards that are consistent with their own faculty's expectations of student performance. School administrators and faculty may choose to use the results to inform curriculum planning to further improve their students' growth in critical thinking skills.

Because there are few established methods for developing standards for constructed-response exams, this three-step process may also provide a promising foundation for standard-setting processes for tests comparable to the CLA.