Ensuring That the Measuring and Improving Student-Centered Learning Toolkit Is Easy to Use and Produces Accurate, Meaningful Results

Photo by Jacob Lund/Adobe Stock

This research brief describes the contents of the MISCL Toolkit and how the RAND team developed and tested it for use in high schools.

Many high schools across the United States are striving to provide student-centered learning (SCL) by engaging students with a meaningful curriculum that supports students' individual needs and allows them to make some decisions about their learning. But how can schools know that they are reaching their SCL-related goals or whether all students in the school have access to the same SCL opportunities?

To help high schools and school systems answer such questions, RAND Corporation researchers developed the Measuring and Improving Student-Centered Learning (MISCL) Toolkit. The MISCL Toolkit is designed to help school practitioners and other school-support providers measure and improve the extent of SCL in high school classrooms. The MISCL Toolkit also helps users understand the degree to which SCL opportunities are equitably distributed across their schools. It is not intended to be used for evaluation or accreditation. Instead, it offers a one-stop shop through which school staff can easily collect, examine, and reflect on data from students, teachers, and district and school leaders about SCL.

The Toolkit defines student-centered learning as, "Students' deep engagement in learning opportunities that are designed to address their goals and interests while at the same time providing appropriate supports and challenge according to their learning needs."

The MISCL Toolkit Measures Five SCL Strategies Using Six Data Collection Instruments

The MISCL Toolkit allows users to measure SCL in their high schools according to five SCL strategies:

1. Learning is personalized to align with students' needs, interests, and pace.
2. Learning is challenging, engaging, and meets students where they are in a competency-based system.
3. Learning can happen anytime, anywhere.
4. Learning opportunities promote student agency and ownership.
5. Learning is informed by data.

Following the MISCL User Guide, users can collect data about their high schools' engagement in SCL from six different data collection instruments:

• school leaders: 20-minute survey
• instructional staff (i.e., teachers and others who provide instruction to students): 30-minute survey
• students: 20-minute survey
• student focus group: 60-minute guided discussion
• district leaders: 20-minute survey
• walkthrough observations: a guide for observing SCL classrooms in action

The Toolkit Was Tested According to Three Criteria

Three key criteria guided the development, testing and revision of the MISCL Toolkit, which took place from 2017 to 2019.

• Validity evidence ensures that Toolkit surveys measure the aspects of SCL as intended.
• Reliability evidence ensures that the Toolkit surveys measure SCL precisely and consistently.
• Usability evidence ensures that the Toolkit is usable and useful for school staff.

Multiple Sources of Data Were Drawn Upon to Develop and Test the Toolkit

Research literature Testing criteria: Validity

Among the 156 studies reviewed by researchers, several studies provided some evidence that aspects of SCL measured by the MISCL Toolkit (e.g., personalization based on students' interests, some competency- or mastery-based learning approaches) may support students' academic achievement and engagement.

Expert input Testing criteria: Validity

The research team's seven-member Advisory Board of practitioners and researchers with SCL expertise provided feedback over the course of Toolkit development and testing.

Iterative pilot-testing Testing criteria: Validity, Reliability, and Usability

Two rounds of pilot-testing helped the research team collect evidence of the reliability and validity of the instruments and improve usability of the entire MISCL Toolkit.

• The first pilot test was conducted in two school systems, which differed in location (suburban and rural), student demographics, and other features. District and school leaders, instructional staff, and students took the surveys and participated in interviews about how they interpreted the survey items. The RAND team revised the surveys after the first pilot test.
• The second pilot test was conducted in two additional school systems—one with extensive SCL experience and one where SCL was a newer focus. Staff at each district used the MISCL Toolkit with support from the research team. The research team visited each school to interview leaders, staff, and students and observe how the Toolkit was being used. Through interviews with educators and students and observations of the Toolkit in use, the research team gathered feedback about how users interpreted survey items, the perceived usefulness of the walkthrough and reflection conversation, and suggestions for improvement.

Supplementary surveys Testing criteria: Validity and Reliability

The district leader, school leader, and instructional staff surveys were fielded to additional samples of leaders and teachers to ensure that the team could test validity and reliability using rigorous methods. The instructional staff survey was fielded to the RAND American Teacher Panel, a nationally representative sample of educators. The district and school leader surveys were fielded to a sample of district and school leaders in five New England states.

When Using the MISCL Toolkit, Keep These Limitations in Mind

The MISCL Toolkit is not an assessment or evaluation tool. The MISCL Toolkit was designed to collect formative data through a collaborative process. The data from the Toolkit can be used for professional learning, reflection, and informing school improvement strategies related to SCL. The validity, reliability, and usability evidence collected support these uses.

The Toolkit is intended for use in high schools. The survey questions are designed to capture elements of SCL implementation and SCL instructional approaches that most often occur in high schools. The Toolkit might not be appropriate for use in other grade levels. The student survey and focus group questions are specifically written at a level appropriate for high school students, and the questions might not be accessible to younger students.

Surveys are only one source of information and need to be interpreted carefully. Although the evidence gathered suggests that the MISCL Toolkit surveys provide good and useful information about SCL, the Toolkit survey questions might not always work as intended. During the process of testing the Toolkit, researchers gathered suggestive evidence that those completing the Toolkit surveys may have felt compelled to indicate that they were engaged in more SCL practices than they actually were using or might have rated their practices at lower levels if they were not sure what a survey question meant. To balance the surveys, the Toolkit includes other ways of collecting data about SCL, such as student focus groups and classroom walkthroughs.

More testing is needed. The RAND team was able to gather evidence that the surveys were reliable and valid by administering them to many different groups. Participants included school staff, leaders, and students who participated in pilot tests; teachers from a national sample; and district and school leaders across New England. This work provided preliminary evidence that the Toolkit can differentiate between schools with high SCL and those with low SCL. That said, the MISCL Toolkit as a whole was tested in only four high schools with relatively small numbers of students, teachers, and leaders. More testing in schools with a variety of different characteristics and experiences with SCL is needed to provide conclusive evidence on how well the MISCL Toolkit can measure whole-school SCL.

Topics

• Academic Achievement
• Educational Program Evaluation
• Personalized Learning
• Secondary Education
• Students