Measuring Hard-to-Measure Student Competencies

A Research and Development Plan

by Brian M. Stecher, Laura S. Hamilton

Download

Download eBook for Free

FormatFile SizeNotes
PDF file 0.6 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Purchase

Purchase Print Copy

 FormatList Price Price
Add to Cart Paperback102 pages $22.00 $17.60 20% Web Discount

Research Questions

  1. What research and development is needed to create high-quality measures of students' interpersonal and intrapersonal competencies?
  2. Which competencies should be addressed first?
  3. Which research and development goals should receive priority for the identified competencies?
  4. How long will the research and development process take, and how much money needs to be committed to support the efforts?
  5. How should the measurement-development process be managed?

Efforts to prepare students for college, careers, and civic engagement have traditionally emphasized academic skills, but a growing body of research suggests that interpersonal and intrapersonal competencies, such as communication and resilience, are important predictors of postsecondary success and citizenship. One of the major challenges in designing educational interventions to support these outcomes is a lack of high-quality measures that could help educators, students, parents, and others understand how students perform and monitor their development over time. This report provides guidelines to promote thoughtful development of practical, high-quality measures of interpersonal and intrapersonal competencies that practitioners and policymakers can use to improve valued outcomes for students.

Key Findings

A Few Guidelines Can Keep Research and Development Directed Appropriately

  • Five broad tasks must be accomplished to develop and implement appropriate measures of interpersonal and intrapersonal competencies: defining and selecting constructs, identifying the intended uses of the measure, developing measures, evaluating the technical quality of measures, and documenting consequences of assessment use.
  • To determine which competencies to address first, start by examining research to understand what measures currently exist across the domains of interest; how good they are from a technical, as well as a practical, perspective; and how difficult it is likely to be to develop new ones.
  • Four kinds of development activities are likely to be needed: (1) conduct basic research to understand the nature of the psychological processes or behavioral manifestations that underlie a construct, (2) develop new measures for a construct that is well understood, (3) assess or improve the quality of an existing measure of a construct, or (4) investigate the consequences of using a measure in the school context.

Recommendations

  • Create a pair of independent research coordinating boards to guide the measurement development process — one for interpersonal competencies and the other for intrapersonal competencies.
  • Each board would create a research and development agenda, receive funding from contributing foundations and agencies, disburse it to developers, monitor the process incrementally, and make midcourse adjustments based on successes. The boards would not be responsible for doing the assessment development or validation work; this would be contracted to others.
  • The boards should have a multiple-year mandate to reflect that fact that the process of development and validation is likely to take multiple years.
  • The boards should be composed of members from key stakeholder groups, including researchers, policymakers, practitioners and funder, and they should represent key academic disciplines, including education, psychology, and psychometrics.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Rationale for Developing New Measures

  • Chapter Three

    Research and Development Guidelines

  • Chapter Four

    Promoting High-Quality Measures: Recommendations and Challenges

  • Chapter Five

    Conclusion

  • Appendix A

    Summary of the White House Workshop on Hard-to-Measure 21st-Century Skills

  • Appendix B

    Experts Who Participated in Meetings and Interviews

The research described in this report was conducted in RAND Education, a unit of the RAND Corporation, under a grant from the William and Flora Hewlett Foundation.

This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.