John F. Pane

@JohnFPane on Twitter

Senior Scientist

Pittsburgh Office

Education

Ph.D. in computer science, Carnegie Mellon University; M.S. in computer science, Carnegie Mellon University; B.S. in electrical and computer engineering and mathematics, Carnegie Mellon University

One Page Bio

Media Resources

This researcher is available for interviews.

To arrange an interview, contact the RAND Office of Media Relations at (310) 451-6913, or email media@rand.org.

More Experts

Overview

John F. Pane is a senior scientist at the RAND Corporation. He studies the implementation and effectiveness of innovations in education, focusing on education technology, personalized/competency-based learning, math, and science. His expertise includes the application of experimental and rigorous quasi-experimental methods. He has led or coled numerous experiments that randomized students, teachers, or schools. Current experiments evaluate the efficacy of a popular digital mathematics curriculum supplement, and the relative efficacy of two ways such software can be used to help students who are behind in mathematics. Prior experiments included a large-scale effectiveness trial of mathematics tutoring software in 147 schools, 51 school districts, and seven states, and an efficacy trial of summer learning programs in five urban school districts. He led the first large-scale evaluation of schoolwide personalized learning. His published methodological collaborations include work on principal stratification, unbiased parameter estimation for experiments with noncompliance or partially-nested designs, and translating standardized effect sizes into more easily interpreted metrics. He codirected the IES postdoctoral training program, Carnegie Mellon and RAND Traineeships in Methodology and Interdisciplinary Research (CMART). From 2015-18 he held RAND’s Distinguished Chair in Education Innovation. He is a frequent participant in the review of scientific research proposals and articles, and sits on advisory boards of several nonprofit education organizations. Major sponsors of his research have included the U.S. Department of Education, the U.S. National Science Foundation, the Bill & Melinda Gates Foundation, and the Wallace Foundation. He holds a Ph.D. in computer science from Carnegie Mellon University.

Research Focus

Previous Positions

Project Scientist, Carnegie Mellon University

Recent Projects

Experimental studies of the implementation and effects of technology-based mathematics curricula
Study of the implementation and outcomes of innovative personalized learning school models
Experimental evaluation of summer learning programs
Study of the implementation of data-driven decisionmaking practices in schools
Evaluation of a school district's one-to-one laptop computer initiative

Selected Publications

Matthew D. Baird and John F. Pane, "Translating Standardized Effects of Education Programs into More Interpretable Metrics," Educational Researcher, 48(4), 2019

Pane, John F., Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped, RAND Corporation (PE-314), 2018

Pane, John F., Elizabeth D. Steiner, Matthew D. Baird, Laura S. Hamilton, and Joseph D. Pane, Informing Progress: Insights on Personalized Learning Implementation and Effects, RAND Corporation (RR-2042), 2017

John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, Rita Karam, "Effectiveness of Cognitive Tutor Algebra I at Scale," Educational Evaluation and Policy Analysis, 36(2), 2014

John F. Pane, Daniel F. McCaffrey, Mary Ellen Slaughter, Jennifer L. Steele, and Gina S. Ikemoto, "An Experiment to Evaluate the Efficacy of Cognitive Tutor Geometry," Journal of Research on Educational Effectiveness, 3(3), 2010

Augustine, Catherine H., Jennifer Sloan McCombs, John F. Pane, Heather L. Schwartz, Jonathan Schweig, Andrew McEachin, and Kyle Siler-Evans, Learning from Summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth, RAND Corporation (RR-1557), 2016

John F. Pane, Daniel F. McCaffrey, Nidhi Kalra, Annie J. Zhou, "Effects of Student Displacement in Louisiana During the First Academic Year After the Hurricanes of 2005," Journal of Education for Students Placed at Risk, 13(2-3), 2008

Marsh, Julie A., John F. Pane, and Laura S. Hamilton, Making Sense of Data-Driven Decision Making in Education: Evidence from Recent RAND Research, RAND Corporation (OP-170), 2006

Honors & Awards

Distinguished Chair in Education Innovation (2015-2018), RAND Education

Recent Media Appearances

Interviews: The 74; Associated Press; The Atlantic; Baton Rouge Advocate; Bloomberg News; Chalkbeat; Christian Science Monitor; District Administration; EdSurge; Education Week; Fast Company; Financial Times; The Hechinger Report; KPCC Pasadena; Louisiana Radio Network; Mindshift by KQED San Francisco; New York Times; NPR Online; Pittsburgh Post-Gazette; Pittsburgh Quarterly; Pittsburgh Tribune-Review; POLITICO; Sirius Satellite Radio; Southern California Public Radio

Commentary: Brookings Brown Center Chalkboard; Education Week; Pittsburgh Post-Gazette; The RAND Blog

Commentary

Educational Program Evaluation

Building an Evidence Base for High School Improvement Will Require Concerted Effort Among Policymakers, School Systems, Researchers, and Funders
The ultimate goal of every education intervention is to help enable students to achieve their fullest potential. Relatively few interventions suited for high schools are supported by evidence of positive effects. Research that is able to provide sound evidence about effectiveness is more important than ever.
May 12, 2021
The RAND Blog
Social and Emotional Learning

Online Doesn't Have to Mean Impersonal
Children's needs extend beyond the purely academic. It is important that their social and emotional well-being is supported as instruction moves online during the COVID-19 pandemic. A whole-child view of what students need could benefit them now more than ever.
Apr 2, 2020
The 74 Million
Educational Program Evaluation

Research Rigor Is Undermined by Translating Into Years of Learning
The research field increasingly recognizes that we need metrics that everyone can understand. Translating the result into years of learning has become a popular approach, but this metric has major flaws. There are better options.
Jul 5, 2019
Education Week
Personalized Learning

What Emerging Research Says About the Promise of Personalized Learning
Personalized learning could lead to improved student outcomes. But those implementing this approach should temper their expectations for how big these benefits will be—and be patient while the benefits emerge. It's also important to consider the challenges of implementation.
Aug 16, 2017
Brookings Institution's Brown Center Chalkboard
Pittsburgh

Helping Pittsburgh Stay on the Cutting Edge of Education Innovation
Local organizations, educators, and leaders are creating remarkable learning experiences for Pittsburgh's young people, and other cities around the country are striving to replicate their innovations.
May 8, 2016
Pittsburgh Post-Gazette
Students

Technology Is Only Part of Personalized Learning
The concept of personalized learning has gained favor in recent years, partly because new technology applications help to facilitate it. But those who characterize this instructional approach as
Mar 4, 2016
The RAND Blog
No Child Left Behind Act of 2001

Making Room for New Approaches: Recommendations for ESEA Reauthorization That Support Innovation
As lawmakers consider the reauthorization of the Elementary and Secondary Education Act, it is critical that in meeting their objectives they do not create unnecessary obstacles to the productive innovations being explored at schools, such as personalized learning.
Jan 27, 2015
The RAND Blog
Students

Helping Displaced Students in the Wake of Hurricane Sandy
If Hurricane Sandy causes extensive disruptions in public schools—particularly in hard-hit New York City—our research shows that choices made by parents and policymakers could significantly limit the negative short-term effects of changing schools under such difficult circumstances, writes John Pane.
Oct 31, 2012
The RAND Blog
When Students Disappear...
Published commentary by RAND staff: When Students Disappear..., in Education Week.
Feb 21, 2007
Education Week

Publications

Report
Approaches and Obstacles to Promoting Media Literacy Education in U.S. Schools
A survey administered to public school teachers indicates that media literacy (ML) instruction is unevenly implemented, at best, and that obstacles to teaching ML (e.g., lack of time, other priorities) are common.
Sep 23, 2021
Report
Media Literacy Education to Counter Truth Decay: An Implementation and Evaluation Framework
The authors outline a framework for implementing and evaluating media literacy (ML) education, a promising approach to slowing Truth Decay. The report provides a framework for teachers and evaluators furthering their work in ML education.
Aug 4, 2021
Report
Media Literacy Standards to Counter Truth Decay
Truth Decay is the diminishing role that facts, data, and analysis play in political and civic discourse. This report, part of a series, synthesizes existing media literacy-related standards into one concise set, specific to curtailing Truth Decay.
Jan 19, 2021
Report
Every Summer Counts: A Longitudinal Analysis of Outcomes from the National Summer Learning Project
This seventh report in a series presents longitudinal findings on the effectiveness of voluntary summer learning programs in five school districts. The authors also offer implications for policy on narrowing the achievement gap between students.
Dec 17, 2020
Report
Building and Sustaining Innovative High Schools: Findings from the Opportunity by Design Study
The Opportunity by Design initiative gave support to small schools of choice to adopt a set of design principles intended to prepare students for college and careers. This final report describes the methods and findings of the five-year study.
Dec 17, 2020
Research Brief
Designing and Sustaining Innovative High Schools: Successes, Challenges, and Student Outcomes
In a five-year study, high schools adopted a set of design principles intended to prepare students for college and careers.
Dec 17, 2020
Journal Article
Switching Cluster Membership in Cluster Randomized Control Trials: Implications for Design and Analysis
With a focus on cluster switching that violates treatment assignment, the goal of this article is to explore the challenges posed for the analysis of clustered RCTs and to propose a potential solution to these challenges.
Jun 16, 2020
Journal Article
Implementing an Adaptive Intelligent Tutoring System as an Instructional Supplement
This article studies the implementation of ALEKS (Assessment and LEarning in Knowledge Spaces) in a blended-learning model. This is the first rigorous experimental evaluation of ALEKS in K–12 in-school settings, and the first to gather extensive implementation data.
Mar 10, 2020
Journal Article
Translating Standardized Effects of Education Programs Into More Interpretable Metrics
Compares four ways to translate standardized effect sizes into metrics that may be easier for educators and policymakers to interpret. Concludes that translating to years of learning is a poor choice; translating to percentile gains is recommended.
May 23, 2019
Report
Implementation and Outcomes of the Appalachian Renaissance Initiative in Eastern Kentucky Schools
The Appalachian Renaissance Initiative provided principals, teachers, and students in rural Eastern Kentucky with opportunities to support and improve teaching and learning. This evaluation examines its implementation and outcomes.
Jan 3, 2019
Report
Strategies for Implementing Personalized Learning While Evidence and Resources Are Underdeveloped
This Perspective draws on theory, learning science principles, and the limited research that exists to offer strategic guidance on the design of personalized learning models for primary and secondary education.
Oct 2, 2018
Report
Privacy and Interoperability Challenges Could Limit the Benefits of Education Technology
This Perspective examines some of the challenges and privacy risks that accompany the potential educational benefits of increased data collection related to technology.
Oct 2, 2018
Report
Evaluation of the Creating College and Career Readiness Initiative in Kentucky
This report presents findings from an evaluation of the Creating College and Career Readiness initiative, which used a suite of personalized software in the kindergarten-12th grade setting in rural Kentucky schools.
Sep 27, 2018
Draft
Controlling for Changes in Test Conditions when Estimating Education Intervention Effects
Examines the assumption that education evaluations that analyze student achievement are administered consistently from pretest to posttest.
May 29, 2018
Research Brief
How Does Personalized Learning Affect Student Achievement?
Describes the concept and implementation of personalized learning and considers achievement findings in a small sample of schools.
Dec 7, 2017
Report
Informing Progress: Insights on Personalized Learning Implementation and Effects
This report describes the concept and implementation of personalized learning, along with some of the challenges and facilitators, and considers achievement findings in a small sample of schools.
Jul 11, 2017
Research Brief
Observations and Guidance on Implementing Personalized Learning
Schools implementing personalized learning were pursuing a wide variety of approaches and students closed achievement gaps relative to national norms. Observed challenges to implementation lead to recommendations for implementers.
Jul 11, 2017
Journal Article
Examining the Implementation of Technology-Based Blended Algebra I Curriculum at Scale
This article examines the implementation of a widely used blended algebra curriculum and the relationship between implementation and student outcomes.
Dec 28, 2016
Research Brief
Kids Who Attend More Benefit More: Voluntary Summer Learning Programs
This brief summarizes the outcomes of two summers of voluntary academic programming (2013 and 2014) on 3,192 low-income urban students accepted into the programs who had completed third grade before the first summer.
Sep 7, 2016
Infographic
Learning from Summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth
This infographic summarizes several student outcomes from an assessment of five voluntary, district-led summer learning programs offered at no cost to low-income, urban elementary students.
Sep 7, 2016
Report
Learning from Summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth
RAND researchers assess voluntary, district-led summer learning programs for low-income, urban elementary students. This third report in a series examines student outcomes after one and two summers of programming.
Sep 7, 2016
Journal Article
Intention-to-treat Analysis in Partially Nested Randomized Controlled Trials with Real-World Complexity
Randomized Control Trials have become the 'gold standard' of education research and an integral part of the 'what works' agenda.
Jun 10, 2016
Report
Continued Progress: Promising Evidence on Personalized Learning
This report explores personalized learning models that were adopted in schools throughout the U.S. It examines school and teacher practices, perceptions of staff and students, and achievement outcomes.
Nov 10, 2015
Report
Continued Progress: Promising Evidence on Personalized Learning: Executive Summary
This Executive Summary recaps the findings in Continued Progress: Promising Evidence on Personalized Learning, a report that explores the personalized learning models adopted in U.S. schools.
Nov 10, 2015
Report
Continued Progress: Promising Evidence on Personalized Learning: Survey Results Addendum
This companion report to Continued Progress: Promising Evidence on Personalized Learning documents results from surveys administered to teachers and students in personalized learning schools in the spring of the 2014-15 school year.
Nov 10, 2015
Research Brief
First Outcomes from the National Summer Learning Study
This brief describes the near-term effects of voluntary summer learning programs provided by five urban school districts to large numbers of struggling low-income elementary students in the summer of 2013.
Feb 13, 2015
Report
Ready for Fall? Near-Term Effects of Voluntary Summer Learning Programs on Low-Income Students' Learning Opportunities and Outcomes
As part of a randomized controlled trial study assessing the effect of district-run voluntary summer programs, this second report in a series looks at how summer programs affected student performance in certain categories in fall 2013.
Dec 16, 2014
Draft
Methods Used to Estimate Achievement Effects in Personalized Learning Schools
Describes the methods RAND used to analyze achievement for 23 personalized learning schools for the 2012-13 through 2013-14 academic years.
Nov 5, 2014
Draft
Personalized Learning Instructional Staff Survey Results (Spring 2014)
Descriptively summarizes instructional staff responses to a survey administered by RAND in 23 personalized learning (PL) schools in Spring 2014.
Nov 5, 2014
Journal Article
Effectiveness of Cognitive Tutor Algebra I at Scale
This article examines the effectiveness of a technology-based algebra curriculum in a wide variety of middle schools and high schools in seven states.
Jun 1, 2014
Draft
Addendum to Effectiveness of Cognitive Tutor Algebra I at Scale
Presents alternative analyses of data from large-scale effectiveness studies of Cognitive Tutor Algebra I in middle schools and high schools.
May 30, 2014
Research Brief
Does an Algebra Course with Tutoring Software Improve Student Learning?
Examines whether the Cognitive Tutor Algebra I (CTAI) algebra curriculum, which includes tutoring software, is effective in improving the math test scores of middle and high school students, relative to traditional algebra instruction.
Nov 18, 2013
Report
Analysis of Costs in an Algebra I Curriculum Effectiveness Study
Examines the costs of implementing Carnegie Learning's Cognitive Tutor Algebra I curriculum and comparison algebra I curricula (published by Prentice Hall, Glencoe, and McDougal Littell).
Jul 12, 2012
Research Brief
No Evidence That Incentive Pay for Teacher Teams Improves Student Outcomes: Results from a Randomized Trial
Analysis of an incentive program that paid teachers bonuses based on their students' growth in achievement in mathematics, English language arts, science, and social studies showed no effect on student test scores in any of the subject areas.
May 30, 2012
Journal Article
How Curriculum and Classroom Achievement Predict Teacher Time on Lecture- and Inquiry-Based Mathematics Activities
This study drew on data from a large, randomized trial of Cognitive Tutor Algebra (CTA) in high-poverty settings to investigate how mathematics curricula and classroom achievement related to teacher reports of time spent on inquiry-based and lecture-based mathematics activities.
2012
Journal Article
Team Pay for Performance: Experimental Evidence from the Round Rock Pilot Project on Team Incentives
This article examines a program in which bonuses were awarded to teams of middle school teachers based on their collective contribution to student test score gains.
2012
Journal Article
An Experiment to Evaluate the Efficacy of Cognitive Tutor Geometry
This randomized, controlled field trial estimated the causal impact of a technology-based geometry curriculum on students' geometry achievement, as well as their attitudes toward mathematics and technology.
2010
Research Brief
Measuring the Progress of the Math Science Partnership of Southwest Pennsylvania
This research brief summarizes RAND's evaluation of the Math Science Partnership of Southwest Pennsylvania's progress toward increasing student achievement, increasing the quality of the educator workforce, and creating sustainable partnerships.
Apr 8, 2009
Report
Math Science Partnership of Southwest Pennsylvania: Measuring Progress Toward Goals
This monograph examines the Math Science Partnership of Southwest Pennsylvania's progress toward its goals of increasing student math and science achievement, increasing the quality of the educator workforce, and creating sustainable partnerships.
Mar 20, 2009
Journal Article
Mobile Technology and Action Teams: Assessing Blackberry Use in Law Enforcement Units
This research explores the effectiveness of mobile wireless information and communication technologies for law enforcement teams.
2009
Report
Effects of Student Displacement in Louisiana During the First Academic Year After the Hurricanes of 2005
In 2005, Hurricanes Katrina and Rita displaced more than one-fourth of Louisiana public school students. This article examines how these students fared in the first academic year following the storms.
Nov 14, 2008
Draft
Supplemental Material for: Effects of Student Displacement in Louisiana During the First Academic Year After the Hurricanes of 2005
The purpose of this document is to publish a complete tabulation of statistical model results reported in an article in the Journal of Education for Students Placed at Risk.
Oct 6, 2008
Draft
Math Science Partnership of Southwest Pennsylvania: Year Four Evaluation Report
Evaluates the success of the Math Science Partnership of Southwest Pennsylvania, a program to unite the efforts of K-12 school districts with science, mathematics, engineering, and education faculties of colleges and universities.
Jun 11, 2007
Draft
Mobile Technology and Action Teams: Assessing BlackBerry Use in Law Enforcement Units
Explores the effectiveness of mobile wireless information and communication technologies for law enforcement teams.
Mar 5, 2007
Research Brief
How the 2005 Hurricanes Affected Students in Louisiana
This research brief summarizes a study of the movements of Louisiana public school students in the aftermath of Hurricanes Katrina and Rita and of the short-term effects of these displaced students on the state's public education system.
Nov 29, 2006
Report
Student Displacement in Louisiana After the Hurricanes of 2005: Experiences of Public Schools and Their Students
Describes the movements of Louisiana public school students in the aftermath of Hurricanes Katrina and Rita and documents many of the short-term effects of these displaced students on the state's public education system.
Nov 27, 2006
Draft
Professional Development Participation and the Role of Administrator Involvement in the Math Science Partnership of Southwest Pennsylvania
Examines how levels of professional development participation in mathematics and science among district and school staff in Pennsylvania were associated with the participation levels of their school-based staff.
Nov 8, 2006
Report
Making Sense of Data-Driven Decision Making in Education: Evidence from Recent RAND Research
Using RAND research this paper examines how schools are utilizing test results and other data to make decisions for improving student success, describes the conditions and policies that support or hinder these efforts, and suggests future research.
Nov 7, 2006
Draft
Math Science Partnership of Southwest Pennsylvania: Year Three Evaluation Report
Investigates the effectiveness of the Math Science Partnership of Southwest Pennsylvania; its impact on institutional practices and policies; and its effect on instruction, student course taking, and student outcomes.
Aug 22, 2006
Draft
Data Driven Decisionmaking in Southwestern Pennsylvania School Districts
This report investigates the preparedness of Southwestern Pennsylvania school districts to meet the challenges of using data to improve instruction, evaluate the effectiveness of schools, teachers and programs, and create reports.
Jan 10, 2006
Draft
Math Science Partnership of Southwest Pennsylvania: Year Two Evaluation Report
Math Science Partnership of Southwest Pennsylvania: Year Two Evaluation Report
Jul 29, 2005
Report
Making Better Use of Bandwidth: Data Compression and Network Management Technologies
Explores how the Army might better use bandwidth, specifically how compression technologies and network management techniques might help to improve information throughput.
2005
Report
Quaker Valley Digital School District: Early Effects and Plans for Future Evaluation
Preliminary evaluation of a district-wide laptop computer initiative and recommended research design for more comprehensive future evaluation.
2003