RAND Study Finds Mexican Teacher Incentive Program Had Negligible Effect on Student Achievement

Mexico's pioneering pay-for-performance teacher incentive program has had no discernible effect on student achievement and should be improved, according to a RAND Corporation study issued today.

RAND researchers found that the program's incentives to raise student test scores, which were used to measure educational quality, had a negligible impact.

“While the Ministry of Education certainly deserves credit for trying out a new program, our study suggests that the bonuses did not have any real effect on student test scores for primary school teachers,” said lead study author Lucrecia Santibañez. “The financial bonuses had only a very modest positive impact on student test scores for secondary school teachers who are trying to get into the incentive program.”

The teacher incentive program, called Carrera Magisterial, was created by the Mexican government and the nation's teacher union in 1992 as a groundbreaking promotional system designed to recognize the teaching profession and provide economic incentives for superior performance.

The pay-for-performance program offers salary incentives to teachers who partake in professional development, undergo peer review, and consent to be evaluated through standardized tests of teacher knowledge and student knowledge

However, although the program costs millions of dollars in salary bonuses each year, it had never been formally evaluated. Mexico's Ministry of Education asked RAND, a nonprofit research organization, in 2004 to examine aspects of the incentive program to determine the effectiveness of the program's assessment tools and its impact on some measures of educational quality.

Santibañez and her co-authors pointed out that other pay-for-performance initiatives in other countries have experienced similar problems. Student test scores rose modestly when a teacher became inducted into the bonus pay program, but the improvements were often short-lived and not always the result of increased efforts by educators.

The study's authors found that the motivation for improving student test scores decreased once teachers were incorporated into the Mexican program or received a promotion to another pay level, since educators are guaranteed the salary bonuses for their entire career.

“We found moderate negative effects on student test scores the year after a teacher was incorporated into the pay-for-performance program or got a promotion,” Santibañez said. “Because of the salary guarantees, teachers might have less of incentive to exert additional effort with their students.”

In order to prevent teachers from returning to a “normal” effort level after securing a bonus or promotion, the researchers suggested that Mexico's policymakers use more than one year of an educator's achievement data to measure his or her performance, or include penalties when performance falls below an acceptable level. This would provide continuous motivation for improvement.

An analysis of the tests given to students and teachers under the pay-for-performance program found that while some of procedures were adequate, other features of the testing process will require more attention to put it on par with internationally accepted standards.

RAND researchers concluded that while the tests given to teachers as part of their performance evaluation showed broad coverage of the national education curriculum, the tests given to teachers focused a good deal on rote memorization rather than higher cognitive skills. Similarly, the student tests – key factors in whether or not a teacher received a bonus – also required rather low skill levels.

“We found that both the tests given to the teachers and those given to the students had some serious technical flaws,” Santibañez said. “We think that the teacher tests were more carefully developed and well monitored, but the tests taken by students were much less carefully created.”

Another problem researchers found was that the tests the teachers took to qualify for a bonus did not increase in difficulty as they rose to higher levels of promotion. In order to ensure continued professional and classroom development, the study suggests that Mexico's policymakers consider developing a teacher test that measures both subject knowledge as well as individual and teaching competencies.

Santibañez and her co-authors — Jose-Felipe Martinez, Ashlesha Datar, Patrick J. McEwan, Claude Messan-Setodji and Ricardo Basurto-Dávila — also suggested that the Ministry of Education revamp its professional development and peer review components. While the bonus pay program places a strong emphasis on professional development, the study found that measure to be only slightly related to improvements in the teacher and student tests or peer review ratings.

“Essentially, the Ministry of Education and the teacher's union need to review the main features of Carrera Magisterial's evaulation system, from the factors it includes to how the results are assessed,” Santibañez concluded. “It needs to be stressed that even the best designed incentive systems might not show improvements in educational quality if the instruments used to measure teachers are flawed.”

The study — “Breaking Ground: The National Teacher Incentive Program ‘Carrera Magisterial,' Analysis of its Assessments and Impact on Educational Quality in Mexico” — is available in English and Spanish at www.rand.org