Download Free Electronic Document

FormatFile SizeNotes
PDF file 0.6 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. Will the intervention have a positive effect on the outcomes in reading comprehension of pupils who are randomly assigned to the intervention condition compared to pupils in the control condition?
  2. Was the intervention implemented with fidelity in the intervention classrooms?
  3. What factors and initial conditions appear to explain variation in fidelity of implementation?
  4. What appear to be the necessary conditions for the success of the intervention?
  5. What were the barriers to delivery?

RAND Europe evaluated the implementation of a technology-based intervention that made use of the Microsoft Immersive Reader (IR) in UK classrooms for eight- and nine-year-old pupils. This is the first evaluation of this intervention that uses a randomised control trial (RCT) design, following promising findings related to the use of IR from a small-scale study in the United States. Microsoft granted funding to Achievement for All (AfA) to deliver the intervention using IR, with the aim of assessing its impact on pupils' reading outcomes. AfA aimed to implement this intervention in approximately 80 classrooms across 20 different primary schools in England. RAND Europe was subcontracted as the independent evaluator by AfA to design and undertake a two-arm RCT supplemented by an implementation process evaluation. The study randomly allocated Year 3 and Year 4 classes in 20 schools to either an intervention condition (receiving the intervention including IR) or a control condition (business-as-usual). Using the standardised Progress in Reading Assessment (PiRA) test, this study compared reading outcomes for pupils in classes randomly allocated to receive the intervention to reading outcomes for pupils in the business-as-usual classes.

The RCT found no significant difference between class-level outcomes of pupils who were allocated to the intervention and those allocated to the control condition. This means that on average, pupils in the intervention classes performed as well as pupils in control classes. With low implementation fidelity, alongside high trial attrition and the infrequent use of the intervention in some schools, it was not possible to determine if the intervention would have influenced the students' reading attainment had it been implemented as intended.

Key Findings

  • The main outcome analysis of this two-arm RCT does not show a statistically significant effect of the intervention using IR on reading outcomes. This suggests that classes that received the intervention performed as well as classes in the 'business-as-usual' condition.
  • The implementation process evaluation found that the intervention was not implemented as intended in participating schools, nor was there uniformity in participating schools' reported practices.
  • Teachers experienced difficulties while using IR due to a lack of resources and IT infrastructure (such as operating systems, computers, tablets, etc.) at their schools and overall considered the intervention difficult to implement in the classroom.

Recommendations

  • Implementation clarity should be established early on, so that schools and teachers are aware of all the implications of participating in and implementing the intervention.
  • To ensure the robustness of results from studies assessing interventions using IR, implementation should be consistent across participating schools, classes and teachers. Future studies should include a measure of implementation fidelity.
  • Future interventions using IR should ensure that support is given to schools and teachers to help them deal with technical difficulties that may impede the implementation of the intervention.

Research conducted by

The research described in this report was prepared for Achievement for All and conducted by RAND Europe.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.