RAND Europe found in a UK randomised controlled trial that classes of 8 and 9 year-old primary school children randomly allocated to receive a technology-based education intervention using Microsoft's Immersive Reader performed as well as business-as-usual classes. Schools did not implement the intervention as expected, and many left the trial, citing difficulties in implementation.
Download Free Electronic Document
| Format | File Size | Notes |
|---|---|---|
| PDF file | 0.6 MB | Use Adobe Acrobat Reader version 10 or higher for the best experience. |
Research Questions
- Will the intervention have a positive effect on the outcomes in reading comprehension of pupils who are randomly assigned to the intervention condition compared to pupils in the control condition?
- Was the intervention implemented with fidelity in the intervention classrooms?
- What factors and initial conditions appear to explain variation in fidelity of implementation?
- What appear to be the necessary conditions for the success of the intervention?
- What were the barriers to delivery?
RAND Europe evaluated the implementation of a technology-based intervention that made use of the Microsoft Immersive Reader (IR) in UK classrooms for eight- and nine-year-old pupils. This is the first evaluation of this intervention that uses a randomised control trial (RCT) design, following promising findings related to the use of IR from a small-scale study in the United States. Microsoft granted funding to Achievement for All (AfA) to deliver the intervention using IR, with the aim of assessing its impact on pupils' reading outcomes. AfA aimed to implement this intervention in approximately 80 classrooms across 20 different primary schools in England. RAND Europe was subcontracted as the independent evaluator by AfA to design and undertake a two-arm RCT supplemented by an implementation process evaluation. The study randomly allocated Year 3 and Year 4 classes in 20 schools to either an intervention condition (receiving the intervention including IR) or a control condition (business-as-usual). Using the standardised Progress in Reading Assessment (PiRA) test, this study compared reading outcomes for pupils in classes randomly allocated to receive the intervention to reading outcomes for pupils in the business-as-usual classes.
The RCT found no significant difference between class-level outcomes of pupils who were allocated to the intervention and those allocated to the control condition. This means that on average, pupils in the intervention classes performed as well as pupils in control classes. With low implementation fidelity, alongside high trial attrition and the infrequent use of the intervention in some schools, it was not possible to determine if the intervention would have influenced the students' reading attainment had it been implemented as intended.
Key Findings
- The main outcome analysis of this two-arm RCT does not show a statistically significant effect of the intervention using IR on reading outcomes. This suggests that classes that received the intervention performed as well as classes in the 'business-as-usual' condition.
- The implementation process evaluation found that the intervention was not implemented as intended in participating schools, nor was there uniformity in participating schools' reported practices.
- Teachers experienced difficulties while using IR due to a lack of resources and IT infrastructure (such as operating systems, computers, tablets, etc.) at their schools and overall considered the intervention difficult to implement in the classroom.
Recommendations
- Implementation clarity should be established early on, so that schools and teachers are aware of all the implications of participating in and implementing the intervention.
- To ensure the robustness of results from studies assessing interventions using IR, implementation should be consistent across participating schools, classes and teachers. Future studies should include a measure of implementation fidelity.
- Future interventions using IR should ensure that support is given to schools and teachers to help them deal with technical difficulties that may impede the implementation of the intervention.
Research conducted by
The research described in this report was prepared for Achievement for All and conducted by RAND Europe.
This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.