- Will the intervention have a positive effect on the outcomes in reading comprehension of pupils who are randomly assigned to the intervention condition compared to pupils in the control condition?
- Was the intervention implemented with fidelity in the intervention classrooms?
- What factors and initial conditions appear to explain variation in fidelity of implementation?
- What appear to be the necessary conditions for the success of the intervention?
- What were the barriers to delivery?
RAND Europe evaluated the implementation of a technology-based intervention that made use of the Microsoft Immersive Reader (IR) in UK classrooms for eight- and nine-year-old pupils. This is the first evaluation of this intervention that uses a randomised control trial (RCT) design, following promising findings related to the use of IR from a small-scale study in the United States. Microsoft granted funding to Achievement for All (AfA) to deliver the intervention using IR, with the aim of assessing its impact on pupils' reading outcomes. AfA aimed to implement this intervention in approximately 80 classrooms across 20 different primary schools in England. RAND Europe was subcontracted as the independent evaluator by AfA to design and undertake a two-arm RCT supplemented by an implementation process evaluation. The study randomly allocated Year 3 and Year 4 classes in 20 schools to either an intervention condition (receiving the intervention including IR) or a control condition (business-as-usual). Using the standardised Progress in Reading Assessment (PiRA) test, this study compared reading outcomes for pupils in classes randomly allocated to receive the intervention to reading outcomes for pupils in the business-as-usual classes.
The RCT found no significant difference between class-level outcomes of pupils who were allocated to the intervention and those allocated to the control condition. This means that on average, pupils in the intervention classes performed as well as pupils in control classes. With low implementation fidelity, alongside high trial attrition and the infrequent use of the intervention in some schools, it was not possible to determine if the intervention would have influenced the students' reading attainment had it been implemented as intended.
- The main outcome analysis of this two-arm RCT does not show a statistically significant effect of the intervention using IR on reading outcomes. This suggests that classes that received the intervention performed as well as classes in the 'business-as-usual' condition.
- The implementation process evaluation found that the intervention was not implemented as intended in participating schools, nor was there uniformity in participating schools' reported practices.
- Teachers experienced difficulties while using IR due to a lack of resources and IT infrastructure (such as operating systems, computers, tablets, etc.) at their schools and overall considered the intervention difficult to implement in the classroom.
- Implementation clarity should be established early on, so that schools and teachers are aware of all the implications of participating in and implementing the intervention.
- To ensure the robustness of results from studies assessing interventions using IR, implementation should be consistent across participating schools, classes and teachers. Future studies should include a measure of implementation fidelity.
- Future interventions using IR should ensure that support is given to schools and teachers to help them deal with technical difficulties that may impede the implementation of the intervention.