Intention-to-treat Analysis in Partially Nested Randomized Controlled Trials with Real-World Complexity

Demands for scientific knowledge of what works in educational policy and practice has driven interest in quantitative investigations of educational outcomes, and randomized controlled trials (RCTs) have proliferated under these conditions. In educational settings, even when individuals are randomized, both experimental and control students are often grouped into particular classrooms and schools and share common learning experiences. Analyses that account for these clusters are common. A less common design involves one clustered experimental arm and one unclustered experimental arm, sometimes called a partially clustered design. Analysts do not always use methods that yield valid statistical inferences for such partially clustered designs. Additionally, published methods for handling partially clustered designs may not be flexible enough to handle real-world complications, including treatment non-compliance. In this paper, we illustrate how models that accommodate partial clustering may be used in educational research. We explore the performance of these models using a series of Monte Carlo simulations informed by data taken from a large-scale RCT studying the impacts of a programme designed to decrease summer learning loss. We find that clustering and non-compliance can have substantial impacts on statistical inferences about intent-to-treat effects, and demonstrate methods that show promise for addressing these complications.