INNOVATIVE THINKING
RIGOROUS APPROACHES
REAL WORLD IMPACT

 

Publication

Evaluation of Research Methods for Handling Clustered Data

Title
Sensitivity Analyses for Clustered Data: An Illustration from a Large-scale Clustered Randomized Controlled Trial in Education
Date
December 2014
Author(s)
Abe, Y. & Gee, K. A.
Publication
Evaluation and Program Planning
Market
Education
Citation
Abe, Y. & Gee, K. A. (2014). Sensitivity analyses for clustered data: An illustration from a large-scale clustered randomized controlled trial in education. Evaluation and Program Planning, 47, 26-34. https://doi.org/10.1016/j.evalprogplan.2014.07.001

As researchers begin to use cluster randomized controlled trials (RCTs) more frequently in their studies, the recurring question of which method of analysis to use becomes more important. This is especially true for clustered data, which groups individuals into higher level units. Failure to select an evaluation method that accounts for the correlation between these grouped individuals in the higher-level units can result in bias. To emphasize the importance of using well-thought-out sensitivity analysis when using clustered data for RCTs, Yasuyo Abe co-authored a paper exploring this topic.

This study used data from a cluster RCT of the Pacific Communities with High Performance in Literacy Development (Pacific CHILD), a teacher professional development program. This evaluation had a sample of 45 elementary schools which were randomly assigned to treatment or control groups and sought to measure student reading achievement. Using the Pacific CHILD evaluation, this study used the four most common analytic methods applied to clustered data: 1) hierarchical linear modeling (HLM); 2) feasible generalized least squares (FGLS); 3) generalized estimating equations (GEE); and 4) ordinary least squares regression (OLS) with cluster-robust (Huber-White) standard errors.

Researchers compared the findings of each method, revealing inconsistent results in sizes and significance of observed effects. These results emphasize that there is no "correct" method of estimation, and that researchers must rely on their judgement and sensitivity analyses when making analytic decisions. The results of this study are especially important for fields that have seen increased use of cluster RCTs in impact evaluation, such as education, public health, and social welfare.