March 19, 2015
A study co-authored by IMPAQ International and REL Midwest at American Institutes for Research demonstrates how the choice of statistical methods can change which schools are identified as “beating the odds,” or having higher academic achievement than schools with similar student demographics and income levels.
The study, released as a REL Midwest report by the Institute of Education Sciences in February 2015, found that different statistical analysis methodologies may lead to wide variations in which schools are considered as “beating the odds.”
The study’s lead author Dr. Yasuyo Abe, Sr. Researcher at IMPAQ International, noted that “less than 50 percent of the schools appearing on a list generated by one statistical method also appeared on a list generated by another method. It’s important for policy makers to understand that performance can be defined in countless ways.”
The study also found that the selection of school performance measures made a large difference in which schools were selected. Selection of school demographic characteristics made some difference, but whether schools were compared by the grade levels they served or across all grade levels made little difference
In its email alert service, the U.S. Department of Education, Institute of Education Sciences, which funded the study, stated that, “The findings of this study highlight the importance of careful consideration of the conceptual basis for the selection criteria for beating-the-odds schools, as well as technical decisions in operationalizing these selection criteria.”
The report is titled “How Methodology Decisions Affect the Variability of Schools Identified as Beating the Odds” and can be read in full at http://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=353.
You can also read a blog post on the study here: http://www.impaqint.com/media-center/blog/which-schools-beat-the-odds
About IMPAQ International
IMPAQ International is a public policy research firm that drives value by utilizing innovative thinking and rigorous approaches to improve our clients’ policies and programs, delivering meaningful, real world impact. IMPAQ’s mission is to provide exemplary research and consulting services through the use of impact evaluation studies, applied research, policy analysis, quantitative and qualitative data analysis, and technical assistance. For more information, visit http://www.impaqint.com.
Dan Kaplan, Director, Corporate Communications