INNOVATIVE THINKING
RIGOROUS APPROACHES
REAL WORLD IMPACT

 

Which Schools “Beat The Odds?” Well, It Depends.

Thursday, March 19, 2015

A study co-authored by IMPAQ International and REL Midwest at American Institutes for Research demonstrates how the choice of statistical methods can change which schools are identified as “beating the odds,” or having higher academic achievement than schools with similar student demographics and income levels.

Some public schools, including some high-poverty schools, outperform others with similar student demographic and socio-economic characteristics. Such schools hold promise because they suggest that academic success can be achieved in challenging school environments.

Policymakers, researchers, and teachers want to learn from these “beating the odds” schools, yet schools identified as high-performing could differ significantly, depending on the choice of statistical methods used.

Methodology Decisions Can Lead To Different Results When Identifying Beating-The-Odds Schools

Using IMPAQ International’s rigorous approach to data collection and analysis, Sr. Researcher Dr. Yasuyo Abe examined a set of student performance data from Michigan’s public schools from August 2007 through November 2010.

The report considered the Michigan Department of Education’s approach to identifying beating-the-odds schools by using two statistical methods.

  • The first method, the prediction method, identifies a school as beating the odds if it outperforms its predicted level of performance.
  • The second method, the comparison method, identifies a school as beating the odds if it outperforms other demographically similar schools.

The results might surprise you.

  • In school years 2009/10 and 2010/11, fewer than a third of schools identified by either method were identified by both methods.
  • Within each method, the list of schools changed from one year to the next, with less than half the schools consistently identified as beating the odds for two consecutive years.
  • Of 184 schools identified by one of the two methods, only 4 were identified by both methods in both 2009/10 and 2010/11.

The IMPAQ Impact

This is important because as Dr. Abe notes, “policy makers should understand that there are many ways to define school performance.”

Our work has helped the U.S. Department of Education to conclude that “The findings of this study highlight the importance of careful consideration of the conceptual basis for the selection criteria for beating-the-odds schools, as well as technical decisions in operationalizing these selection criteria.”

Although some variation across the two methods was expected in the schools identified as beating the odds, the variation was larger than expected. This raised concerns among DOE Research Alliance members about the consistency of results and whether these methods can adequately identify schools that exceed performance expectations over time.

Schools identified based on a single-year spike in achievement or common year-to-year fluctuations in achievement might be less likely to yield useful lessons on practices associated with school improvement. 

Research alliance members have expressed interest in reviewing the state’s technical approach to identifying beating-the-odds schools. In response, REL Midwest examined Michigan’s approach, documenting how the identification results can vary because of decisions about statistical methods, technical specifications (for example, performance measures, school sample configuration, and school characteristics), and time periods examined.

Although this study focused on Michigan, the challenges of identifying beating-the-odds schools are not unique to one state. The study may also provide information that can assist other states and districts in developing or revising their technical approaches to identifying schools that exceed performance expectations and therefore the implications of the choices they make.

Further Reading

The full report is titled How Methodology Decisions Affect the Variability of Schools Identified as Beating the Odds, and is available on the U.S. Department of Education, Institute of Education Sciences website: http://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=353