Effects of Visual Representations and Associated Interactive Features on Student Performance on National Assessment of Educational Progress (NAEP) Pilot Science Scenario-Based Tasks

The National Assessment of Educational Progress’s (NAEP’s) transition to an entirely digitally based assessment (DBA) began in 2017. As part of this transition, new types of NAEP items have begun to be developed that leverage the DBA environment to measure a wider range of knowledge and skills. These new item types include the science scenario-based tasks (SBTs) that are the focus of this report.

The study involved a cognitive lab investigation of 31 eighth-grade students performing five science SBTs that had been used in the 2015 NAEP pilot assessment. The main research question was:

• Which key visual and associated interactive features of NAEP science SBTs used in the 2015 pilot assessment might inhibit or enable the ability of students to accurately demonstrate their actual level of mastery of target knowledge and skills?

The analysis of students’ performance, retrospective think-alouds, and video-recorded actions supported conclusions that, with relatively few exceptions, students in our small sample generally understood the visual and interactive features of SBTs as intended, suggesting that NAEP science SBT development procedures are working well. That said, the study also suggests that better formulated principles, or guidelines, for visual and interactive features should be developed and that quality control focused on those principles should be integrated into the critical path. The exact means for defining such principles remain to be developed, but the authors suggest that close-in investigations of students’ interactions with these types of features could be a component, particularly if these investigations were informed by previous research on multimedia principles as applied to assessments.

The investigation was based on pilot versions of the SBTs. Based on the results from the 2015 pilot, as well as interim feedback from our study, many of the SBTs were significantly revised, and many of the issues identified in the study identified were corrected or became irrelevant.