Svetina, D., Dai, S., & Wang, X. (2017). Use of cognitive diagnostic model to study differential item functioning in accommodations . Behaviormetrika , 44 (2), 313–349. https://doi.org/10.1007/s41237-017-0021-0

This study analyzed extant data for students using a single accommodation—extended time, oral delivery, or small group administration—for each comparison group, in order to examine the application of a model for investigating the impact of accommodations on individual assessment items. These accommodations—the most commonly used by students in the data set—were compared with one another and with an unaccommodated assessment data set for their impacts.

A national (U.S.) extant data set from 2007 of grade 8 students with various disabilities was analyzed using different types of item performance effect approaches. Information about the students' disabilities, and other demographic or other characteristics, was not reported as it was not specifically the focus of the investigation. Comparison groups were constructed with data for students each using only one of the most common accommodations: approximately 300 students using extended time, 100 using oral delivery, and 100 using small group administration. Performance data from an additional random sample of approximately 3400 students not using accommodations were also analyzed.

The item-level responses for 53 released items—actual performance data—for the 2007 National Assessment of Educational Progress (NAEP) in grade 8 mathematics comprised the data source. The academic content included several areas for this grade level (eighth), including algebra; geometry; measurement; numbers and operations; and data analysis, statistics, and probability. The response types were multiple choice, short constructed response, and extended constructed response; many items were already dichotomous, and the polytomous items were dichotomized (correct-incorrect) for the purpose of applying the analysis approaches. The item difficulties and the skills needed for answering each item were examined and described. The research questions involved "examining the effect of accommodations on NAEP Mathematics" (p. 322) and "examination of item demands as potential sources of DIF" (p. 322); the latter question sought to uncover whether items require similar skills for different student groups. For the first research question, the researchers applied generalized logistic regression (GLR) to test both uniform and nonuniform differential item functioning (DIF). For the second research question, the researchers employed the reparameterized unified model (RUM) using a Q-matrix.

Twenty-five of the 53 NAEP math items were identified with DIF, and the researchers specified that 14 had nonuniform DIF, 9 had uniform DIF, and 2 had both. Examining these 25 items further, the researchers determined that the item demands required mastery of 24 different skills for correct math test answers, such as number sense and estimation. On average, the nonaccommodated group of students had higher math performance than the accommodated groups, indicating that they had "higher mastery probabilities as well as higher proportions of students with mastery across all [item] attributes" (p. 333), with a 33% difference in mastery level performance. The groups using single accommodations (extended time, oral delivery, or small group administration) had similar performance success as one another across item attributes, with students using small group administration having a larger proportion of students reaching performance mastery. The nonaccommodated group had much higher proportions of students demonstrating skill mastery, with more than half of its members mastering 20 skills. In comparison, one skill—unit conversion—was mastered by more than half of each of the accommodated groups. A couple skills were mastered by more than half of the group receiving small group administration: evaluating/verifying options and word problems; for both skills, the other accommodated groups had 50% or fewer of its members showing mastery. The group using oral delivery had 51% of its members with mastery of quantitative reading, yet the remaining 22 skills were mastered by fewer than half of its members. The group using extended time had half or fewer of its members showing mastery on 23 skills. In contrast, both nonaccommodated and accommodated groups scored similarly on the item skills of quantitative reading and logical reasoning. The researchers concluded that this approach to investigate item analysis is a viable alternative. Limitations of the study were reported, and future research directions were suggested.