Russell, M. (1999). Testing on computers: A follow-up study comparing performance on computer and on paper . Education Policy Analysis Archives , 7 (20). https://doi.org/10.14507/epaa.v7n20.1999

Mode of administration—computer versus paper-and-pencil—was the accommodation used on open-ended SAT-9 test item administrations.

A total of 382 grade 8 students from Massachusetts participated. Due to various limitations, 229 participants' scores were used in the analyses. The numbers of students with and without disabilities were not reported.

Open-ended test items, requiring constructed responses, were drawn from the grade 7 Stanford Achievement Test version 9 (SAT 9) for language arts, mathematics, and science—which worked out to 6 items in each subject. Participants also completed a keyboarding skill test and a computer use survey. Participants' previous total SAT 9 score (including selected response or multiple choice items), were also used. Released items from the Massachusetts Comprehensive Assessment System (MCAS) and the National Assessment of Educational Progress (NAEP) were used for the assessments.

This study suggests that for students who keyboard about 20 words per minute or more, performing open-ended language arts tests on paper substantially underestimates their level of achievement. However, for slower keyboarders, performing open-ended tests on computer adversely affects their performance. To provide more accurate estimates of students' achievement, these findings suggest that students who can keyboard at a moderate level should be allowed to compose their responses to open-ended items on computers. Conversely, students with weak keyboarding speed should compose their responses on paper.This study also demonstrates that for math tests, performance on computer underestimates students' achievement regardless of their level of keyboarding speed. This occurred despite efforts to include items that did not require students to draw pictures or graphs to receive credit. Nonetheless, about 20% of the students who performed the math test on computer indicated that they had difficulty showing their work and/or needed scrap paper to work out their solutions. For these reasons, it is likely that the negative effect found in this study underestimates the effect that would occur if a full range of open- ended math items were included.