Russell, M. (1999). Testing on computers: A follow-up study comparing performance on computer and on paper . Education Policy Analysis Archives , 7 (20). https://doi.org/10.14507/epaa.v7n20.1999

Mode of administration—computer versus paper-and-pencil—was the accommodation used on open-ended SAT-9 test item administrations.

A total of 382 grade 8 students from Massachusetts (U.S.) were recruited; due to various limitations, 229 participants' scores were used in the analyses. The numbers of students with and without disabilities were not reported.

Open-ended test items, requiring constructed responses, were drawn from the grade 7 Stanford Achievement Test version 9 (SAT 9) for language arts, mathematics, and science—which worked out to 6 items in each subject. Participants also completed a keyboarding skill test and a computer use survey. Participants' previous total SAT 9 score (including selected response or multiple choice items), were also used. Released items from the Massachusetts Comprehensive Assessment System (MCAS) and the National Assessment of Educational Progress (NAEP) were used for the assessments.

For participating students who keyboarded about 20 words or more per minute, performing open-ended language arts tests on paper substantially underestimated their level of achievement. However, for slower keyboarders, performing open-ended tests on computer adversely affected their performance. To provide more accurate estimates of students' achievement, the researcher suggested that students who can keyboard at a moderate level should be allowed to compose their responses to open-ended items on computers. Conversely, students with weak keyboarding speed should compose their responses on paper. On math tests, performance on computer underestimated students' achievement regardless of their level of keyboarding speed. This occurred despite efforts to include items that did not require students to draw pictures or graphs to receive credit. Nonetheless, about 20% of the students who performed the math test on computer indicated that they had difficulty showing their work and/or needed scrap paper to work out their solutions. For these reasons, the researcher noted that the negative effect found likely underestimated the effect that would have occurred if a full range of open-ended math items were included.