Carpenter, R., & Alloway, T. (2019). Computer versus paper-based testing: Are they equivalent when it comes to working memory ? Journal of Psychoeducational Assessment , 37 (3), 382–394. https://doi.org/10.1177/0734282918761496

First published online (3/14/18)

The effects of a digital (computer-based) and traditional (paper-and-pencil) administration of a working memory assessment on student performance were investigated.

A nationally representative demographic sample of students (n=1339) from schools throughout England participated. Students ranged in age between 4 and 11 years old, from three age bands or ranges: 4–6, 7–8, and 9–11. Demographic information including sex (male/female) was also reported. No information on participants' disability statuses were gathered, and no comparisons were made based on participants either having or not having disabilities. Students who had average performance on national assessments in English, mathematics, and science at the end of primary education were selected for participation.

The Automated Working Memory Assessment (AWMA; Alloway, 2007) or the computer-based AWMA, were administered to students. The AWMA contained scales of verbal short-term memory, verbal working memory, and visuospatial memory. Verbal short-term memory items examined digit recall, word recall, and nonword recall. Verbal working memory items addressed listening recall, backward digit recall, and counting recall. Visuospatial short-term memory items incorporated memory for mazes and for sequence of blocks. Across the assessment, items varied in task demands requiring a range of cognitive load or effort. In both administration versions, test-takers were not permitted to review nor change their answers to each item.

Participants performed higher on nonword recall—a verbal short-term memory skill—on the paper version of the AWMA than participants performed on the computer-administered AWMA in all age bands, with the highest average score for participants in the oldest age band. Participants testing on the paper version also performed higher on the listening and backward digit recall skills—that is, verbal working memory—than participants scored on the computer-administered AWMA in all age bands, with the highest average score for participants in the oldest age band. Participants scored higher on average on the paper version in visuospatial short-term memory skills as a whole than did participants on the computer-administered assessment, with the same trend by age band. Similar analyses with sex as a variable indicated higher scores for participants on the paper version of the three AWMA scales than the participants scored on the computer-administered assessment, with only some sex-based differences: female participants scored better on average than males on verbal short-term memory skills and the reverse was true on the visuospatial memory, mazes subtest. The researchers suggested possible explanations for the consistent higher performance on the paper version of the assessment on working memory, including: (a) increased cognitive load and decreased capacity of working memory for the computer-administered assessment; (b) socioeconomic status-related differences on high tech versions; and (c) biological sex and developmental differences. Due to students not being permitted to return and review answers to items, because of the test measuring working memory, the researchers also indicated that underperforming students might have been disadvantaged because of missing the opportunity to take additional time to check response accuracy. The researchers also noted that issues such as screen fatigue and easier movement through paper test forms might have influenced these performance differences.