The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences

Hedge, Craig ORCID: https://orcid.org/0000-0001-6145-3319, Powell, Georgina ORCID: https://orcid.org/0000-0001-6793-0446 and Sumner, Petroc ORCID: https://orcid.org/0000-0002-0536-0510 2018. The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods 50 (3) , pp. 1166-1186. 10.3758/s13428-017-0935-1

Preview

PDF - Published Version Available under License Creative Commons Attribution. Download (1MB) | Preview

Abstract

Individual differences in cognitive paradigms are increasingly employed to relate cognition to brain structure, chemistry, and function. However, such efforts are often unfruitful, even with the most well established tasks. Here we offer an explanation for failures in the application of robust cognitive paradigms to the study of individual differences. Experimental effects become well established – and thus those tasks become popular – when between-subject variability is low. However, low between-subject variability causes low reliability for individual differences, destroying replicable correlations with other factors and potentially undermining published conclusions drawn from correlational relationships. Though these statistical issues have a long history in psychology, they are widely overlooked in cognitive psychology and neuroscience today. In three studies, we assessed test-retest reliability of seven classic tasks: Eriksen Flanker, Stroop, stop-signal, go/no-go, Posner cueing, Navon, and Spatial-Numerical Association of Response Code (SNARC). Reliabilities ranged from 0 to .82, being surprisingly low for most tasks given their common use. As we predicted, this emerged from low variance between individuals rather than high measurement variance. In other words, the very reason such tasks produce robust and easily replicable experimental effects – low between-participant variability – makes their use as correlational tools problematic. We demonstrate that taking such reliability estimates into account has the potential to qualitatively change theoretical conclusions. The implications of our findings are that well-established approaches in experimental psychology and neuropsychology may not directly translate to the study of individual differences in brain structure, chemistry, and function, and alternative metrics may be required.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Psychology Cardiff University Brain Research Imaging Centre (CUBRIC)
Additional Information:	This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Publisher:	Springer
ISSN:	1554-351X
Funders:	Wellcome Trust, ESRC
Date of First Compliant Deposit:	6 July 2017
Date of Acceptance:	20 June 2017
Last Modified:	05 May 2023 19:09
URI:	https://orca.cardiff.ac.uk/id/eprint/102138

Citation Data

Cited 502 times in Scopus. View in Scopus. Powered By Scopus® Data

Actions (repository staff only)

Edit Item