Validity and transparency in quantifying open-ended data

Conry-Murray, Clare, Waltzer, Tal, DeBernardi, Fiona C., Fossum, Jessica L., Haasova, Simona, Matthews, Michael S., O’Mahony, Aoife, Moreau, David, Baum, Myriam A., Karhulahti, Veli-Matti, McCarthy, Randy J., Paterson, Helena M., McSweeney, Kara and Elsherif, Mahmoud M. 2024. Validity and transparency in quantifying open-ended data. Advances in Methods and Practices in Psychological Science 7 (4) , pp. 1-20. 10.1177/25152459241275217

PDF - Published Version Available under License Creative Commons Attribution Non-commercial. Download (762kB)

Abstract

Quantitatively coding open-ended data (e.g., from videos, interviews) can be a rich source of information in psychological research, but reporting practices vary substantially. We provide strategies for improving validity and reliability of coding open-ended data and investigate questionable research practices in this area. First, we systematically examined articles in four top psychology journals (N = 956) and found that 21% included open-ended data coded by humans. However, only about one-third of those articles reported sufficient details to replicate or evaluate the validity of the coding process. Next, we propose multiphase guidelines for transparently reporting on the quantitative coding of open-ended data, informed by concerns with replicability, content validity, and statistical validity. The first phase involves research design, including selecting data and identifying units reliably. The second phase includes developing a coding manual and training coders. The final phase outlines how to establish reliability. As part of this phase, we used data simulations to examine a common statistic for testing reliability on open-ended data, Cohen’s κ, and found that it can become inflated when researchers repeatedly test interrater reliability or manipulate categories, such as by including a missing-data category. Finally, to facilitate transparent and valid coding of open-ended data, we provide a preregistration template that reflects these guidelines. All of the guidelines and resources provided in this article can be adapted for different types of studies, depending on context.

Item Type:	Article
Date Type:	Published Online
Status:	Published
Schools:	Psychology
Additional Information:	License information from Publisher: LICENSE 1: URL: https://creativecommons.org/licenses/by-nc/4.0/, Start Date: 2024-11-15
Publisher:	SAGE Publications
ISSN:	2515-2459
Date of First Compliant Deposit:	18 November 2024
Date of Acceptance:	19 July 2024
Last Modified:	09 Dec 2024 16:30
URI:	https://orca.cardiff.ac.uk/id/eprint/174121

Actions (repository staff only)

Edit Item