The accuracy and reliability of crowdsource annotations of digital retinal images

Mitry, Danny, Zutis, Kris, Dhillon, Baljean, Peto, Tunde, Hayat, Shabina, Khaw, Kay-Tee, Morgan, James E. ORCID: https://orcid.org/0000-0002-8920-1065, Moncur, Wendy, Trucco, Emanuele and Foster, Paul J. 2016. The accuracy and reliability of crowdsource annotations of digital retinal images. Translational Vision Science & Technology 5 (5) , 6. 10.1167/tvst.5.5.6

Preview

PDF - Published Version Available under License Creative Commons Attribution. Download (620kB) | Preview

Abstract

Purpose: Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification. Methods: We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy. Results: In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%–74%) and 87% (95% CI, 86%–88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91–0.96). For image annotation, a maximal Dice coefficient (∼0.6) was achieved with a consensus threshold of 0.25. Conclusions: This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group. Translational Relevance: The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Optometry and Vision Sciences
Subjects:	R Medicine > RE Ophthalmology
Uncontrolled Keywords:	retina, image analysis, crowdsourcing
Publisher:	ARVO
ISSN:	2164-2591
Funders:	Fight for Sight, College of Optometrists, NIHR
Date of First Compliant Deposit:	30 March 2017
Date of Acceptance:	7 July 2016
Last Modified:	06 May 2023 00:44
URI:	https://orca.cardiff.ac.uk/id/eprint/99553

Actions (repository staff only)

Edit Item