Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images.

Chen, Ziman, Chambara, Nonhlanhla ORCID: https://orcid.org/0000-0002-3183-883X, Wu, Chaoqun, Lo, Xina, Liu, Shirley Yuk Wah, Gunda, Simon Takadiyi, Han, Xinyang, Qu, Jingguo, Chen, Fei and Ying, Michael Tin Cheung 2024. Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images. Endocrine 10.1007/s12020-024-04066-x

PDF - Published Version Available under License Creative Commons Attribution. Download (2MB)

Abstract

Large language models (LLMs) are pivotal in artificial intelligence, demonstrating advanced capabilities in natural language understanding and multimodal interactions, with significant potential in medical applications. This study explores the feasibility and efficacy of LLMs, specifically ChatGPT-4o and Claude 3-Opus, in classifying thyroid nodules using ultrasound images. This study included 112 patients with a total of 116 thyroid nodules, comprising 75 benign and 41 malignant cases. Ultrasound images of these nodules were analyzed using ChatGPT-4o and Claude 3-Opus to diagnose the benign or malignant nature of the nodules. An independent evaluation by a junior radiologist was also conducted. Diagnostic performance was assessed using Cohen's Kappa and receiver operating characteristic (ROC) curve analysis, referencing pathological diagnoses. ChatGPT-4o demonstrated poor agreement with pathological results (Kappa = 0.116), while Claude 3-Opus showed even lower agreement (Kappa = 0.034). The junior radiologist exhibited moderate agreement (Kappa = 0.450). ChatGPT-4o achieved an area under the ROC curve (AUC) of 57.0% (95% CI: 48.6-65.5%), slightly outperforming Claude 3-Opus (AUC of 52.0%, 95% CI: 43.2-60.9%). In contrast, the junior radiologist achieved a significantly higher AUC of 72.4% (95% CI: 63.7-81.1%). The unnecessary biopsy rates were 41.4% for ChatGPT-4o, 43.1% for Claude 3-Opus, and 12.1% for the junior radiologist. While LLMs such as ChatGPT-4o and Claude 3-Opus show promise for future applications in medical imaging, their current use in clinical diagnostics should be approached cautiously due to their limited accuracy. [Abstract copyright: © 2024. The Author(s).]

Item Type:	Article
Date Type:	Published Online
Status:	In Press
Schools:	Healthcare Sciences
Publisher:	Springer
ISSN:	1355-008X
Date of First Compliant Deposit:	25 October 2024
Date of Acceptance:	2 October 2024
Last Modified:	25 Oct 2024 09:45
URI:	https://orca.cardiff.ac.uk/id/eprint/173347

Actions (repository staff only)

Edit Item