Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images

Chen, Ziman, Chambara, Nonhlanhla ORCID: https://orcid.org/0000-0002-3183-883X, Wu, Chaoqun, Lo, Xina, Liu, Shirley Yuk Wah, Gunda, Simon Takadiyi, Han, Xinyang, Qu, Jingguo, Chen, Fei and Ying, Michael Tin Cheung 2025. Assessing the feasibility of ChatGPT-4o and Claude 3-Opus in thyroid nodule classification based on ultrasound images. Endocrine 87 (3) , pp. 1041-1049. 10.1007/s12020-024-04066-x

Preview

PDF - Published Version Available under License Creative Commons Attribution. Download (2MB) | Preview

Abstract

Purpose: Large language models (LLMs) are pivotal in artificial intelligence, demonstrating advanced capabilities in natural language understanding and multimodal interactions, with significant potential in medical applications. This study explores the feasibility and efficacy of LLMs, specifically ChatGPT-4o and Claude 3-Opus, in classifying thyroid nodules using ultrasound images. Methods: This study included 112 patients with a total of 116 thyroid nodules, comprising 75 benign and 41 malignant cases. Ultrasound images of these nodules were analyzed using ChatGPT-4o and Claude 3-Opus to diagnose the benign or malignant nature of the nodules. An independent evaluation by a junior radiologist was also conducted. Diagnostic performance was assessed using Cohen’s Kappa and receiver operating characteristic (ROC) curve analysis, referencing pathological diagnoses. Results: ChatGPT-4o demonstrated poor agreement with pathological results (Kappa = 0.116), while Claude 3-Opus showed even lower agreement (Kappa = 0.034). The junior radiologist exhibited moderate agreement (Kappa = 0.450). ChatGPT-4o achieved an area under the ROC curve (AUC) of 57.0% (95% CI: 48.6–65.5%), slightly outperforming Claude 3-Opus (AUC of 52.0%, 95% CI: 43.2–60.9%). In contrast, the junior radiologist achieved a significantly higher AUC of 72.4% (95% CI: 63.7–81.1%). The unnecessary biopsy rates were 41.4% for ChatGPT-4o, 43.1% for Claude 3-Opus, and 12.1% for the junior radiologist. Conclusion: While LLMs such as ChatGPT-4o and Claude 3-Opus show promise for future applications in medical imaging, their current use in clinical diagnostics should be approached cautiously due to their limited accuracy.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Healthcare Sciences
Publisher:	Springer
ISSN:	1355-008X
Date of First Compliant Deposit:	25 October 2024
Date of Acceptance:	2 October 2024
Last Modified:	16 Apr 2025 09:11
URI:	https://orca.cardiff.ac.uk/id/eprint/173347

Actions (repository staff only)

Edit Item