Enhancing clinical decision support with LLMs: a feasibility study integrating CoT, RAG, and QLoRA

Matthew, George and Hicks, Yulia ORCID: https://orcid.org/0000-0002-7179-4587 2025. Enhancing clinical decision support with LLMs: a feasibility study integrating CoT, RAG, and QLoRA. Presented at: 29th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems, Osaka, Japan, 10-12 September 2025. Procedia Computer Science. , vol.270 Elsevier, pp. 4946-4956. 10.1016/j.procs.2025.09.621

PDF - Published Version Download (590kB)

Abstract

The integration of artificial intelligence (AI) into healthcare holds significant promise for enhancing clinical decision-making and improving patient outcomes. However, general-purpose large language models (LLMs) frequently exhibit limitations such as hallucinations, lack of domain-specific accuracy, and opaque reasoning processes, posing risks in clinical applications. This study addressed these challenges by proposing and exploring an innovative integration of Chain-of-Thought (CoT) prompting, Retrieval-Augmented Generation (RAG), and parameter-efficient fine-tuning using Quantized Low-Rank Adaptation (QLoRA) specifically tailored for medical use. A distilled 14-billion-parameter variant of the DeepSeek R1 model was fine-tuned using a structured clinical dataset that emphasizes step-by-step reasoning. Additionally, external medical references were incorporated through RAG, employing embedding models for precise context retrieval. The combination of these techniques was systematically evaluated on a challenging set of open-ended medical questions, resulting in accuracy improvements—from a baseline accuracy of 55% to a final performance of 81%. Further qualitative evaluation involving three practising General Practitioners (GPs) and three fourth-year medical students from Cardiff University underscored the proposed system’s clinical utility and transparent reasoning capabilities while also identifying areas for improvement, such as conciseness and explicit adherence to national-specific clinical guidelines. This research demonstrated that integrating CoT, RAG, and QLoRA provides a practical pathway toward reliable, transparent, and clinically relevant AI support for healthcare professionals. Recommendations for future work include scaling models, incorporating comprehensive patient data, and enhancing customization for clinical application contexts.

Item Type:	Conference or Workshop Item (Paper)
Date Type:	Published Online
Status:	Published
Schools:	Schools > Engineering
Publisher:	Elsevier
ISSN:	1877-0509
Date of First Compliant Deposit:	15 December 2025
Last Modified:	15 Dec 2025 15:00
URI:	https://orca.cardiff.ac.uk/id/eprint/183243

Actions (repository staff only)

Edit Item