Towards responsible AI for IoT network security auditing using knowledge graph and RAGAS

Briliyant, Obrina ORCID: https://orcid.org/0000-0002-1054-8112, Javed, Amir ORCID: https://orcid.org/0000-0001-9761-0945 and Cherdantseva, Yulia ORCID: https://orcid.org/0000-0002-3527-1121 2026. Towards responsible AI for IoT network security auditing using knowledge graph and RAGAS. Journal of Cybersecurity and Privacy 6 (3) , 98. 10.3390/jcp6030098

PDF - Published Version Available under License Creative Commons Attribution. Download (1MB)

Abstract

The trustworthiness of AI-powered network security auditing depends not only on detection accuracy but on the faithfulness of the explanations that support compliance verdicts. In IoT network security, Large Language Models (LLMs) are increasingly utilized to produce natural-language security assessments from raw network traffic, yet the extent to which these explanations are grounded in retrieved evidence is rarely measured. This paper presents the Retrieval-Augmented Generation Assessment Suite (RAGAS) as an evaluation framework that compares three retrieval paradigms—rule-based heuristic scoring, dense vector retrieval, and knowledge graph traversal—on the task of explaining network compliance against ETSI EN 303 645 IoT cybersecurity provisions. Using 30 human expert-validated compliance scenarios derived from the CIC-IoT2023 dataset and three LLMs (DeepSeek-R1, Qwen-2.5, Llama-3.2), we find that graph-based retrieval achieves the highest faithfulness (0.570), outperforming rule-based (0.524) and vector retrieval (0.509). All methods, however, exhibit low context recall (≤22.4%), and we highlight that high detection F1 scores do not guarantee faithful explanations; over 40% of statements in compliance answers are unsupported by retrieved evidence. A proof-of-concept prototype, Security Audit Compliance Agent (SACA), demonstrates how knowledge graph traversal can be integrated with interactive visualization to support human auditor oversight. We argue that, in adherence to responsible AI principles, faithfulness measurement should become a standard complement to accuracy reporting for an AI-driven network audit or forensic analysis.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Additional Information:	License information from Publisher: LICENSE 1: URL: https://creativecommons.org/licenses/by/4.0/, Start Date: 2026-06-06
Publisher:	MDPI
Date of First Compliant Deposit:	24 June 2026
Date of Acceptance:	4 June 2026
Last Modified:	04 Aug 2026 21:36
URI:	https://orca.cardiff.ac.uk/id/eprint/187726

Actions (repository staff only)

Edit Item