Diagram2Structure: Unlocking LLMs’ diagram comprehension through DiagramDiff, a framework for structuring offline diagrams

Hu, Haoxiang, Li, Yaokun, Huang, Zeyuan, Gao, Cangjun, He, Qiang, Li, Qingkun, Deng, Xiaoming, Ma, Cuixia, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Liu, Yong-Jin and Wang, Hongan 2026. Diagram2Structure: Unlocking LLMs’ diagram comprehension through DiagramDiff, a framework for structuring offline diagrams. Presented at: The IEEE/CVF Conference on Computer Vision and Pattern Recognition 2026, Denver, CO, USA, 3-7 June 2026. IEEE, Item availability restricted.

	PDF (Provisional file) - Accepted Post-Print Version Download (17kB)
	PDF - Accepted Post-Print Version Restricted to Repository staff only Download (16MB)

Abstract

Diagrams are widely used in daily life. However, offline diagrams typically exist in the form of images, lacking structured data representation, which significantly limits their reusability and editability. Current research mainly focuses on supporting basic query tasks for online diagrams and does not meet the semantic understanding and interaction requirements for complex offline diagrams. Although large language models (LLMs) possess powerful reasoning and knowledge integration capabilities, their performance in processing offline diagrams is unsatisfactory due to the inability to accurately understand the structure and content of offline diagrams. To address these issues, we propose DiagramDiff, a framework consisting of a high-precision diagram reconstruction model and an instance-level diagram element recognition model. The framework converts offline diagrams into standardized data structures, enabling LLMs to transition from being unable to understand offline diagrams to intelligent assistants capable of semantic reasoning, logical validation, and efficient diagram editing. To deal with the lack of the dataset, we constructed a dataset containing diagrams, and their corresponding question and answer (Q&A) and editing tasks. Experiments demonstrate that DiagramDiff achieves state-of-the-art performance in diagram reconstruction and recognition tasks, significantly enhancing LLMs’ understanding and interaction capabilities with offline diagrams.

Item Type:	Conference or Workshop Item - published (Paper)
Status:	In Press
Schools:	Schools > Computer Science & Informatics
Publisher:	IEEE
Date of First Compliant Deposit:	13 May 2026
Date of Acceptance:	21 February 2026
Last Modified:	15 Jul 2026 12:54
URI:	https://orca.cardiff.ac.uk/id/eprint/186973

Actions (repository staff only)

Edit Item