Evaluating and improving graph-to-text generation with large language models

He, Jie, Yang, Yijun, Long, Wanqiu, Xiong, Deyi, Gutierrez Basulto, Victor ORCID: https://orcid.org/0000-0002-6117-5459 and Pan, Jeff Z. 2025. Evaluating and improving graph-to-text generation with large language models. Presented at: 2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Albuquerque, New Mexico, 29 April - 4 May 2025. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Albuquerque, New Mexico: ACL, pp. 10219-10244. 10.18653/v1/2025.naacl-long.513

Preview

PDF - Published Version Available under License Creative Commons Attribution. Download (726kB) | Preview

Abstract

Large language models (LLMs) have demonstrated immense potential across various tasks. However, research for exploring and improving the capabilities of LLMs in interpreting graph structures remains limited. To address this gap, we conduct a comprehensive evaluation of prompting current open-source LLMs on graph-to-text generation tasks. Although we explored the optimal prompting strategies and proposed a novel and effective diversity-difficulty-based few-shot sample selection method, we found that the improvements from tuning-free approaches were incremental, as LLMs struggle with planning on complex graphs, particularly those with a larger number of triples. To further improve LLMs in planning with graph sequences and grounding in truth, we introduce a new graph-to-text dataset, PlanGTG, annotated with two sub-tasks: reordering and attribution. Through extensive automatic and human evaluations, we demonstrate significant improvements in the quality of generated text from both few-shot learning and fine-tuning perspectives using the PlanGTG dataset. Our study paves the way for new research directions in graph-to-text generation.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	ACL
ISBN:	979-8-89176-189-6
Date of First Compliant Deposit:	12 February 2025
Date of Acceptance:	22 January 2025
Last Modified:	27 Jan 2026 10:15
URI:	https://orca.cardiff.ac.uk/id/eprint/176140

Actions (repository staff only)

Edit Item