A meta-evaluation of automatic metrics for elaborative simplification

Alshatti, Abdullah, Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 and Alva Manchego, Fernando ORCID: https://orcid.org/0000-0001-6218-8377 2026. A meta-evaluation of automatic metrics for elaborative simplification. Presented at: LREC 2026 Workshop, Mallorca, Spain, 11-16 May 2026. Published in: Shardlow, M., François, T., Amaro, R., Baptista, J., Cardon, R., Ribeiro, E., Saggion, H., Stodden, R., Todirascu, A. and Wilkens, R. eds. Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026. LREC, pp. 193-209. 10.63317/3bhnb2uoif7o

PDF - Accepted Post-Print Version Available under License Creative Commons Attribution. Download (694kB)

Abstract

Elaborative simplification aims to improve the readability of texts by adding content that helps the readers. However, evaluating these elaborations remains challenging due to their subjective nature and the lack of suitable annotated datasets. To support the evaluation of elaborative simplification models, we introduce a new dataset with human ratings of elaborations generated by Large Language Models (LLMs), focusing on two quality criteria: cohesion and informativeness. Using these human judgments as a reference, we conduct a meta-evaluation of existing automatic evaluation approaches, with a focus on LLM-as-a-judge strategies. Our experiments suggest that evaluations made by smaller LLMs correlate poorly with human judgments, while larger models with structured prompting exhibit higher agreement. Informativeness evaluation proved to be challenging due to its subjectivity, as evidenced by the low inter-annotator agreement compared to cohesion.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Additional Information:	RRS policy applied
Publisher:	LREC
Date of First Compliant Deposit:	7 July 2026
Last Modified:	05 Aug 2026 13:15
URI:	https://orca.cardiff.ac.uk/id/eprint/187993

Actions (repository staff only)

Edit Item