Annotating training data for conditional semantic textual similarity measurement using large language models

Zhang, Gaifan, Zhou, Yi and Bollegala, Danushka 2025. Annotating training data for conditional semantic textual similarity measurement using large language models. Presented at: 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, 4-9 November 2025. Published in: Christodoulopoulos, Chistos, Chakraborty, Tanmoy, Rose, Carolyn and Peng, Violet eds. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 27003-27015. 10.18653/v1/2025.emnlp-main.1373

PDF - Published Version Available under License Creative Commons Attribution. Download (497kB)

Abstract

Semantic similarity between two sentences depends on the aspects considered between those sentences. To study this phenomenon, Deshpande et al. (2023) proposed the Conditional Semantic Textual Similarity (C-STS) task and annotated a human-rated similarity dataset containing pairs of sentences compared under two different conditions. However, Tu et al. (2024) found various annotation issues in this dataset and showed that manually re-annotating a small portion of it leads to more accurate C-STS models. Despite these pioneering efforts, the lack of large and accurately annotated C-STS datasets remains a blocker for making progress on this task as evidenced by the subpar performance of the C-STS models. To address this training data need, we resort to Large Language Models (LLMs) to correct the condition statements and similarity ratings in the original dataset proposed by Deshpande et al. (2023). Our proposed method is able to re-annotate a large training dataset for the C-STS task with minimal manual effort. Importantly, by training a supervised C-STS model on our cleaned and re-annotated dataset, we achieve a 5.4% statistically significant improvement in Spearman correlation. The re-annotated dataset is available at https://LivNLP.github.io/CSTS-reannotation.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Engineering
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-332-6
Date of First Compliant Deposit:	16 June 2026
Last Modified:	16 Jun 2026 09:45
URI:	https://orca.cardiff.ac.uk/id/eprint/187574

Actions (repository staff only)

Edit Item