Evaluating cultural knowledge and reasoning in LLMs through Persian allusions

Nobakhtian, Melika, Yaghoobzadeh, Yadollah and Pilehvar, Mohammad T. 2025. Evaluating cultural knowledge and reasoning in LLMs through Persian allusions. Presented at: EMNLP 2025, Suzhou, China, 4 - 9 November 2025. Published in: Christodoulopoulos, Christos, Chakraborty, Tanmoy, Rose, Carolyn and Peng, Violet eds. Findings of the Association for Computational Linguistics. Association for Computational Linguistics, pp. 25725-25737. 10.18653/v1/2025.findings-emnlp.1403

Full text not available from this repository.

Abstract

Allusion recognition—a task demanding contextual activation of cultural knowledge—serves as a critical test of LLMs’ ability to deploy stored information in open-ended, figurative settings. We introduce a framework for evaluating Persian literary allusions through (1) classical poetry annotations and (2) LLM-generated texts incorporating allusions in novel contexts. By combining knowledge assessments, multiple-choice tasks, and open-ended recognition, we analyze whether failures stem from knowledge gaps or activation challenges. Evaluations across eleven LLMs highlight a notable observation: models exhibit strong foundational knowledge and high multiple-choice accuracy, yet performance drops substantially in open-ended tasks, especially for indirect references. Reasoning-optimized models generalize better to novel contexts, whereas distilled models show marked degradation in cultural reasoning. The gap underscores that LLMs’ limitations arise not from missing knowledge but from difficulties in spontaneously activating cultural references without explicit cues. We propose allusion recognition as a benchmark for contextual knowledge deployment, highlighting the need for training paradigms that bridge factual recall and culturally grounded reasoning. Our code, datasets and results are available at https://github.com/MelikaNobakhtian/Allusion

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-335-7
Last Modified:	10 Feb 2026 12:26
URI:	https://orca.cardiff.ac.uk/id/eprint/184571

Actions (repository staff only)

Edit Item