There’s no such thing as simple reasoning for LLMs

Ariyani, Nurul, Bouraoui, Zied, Booth, Richard ORCID: https://orcid.org/0000-0002-6647-6381 and Schockaert, Steven ORCID: https://orcid.org/0000-0002-9256-2881 2025. There’s no such thing as simple reasoning for LLMs. Presented at: The 63rd Annual Meeting of the Association for Computational Linguistics, Vienna, Austria, 27 July - 1 August 2025. Published in: Waxiang, C., Nabende, J., Shutova, E. and Pilehvar, M. T. eds. Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, pp. 4503-4514. 10.18653/v1/2025.findings-acl.232

Preview

PDF - Accepted Post-Print Version Download (484kB) | Preview

Abstract

Large Language Models (LLMs) have been widely found to struggle with logical reasoning, where even fine-tuned models fail dramatically on out-of-distribution problems. However, existing work has focused on relatively complex “many-hop” reasoning problems. In this paper, we analyse the performance of fine-tuned LLMs on simple reasoning problems, all of which can be solved in at most three inference steps. Due to the simplicity of these problems, the model cannot encounter test problems that are fundamentally different from those it has seen during training. Unfortunately, however, we find that the models remain highly brittle, being susceptible to seemingly innocent perturbations, such as the addition of duplicates to the set of premises and shuffling the order in which the premises are presented.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	9798891762565
Related URLs:	https://2025.aclweb.org/
Date of First Compliant Deposit:	22 July 2025
Date of Acceptance:	15 May 2025
Last Modified:	17 Jun 2026 11:00
URI:	https://orca.cardiff.ac.uk/id/eprint/179568

Actions (repository staff only)

Edit Item