Beyond accuracy: revisiting out-of-distribution generalization in NLI models

Delbari, Zahra and Pilehvar, Mohammad Taher 2025. Beyond accuracy: revisiting out-of-distribution generalization in NLI models. Presented at: 29th Conference on Computational Natural LanguageLearning, Vienna, Austria, 31 July - 1 August 2025. Published in: Boleda, Gemma and Roth, Michael eds. Proceedings of the 29th Conference on Computational Natural Language Learning. Association for Computational Linguistics, pp. 557-570. 10.18653/v1/2025.conll-1.36

PDF - Published Version Available under License Creative Commons Attribution. Download (6MB)

Abstract

This study investigates how well discriminative transformers generalize in Natural Language Inference (NLI) tasks. We specifically focus on a well-studied bias in this task: the tendency of models to rely on superficial features and dataset biases rather than a true understanding of language. We argue that the performance differences observed between training and analysis datasets do not necessarily indicate a lack of knowledge within the model. Instead, the gap often points to a misalignment between the decision boundaries of the classifier head and the representations learned by the encoder for the analysis samples. By investigating the representation space of NLI models across different analysis datasets, we demonstrate that even when the accuracy is nearly random in some settings, still samples from opposing classes remain almost perfectly linearly separable in the encoder’s representation space. This suggests that, although the classifier head may fail on analysis data, the encoder still generalizes and encodes representations that allow for effective discrimination between NLI classes.

Item Type:	Conference or Workshop Item - published (Paper)
Date Type:	Publication
Status:	Published
Schools:	Schools > Computer Science & Informatics
Publisher:	Association for Computational Linguistics
ISBN:	979-8-89176-271-8
Date of First Compliant Deposit:	16 June 2026
Last Modified:	02 Aug 2026 01:19
URI:	https://orca.cardiff.ac.uk/id/eprint/187572

Actions (repository staff only)

Edit Item